registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these...
Transcript of registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these...
![Page 1: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/1.jpg)
Testing the entire npm registry
nodechecker.com
![Page 2: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/2.jpg)
● Pedro Dias○ @pedromdias○ http://abru.pt
● Fullstack engineer at ptisp.pt● Lecturer at ipt.pt● Petrolhead● Hiker● Soon to be father (if I leave this room
running, now you know why)
Who am i
![Page 3: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/3.jpg)
Why
● Autonomously test all modules in npm
registry○ Will results have statistical weight?
○ Is it practicable?
■ Resource and time wise
○ Could it be done autonomously?
I had no idea... that made me curious...
![Page 4: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/4.jpg)
Disclaimer
● Do not make judgements on these results.
● This isn’t a scientific metric.
● All data is available.
● So far only was debugged and fixed what
was needed for this to run.
![Page 5: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/5.jpg)
1st iteration
Like all nightly proof-of-concepts ...
![Page 6: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/6.jpg)
So...
1. Read npm’s couchdb dump
2. Iterate all modulesa. git clone
b. npm install
c. npm test
3. Save exit codes in Redis
![Page 7: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/7.jpg)
Issues...
● npm○ npm init
○ npm test
● Outdated repositories
![Page 8: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/8.jpg)
Weird things started happening...
● Redis started gaining life.
● But it got worse...
![Page 9: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/9.jpg)
People really do crazy shit in their tests without
any warning.
Next time you do a ‘npm test’ think about this...
![Page 10: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/10.jpg)
2nd iteration
● Test environment needed to be sandboxed.
● Switching from repos to tarballs
![Page 11: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/11.jpg)
Docker
● Why docker○ Docker isn’t trying to be what it isn’t.
○ Good toolset (try using plain lxc or openvz)
○ API
○ Execution is easy
● Test standardization/repeatability
● Process isolation
![Page 12: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/12.jpg)
Docker image
● https://index.docker.
io/u/apocas/nodechecker
● Installed○ Node.js v0.10.15
○ Git
○ wget
○ Redis (recent)
![Page 13: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/13.jpg)
Container lifecycle
1. Create a container
2. Attach to container
3. Start the container
4. Wait for it to end
5. Destroy it
![Page 14: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/14.jpg)
Container execution
● https://github.com/dotcloud/docker/wiki/Docker-run-improvements○ Giving ‘docker run’ a process’ish like behavior.
● http://docs.docker.io/en/latest/api/docker_remote_api_v1.3/#inside-docker-run○ How ‘docker run’ works.
![Page 15: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/15.jpg)
3rd iteration
● Improving container lifecycle○ Attach to it
○ Control containers IDs
○ Remote API
■ http://docs.docker.
io/en/latest/api/docker_remote_api_v1.3
● Multiple module sources
![Page 16: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/16.jpg)
Module sources
● Module tarball○ From npm registry
○ Tests in .npmignore
● Repository○ Specified in package.json
○ May be outdated/invalid
○ May be unstable
![Page 17: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/17.jpg)
.npmignore
● Scenarioa. Module’s package.json specifies a test script.
b. Dev added his test code to .npmignore
![Page 18: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/18.jpg)
Cycle● Module has test script in package.json?
○ No■ NOTTESTED
○ Yes■ It’s a dummy script?
● Yes○ NOTTESTED
● No○ Have been tested before?
■ No● Test using tarball
○ Exit code 0■ OK
○ Exit code != 0■ NOK
■ Yes● Test using repository
○ ...
Diagrams are for pussies...
![Page 19: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/19.jpg)
Container limits
● CPU is not a problem.
● Memory is!○ Modules using 1gb+
○ Container memory limit
■ Stuck containers
![Page 20: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/20.jpg)
Funny
● https://npmjs.org/package/ifyouwanttogetthesumoftwonumberswherethosetwonumbersarechosenbyfindingthelargestoftwooutofthreenumbersandsquaringthemwhichismultiplyingthembyitselfthenyoushouldinputthreenumbersintothisfunctionanditwilldothatforyou
● https://npmjs.org/package/dos-fork-bomb
![Page 21: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/21.jpg)
Weird
● 404’s in some packages - Fixed. Isaac did his magic.
● Empty npm info - Yet to debug/question.
![Page 22: registry Testing the entire npm · 2013. 8. 14. · Disclaimer Do not make judgements on these results. This isn’t a scientific metric. All data is available. So far only was debugged](https://reader036.fdocuments.net/reader036/viewer/2022071212/602780de5f1ec016901a8c39/html5/thumbnails/22.jpg)
Source
● nodechecker-test○ Container abstraction○ https://github.com/apocas/nodechecker-tester
● nodechecker-engine○ Uses nodechecker-test○ Iterates the npm registry○ https://github.com/apocas/nodechecker-engine
● nodechecker.com○ Website○ API○ https://github.com/apocas/nodechecker.com