Sylvestre Ledru – [email protected] Léo Cavaillé – leo ... · January, 19th 2014 The...
Transcript of Sylvestre Ledru – [email protected] Léo Cavaillé – leo ... · January, 19th 2014 The...
![Page 1: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/1.jpg)
Sylvestre Ledru – [email protected]éo Cavaillé – [email protected]
![Page 2: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/2.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Debian
+ 20 000 source packages~13 architectures
3 kernels
The biggest database of FLOSS code (?)
![Page 3: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/3.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
How to:
● Build all these packages with a new compiler ● Run static analyzers on their codes● Provide reports at the same place for both upstreams and packagers
![Page 4: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/4.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
For now, good luck with that !
![Page 5: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/5.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Started as a requirement for the clang transparent builds and scan-build static analyzers (Léo
Cavaillé's GsoC 2013)
Not trying to redevelop yet a new build system
Also related to the work presented by Zack this afternoon on sources.debian.net
![Page 6: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/6.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
To build packages in Debian:
● wanna-build / buildd (the official services)http://buildd.debian.org/
● http://www.debian-ports.org/ (also based on wanna-build / buildd)● Rebuildd● Open Build Service● ...
![Page 7: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/7.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Decided to work on debuild.mewith Paul Tagliamonte
Python + flask + sqlalchemy
Goal : easy to hack / lightweight / expendable
Pluggable in the fedmsg bus
![Page 8: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/8.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Renamed to Debile
After propositions by the DPL cabals (Lucas + Zack)
![Page 9: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/9.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
ArchitectureInfrastructure
![Page 10: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/10.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Basic architecture
PostgreSQL database
Image credits :Database designed by Stefan Parnarov, Server designed by aLf from the Noun Project
debile master
incoming/
debile slave farm
debile-webfirewoes
signed source packages
● One slave = a docker instance of debile-slave● A slave executes one job (build, static analysis…)
at a time on binaries/sources.● Build jobs upload their results to incoming/
flask
SQLalchemy
python
python
![Page 11: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/11.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Devices :
Master node (~ 8 To of HD)
M1000e with :4 blades (8 cores, 16G)12 blades (4 cores, 16G)
![Page 12: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/12.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Plus :
7 various powerful Dell servers
Thanks to Google and IRILL
Hosted by IRILL on Renater
![Page 13: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/13.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
● Build nodes will run Debian jessie● Installation done through PXE ● Configuration done with Salt Stack
![Page 14: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/14.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Results
![Page 15: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/15.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Provides build workers:● Normal (ie gcc) builds● Clang● gcc snapshot ?
With repository of clang-built packages
Uploads binaries to debile's own APT repository
![Page 16: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/16.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Provides analyzer workers:● Scan-build (C, C++, Objective-C)● Lintian (Debian packages)● Coccinelle (C, C++)● Cppcheck (C++)● Findbugs (Java)● ...
![Page 17: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/17.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
![Page 18: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/18.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
![Page 19: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/19.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
![Page 20: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/20.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Next steps
![Page 21: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/21.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Next steps :
● Finalize the Debian packages
● Deploy on the new servers
● Relaunch all the workers on the whole Debian Archive
● Send the reports to a visualisation interfacebased on Firehose, a Fedora XML formatOn going work with Zack & Matthieu
![Page 22: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/22.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Future
![Page 23: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/23.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
TODO in Debile v2.0:
● Debian PPA● Support others archs (example : arm)● Plug new analyzers (flake8, etc)● Ship the slaves as Docker instances● Reverse dependencies rebuild● Make it an official service
![Page 24: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/24.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Want to contribute ?
https://alioth.debian.org/projects/pkg-debile/http://anonscm.debian.org/gitweb/?p=pkg-debile/debile.git;a=summary
http://anonscm.debian.org/gitweb/?p=pkg-debile/debile-web.git;a=summary
irc://irc.oftc.net/#debile
![Page 25: Sylvestre Ledru – sylvestre@debian.org Léo Cavaillé – leo ... · January, 19th 2014 The Debile Project Sylvestre Ledru & Léo Cavaillé Started as a requirement for the clang](https://reader030.fdocuments.net/reader030/viewer/2022040612/5edd6bd9ad6a402d6668824e/html5/thumbnails/25.jpg)
January, 19th 2014 The Debile ProjectSylvestre Ledru & Léo Cavaillé
Any questions ? Remarks ?