Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum [email protected] [email protected] Debian...

30
Debian Quality Assurance Lucas Nussbaum [email protected] [email protected] Debian Quality Assurance 1 / 24

Transcript of Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum [email protected] [email protected] Debian...

Page 1: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Debian Quality Assurance

Lucas [email protected]

[email protected] Debian Quality Assurance 1 / 24

Page 2: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Debian

Well known and respected for :

Its long and bumpy release cycles

Its quality

But it was not a given :>1000 Debian Developers

Mostly volunteers15000 software packages

Some very popular packagesA lot of niche packages

[email protected] Debian Quality Assurance 2 / 24

Page 3: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Debian

Well known and respected for :Its long and bumpy release cycles

Its quality

But it was not a given :>1000 Debian Developers

Mostly volunteers15000 software packages

Some very popular packagesA lot of niche packages

[email protected] Debian Quality Assurance 2 / 24

Page 4: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Debian

Well known and respected for :Its long and bumpy release cyclesIts quality

But it was not a given :>1000 Debian Developers

Mostly volunteers15000 software packages

Some very popular packagesA lot of niche packages

[email protected] Debian Quality Assurance 2 / 24

Page 5: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Quality factors

Culture : "We release when it’s ready"Many DDs won’t compromise on this

Packages ownership : individuals or teamsAbout 600 active developersMany of them experts of their packages

Release Critical (RC) bugsPrevent packages from being part of a releaseEverybody is welcomed to fix them

Non-Maintainer Uploads

Debian Quality Assurance team

[email protected] Debian Quality Assurance 3 / 24

Page 6: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Quality factors

Culture : "We release when it’s ready"Many DDs won’t compromise on this

Packages ownership : individuals or teamsAbout 600 active developersMany of them experts of their packages

Release Critical (RC) bugsPrevent packages from being part of a releaseEverybody is welcomed to fix them

Non-Maintainer Uploads

Debian Quality Assurance team

[email protected] Debian Quality Assurance 3 / 24

Page 7: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Quality factors

Culture : "We release when it’s ready"Many DDs won’t compromise on this

Packages ownership : individuals or teamsAbout 600 active developersMany of them experts of their packages

Release Critical (RC) bugsPrevent packages from being part of a releaseEverybody is welcomed to fix them

Non-Maintainer Uploads

Debian Quality Assurance team

[email protected] Debian Quality Assurance 3 / 24

Page 8: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Quality factors

Culture : "We release when it’s ready"Many DDs won’t compromise on this

Packages ownership : individuals or teamsAbout 600 active developersMany of them experts of their packages

Release Critical (RC) bugsPrevent packages from being part of a releaseEverybody is welcomed to fix them

Non-Maintainer Uploads

Debian Quality Assurance team

[email protected] Debian Quality Assurance 3 / 24

Page 9: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Quality factors

Culture : "We release when it’s ready"Many DDs won’t compromise on this

Packages ownership : individuals or teamsAbout 600 active developersMany of them experts of their packages

Release Critical (RC) bugsPrevent packages from being part of a releaseEverybody is welcomed to fix them

Non-Maintainer Uploads

Debian Quality Assurance team

[email protected] Debian Quality Assurance 3 / 24

Page 10: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Debian QA team

Goal : Improve the quality of Debian as a whole

Not really a team (no membership), more a central place todiscuss and work on QA

IRC : #[email protected] (OFTC)

Mailing list : [email protected]

[email protected] Debian Quality Assurance 4 / 24

Page 11: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Debian QA team : Tasks

Maintain infrastructure

Run archive-wide checks and mass bug filings

Take care of the dirty areas of Debian

[email protected] Debian Quality Assurance 5 / 24

Page 12: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Maintaining infrastructure

[email protected] Debian Quality Assurance 6 / 24

Page 13: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Infrastructure : mitigating the data hell

Debian : aggregation of loosely-connected servicesNo Launchpad !

Data everywhereA dozen of places to get interesting data

⇒ Tools required to gather data in a central place

[email protected] Debian Quality Assurance 7 / 24

Page 14: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Package Tracking System (PTS)

Source package-centric viewhttp://packages.qa.debian.org/package

[email protected] Debian Quality Assurance 8 / 24

Page 15: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Developer Packages Overview (DDPO)

Developer-centric viewhttp://qa.debian.org/developer.php

[email protected] Debian Quality Assurance 9 / 24

Page 16: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Ultimate Debian Database (UDD)Gather everything into a single SQL DBPerfect tool for data-mining Debian

Currently imported in UDD :Debian Sources and Packages, bugs, popcon, testing migrations, uploadhistory, orphaned packages, carnivore, lintian, debtags, translations, NEWqueue, screenshots, DEHS, ldap, wanna-build, removals, Ubuntu Sourcesand Packages, Ubuntu bugs, Ubuntu popcon

Possible questions you can answer with UDD :

RC buggy packages in testing, sorted by popcon ?

Packages for which the last 4 uploads were NMUs ?

Packages not maintained by official DDs ?

http://udd.debian.org/

[email protected] Debian Quality Assurance 10 / 24

Page 17: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Archive-wide checks

and Mass bug filings

[email protected] Debian Quality Assurance 11 / 24

Page 18: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Archive-wide checks and MBF

Developers are volunteers :Focus on interesting thingsManual testing is boring⇒ not done

Need for automated tests

Other advantage : treats all packages equallyNiche packages are very important to some users

[email protected] Debian Quality Assurance 12 / 24

Page 19: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Archive rebuilds

Rebuild all packages from source7 hours on 40 8-cores build nodes (Thanks OpenOffice.Org)

Detect packages that Fail To Build From Source (FTBFS)Bugs in packagesToolchain bugsFailures caused by changes in other packages

Ran every 2-3 weeks

Scripts to analyze failures and file bugs semi-automatically(about 200 bugs filed per hour)

Also used for test rebuilds with new compilers, linkers, . . .⇒ Valuable information for upstream

[email protected] Debian Quality Assurance 13 / 24

Page 20: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

DebcheckCheck (statically) that dependencies can be satisfied

Not trivial : versioned dependencies, conflicts

2 versions :"Simple one", in PHPhttp://qa.debian.org/debcheck.php

Better one : edos-debcheck (EDOS project)Can also work with RPMshttp://edos.debian.net/

[email protected] Debian Quality Assurance 14 / 24

Page 21: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Piuparts

Testing package installation, upgrade and removal

Find bugs in maintainer scripts(Shell) scripts executed during package installation/removal

Developers don’t run it on their systems⇒ Archive-wide setup

http://piuparts.debian.org

[email protected] Debian Quality Assurance 15 / 24

Page 22: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Lintian

Static analysis of packages

Usually run locally by the developer, before uploading

Archive-wide setup for continuous testing

Packages now rejected at upload time for some errors

http://lintian.debian.org

[email protected] Debian Quality Assurance 16 / 24

Page 23: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

DEHS - Debian External Health StatusMonitor upstream version of packages

Detect when the Debian package is outdated

HTML scraping + regular expressionsconfigured in debian/watch

http://dehs.alioth.debian.org

[email protected] Debian Quality Assurance 17 / 24

Page 24: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Taking care of the dirty areas of Debian

[email protected] Debian Quality Assurance 18 / 24

Page 25: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Taking care of the dirty areas of Debian

Two complementary approaches :

Focus on packages neglected by their maintainers

Focus on maintainers neglecting their packages

[email protected] Debian Quality Assurance 19 / 24

Page 26: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Neglected packages

Already orphaned :Do minimalistic maintenanceProblem : > 500 orphaned packages

Not (yet) orphaned :Find them, orphan or remove themGoal : Put maintenance in the hands of people with time

[email protected] Debian Quality Assurance 20 / 24

Page 27: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Bapase : finding neglected packages

Multi-criteria search for neglected packages

Based on Ultimate Debian Database

http://udd.debian.org/cgi-bin/bapase.cgi

[email protected] Debian Quality Assurance 21 / 24

Page 28: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Inactive maintainers

Missing In Action (MIA) team

Find them, track them, orphan their packagesMostly based on reporting by users or fellow developersEchelon : records activity of DDs ; Carnivore : tracks identities

Collaboration with the Account Managers :possible removal of their account(Unused accounts⇒ possible security problems)

[email protected] Debian Quality Assurance 22 / 24

Page 29: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Conclusion

Debian Quality Assurance :Mosty archive quality work

Lots of tools and infrastructure developed within Debian

What we don’t do :Focus on specific packages

Bug triaging

Manual testing to find bugsReliance on users to find/file bugs

Join the QA team !

[email protected] Debian Quality Assurance 23 / 24

Page 30: Lucas Nussbaum lucas@debian - LORIA · Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 24. Debian Well known and respected for : Its long and bumpy release

Debian Quality Assurance

Lucas [email protected]

[email protected] Debian Quality Assurance 24 / 24