RelEng Docs Documentation · Determine L10N changesets Finalize and ship L10N Initiate release...
65
RelEng Docs Documentation Release 1.0 catlee and a cast of tens! March 09, 2015
Transcript of RelEng Docs Documentation · Determine L10N changesets Finalize and ship L10N Initiate release...
RelEng Docs DocumentationRelease 1.0
catlee and a cast of tens!
March 09, 2015
Contents
1 Flows 31.1 Firefox builds: from checkin to TBPL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Release Workflows with Inter-Team Handoffs 52.1 Firefox GA Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Firefox Beta Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3 Release Engineering Procedures 93.1 Tree Closing Window Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Tree Closing Window Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4 Hosts 134.1 hg.mozilla.org . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5 Software 155.1 Buildbot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.2 hg pushlog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165.3 buildbot schedulers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165.4 HgPoller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165.5 postrun.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175.6 Mozharness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175.7 TBPL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.8 buildapi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285.9 Cloud tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285.10 VCS Sync tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285.11 Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6 Scheduler Database 296.1 build requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296.2 pending jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306.3 running jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
7 Status Database 317.1 finished jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
8 Releng 101 338.1 What is this? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338.2 What this is not . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338.3 I hope this provides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
i
8.4 Releng in a Nutshell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338.5 The Walk-through series at a glance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
9 Coding Guidelines 359.1 Code Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359.2 Scripting Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359.3 Configuration Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369.4 Web Application Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369.5 Futures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
10 Debugging “jobs aren’t starting” 3710.1 pending jobs aren’t starting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3710.2 jobs aren’t being scheduled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
11 Adding a new repo to Releng Account on Read-The-Docs 3911.1 Pre-requisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3911.2 Enable RTFD updates on github . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3911.3 Add repo to Releng Account on RTFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3911.4 Assign maintainers to the repo on RTFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
12 Adding Sphinx docs to Existing Repositories 4112.1 Adding Sphinx to a project with no docs yet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4112.2 Adding autodoc to a project already using Sphinx . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4112.3 Tweaking the conf.py file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
13 Maintaining this Documentation 4313.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4313.2 Documenting Source Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
14 From a change on a repo to builds being triggered 4514.1 Recap – the full code from examples above . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
15 Topics needing authors 57
16 Indices and tables 59
ii
RelEng Docs Documentation, Release 1.0
Here is Mozilla’s Release Engineering Group’s more technical documentation. You might also be interested in ourmain web site.
Contents:
Contents 1
RelEng Docs Documentation, Release 1.0
2 Contents
CHAPTER 1
Flows
1.1 Firefox builds: from checkin to TBPL
A developer checks in code to hg.mozilla.org using mercurial.
Buildbot is polling the hg pushlog for each of the code repositories we do builds for using an HgPoller.
When the poller detects the new push, the information about the change is written into the Scheduler Database forlater processing by the buildbot schedulers.
Once the schedulers have detected a new change and created new build requests, the jobs are now pending on TBPL.At this point we may start or create new VMs in the cloud to service this pending work.
Once the job starts, it is now considered a running job on TBPL.
When the job finishes, the log of all its output is collected by postrun.py, and uploaded to FTP. The Status Databaseis then updated with the job’s final status. It is now considered finished jobs on TBPL.
Note that because there are several asynchronous processes at play here, a job can disappear from the running listbefore it appears in the finished list.
3
RelEng Docs Documentation, Release 1.0
4 Chapter 1. Flows
CHAPTER 2
Release Workflows with Inter-Team Handoffs
Contents:
2.1 Firefox GA Release
See also:
Disclaimer
Name DescriptionDetermine L10Nchangesets
Finalize and ship L10N
Initiate release Start release via Ship-It! applicationBuild everything Automation will build installers and updaters for all locales and all platforms. (Progress
emails are sent, some of which enable QE to begin phases of testing. That level of detail isnot shown in this diagram.)
functional &updater testing
initiated by automated email QE tests all produced artifacts, obtained via internal links.
Decide build willbe a GO
Manual email from QE initiates Decide if this build is acceptable, or another is needed.Restart process for new build.
push installers tomirrors
Manual email from RelMgmt initiates Push Installers and updaters to Mirrors
verify updater links initiated by automated email QE verifies installers are properly accessible, and updatesare served via normal mechanisms.
Set throttle % & gofor live updaters
Manual email from QE initiates Decide when the release should become visible to endusers.
configure update % Manual email from RelMgmt initiates Apply throttlingpush updaters live Manual email from RelMgmt initiates Deploy the updater artifacts to the production
release site. End users will be offered updates at this point.verify throttlingand live updaters
Manual email from RelEng initiates QE verifies that update throttling is at the correctlevel, and end users will be served accordingly.
make fully visible Manual email from QE initiates Do final clean up of the release, including makingvisible on the FTP servers.
Party! Everything completed for this release.
Note: All “RelEng” steps in the “Description” column above are taken from our checklist for Firefox Releases.
5
RelEng Docs Documentation, Release 1.0
Figure 2.1: GA Releases6 Chapter 2. Release Workflows with Inter-Team Handoffs
RelEng Docs Documentation, Release 1.0
2.2 Firefox Beta Release
See also:
Disclaimer
Name DescriptionDetermine L10Nchangesets
Finalize and ship L10N
Initiate release Start release via Ship-It! applicationBuild everything Automation will build installers and updaters for all locales and all platforms. (Progress
emails are sent, some of which enable QE to begin phases of testing. That level of detail isnot shown in this diagram.)
functional & localupdater testing
initiated by automated email “Updates available on beta-localtest” QE tests all producedartifacts, obtained via internal links.
Decide build willbe a GO
Manual email from QE initiates Decide if this build is acceptable, or another is needed.Restart process for new build.
cdn updatertesting
initiated by automated email “Updates available on beta-cdntest” QE verifies installersare properly accessible, and updates are served via normal mechanisms.
push installers tomirrors
automatic for beta 2 on Push Installers and updaters to Mirrors
push updaters live Manual email from QE initiates Deploy the updater artifacts to the production release site.End users will be offered updates at this point.
verify liveupdaters
Manual email from RelEng initiates QE verifies that updates are available to end users.
make fully visible Manual email from QE initiates Do final clean up of the release, including making visibleon the FTP servers.
Success! Everything completed for this release.
Note: All “RelEng” steps in the “Description” column above are taken from our checklist for Firefox Releases.
Caution: NOT AUTHORITATIVEThe authoritative procedures are maintained by each team. Where any discrepancies are noticed, the officialprocedures (which may include release specific changes) are authoritative.
For Release Engineering, the authoritative procedures are listed as checklists in the main wiki.
2.2. Firefox Beta Release 7
RelEng Docs Documentation, Release 1.0
Figure 2.2: Beta Releases
8 Chapter 2. Release Workflows with Inter-Team Handoffs
CHAPTER 3
Release Engineering Procedures
Procedures that don’t easily fit elsewhere.
3.1 Tree Closing Window Planning
The purpose of the Planning Procedure below is to state what I believe to already be the de facto agreement on theprocess for Tree Closing Windows (TCW). A few enhancements will be proposed along the way in italics.
Once we’re in agreement, we can make it more public, to avoid surprises all around.
3.1.1 Planning Procedure
The goal is to have all the workload, durations, and personnel assignments done at least one week in advance of theTCW.
• All requests for work during a TCW must be submitted to CAB no later than 2 meetings prior to the TCW. (Thisis typically 10 days.)
– Late requests that extend the TCW will be denied unless they are emergency.
– Late requests that fit within the planned window will be denied unless there is good reason for the lateness.Even so, they may be denied if personnel changes would be needed.
– Late requests that do not require a tree closure are up to IT to decide.
• All requests must include:
– Steps and duration for “all goes well”.
– Steps and duration for rollback.
– Assistance needed (specifically). E.g. “zeus expert” or “buildapi expert”
• IT will keep the Engineering informed of TCW status via the Engineering meeting run by Lawrence Mandel.(Typical Agenda)
• Announcements of TCW times will occur in EngOps meeting.
• A calendar of next two TCWs will be maintained, including tracker bugs.
• No later than 2 Fridays preceding the TCW (8 days), the composite time line will be established and documentedin an ether pad. This ensures accurate information can be provided at various engineering meetings the weekprior to the TCW.
9
RelEng Docs Documentation, Release 1.0
Note: Tracking Bug
The tracking bug summary should be updated with the planned start and completion times of theTCW.
Note: Releng Access
Not all RelEng folks are able to access the infra etherpads. The etherpad should be password pro-tected, and internet viewable, prior to the TCW.
Note: Etherpad Access
If there is a chance that Etherpad will be affected by the TCW, a Google Doc should be set up instead.
• No later than end-of-day Wednesday preceding the downtime, email announcement, cross posted to the appro-priate engineering lists, will be sent out. (Typically, that is dev-planning, dev-tree-management, dev-platform,dev-b2g, dev-gaia.)
3.1.2 Execution of TCW
We’ve had the most consistent success when the following practices were followed:
• Ether pad with name of person on the hook for each bug and support group created. Links and any specialcontact info should be in the pad.
• Planned timeline in the ether pad.
• Communication channels defined. Usually Vidyo (NOC), irc (#it, #infra), backup irc (freenode #mozilla-it).
• Designated person to run the TCW. Usually the oncall SE (if they don’t have too much work to do during TCW).
• Etherpad kept current with status, including times.
• All people involved with a tree closing part of the window remain on call until the trees are reopened.
3.2 Tree Closing Window Execution
The checklist below is for before, during, and after a Tree Closing Window (TCW). These are the actions for almostany TCW. Most will benefit from specific additions.
3.2.1 Wednesday Before
Make sure all communications have gone out from the Planning Procedure.
Double check all bugs to be included, make sure you know how to recover from potential issues. The CAB list is the“source of truth”.
3.2.2 Day of TCW
Check and screenshot various dashboards to see what is current “normal”.
• Check nagios service dashboard
10 Chapter 3. Release Engineering Procedures
RelEng Docs Documentation, Release 1.0
• Screenshot nagios tacktical dashboard
• Screenshot slavehealth
(optional) post message in IRC channels in advance. Usually #mobile, #b2g, #developers, #releng, and #gaia. Sample:
REMINDER - Trees close in about 1 hour for https://bugzil.la/1087431
Pull up local copies of all bugs and the etherpad, in case of network issues (planned, or unplanned)
Log in to the primary and backup IRC channels, see IT IRC usage, make sure you have latest passwords.
Touch base with the MOC “on duty” person about 15 minutes before scheduled start of TCW.
Close the trees with the tracker bug URL mentioned. (For “non-tree closing” TCW, select all the open branches andadd the message “TCW in process, devs need to handle their own restarts”, and “open” them saving state.)
3.2.3 When TCW Done, Before Opening Trees
Check nagios dashboard that all is as expected.
Check build API for pending, running, and recent to ensure those system are up.
Check tbpl to ensure it is up.
3.2. Tree Closing Window Execution 11
RelEng Docs Documentation, Release 1.0
12 Chapter 3. Release Engineering Procedures
CHAPTER 4
Hosts
4.1 hg.mozilla.org
Hooks: http://hg.mozilla.org/hgcustom/hghooks/
13
RelEng Docs Documentation, Release 1.0
14 Chapter 4. Hosts
CHAPTER 5
Software
5.1 Buildbot
5.1.1 Buildbot Overview
If you are new to Buildbot, please follow this excellent tutorial: Buildbot in 5 min. If you end up lost or want moreinformation on Buildbot itself, check out the full Buildbot Docs.
Buildbot automates the following repetitive process:
1. recognizing when changes land in your application’s source code
2. building/installing your software against changed source code across all supported platforms
3. running tests on the newly build software
4. storing the output and results (status) of how everything went.
By no means is it restricted to this but that’s a general use case of Buildbot.
Now let’s take a practical example in Mozilla where this would apply:
1. A developer pushes a commit to the mozilla-central repo.
2. Firefox is then installed on all our supported versions of Windows, Mac os x, Linux, and Android.
3. All tests and profiling suites (mochitests, reftests, talos, etc) are then ran against each newly build Firefox.
4. Logs are uploaded to TBPL, with status of how everything went.
Buildbot has a concept of masters and slaves. As the names imply, the masters are the brains, and the slaves are theheadless chickens who are told what to do.
Let’s take a simple scenario. You have a few machines with Buildbot installed and you construct them as “BuildbotSlaves”. Then, on another machine, you construct a “Buildbot Master”. The master and slaves connect and the masterwill eventually do things like ‘hey slave, install Firefox against this revision of source’. Slaves don’t know how todo this and it will be up to the master to communicate how that’s done with specific commands(steps). Generally,Buildbot Masters are configured so that they know how to do everything: what repos to watch, how to prioritize andschedule builds, what slaves it has in its control and what builds they are capable of building.
5.1.2 Mozilla Releng and Buildbot
RelEng is currently using a patched version of buildbot 0.8.2. Our repository is located athttp://hg.mozilla.org/build/buildbot.
15
RelEng Docs Documentation, Release 1.0
Upstream buildbot is currently located at https://github.com/buildbot/buildbot/.
universal master; scheduler master; build/test master
slave deployment
https://wiki.mozilla.org/ReleaseEngineering/Landing_Buildbot_Master_Changes
5.2 hg pushlog
Mozilla maintains a record of when changes are pushed into various repositories on hg.mozilla.org. This is currentlyimplemented in the pushlog.py hook
The pushlog has several interfaces of interest, the html view which is useful to developers, and the json format whichis more useful for automated systems.
5.3 buildbot schedulers
Buildbot schedulers are objects that are responsible for creating new build requests. Some usual examples of this arecreating the requests to do builds in response to a developer push, or doing nightly builds on a timer.
Schedulers themselves are run sequentially in a loop on the buildbot master. They are generally run on a timer, or if anew buildbot change is added to the master.
Schedulers that use push information generally look in the Scheduler Database for new changes since last time thescheduler ran. If there are any new changes that are applicable, the scheduler then creates new rows in the buildrequeststable of the Scheduler Database.
In addition to the built-in schedulers, RelEng maintains many custom buildbot schedulers.
aggregating scheduler
pgo scheduler
per-product scheduling
coalescing
5.3.1 Commonly Used Built-In Schedulers
Periodic: Runs a build at a certain interval, starting at time of builder creation (scheduler startup).
Nightly: Runs a build at a specific time.
5.4 HgPoller
http://hg.mozilla.org/build/buildbotcustom/file/a9f6adc7dcbb/changes/hgpoller.py
l10n
tipsonly
mergepushchanges
maxchanges
16 Chapter 5. Software
RelEng Docs Documentation, Release 1.0
5.5 postrun.py
postrun.py is run after most jobs. It is reponsible for creating the text log for the job, uploading it to ftp, updating theStatus Database and pushing the final notification events to pulse.
http://hg.mozilla.org/build/buildbotcustom/file/default/bin/postrun.py
5.6 Mozharness
Mozharness is a configuration driven script harness. It is a Script harness in that it knows how to automate a set oftasks. The scripts tend to not to need to know if you are running on Windows or Linux, nor does it know much aboutwhat tests or commands you need to run. The scripts get those varying details from a corresponding config (hence itbeing driven by configuration).
Let’s take a more concrete example of why you might use Mozharness. Let’s say you have a new test suite that youwant to start running against every new checkin of Firefox desktop across our continous integration for our variousrepositories. You know at a high level you need to do a number of things each time you run the tests:
1. Clear a work space so you are starting off fresh
2. Clone some repository that provides you with the tests you are going to call
3. Download a binary of Firefox to test against
4. The tests are in python and have some dependencies so you need to create a virtualenv and install some modules
5. Run the tests against the binary
6. Parse the output and interpret the return code
7. Log the results and and report some sort of overall status
Doing this for 10.8 OS X with mozilla-central in your local machine and with known static paths/packages might bepretty straight forward, however, this becomes a bit more complicated when you need to support X different platforms,over a dozen repositories, and a varying set of build types (e.g. pgo, debug, asan, etc).
Supporting all those variants can quickly make a script harness turn into a bag of snakes. I am going to prove this bycreating a new script for Mozharness which can simplify this and, through the process, provide an outline of the coremodules of mozharness and how you might go about creating or adding to an existing script for your own needs. Iencourage you to follow along and identify the core concepts with some mozharness coding.
So grab a copy of mozharness and let’s begin: The official repository location: http://hg.mozilla.org/build/mozharnessA mirror of the repo for git users: https://github.com/mozilla/build-mozharness
The File Structure:
|-- configs -> where all config files live that are used against individual scripts|-- docs|-- examples|-- external_tools|-- mozfile|-- mozharness| |-- base -> the core of mozharness and common tools for extending scripts| ‘-- mozilla -> common tools for extending scripts that are specific to Mozilla’s needs|-- mozinfo|-- mozprocess|-- scripts -> where all the callable scripts go‘-- test
Before we get to the scripts, I’d like to cover three classes that each correspond to a critical part of Mozharness:
5.5. postrun.py 17
RelEng Docs Documentation, Release 1.0
BaseLogger -> mozharness/base/log.py
BaseConfig -> mozharness/base/config.py
BaseScript -> mozharness/base/script.py
5.6.1 BaseLogger
BaseLogger provides a consistent logging for script runs:
13:13:19 INFO - #####13:13:19 INFO - ##### Running clobber step.13:13:19 INFO - #####13:13:19 INFO - Running main action method: clobber13:13:19 INFO - retry: Calling <bound method FxDesktopBuild.run_command of <__main__.FxDesktopBuild object at 0x23c38d0>> with args: [[’/builds/slave/ash-l64-0000000000000000000000/scripts/external_tools/clobberer.py’, ’-s’, ’scripts’, ’-s’, ’logs’, ’-s’, ’buildprops.json’, ’-s’, ’token’, ’-t’, ’168’, ’http://clobberer.pvt.build.mozilla.org/index.php’, u’ash’, u’Linux x86-64 ash build’, ’ash-l64-0000000000000000000000’, u’b-linux64-ix-0002’, u’http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/’]], kwargs: {’error_list’: [{’substr’: ’Error contacting server’, ’explanation’: ’Error contacting server for clobberer information.’, ’level’: ’error’}], ’cwd’: ’/builds/slave’}, attempt #113:13:19 INFO - Running command: [’/builds/slave/ash-l64-0000000000000000000000/scripts/external_tools/clobberer.py’, ’-s’, ’scripts’, ’-s’, ’logs’, ’-s’, ’buildprops.json’, ’-s’, ’token’, ’-t’, ’168’, ’http://clobberer.pvt.build.mozilla.org/index.php’, u’ash’, u’Linux x86-64 ash build’, ’ash-l64-0000000000000000000000’, u’b-linux64-ix-0002’, u’http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/’] in /builds/slave13:13:19 INFO - Copy/paste: /builds/slave/ash-l64-0000000000000000000000/scripts/external_tools/clobberer.py -s scripts -s logs -s buildprops.json -s token -t 168 http://clobberer.pvt.build.mozilla.org/index.php ash "Linux x86-64 ash build" ash-l64-0000000000000000000000 b-linux64-ix-0002 http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/13:13:22 INFO - Checking clobber URL: http://clobberer.pvt.build.mozilla.org/index.php13:13:22 ERROR - can not reach clobber URL13:13:22 INFO - b2g_b2g-in_ham_dep-00000000000:Our last clobber date: None
We always start with a timestamp and then a log level for each line. Log levels, by default, are debug, info, warning,error, critical, and fatal. Logs are outputted via BaseLogger.log_message()
I am going to show you a snippet of code from the BaseLogger only to show you what happens under the hood, Thisclass is rarely directly reached by Mozharness scripts.
snippet of BaseLogger mozharness/base/log.py:
class BaseLogger(object):"""Create a base logging class.LEVELS = {
DEBUG: logging.DEBUG,INFO: logging.INFO,WARNING: logging.WARNING,ERROR: logging.ERROR,CRITICAL: logging.CRITICAL,FATAL: FATAL_LEVEL
}
def __init__(self, log_level=INFO,log_format=’%(message)s’,log_date_format=’%H:%M:%S’,log_name=’test’,log_to_console=True,log_dir=’.’,log_to_raw=False,logger_name=’’,append_to_log=False,
):
self.all_handlers = []self.log_files = {}
self.create_log_dir()
def create_log_dir(self):if os.path.exists(self.log_dir):
18 Chapter 5. Software
RelEng Docs Documentation, Release 1.0
if not os.path.isdir(self.log_dir):os.remove(self.log_dir)
if not os.path.exists(self.log_dir):os.makedirs(self.log_dir)
self.abs_log_dir = os.path.abspath(self.log_dir)
def new_logger(self, logger_name):"""Create a new logger.By default there are no handlers."""self.logger = logging.getLogger(logger_name)self.logger.setLevel(self.get_logger_level())self._clear_handlers()if self.log_to_console:
self.add_console_handler()if self.log_to_raw:
self.log_files[’raw’] = ’%s_raw.log’ % self.log_nameself.add_file_handler(os.path.join(self.abs_log_dir,
self.log_files[’raw’]),log_format=’%(message)s’)
def log_message(self, message, level=INFO, exit_code=-1, post_fatal_callback=None):if level == IGNORE:
returnfor line in message.splitlines():
self.logger.log(self.get_logger_level(level), line)if level == FATAL:
if callable(post_fatal_callback):self.logger.log(FATAL_LEVEL, "Running post_fatal callback...")post_fatal_callback(message=message, exit_code=exit_code)
self.logger.log(FATAL_LEVEL, ’Exiting %d’ % exit_code)raise SystemExit(exit_code)
So how do we avail of this if we don’t call methods from it? LogMixin class provides helper methods for things likelog(msg, level) or, even simpler, self.{level}(msg) as in: self.info(msg) or self.error(msg) and BaseLogger inheritsthose methods. These are the ones you will likely use the most.
• A note about self.fatal(msg) or self.log(msg, FATAL): these methods will also cause the script to halt and exit
snippet of LogMixin mozharness/base/log.py:
class LogMixin(object):
def log(self, message, level=INFO, exit_code=-1):if self.log_obj:
return self.log_obj.log_message(message, level=level,exit_code=exit_code,post_fatal_callback=self._post_fatal,
)if level == INFO:
if self._log_level_at_least(level):self._print(message)
elif level == DEBUG:if self._log_level_at_least(level):
self._print(’DEBUG: %s’ % message)elif level in (WARNING, ERROR, CRITICAL):
if self._log_level_at_least(level):self._print("%s: %s" % (level.upper(), message), stderr=True)
5.6. Mozharness 19
RelEng Docs Documentation, Release 1.0
elif level == FATAL:if self._log_level_at_least(level):
self._print("FATAL: %s" % message, stderr=True)raise SystemExit(exit_code)
def debug(self, message):self.log(message, level=DEBUG)
def info(self, message):self.log(message, level=INFO)
def warning(self, message):self.log(message, level=WARNING)
# ... etc
One final thing worth mentioning here is that mozharness can also save a single log file, or even split your log intomultiple log files based on individual log levels. Since splitting the log into multiple files is the most common, I’llmention how that works. MultiFileLogger is a subclass of BaseLogger and does this work for you. The split logic willtake a script run and save the following:
logs/ -> default log path unless you overwrite itlog_info.log -> contains every single line of outputlog_warning.log -> contains only warning and worse (error, crit, fatal) lines of outputlog_error.log -> contains only error and worse...log_critical.log -> ... and so onlog_fatal.log
So how do you add logging to your script? Adding the logging module to your script is already done for you if youavail of BaseScript (you pretty much always will want to). BaseScript connects all core parts of Mozharness and wewill dive into that shortly.
5.6.2 BaseConfig
This is the class that will interpret all of your configuration from many different sources. These sources could be CLIarguments, json or python (dict) files, remote url files, or a static configuration (dict) inputted directly from the script.
BaseConfig provides a constant hierarchy across your scripts so if you have multiple duplicate keys from varioussources, a precedence will decide what you end up with.
• the hierarchy from highest to lowest: CLI options -> config files -> static from script
Not sure what to put in a config file or how to extend CLI options? Don’t worry, we will be doing a full exampleshortly.
After BaseConfig constructs what your config will ultimately look like against a given script run, it will lock the itemsso it becomes an immutable dict. This expresses how a config is what drives the job, not the script. It is essentiallyread only dict so feel free to use config[key] and config.get(key, default) syntax.
Finally, BaseConfig also interprets what Actions (steps of the job) that will be run. Defining and understanding actionswill be explained in the BaseScript section.
snippet of BaseConfig mozharness/base/config.py:
class BaseConfig(object):"""Basic config setting/getting."""def __init__(self, config=None, initial_config_file=None, config_options=None,
all_actions=None, default_actions=None,
20 Chapter 5. Software
RelEng Docs Documentation, Release 1.0
volatile_config=None, option_args=None,require_config_file=False, usage="usage: %prog [options]"):
# ...# ...if initial_config_file:
initial_config = parse_config_file(initial_config_file)self.all_cfg_files_and_dicts.append(
(initial_config_file, initial_config))self.set_config(initial_config)
if config_options is None:config_options = []
# CREATES AN OPTION PARSER FOR OUR cli ARGSself._create_config_parser(config_options, usage)# PARSE THE ARGS THAT WERE GIVEN FOR THE CURRENT SCRIPT CALL# AND INTERPRET ANY CONFIG FILES USEDself.parse_args(args=option_args)
def get_read_only_config(self):return ReadOnlyDict(self._config)
def _create_config_parser(self, config_options, usage):self.config_parser = ExtendedOptionParser(usage=usage)self.config_parser.add_option(
"--work-dir", action="store", dest="work_dir",type="string", default="build",help="Specify the work_dir (subdir of base_work_dir)"
)# ...# ... more default options for your scripts# ...
def parse_args(self, args=None):self.command_line = ’ ’.join(sys.argv)if not args:
args = sys.argv[1:](options, args) = self.config_parser.parse_args(args)
defaults = self.config_parser.defaults.copy()
if not options.config_files:# SOMETIMES WE DON’T ALWAYS NEED A CONFIG FILEif self.require_config_file:
# BUT WE CAN FORCE THE REQUIREMENT TO HAVE ONEif options.list_actions:
self.list_actions()print("Required config file not set! (use --config-file option)")raise SystemExit(-1)
else:# INTERPRET THE CONFIG FILE(S) AND THEN ADDD THAT TO SELF.CONFIGself.all_cfg_files_and_dicts.extend(self.get_cfgs_from_files(
# append opt_config to allow them to overwrite previous configsoptions.config_files + options.opt_config_files, parser=options
))config = {}for i, (c_file, c_dict) in enumerate(self.all_cfg_files_and_dicts):
config.update(c_dict)self.set_config(config)
5.6. Mozharness 21
RelEng Docs Documentation, Release 1.0
# MAKE SURE THAT DEFAULT OPTIONS ARE OVERRIDDEN BY CONFIG FILE OPTIONS AND PARSER OPTIONSfor key in defaults.keys():
value = getattr(options, key)if value is None:
continue# Don’t override config_file defaults with config_parser defaultsif key in defaults and value == defaults[key] and key in self._config:
continueself._config[key] = value
# ...# ...# determine action details from configuration. more on that later# ...# ...
self.options = optionsself.args = argsreturn (self.options, self.args)
Like BaseLogger, BaseScript will instantiate BaseConfig and attach itself as an attr so you won’t have to call BaseC-onfig directly.
• Wondering what your config will look like if you only inherit from BaseScript and don’t extend your script withany CLI or config files:
# defaults{’append_to_log’: False, # whether you want to start your log files cleanly or append to prev run’base_work_dir’: ’~/devel/mozilla/dirtyRepos/mozharness_jlund’, # path you call the script from’log_level’: ’info’, # what default level you want to start at’log_to_console’: True,’opt_config_files’: (), # a list of config files passed for the run’volatile_config’: {’actions’: None, ’add_actions’: None, ’no_actions’: None},# used by BaseConfig to determine what actions to run’work_dir’: ’build’ # the dirname of where you will put and run things. e.g. downloads/src/artifacts}
5.6.3 BaseScript
You may have an idea now that BaseScript is where everything comes together. By inheriting and instantiatingBaseScript, you get your logging obj (self.log_obj), your configuration (self.config), and your actions used for thescript (self.actions). You should be familiar about logging and configuration so let’s discuss actions.
Actions express the list of steps for a job on a given run. Think ‘remove tree’, ‘clone something’, ‘run this test suite’,‘clean up’. Essetially self.actions is a list:
[’clobber’, ‘clone’, ‘run-tests’, ‘clean-up’]
What happens is when you call BaseScript.run_and_exit(), Mozharness will run through each action in the list and lookfor a corresponding method within scope of your script class. e.g. when we get to the ‘clone’ action in self.actions,BaseScript will look for self.clone() and execute that method.
• note about actions names: when the action name uses a hyphen, e.g. ‘run-tests’, BaseScript will replace the ‘-‘with a ‘_’ so it will look for self.run_tests().
In addition to running actions, BaseScript also has an overall status: self.return_code. This value can be manipulatedas the script runs so you can keep track of how your script did if you do not want to halt early or the overall returnvalue is swallowed downstream.
22 Chapter 5. Software
RelEng Docs Documentation, Release 1.0
BaseScript has a few ‘helper’ methods itself but it leverages from one of the more powerful Mixins in Mozharness:BaseMixin. BaseMixin is aimed to provide you with a set of tools for doing common tasks: e.g. sys admin, networking,subprocess commands. It does so but aims to be platform agnostic while incorporating Mozharness’s self.log_obj andself.config.
BaseScript mozharness/base/script.py:
class BaseScript(ScriptMixin, LogMixin, object):def __init__(self, config_options=None, ConfigClass=BaseConfig,
default_log_level="info", **kwargs):super(BaseScript, self).__init__()
self.return_code = 0# HERE IS WHERE WE INSTANTIATE THE CONFIG (99% OF THE TIME bASEcONFIG)rw_config = ConfigClass(config_options=config_options, **kwargs)self.config = rw_config.get_read_only_config()# WE DERIVE OUR LIST OF ACTIONS WE WANT TO USE FOR SCRIPT CALLself.actions = tuple(rw_config.actions)# here is where we create our log_obj (a subclass of BaseLogger)self.log_obj = Noneself.new_log_obj(default_log_level=default_log_level)
# ADD A DECORATOR METHOD THAT WE CAN USE IN OUR SCRIPT IF WE WANT TO CHANGE# SELF.CONFIG BEFORE LOCKING IT FOR GOODself._pre_config_lock(rw_config)# SET SELF.CONFIG TO READ-ONLY.self._config_lock()
def run_and_exit(self):"""Runs the script and exits the current interpreter."""sys.exit(self.run())
def run(self):# VERY SIMPLIFIEDtry:
for action in self.all_actions:self.run_action(action)
except Exception:self.fatal("Uncaught exception: %s" % traceback.format_exc())
if self.config.get("copy_logs_post_run", True):self.copy_logs_to_upload_dir()
return self.return_code
def run_action(self, action):# AGAIN SIMPLIFIED DRAMATICALLYif action not in self.actions:
self.action_message("Skipping %s step." % action)return
method_name = action.replace("-", "_")try:
self._possibly_run_method(method_name, error_if_missing=True)
5.6. Mozharness 23
RelEng Docs Documentation, Release 1.0
5.6.4 Mozharness Example
Before we dive into the example, I’d like to outline some common built in CLI args you can use to explore the conceptsmentioned above:
With any script, you can run –help to see a list of options you can pass:
for modifying/listing self.config:
--work-dir=WORK_DIR Specify the work_dir (subdir of base_work_dir)--base-work-dir=BASE_WORK_DIR
Specify the absolute path of the parent of the workingdirectory
-c CONFIG_FILES, --config-file=CONFIG_FILES, --cfg=CONFIG_FILESSpecify the config files
-C OPT_CONFIG_FILES, --opt-config-file=OPT_CONFIG_FILES, --opt-cfg=OPT_CONFIG_FILESSpecify the optional config files
--dump-config List and dump the config generated from this run to aJSON file.
--dump-config-hierarchyLike dump config but will list and dump which configfiles were used making up the config and specify theirown keys/values that were not overwritten by anothercfg -- held the highest hierarchy.
for modifying self.log_obj:
--log-level=LOG_LEVELSet log level(debug|info|warning|error|critical|fatal)
-q, --quiet Don’t log to the console--append-to-log Append to the log--multi-log Log using MultiFileLogger--simple-log Log using SimpleFileLogger
for modifying/listing self.actions:
--list-actions List all available actions, then exit--add-action=ACTIONS
Add action [’clobber’, ’nap’, ’ship-it’] to the listof actions
--no-action=ACTIONSDon’t perform action
--{action} for any action the script knows about, pass it explicitly and the scriptwill only run that action
• pro learning tip: use –list-actions –dump-config and –dump-config-hierarchy
They are all great ways of interpreting what actions will be called or what self.config will look like based on theoptions and config files passed to a script run. Running any one of those three against a script + other options won’tcause any actions to be run so they are not dangerous.
Ok, so how can we put all these concepts together in some trivial script? Lucky for us, there is already a committedexample we can use in the mozharness repo
ActionsConfigExample examples/action_config_script.py:
sys.path.insert(1, os.path.dirname(sys.path[0]))# MESSING WITH SYS.PATH LIKE ABOVE IS A NORMAL IDIOM SO WE CAN REACH MOZHARNESS/* BELOWfrom mozharness.base.script import BaseScript
24 Chapter 5. Software
RelEng Docs Documentation, Release 1.0
# ActionsConfigExample {{{1class ActionsConfigExample(BaseScript): # HERE IS WHERE WE INHERIT BASESCRIPT INTO OUR CLASS
config_options = [[ # WE ADD SOME OF OUR OWN OPTIONS IN ADDITION TO WHAT WE GET FROM DEFAULT[’--beverage’, ],{"action": "store","dest": "beverage","type": "string","help": "Specify your beverage of choice",}
], [[’--ship-style’, ],{"action": "store","dest": "ship_style","type": "choice","choices": ["1", "2", "3"],"help": "Specify the type of ship",}
], [[’--long-sleep-time’, ],{"action": "store","dest": "long_sleep_time","type": "int","help": "Specify how long to sleep",}
]]
def __init__(self, require_config_file=False):# OUR ActionsConfigExample MERELY INSTANTIATES BaseScripts __init__super(ActionsConfigExample, self).__init__(
config_options=self.config_options, # PASS IN THE ADDITIONAL CLI OPTIONS# THESE ARE ALL THE ACTIONS THAT ARE POSSIBLE TO RUN. BaseScript WILL VERIFY IT# CAN SEE A METHOD FOR EACH OF THESEall_actions=[
’clobber’,’nap’,’ship-it’,
],# IF WE DON’T SPECIFY WHICH ACTIONS TO RUN IN A CONFIG OR CLI, DO THESE DEFAULT ONESdefault_actions=[
’clobber’,’nap’,’ship-it’,
],# IF YOUR SCRIPT REQUIRES A CONFIG FILE, YOU CAN USE REQUIRE_CONFIG_FILErequire_config_file=require_config_file,# this is our default config (what will be added to self.config)# remember keys from config files will take precedence over these defaults# CLI options like the cooresponding --beverage will take precendence over allconfig={
’beverage’: "kool-aid",’long_sleep_time’: 3600,’ship_style’: "1",
})
# HELPER METHODS USED BY MAIN ACTIONS. FOR NOW LET’S SKIP OVER _SLEEP AND THE SHIP() METHODSdef _sleep(self, sleep_length, interval=5):
self.info("Sleeping %d seconds..." % sleep_length)
5.6. Mozharness 25
RelEng Docs Documentation, Release 1.0
counter = 0while counter + interval <= sleep_length:
sys.stdout.write(".")try:
time.sleep(interval)except:
printself.error("Impatient, are we?")sys.exit(1)
counter += intervalprintself.info("Ok, done.")
def _ship1(self):self.info("""
_~_~ )_)_~)_))_))_)_!__!__!_\______t/
~~~~~~~~~~~~~""")
# ... ship2() impl# ... ship3() impl
# AH, NAP() OUR FIRST DEFINED ACTION! But where’s clobber() you might ask? Remember actions# only need to be part of self. We have base impl of clobber() in BaseScript. If that impl# suits your needs, no need to overwrite in this class :)def nap(self):
for var_name in self.config.keys():if var_name.startswith("random_config_key"):
# LOOK, OUR FIRST USE OF SELF.LOG_OBJ. REMEMBER, THIS IS A CONVENIENCE METHOD# self.info(msg)self.info("This is going to be %s!" % self.config[var_name])
# HERE WE ARE POLLING SELF.CONFIG. REMEMBER IT IS READ ONLY AND AVAILABLE AFTER# BaseScript.__init__()sleep_time = self.config[’long_sleep_time’]if sleep_time > 60:
self.info("Ok, grab a %s. This is going to take a while." % self.config[’beverage’])else:
self.info("This will be quick, but grab a %s anyway." % self.config[’beverage’])self._sleep(self.config[’long_sleep_time’])
# NAP TIME IS OVER, TIME TO SHIP IT! OUR FINAL ACTION.def ship_it(self):
name = "_ship%s" % self.config[’ship_style’]if hasattr(self, name):
# WE USE SELF.CONFIG[’SHIP_STYLE’] TO CALL THE APPROPRIATE HELPER METHOD.# take a moment to figure out what this will be if you do not pass --ship-style# or pass a separate config file with ’ship_style’ in itgetattr(self, name)()
# __main__ {{{1if __name__ == ’__main__’:
actions_config_example = ActionsConfigExample()# AHA, THIS IS THAT METHOD I MENTIOND AND ANOTHER COMMON IDIOM ON HOW WE KICK OFF OUR SCRIPTS
26 Chapter 5. Software
RelEng Docs Documentation, Release 1.0
actions_config_example.run_and_exit()
Once you’ve had a read through, play with this script. It’s harmless. :)
Try the following calls and see if you can determine why you got the results you did:
# what actions ran with these calls? How did the behaviour of the actions change? Notice how# the script code itself has very minimal conditions and never changes. The config decides all
# the default nap time is an hour, you might want to ctrl-c this one after a few seconds :)python examples/action_config_script.py
python examples/action_config_script.py --long-sleep-time 5
python examples/action_config_script.py --long-sleep-time 3 --ship-style 2
python examples/action_config_script.py --ship-it --ship-style 2
# this one requires you to make a dummy config file. create a py file that just has the# following contents: config = {’ship_style’: 3, ’default_actions’: [’ship-it’]}python examples/action_config_script.py --cfg path/to/your/dummy_config.py
Congratulations! If you have followed along, you pretty much understand the core required to read any mozharnessscript.
Where do you go from here you might ask?
Here’s some things you can do:
1. BaseMixin provides a number of great helper methods and default actions. Poke through them e.g.run_command is your subprocess friend; it may be long but it should be able to handle all of your externalcommands you need to run
2. outside of BaseMixin, there is a ton of other mixins and base classes you can use for extending the actions at yourdisposal. Peek in mozharness/base/* and mozharness/mozilla/* . You’ll likely find methods for achieving yourrequirements so you don’t even need to impl any new actions. e.g. create_virtualenv(), clone(), setup_mock(),download_file(), make_gaia()
3. check out the actual scripts/* . Mozharness is used for ~90% of all our jobs done in tbpl/treeherder. Want tocorrelate a script to one of those jobs? Open up a log, and grep the mozharness call, likely ‘scripts/’ or look forthe starting log output (timestamp and log level at the start of each line)
4. take a look at the mozharness FAQ https://developer.mozilla.org/en-US/docs/Mozharness_FAQ
5. read some blog posts http://escapewindow.dreamwidth.org/tag/mozharness
6. ready to contribute or work on this project? Check out https://wiki.mozilla.org/Auto-tools/Projects/Mozharness
There is a lot more we can discuss, e.g. using pre and post listeners for setting up or tearing down actions, decoratingpre_config_lock() to manipulate self.config before it locks for the whole script run, and passing more than one configfile to a single script run. However I think that goes beyond the requirements for navigating or contributing to theMozharness code base.
5.7 TBPL
https://wiki.mozilla.org/Sheriffing/TBPL
https://tbpl.mozilla.org/
https://wiki.mozilla.org/Tinderboxpushlog
5.7. TBPL 27
RelEng Docs Documentation, Release 1.0
https://hg.mozilla.org/webtools/tbpl/file/tip/README
https://wiki.mozilla.org/TbplWebsiteDoc
?showall=1 ?jobname=foo
https://hg.mozilla.org/webtools/tbpl/
treeherder
5.8 buildapi
http://hg.mozilla.org/build/buildapi (buildapi docs)
https://wiki.mozilla.org/ReleaseEngineering/BuildAPI
5.9 Cloud tools
http://hg.mozilla.org/build/cloud-tools/
5.9.1 aws_watch_pending
http://hg.mozilla.org/build/cloud-tools/file/1e02720fa840/aws/aws_watch_pending.py
5.9.2 aws_stop_idle
http://hg.mozilla.org/build/cloud-tools/file/1e02720fa840/aws/aws_stop_idle.py
5.10 VCS Sync tools
5.10.1 legacy
The legacy (first implementation) code is in: http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/
The legacy configurations are in: http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-configs/
Documentation is in the code repository, a rendered version of the latest is athttps://people.mozilla.org/~hwine/tmp/vcs2vcs/index.html
5.11 Databases
28 Chapter 5. Software
CHAPTER 6
Scheduler Database
http://hg.mozilla.org/build/buildbot/file/b4673f1f2a86/master/buildbot/db/schema/tables.sql
https://wiki.mozilla.org/ReleaseEngineering/Buildbot_Database_Schemas
6.1 build requests
The buildrequests table in the schedulerdb is one of the core tables for buildbot. buildbot masters use this table to findnew jobs to run (with get_unclaimed_buildrequests)
Column documentation:
• buildername: this corresponds directly to the buildbot buildername. masters poll the table looking forpending jobs for builders they have enabled.
• priority: this affects the order in which pending build requests are processed. See builderPriority
TodoWrite up builder priority Include link to definitions, and intended usage.
• claimed_at: this is a unix timestamp. masters update this field periodically as a job is running. It is notwhen the job starts. If a master dies or hangs, it will no longer be updating this field, and so other masterswill be free to steal the job.
• claimed_by_name: The master’s hostname and path that has this job claimed. This andclaimed_by_incarnation are used to determine when requests can be stolen.
• claimed_by_incarnation: The master’s process id and timestamp of when the process started. If a mastersees requests that are claimed by itself (claimed_by_name), but a different claimed_by_incarnation, thenit knows that it can immediately steal the request instead of waiting for the timeout.
• complete: Set to 1 if the build is complete
• results: The result code of the build. RETRY doesn’t appear here, rather the build is set as unclaimedagain.
• submitted_at: Timestamp when this request was submitted.
• complete_at: Timestamp when this request finished.
29
RelEng Docs Documentation, Release 1.0
6.2 pending jobs
Jobs are pending when they are marked as not complete in the DB, and no master currently has them claimed.
TBPL fetches pending jobs from https://secure.pub.build.mozilla.org/builddata/buildjson/builds-pending.js, which ispopulated from https://secure.pub.build.mozilla.org/buildapi/pending?format=json
6.3 running jobs
Jobs are running when they are currently claimed by a master and are running on a slave.
TBPL fetches pending jobs from https://secure.pub.build.mozilla.org/builddata/buildjson/builds-running.js, which ispopulated from https://secure.pub.build.mozilla.org/buildapi/running?format=json
30 Chapter 6. Scheduler Database
CHAPTER 7
Status Database
http://hg.mozilla.org/build/buildbotcustom/file/a9f6adc7dcbb/status/db/model.py https://wiki.mozilla.org/ReleaseEngineering/Buildbot_Database_Schemas
7.1 finished jobs
When jobs complete, their information is stored into the status db.
Status of finished jobs is published to http://builddata.pub.build.mozilla.org/buildjson/ from buildapi. TBPL consumeshttp://builddata.pub.build.mozilla.org/buildjson/builds-4hr.js.gz to find when jobs have completed.
31
RelEng Docs Documentation, Release 1.0
32 Chapter 7. Status Database
CHAPTER 8
Releng 101
8.1 What is this?
• a primer for understanding the Mozilla Release Engineering system. It strips down our infrastructure into bareessentials
• a series of ‘walk-throughs’ to help connect our concepts, technologies, environments, and src code
• provides stripped down snippets that represent real source from our repos.
8.2 What this is not
• documentation. We are moving with a strong emphasize to documenting our logic in the src itself. These aretutorials that can not be expressed in the code
• explanation of technologies we use. Each walk-through will explain how we use technologies but will point toSoftware for overviews
• an in depth coverage of all our special edge cases, variants, conditions, and the like.
8.3 I hope this provides
1. a life jacket for Mozilla Releng contributors and new-hires
2. a bridge between the abstract concepts and the physical code
3. confidence for exploring every dark corner of our releng universe
8.4 Releng in a Nutshell
To get the ball rolling, below are a collection of materials that will help provide a Releng Overview
• Release Engineering as a Force Multiplier – John O’Duinn’s Keynote at ICSE 2013
• Keep Calm and Ship It – Mozilla Releng through John Zeller’s 2012 intern presentation
• Mozilla’s cloud and in-house continuous integration – Armen Zambrano’s Releng Conf 2014 talk on recentcloud integration
33
RelEng Docs Documentation, Release 1.0
• Planet Releng – an aggregate of all Mozilla Releng’s blog sites
8.5 The Walk-through series at a glance
1. From a change on a repo to builds being triggered - a tour through Buildbot (latest update if your not readingthis online.)
TodoWrite the rest of these.
2. Building Firefox in automation - harnessing Mozharness
3. Setting up and configuring our machines - becoming the master of Puppet
4. Monitoring and nagging when things go wrong - keeping an eye on our machines with BuildAPI, SlaveAPI,Slave Health, and Nagios
5. Integrating the Cloud - a look at our Cloud Tools
6. Release Process - explains why we hold the title of ‘Release Engineers’
7. Handling application updates - Balrog to the rescue
8. Serving your own machines - loaning and allocating our machines through Selve-serve
9. Syncing with HG and Git - understanding vcs-sync
10. Where to go from here - tips on exploring our infrastructure: wikis, bugzilla, mxr, emails, irc, etc
11. A look into the future - what’s up and coming: taskcluster, relengAPI, etc
Template for new pages:
• Software:
• Repos:
• Purpose of walk-through:
34 Chapter 8. Releng 101
CHAPTER 9
Coding Guidelines
This section describes current (2014 May) practices. It will be the basis for future changes.
All of the guidelines below are for new projects. For existing projects, follow the current style or migrate the entireproject to a newer approach.
9.1 Code Storage
The most important thing is to have your code in VCS and somewhere other than your laptop. Beyond that, you’re freeto develop wherever you want using the tools you want. As something moves towards being deployable, the master(Repository of Record - RoR) repository needs to meet the following criteria, mostly based on deployment tool needs:
Deployed via repository checkout:
• Must be in either hg.mozilla.org (historical home), or git.mozilla.org (if git client installed ontarget).
• Examples include buildbot code.
Deployed via puppet:
• Source should be in separate repository with documentation, tests, and other goodness.
• Source repository should be *.mozilla.org server.
– Historically, some were (only) in end user repos on github.
• Only a file (or other artifact) produced from the source repository should be deployed by puppet.Puppet only changes are discouraged.
• Examples include tooltool script, buildapi sdist tarballs, custom MRTG tests.
Other cases:
• Handle as under “Deployed via puppet”
• Examples? unknown
9.2 Scripting Languages
1. Python is our tool of choice for anything significant or long lived.
• PEP-8 is our guideline, except for line length.
35
RelEng Docs Documentation, Release 1.0
• if mozharness makes sense, prefer that.
• prefer modules with entry points over scripts (works on Windows)
2. Bash scripts are acceptable for “pure unix” items like cronjobs.
3. Various legacy systems are in PHP or Perl. Let’s not do that again without consent.
9.3 Configuration Formats
At the moment, the bulk of our tooling uses:
• python syntax & constructs (buildbot)
• JSON (for host descriptions, web APIs)
• INI format (git, hg)
9.4 Web Application Frameworks
Ideally, “everything is a web service”, so we can maximize self-service opportunities.
• Flask is preferred for new work.
– If your app works as a blueprint, we have more flexibility in deployment options. That is a good thing.
– Examples include relengapi, ship it, balrog
– common flask extensions include:
* flask-wtf for form validation
• Pylons has been used.
– Examples include buildapi
• Twisted
– Examples include buildbot
9.5 Futures
Here are some ideas we’d like to add in the future. If you need something like this, ask around.
• Vagrant/docker development (at least) environments.
– Ideally part of source code repository.
– Still working on how to set up multi-app environments.
• Deployment Checklist
– Build/test on commit requirements
– Operations documentation.
36 Chapter 9. Coding Guidelines
CHAPTER 10
Debugging “jobs aren’t starting”
One of the more common types of problems we encounter is when jobs aren’t starting properly. This document is anattempt to guide debugging efforts.
“jobs not starting” can mean a few things.
1. jobs are pending, but not running. In this case the jobs are visible as pending on TBPL or elsewhere.
2. jobs aren’t being scheduled properly. In this case, the jobs aren’t visible as pending anywhere. Their triggersappear to be working, but the jobs themselves aren’t being created.
10.1 pending jobs aren’t starting
In the first case, the jobs are being created in the database properly, but aren’t being started.
This could be due to invalid buildbot configuration or reconfig.
Some possible issues:
• Were the scheduler masters reconfigured but not the build/test masters? Perhaps the reconfig of the build/testmasters failed?
• Does any master actually run those builders? Check allthethings.json. Maybe production-masters.json wasn’tupdated with the appropriate platforms or branches?
If buildbot has the builder enabled, but jobs aren’t starting, next check twistd.log on the master. Some things to lookfor:
• “prioritizeBuilders: ... removed builder XXX with no allocated slaves available”. This means there were pend-ing jobs for XXX, but no slaves attached to the master that could do it.
• “<Builder ‘’XXX” at YYYYYY>: want to start build, but we don’t have a remote”. Similar to above. No slaveswere found to do the job.
These messages can mean that there are no slaves attached and idle to do the job, but it can also mean that the slaveselection function (aka “nextSlave”) isn’t selecting one of the connected slaves for some reason. Other things to watchout for:
• Is the builder in a jacuzzi or not? Check the list of jacuzzi’ed builders. See alsohttps://wiki.mozilla.org/ReleaseEngineering:Jacuzzis for debugging jacuzzis. Perhaps the allocation istoo small, in which case all the specified slaves would be online and busy. To address this, increase theallocation for this builder.
Perhaps the allocated slaves aren’t functional for some reason. Check that they’re being started correctly inaws_watch_pending.log and the AWS console.
37
RelEng Docs Documentation, Release 1.0
• “nextAWSSlave Waiting for inhouse slaves to show up”. There’s a short (60s) timeout where we’ll try and waitfor any in-house slaves to take the job before giving it to an AWS slaves.
Jobs not starting can also be a symptom of general capacity issues.
First thing to do is identify what kinds of machines aren’t starting, and if they have anything in common. For example,is the problem specific to AWS machines? AWS machines of a particular type? Test slaves? Tegras? Be on the lookoutfor exceptions to your hypothesis, as the exceptions often indicate where the root of the problem is.
AWS slaves not starting? Try checking:
• aws_watch_pending.log. Any unhandled errors? We could be at our capacity limits. Perhaps spot prices haverisen everywhere and we aren’t able to get enough instances. We have self-imposed limits in watch_pending.cfg.In addition, each AWS slave needs a buildbot name assigned to it, so the list of machines in slavealloc andbuildbot-configs needs to be large enough.
• AWS console. Particularly check the spot instances section. Errors creating spot instances aren’t immediatelyapparent in aws_watch_pending.log
aws_watch_pending.log is available on aws-manager1.srv.releng.scl3.mozilla.com:/builds/aws_manager/aws_watch_pending.log
10.2 jobs aren’t being scheduled
The most likely thing here is a scheduling issue. Perhaps the jobs were inadvertently removed by a recent patch. Inthis case one particular type of job will be missing, but other types will be fine.
The other likely culprit is the scheduler masters are hung or really busy. In this case, new jobs will be very slow to bescheduled. Possible causes for this include:
• Network between scheduler masters and DB is bad
• DB is slow, or blocked by some long running queries
• Scheduler masters are stuck processing lots of old changes. This can happen if we re-enable a scheduler that hasbeen disabled for a while. Normally these should be cleaned up, but it’s possible the cleanup process missed it,or it wasn’t old enough to be deleted, but was old enough to cause problems.
38 Chapter 10. Debugging “jobs aren’t starting”
CHAPTER 11
Adding a new repo to Releng Account on Read-The-Docs
At the moment, our approach is to:
• Enable rtfd.org updates on the github copy of the repo.
• Add repo to Releng Account on rtfd.org.
• Assign maintainers to the repo on rtfd.org.
11.1 Pre-requisites
Before you can add a repo you should:
• Already have configured docs in the repo. (See XXX)
• Successfully built docs locally. (See XXX)
You also need:
• Access to RelEng passwords.
• Admin access for repo to be added on github.
11.2 Enable RTFD updates on github
• Visit the ‘settings’ link for the repository.
• Choose the “Webhooks & Services” link.
• Click the “Configure services” button.
• Select “ReadTheDocs” from the service list.
• Enable the “Active” checkbox.
• Click the “Update Settings” button.
• Click the “Configure services” button again, and verify that “ReadTheDocs” is checked.
11.3 Add repo to Releng Account on RTFD
Set up the doc project for the repository:
39
RelEng Docs Documentation, Release 1.0
• Log in to rtfd.org as user moz_releng.
• Click the “Import” button.
• Fill in a name – names are globally unique on all of rtfd.org, so we’ve been prefixing with “moz releng”.
• Fill in the URL it should pull from.
• Click the “Create” button. You should see a message that your documentation is being built.
Add the new doc project as a subproject to the main project:
• Click the “Dashboard” link.
• Click the “Admin” button for the “moz releng docs” project.
• Follow the “Subprojects” link.
• Add the new project as a subproject, and click “Submit”.
11.4 Assign maintainers to the repo on RTFD
• Log in to rtfd.org as user moz_releng (or yourself, if you’re already a maintainer of the repository in ques-tion).
• From the dashboard, click the “Admin” link for the repository.
• Follow the “Maintainers” link.
• Add the rtfd.org user names to be admins of this repo’s documents
40 Chapter 11. Adding a new repo to Releng Account on Read-The-Docs
CHAPTER 12
Adding Sphinx docs to Existing Repositories
So, you need to add some docs for code that we’ve already written. You’ve come to the right place!
There are two common “setup” scenarios:
• Project with no Sphinx docs
• Project with existing Sphinx docs, but no docs from code
No mater where you start, there are some common enhancements to improve the generated output.
In a nutshell, we use Sphinx with documentation placed in the docs directory off the root of the project.
12.1 Adding Sphinx to a project with no docs yet
Make sure Sphinx is installed on your machine (it shouldn’t be in your project’s virtual environment). From the toplevel project directory, run:
sphinx-apidoc -F -A "RelEng Team" -V "0.1" -o docs $python_module_name
If you happen to have an early stage project, without a python module directory yet, GOOD FOR YOU!!! You’ll justdo the normal sphinx-quickstart. Please keep the defaults, except for ‘autodoc’ and ‘viewcode’ (say ‘Y’es toboth):
sphinx-quickstart docs
No matter which way you add Sphinx, don’t forget to add docs/_build to the project’s .hgignore or.gitignore file.
12.2 Adding autodoc to a project already using Sphinx
The following command should maintain you existing docs, while adding the generated code documentation. Fromthe top directory:
sphinx-apidoc -o docs $python_module_name
This will produce a file modules.rst which you’ll need to manually add into your existing index.rst file.
41
RelEng Docs Documentation, Release 1.0
12.3 Tweaking the conf.py file
After you’ve added generated documentation to your Sphinx project, you’ll also need to tweak the generateddocs/conf.py file needs a couple of edits to work properly. Around line 21, add the following lines:
sys.path.insert(0, os.path.abspath(’..’))import mockMOCK_MODULES = []for mod_name in MOCK_MODULES:
sys.modules[mod_name] = mock.Mock()
The first line ensures the module under development is found. The remaining lines are a handy framework for satisfy-ing import requirements for your module.
Note: If you want to import the real code during document generation, you’ll need to add it to the read-the-docsrequirements file, if it isn’t already in your project’s requirements.txt file..
42 Chapter 12. Adding Sphinx docs to Existing Repositories
CHAPTER 13
Maintaining this Documentation
13.1 Overview
Release Engineering has a wide variety of documentation, served from various locations. A rough breakdown is:
TodoList documentation types from PDX team week talk
13.2 Documenting Source Code
We use Sphinx to generate our code documentation, and host it on Read the Docs. There are two major phases to this:
• Adding Sphinx docs to Existing Repositories
• Adding a new repo to Releng Account on Read-The-Docs
43
RelEng Docs Documentation, Release 1.0
44 Chapter 13. Maintaining this Documentation
CHAPTER 14
From a change on a repo to builds being triggered
• Software:
– Buildbot - If you are new to Buildbot, please see Buildbot
• Repos:
– http://hg.mozilla.org/build/buildbotcustom/
– http://hg.mozilla.org/build/buildbot-configs/
– http://hg.mozilla.org/build/mozharness/ <- no code examples but we will see Buildbot mention scripts from
• Purpose of walk-through:
– expanding on flow, we are going to look at how we configure Buildbot. From monitoring changes pushedto known repos to assigning slaves on each of our platforms the job of compiling specified revisions ofsource, this walk-through will show you the core parts of our buildbot logic. Note this only looks at the‘compile/build’ jobs we do from TBPL and not the test jobs.
First, let’s create a Buildbot Master:
We have scripts that will set up a machine to be a fully functioning master. They will install deps, set up environments,and link files from the above repos to a buildbot master dir. Which links it makes depends on the master we are settingup. For example, in our production environment, we have some masters that only deal with getting slaves to run testson one platform like Linux. We have other masters that only schedule and prioritize which builds need to be run.There are many more masters with specific tasks.
Rather than looking at each of these, we will be touching on a master from our staging setup. A staging masterencompasses all our logic in each from our production masters. You can think of this as the universal setup and iseasier to grep while learning. In fact, you can mimic this setup on your own machine locally. I highly recommenddoing this so you can navigate the code yourself and veer off track as you get curious.
• WARNING: If you follow along with files in our repos, you will notice they are much more complex. Theycontain logic for handling builders and schedulers that are beyond our ‘build a generic version of Firefox oneach one of our platforms’ releng 101 session. For example: we ignore all our debug, nightly, l10n, PGO,non-unified, no-profiling, valgrind, and xulrunner build variants.
To set up a staging master on your own machine, follow these instructions: set up a staging local master
The above script will setup a virtual python environment, install Buildbot, and create a build, test, and try master allon one machine. Ignoring the ‘try’ master, you can imagine the Build Master handles all the compiling/installing jobswhile the Test Master handles all the tests jobs.
Let’s look at the Build Master dir:
45
RelEng Docs Documentation, Release 1.0
[dev_master] jlund@Hastings163:~VIRTUAL_ENV/build-master> lltotal 3176lrwxr-xr-x 1 jlund staff 80B 22 Feb 21:26 staging_config.py -> /Users/jlund/devel/mozilla/dev_master/buildbot-configs/mozilla/staging_config.pydrwxr-xr-x 6 jlund staff 204B 22 Feb 21:26 public_htmllrwxr-xr-x 1 jlund staff 82B 22 Feb 21:26 project_branches.py -> /Users/jlund/devel/mozilla/dev_master/buildbot-configs/mozilla/project_branches.pylrwxr-xr-x 1 jlund staff 83B 22 Feb 21:26 production_config.py -> /Users/jlund/devel/mozilla/dev_master/buildbot-configs/mozilla/production_config.pylrwxr-xr-x 1 jlund staff 86B 22 Feb 21:26 preproduction_config.py -> /Users/jlund/devel/mozilla/dev_master/buildbot-configs/mozilla/preproduction_config.py-rw-r--r-- 1 jlund staff 753B 22 Feb 21:26 passwords.pylrwxr-xr-x 1 jlund staff 20B 22 Feb 21:26 master_localconfig.py -> build_localconfig.pylrwxr-xr-x 1 jlund staff 79B 22 Feb 21:26 master_common.py -> /Users/jlund/devel/mozilla/dev_master/buildbot-configs/mozilla/master_common.pylrwxr-xr-x 1 jlund staff 17B 22 Feb 21:26 localconfig.py -> staging_config.pylrwxr-xr-x 1 jlund staff 72B 22 Feb 21:26 config.py -> /Users/jlund/devel/mozilla/dev_master/buildbot-configs/mozilla/config.pylrwxr-xr-x 1 jlund staff 81B 22 Feb 21:26 builder_master.cfg -> /Users/jlund/devel/mozilla/dev_master/buildbot-configs/mozilla/builder_master.cfg-rw-r--r-- 1 jlund staff 1.2K 22 Feb 21:26 buildbot.taclrwxr-xr-x 1 jlund staff 83B 22 Feb 21:26 build_localconfig.py -> /Users/jlund/devel/mozilla/dev_master/buildbot-configs/mozilla/build_localconfig.py-rw-r--r-- 1 jlund staff 611B 22 Feb 21:26 BuildSlaves.pylrwxr-xr-x 1 jlund staff 55B 22 Feb 21:26 master.cfg -> ../buildbot-configs/mozilla/universal_master_sqlite.cfg-rw-r--r-- 1 jlund staff 322B 22 Feb 21:31 master_config.json-rw-r--r-- 1 jlund staff 224K 19 Apr 17:47 state.sqlite-rw-r--r-- 1 jlund staff 977K 19 Apr 17:52 twistd.log
• Note: I am not including the directories for all the ‘builders’ the master knows how to run or any release* b2g*thunderbird* stuff.
All Buildbot Masters have a ‘master.cfg’ file. This file’s content boils down to a “BuildmasterConfig” dict that tellsBuildbot everything it should do and how. From the above dir tree output, you can see ‘master.cfg’ is a link toa file from one of our two Buildbot repos: ‘../buildbot-configs/mozilla/universal_master_sqlite.cfg’. <- our stagingmaster.cfg
• Navigation tip: buildbot-configs/mozilla/* represents all our Build Master logic while buildbot-configs/mozilla-tests/* holds Test Master logic. ‘buildbot-configs/mozilla2’ and ‘buildbot-configs/calendar’ and ‘buildbot-configs/seamonkey’ can largely be ignored for learning purposes.
universal_master_sqlite.cfg will be our first file we look at.
First things first, let’s start populating Buildmasterconfig:
from master_common import BuildmasterConfigc = BuildmasterConfig
We have some items in master_localconfig that we will copy over. master_localconfig is a link to ‘buildbot-configs/mozilla/build_localconfig.py’ build_localconfig will define some BuildmasterConfig items that are unique toour locally specific master setup. eg: these keys could be for what url and port the master uses to connect with slaves.Let’s grab those items:
for key, value in master_localconfig.BuildmasterConfig.items():if key in c:
c[key].extend(value)else:
c[key] = value
Now let’s grab the bread and butter, our main config file. ‘config.py’ separates all of the differences between buildingMozilla products across each platform within each branch (branch being a separate repo in most cases).
Again, it’s worth noting that the config below in this circumstance will represent our Build Master’s config. This isa link to ‘buildbot-configs/mozilla/config.py’. That link will differ for our Test Masters but the logic flow will staylargely the same:
46 Chapter 14. From a change on a repo to builds being triggered
RelEng Docs Documentation, Release 1.0
from config import BRANCHES, PROJECTS
an extremely simple example of what BRANCHES will look like:
BRANCHES = {’mozilla-central’: {
’some-branch-specific-item’: ’foo’,# ...’platforms’: {
’linux’: {’product_name’: ’Firefox’, # the product this will be’base_name’: ’Linux mozilla-central’, # the buildername this will be# mozharness_config will be the script the slave will run to compile/install Firefox with.# This script and mozharness will be looked at later’mozharness_config’: {
’script_name’: ’scripts/fx_desktop_build.py’,’extra_args’: [
’--config’, ’builds/releng_base_linux32_builds.py’,],
’linux64’: { # contains similar values as linux as above},’win32’: {},’macosx64’: {},# ... more platforms
},},’mozilla-aurora’: {
’platforms’: {’win32’: {},’linux’: {},’linux64’: {},’macosx64’: {},# ... more platforms
},},# ... more branches
},
We will look at ‘buildbot-configs/mozilla/config.py’ in more detail later, but if you are curious, config.py is runnabledirectly outside of buildbot via printing config.py
There is also thunderbird_config and b2g_config that behave similarly and possess their own BRANCHES.
Earlier we took master_localconfig’s BuildmasterConfig for specific master config items. master_localconfig alsodictates which BRANCHES we will use to install/compile against. Unlike config.py, where every branch that isknown to releng resides, build_localconfig.py will dictate which branches are enabled and which are disabled for thespecific Master. build_localconfig will decide this by either its set of defaults or by referencing against a JSON filecalled master_config.json. master_config.json is not inside our repos but is generated during set up a staging localmaster. You can see it in our dir tree from above. Let’s grab the branches it considers enabled (active) so the masterknows what to use:
from master_localconfig import ACTIVE_BRANCHES, ACTIVE_PROJECTS, SLAVES
ACTIVE_BRANCHES and ACTIVE_PROJECTS are just a list of strings representing what is enabled. SLAVES isa list of dicts representing what ‘slaves’ this master will know it can use at its disposal for running certain builders.Again we are only worrying about ACTIVE_BRANCHES.
We will now create an object to track all the builders, status, change_source, and schedulers that makes up our BuildMaster. These are the core concepts in Buildbot that should be familiar after going over Buildbot in 5 min.
This obj will be called buildObjects:
47
RelEng Docs Documentation, Release 1.0
buildObjects = {’builders’: [], ’status’: [], ’change_source’: [], ’schedulers’: []}
buildObjects is extended via generating methods. Using config.py’s BRANCHES, we pass only the ones that areenabled via master_localconfig’s ACTIVE_BRANCHES to generateBranchObjects() and generateBranchObjects()will create builders, schedulers, etc based upon those BRANCHES[branch] being passed:
for branch in ACTIVE_BRANCHES:branchObjects = generateBranchObjects(BRANCHES[branch], branch,
getattr(passwords, ’secrets’, None))buildObjects = mergeBuildObjects(buildObjects, branchObjects)
mergeBuildObjects is a glorified dict.update() that updates buildObjects as we iterate. Again note that in the fulluniversal_master_sqlite.py, buildObjects also takes B2G and Thunderbird items in a similar fashion.
It is worth stepping into generateBranchObjects() as it traverses through buildbot-configs and figures out the appropri-ate buildbot configuration. It is imported from misc which can be found at ‘buildbotcustom/misc.py’:
def generateBranchObjects(config, name, secrets=None):"""name is the name of branch which is usually the last part of the path
to the repository. For example, ’mozilla-central’, ’mozilla-aurora’, or’mozilla-1.9.1’.config is a dictionary containing all of the necessary configurationinformation for a branch. The required keys depends greatly on what’senabled for a branch (unittests, xulrunner, l10n, etc). The best wayto figure out what you need to pass is by looking at existing configsand using ’buildbot checkconfig’ to verify.
"""# We return this at the endbranchObjects = {
’builders’: [],’change_source’: [],’schedulers’: [],’status’: []
}# List of all the per-checkin buildersbuilders = []
First let’s iterate over all platforms we have enabled:
# This section is to make it easier to disable certain products.# Ideally we could specify a shorter platforms key on the branch,# but that doesn’t workenabled_platforms = []for platform in sorted(config[’platforms’].keys()):
pf = config[’platforms’][platform]if pf[’stage_product’] in config[’enabled_products’]:
enabled_platforms.append(platform)
# generate a list of builders, nightly builders (names must be different)# for easy accessfor platform in enabled_platforms:
pf = config[’platforms’][platform]builder_name = ’%s build’ % pf[’base_name’]
now we give a name to our builder based on platform and add it to a given product (eg: Firefox):
buildersByProduct.setdefault(pf[’stage_product’], []).append(builder_name)
48 Chapter 14. From a change on a repo to builds being triggered
RelEng Docs Documentation, Release 1.0
we then set up our change_source so that every time a cset is pushed to the current repo of which was passed togenerateBranchObjects (eg: config[’repo_path’] == hg.m.o/projects/cedar), our schedulers we define can pick up thechange and start the appropriate builds (c[’builders’][’the appropriate build’])
to do this, we use HgPoller mentioned in flow:
branchObjects[’change_source’].append(HgPoller(hgURL=config[’hgurl’],branch=config.get("poll_repo", config[’repo_path’]),tipsOnly=tipsOnly,maxChanges=maxChanges,repo_branch=repo_branch,pollInterval=pollInterval,
))
time for the schedulers! Here we are basically saying when there is a push to the repo matching the Scheduler()’s‘branch’, trigger all the builders with the names from the Scheduler’s ‘builderNames’:
# schedulers# this one gets triggered by the HG Pollerfor product, product_builders in buildersByProduct.items():
branchObjects[’schedulers’].append(Scheduler(name=scheduler_name_prefix + "-" + product,branch=config.get("poll_repo", config[’repo_path’]),builderNames=product_builders,fileIsImportant=fileIsImportant,
**extra_args))
note - check here for more on our buildbot schedulers.
last but not least, the ‘builders’. Above we defined the names (strings) of the builders. Now we will create actualbuildbot builders that are associated with those names so the schedulers will actually have a builder to call:
for platform in enabled_platforms:branchObjects[’builders’].extend(
generateDesktopMozharnessBuilders(name, platform, config
))
return branchObjects
we can briefly look at generateDesktopMozharnessBuilders:
def generateDesktopMozharnessBuilders(name, platform, config):desktop_mh_builders = []
pf = config[’platforms’][platform]
if you recall above when we gave a crude example of what BRANCHES from buildbot-configs/mozilla/config.pywould look like, we defined a mozharness_config at the platform level. Below we use that to define what our builderdoes:
base_extra_args = pf[’mozharness_config’].get(’extra_args’, [])# let’s grab the extra args that are defined at misc levelbranch_and_pool_args = []branch_and_pool_args.extend([’--branch’, name])if config.get(’staging’):
branch_and_pool_args.extend([’--build-pool’, ’staging’])else: # this is production
49
RelEng Docs Documentation, Release 1.0
branch_and_pool_args.extend([’--build-pool’, ’production’])base_extra_args.extend(branch_and_pool_args)base_builder_dir = ’%s-%s’ % (name, platform)
Buildbot Builders are made up of a series of cmds (build steps). That series (a factory) is associated with a Builder.So you can think of a Builder as something with a name that is a string that cooresponds with a buildername from ascheduler, a factory, and some other important data like what slaves are capable of running the respective builder.
let’s look at the factory:
factory = makeMHFactory(config, pf, signingServers=dep_signing_servers,extra_args=base_extra_args)
# and our factory creating methoddef makeMHFactory(config, pf, extra_args=None, **kwargs):
factory_class = ScriptFactorymh_cfg = pf[’mozharness_config’]
scriptRepo = config.get(’mozharness_repo_url’,’%s%s’ % (config[’hgurl’], config[’mozharness_repo_path’]))
factory = factory_class(scriptRepo=scriptRepo,interpreter=mh_cfg.get(’mozharness_python’),scriptName=mh_cfg[’script_name’],reboot_command=mh_cfg.get(’reboot_command’),extra_args=extra_args,script_timeout=pf.get(’timeout’, 3600),script_maxtime=pf.get(’maxTime’, 4 * 3600),
**kwargs)return factory
For our factory, we use the ScriptFactory class to set out a few setup cmds, the main script we want to call, and thensome tear down cmds. Remember cmds being BuildSteps in Buildbot world.
Let’s look at a snippet of ScriptFactory Quickly. You can find it where we keep other factories: buildbotcus-tom/process/factory.py
Remember factories encapsulate a series of pre-defined cmds that a buildbot master will tell a buildbot slave to runsequentially, once a change_source (cset lands on a repo), triggers a scheduler to trigger a builder with that factory:
class ScriptFactory(RequestSortingBuildFactory):
def __init__(self, scriptRepo, scriptName, cwd=None, interpreter=None):BuildFactory.__init__(self)self.platform = platformself.env = env.copy()self.cmd = [scriptName]
if extra_args:self.cmd.extend(extra_args)
we set some initial steps like the basedir that we will run commands and work from on the slave:
self.addStep(SetProperty(name=’get_basedir’,property=’basedir’,command=self.get_basedir_cmd,workdir=’.’,
50 Chapter 14. From a change on a repo to builds being triggered
RelEng Docs Documentation, Release 1.0
haltOnFailure=True,))
then we will need to tell the slave to clone the repo that is home to the script we are going to get the slave to call (inthis case it will be cloning Mozharness):
self.addStep(MercurialCloneCommand(name="clone_scripts",command=[hg_bin, ’clone’, scriptRepo, ’scripts’],workdir=".",haltOnFailure=True,retry=False,log_eval_func=rc_eval_func({0: SUCCESS, None: RETRY}),
))self.runScript()self.addCleanupSteps()self.reboot()
then we define how the script will be called by the slave:
def runScript(self):self.preRunScript()self.addStep(MockCommand(
name="run_script",command=self.cmd,env=self.env,timeout=self.script_timeout,maxTime=self.script_maxtime,log_eval_func=self.log_eval_func,workdir=".",haltOnFailure=True,warnOnWarnings=True,mock=self.use_mock,target=self.mock_target,
))
finally we tell the slave to reboot itself:
def reboot(self):self.addStep(DisconnectStep(
name=’reboot’,flunkOnFailure=False,warnOnFailure=False,alwaysRun=True,workdir=’.’,description="reboot",command=self.reboot_command,force_disconnect=do_disconnect,env=self.env,
))
and that’s it for the factory and list of cmds. We pass that factory to the builder we are defining and that builder getsextended to buildObjects[’builders’]:
generic_builder = {’name’: ’%s build’ % pf[’base_name’],’builddir’: base_builder_dir,’slavebuilddir’: normalizeName(base_builder_dir),’slavenames’: pf[’slaves’],
51
RelEng Docs Documentation, Release 1.0
’nextSlave’: next_slave,’factory’: factory,’category’: name,’properties’: mh_build_properties.copy(),
}desktop_mh_builders.append(generic_builder)
# finally let’s return which builders we did so we know what’s left to do!return desktop_mh_builders
We have reached the end of misc.py’s generateBranchObjects()
Back in our universal_master_sqlite.py, we finish up with adding logic to how we define the steps to run after a jobcompletes. This will contain logic to parsing if the job was a success, failure, etc and also concat the job’s steps intoone log that is uploaded and fed to TBPL. These post run steps are explained in postrun.py. Notice we add this to ourstatus key
Here we also mention our QueueDir objs. To understand that, see queue directories:
# Create our QueueDir objects# This is reloaded in buildbotcustom.miscfrom mozilla_buildtools.queuedir import QueueDircommandsQueue = QueueDir(’commands’, ’%s/commands’ % master_localconfig.QUEUEDIR)from buildbotcustom.status.queued_command import QueuedCommandHandlerbuildObjects[’status’].append(QueuedCommandHandler(
command=[sys.executable, os.path.join(os.path.dirname(buildbotcustom.__file__), ’bin’, ’postrun.py’), ’-c’, os.path.abspath(os.path.join(os.curdir, ’postrun.cfg’))],queuedir=commandsQueue,
))
We can finish up by extending our BuildmasterConfig with all the ‘builders’ ‘status’ ‘change_source’ and ‘schedulers’we generated from generateBranchObjects():
c[’builders’].extend(buildObjects[’builders’])c[’status’].extend(buildObjects[’status’])c[’change_source’].extend(buildObjects[’change_source’])c[’schedulers’].extend(buildObjects[’schedulers’])
Phew! That’s the end of that file. We can consider Buildbot to be ‘configured’. All that is left to do is to start aBuildbot Master with this configuration on a machine and connect Buildbot Slaves to it.
You might be thinking “wait, I still haven’t seen any of our logic for actually ‘compiling’ Firefox from source.”
And that’s true! Up to this point, we have only gone over the logic from ‘a user checking in a cset’ to ‘a buildbot mastertriggering build jobs on a slave from each of our platforms.’. Everything involved on with how to build firefox (thescript we defined in ScriptFactory) we have yet to see. But that is for walk-through 2: Building Firefox in automation
14.1 Recap – the full code from examples above
buildbot-configs/mozilla/universal_master_sqlite.cfg:
from master_common import BuildmasterConfigc = BuildmasterConfig
for key, value in master_localconfig.BuildmasterConfig.items():if key in c:
c[key].extend(value)else:
c[key] = value
52 Chapter 14. From a change on a repo to builds being triggered
RelEng Docs Documentation, Release 1.0
# Create our QueueDir objects# This is reloaded in buildbotcustom.miscfrom mozilla_buildtools.queuedir import QueueDircommandsQueue = QueueDir(’commands’, ’%s/commands’ % master_localconfig.QUEUEDIR)from buildbotcustom.status.queued_command import QueuedCommandHandlerc[’status’].append(QueuedCommandHandler(
command=[sys.executable, os.path.join(os.path.dirname(buildbotcustom.__file__), ’bin’, ’postrun.py’), ’-c’, os.path.abspath(os.path.join(os.curdir, ’postrun.cfg’))],queuedir=commandsQueue,
))
from config import BRANCHES, PROJECTS
from master_localconfig import ACTIVE_BRANCHES, ACTIVE_PROJECTS, SLAVES
buildObjects = {’builders’: [], ’status’: [], ’change_source’: [], ’schedulers’: []}
for branch in ACTIVE_BRANCHES:branchObjects = generateBranchObjects(BRANCHES[branch], branch,
getattr(passwords, ’secrets’, None))buildObjects = mergeBuildObjects(buildObjects, branchObjects)
c[’builders’].extend(buildObjects[’builders’])c[’status’].extend(buildObjects[’status’])c[’change_source’].extend(buildObjects[’change_source’])c[’schedulers’].extend(buildObjects[’schedulers’])
buildbotcustom/misc.py:
def generateBranchObjects(config, name, secrets=None):"""name is the name of branch which is usually the last part of the path
to the repository. For example, ’mozilla-central’, ’mozilla-aurora’, or’mozilla-1.9.1’.config is a dictionary containing all of the necessary configurationinformation for a branch. The required keys depends greatly on what’senabled for a branch (unittests, xulrunner, l10n, etc). The best wayto figure out what you need to pass is by looking at existing configsand using ’buildbot checkconfig’ to verify.
"""# We return this at the endbranchObjects = {
’builders’: [],’change_source’: [],’schedulers’: [],’status’: []
}# List of all the per-checkin buildersbuilders = []
# This section is to make it easier to disable certain products.# Ideally we could specify a shorter platforms key on the branch,# but that doesn’t workenabled_platforms = []for platform in sorted(config[’platforms’].keys()):
pf = config[’platforms’][platform]if pf[’stage_product’] in config[’enabled_products’]:
enabled_platforms.append(platform)
# generate a list of builders, nightly builders (names must be different)
14.1. Recap – the full code from examples above 53
RelEng Docs Documentation, Release 1.0
# for easy accessfor platform in enabled_platforms:
pf = config[’platforms’][platform]builder_name = ’%s build’ % pf[’base_name’]
buildersByProduct.setdefault(pf[’stage_product’], []).append(builder_name)
branchObjects[’change_source’].append(HgPoller(hgURL=config[’hgurl’],branch=config.get("poll_repo", config[’repo_path’]),tipsOnly=tipsOnly,maxChanges=maxChanges,repo_branch=repo_branch,pollInterval=pollInterval,
))
# schedulers# this one gets triggered by the HG Pollerfor product, product_builders in buildersByProduct.items():
branchObjects[’schedulers’].append(scheduler_class(name=scheduler_name_prefix + "-" + product,branch=config.get("poll_repo", config[’repo_path’]),builderNames=product_builders,fileIsImportant=fileIsImportant,
**extra_args))
for platform in enabled_platforms:# shorthandpf = config[’platforms’][platform]branchObjects[’builders’].extend(
generateDesktopMozharnessBuilders(name, platform, config
))
return branchObjects
def generateDesktopMozharnessBuilders(name, platform, config):desktop_mh_builders = []
pf = config[’platforms’][platform]
base_extra_args = pf[’mozharness_config’].get(’extra_args’, [])# let’s grab the extra args that are defined at misc levelbranch_and_pool_args = []branch_and_pool_args.extend([’--branch’, name])if config.get(’staging’):
branch_and_pool_args.extend([’--build-pool’, ’staging’])else: # this is production
branch_and_pool_args.extend([’--build-pool’, ’production’])base_extra_args.extend(branch_and_pool_args)base_builder_dir = ’%s-%s’ % (name, platform)
factory = makeMHFactory(config, pf, signingServers=dep_signing_servers,extra_args=base_extra_args)
generic_builder = {
54 Chapter 14. From a change on a repo to builds being triggered
RelEng Docs Documentation, Release 1.0
’name’: ’%s build’ % pf[’base_name’],’builddir’: base_builder_dir,’slavebuilddir’: normalizeName(base_builder_dir),’slavenames’: pf[’slaves’],’nextSlave’: next_slave,’factory’: factory,’category’: name,’properties’: mh_build_properties.copy(),
}desktop_mh_builders.append(generic_builder)
# finally let’s return which builders we did so we know what’s left to do!return desktop_mh_builders
# and our factory creating methoddef makeMHFactory(config, pf, extra_args=None, **kwargs):
factory_class = ScriptFactorymh_cfg = pf[’mozharness_config’]
scriptRepo = config.get(’mozharness_repo_url’,’%s%s’ % (config[’hgurl’], config[’mozharness_repo_path’]))
factory = factory_class(scriptRepo=scriptRepo,interpreter=mh_cfg.get(’mozharness_python’),scriptName=mh_cfg[’script_name’],reboot_command=mh_cfg.get(’reboot_command’),extra_args=extra_args,script_timeout=pf.get(’timeout’, 3600),script_maxtime=pf.get(’maxTime’, 4 * 3600),
**kwargs)return factory
buildbotcustom/process/factory.py:
class ScriptFactory(RequestSortingBuildFactory):
def __init__(self, scriptRepo, scriptName, cwd=None, interpreter=None):BuildFactory.__init__(self)self.platform = platformself.env = env.copy()self.cmd = [scriptName]
if extra_args:self.cmd.extend(extra_args)
self.addStep(SetProperty(name=’get_basedir’,property=’basedir’,command=self.get_basedir_cmd,workdir=’.’,haltOnFailure=True,
))
self.addStep(MercurialCloneCommand(name="clone_scripts",command=[hg_bin, ’clone’, scriptRepo, ’scripts’],
14.1. Recap – the full code from examples above 55
RelEng Docs Documentation, Release 1.0
workdir=".",haltOnFailure=True,retry=False,log_eval_func=rc_eval_func({0: SUCCESS, None: RETRY}),
))self.runScript()self.addCleanupSteps()self.reboot()
def runScript(self):self.preRunScript()self.addStep(MockCommand(
name="run_script",command=self.cmd,env=self.env,timeout=self.script_timeout,maxTime=self.script_maxtime,log_eval_func=self.log_eval_func,workdir=".",haltOnFailure=True,warnOnWarnings=True,mock=self.use_mock,target=self.mock_target,
))
def reboot(self):self.addStep(DisconnectStep(
name=’reboot’,flunkOnFailure=False,warnOnFailure=False,alwaysRun=True,workdir=’.’,description="reboot",command=self.reboot_command,force_disconnect=do_disconnect,env=self.env,
))
56 Chapter 14. From a change on a repo to builds being triggered
CHAPTER 15
Topics needing authors
Right now, there are a number of items that are “links to no where”. Well, now there’s here! If you landed here,chances are you followed a link that doesn’t yet have documentation written. Please feel free to ask questions in#releng and write some! Then, delete the reference to your newly written section in this source.
TodoList documentation types from PDX team week talk
(The original entry is located in /var/build/user_builds/hwine-test-releng-top-level/checkouts/latest/maintaining_these_docs.rst, line 13.)
TodoWrite the rest of these.
(The original entry is located in /var/build/user_builds/hwine-test-releng-top-level/checkouts/latest/releng101/index.rst, line 44.)
TodoWrite up builder priority Include link to definitions, and intended usage.
(The original entry is located in /var/build/user_builds/hwine-test-releng-top-level/checkouts/latest/schedulerdb.rst,line 29.)
57
RelEng Docs Documentation, Release 1.0
58 Chapter 15. Topics needing authors
CHAPTER 16
Indices and tables
• Maintaining this Documentation
• genindex
• modindex
• search
• Topics needing authors
59
RelEng Docs Documentation, Release 1.0
60 Chapter 16. Indices and tables
Index
BbuilderPriority, 29
Cchecklist
Tree Closing Window, 10
TTCW, see Tree Closing WindowTree Closing Window, 9, 10
checklist, 10
Vvcs2vcs
legacy, 28
61
