Kohsuke Aoyanagi- Creation of a brane universe in String/M-theory
Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson...
Transcript of Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson...
![Page 1: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/1.jpg)
Continuous Integrationin the Cloudwith HudsonKohsuke KawaguchiJesse GlickSun Microsystems, Inc.Hudson committers
![Page 2: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/2.jpg)
Rise of Continuous Integration> Offload from people, push to computers
2
$
time
computers
us
![Page 3: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/3.jpg)
Spend more CPU power to help you> … even if it only helps a little
> Ever bloating IDEs> Static code analysis tools> More frequent build/test executions
AKA “Continuous Integration”
3
![Page 4: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/4.jpg)
Hudson> Open-source CI server at java.net
> Emphasis on ease of installation and use GUI for human users REST API for program users
> Extensibility 130+ community-developed public plugins By 120+ contributors
> Estimated 20,000 installations4
http://hudson-ci.org/
![Page 5: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/5.jpg)
It basically does builds and tests> Check out the source code
Subversion, Perforce, Git, Mercurial, CVS, …> Do builds and/or tests
Java, .NET, shell script, …> Record results
Binary, test results, code coverage, static analysis> Notify people
E-mail, IM, RSS, tray apps, IDEs
5
![Page 6: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/6.jpg)
Localized to 8 languages
6
![Page 7: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/7.jpg)
Adoption in all kinds of businesses
7
![Page 8: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/8.jpg)
Before we talk about clouds…> When I talk to people, they have computers
Lots of them, lying around, under-utilized Just lacking software to use them effectively
> Let’s use lots of computers effectively first> Then we’ll talk about EC2
8
![Page 9: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/9.jpg)
Going distributed> You need to use multiple computers because…
You need different environments You need isolation One computer can’t keep up with all the loads
9
![Page 10: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/10.jpg)
Distributed builds with Hudson> Master
Serves HTTP requests Stores all important info
> Slaves 170KB single jar Assumed to be unreliable Scale to at least 100
> Link Single bi-di byte stream No other requirements
10
![Page 11: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/11.jpg)
How master and slaves start talking> Via sshd
Master talks to sshd on a slave Send slave.jar and “java -jar slave.jar” SSH session becomes bi-di byte stream
11
![Page 12: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/12.jpg)
How master and slaves start talking> Via JNLP
Java Web Start on slave initiates the session Hudson sends JNLP file and jar files A separate socket connection is made
12
![Page 13: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/13.jpg)
How master and slaves start talking> Once started, can be installed as Windows
service
13
![Page 14: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/14.jpg)
How master and slaves start talking> Specifically for Windows
Hudson speaks DCOM to talk to Windows Remotely install a service and starts it No manual intervention needed
14
![Page 15: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/15.jpg)
Heterogeneous Cluster Challenge> Your builds/tests need to run on specific
environment> Dependency to individual nodes hurts utilization
15
WombatWindows test
Hudson Windows test
Windows #1
jobs slaves
GlassFishWindows test
Windows #2
Solaris#1
Hudson Solaris test
![Page 16: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/16.jpg)
Labels to rescue> Label is a group of slaves> Tie jobs to labels
16
WombatWindows test
Hudson Windows test
Windows #1
jobs slaves
GlassFishWindows test
Windows #2
Solaris#1
Hudson Solaris test
Windows
Solaris
![Page 17: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/17.jpg)
Setting up slaves> Keeping slaves look alike is a good thing
Particularly hard on heterogeneous environment> General system administration tasks
Network configuration Package installations for native tools Tools like Puppet or cfEngine are supposed to help
> Install build tools in the cluster Prepare tools on one file system rsync to everywhere
> This part of Hudson needs improvements17
![Page 18: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/18.jpg)
Forecasting failures> Hudson monitors key health metrics of slaves
Low disk space, insufficient swap Clock out of sync Extensible
> Slaves put offline automatically> Catch problems before it breaks builds
18
![Page 19: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/19.jpg)
Installing new slaves> For first 20 or so slaves, we did it manually
Insert CD, click, type, click, type, click, … But that doesn’t scale
> Then we automated
19
![Page 20: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/20.jpg)
Automated System Installations
20
> Slaves Power on, hit F12 PC boots from network (PXE) Choose OS from menu Chain boot into OS Installs non-interactively
> Hudson + pxe-install plugin BOOTP proxy TFTP pxelinux
Your corporate IT guy & his DHCP server
![Page 21: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/21.jpg)
Automated System Installations> Trivial with most Linux and Solaris> Works with Windows, too
Called “Windows Deployment Service” Needs a Windows 2003 server Vista (easy) or XP (hard) deployment
> Turns out quite useful outside Hudson, too No more broken CD drives No more CD-Rs
21
![Page 22: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/22.jpg)
System Utilization Monitoring> Showing about 25% utilization
22
![Page 23: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/23.jpg)
When it’s time to add more slaves> There’s almost always something in the queue
23
![Page 24: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/24.jpg)
Hudson made this extensible> Hudson detects excessive workload
With exponential decay to filter out noise> Hudson notifies plugins> Plugins can provision more slaves
24
![Page 25: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/25.jpg)
25
![Page 26: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/26.jpg)
Amazon EC2: The Good> Pay as you go (15¢/h or so)> Programmable API> Instances launch pretty quickly (esp. Linux)> EC2 instances are forgetful> Good fit with Hudson
Loads on Hudson tend to be spiky Tests are embarrassingly parallel
At least in theory
26
![Page 27: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/27.jpg)
Amazon EC2: The Bad> Your data is still inside your firewall
Takes time to check out code … or to archive build artifacts Some data just can’t be moved
> EC2 instances are forgetful> Your build/test may depend on your environment> Can your tests run in parallel?
27
![Page 28: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/28.jpg)
Hudson EC2 plugin> Built on top of typica*> What does it do?
Automatically provision slaves on EC2 on demand Pick the right AMI depending on demand Connect and install JDK on demand Shut down unused instances
28* http://code.google.com/p/typica/
![Page 29: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/29.jpg)
Hudson EC2 plugin usage> Tell Hudson your AWS account information
29
![Page 30: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/30.jpg)
Hudson EC2 plugin usage> Tell Hudson what AMIs you want to start
30
![Page 31: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/31.jpg)
Putting it all together
31
time
# of
exe
cuto
rs
capacity
usage
![Page 32: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/32.jpg)
Hudson “Appliance” on EC2> Run the master in the cloud too, if you like
Hudson on stock OpenSolaris AMI Data stored persistently in Elastic Block Storage
Dynamically expandable thanks to ZFS Online, too
> Packaged as a wizard
32
![Page 33: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/33.jpg)
33
![Page 34: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/34.jpg)
34
![Page 35: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/35.jpg)
> Hudson Hadoop plugin Exactly two mouse clicks to install Turn every Hudson slave into a Hadoop node
> Distributed file system Automatic data replication (fail tolerant) Nice for storing old artifacts, logs, test records, …
> Map/reduce framework Large scale test results analysis / datamining More interesting work to be done in the future
35
![Page 36: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/36.jpg)
Selenium Grid> Use Hudson slaves as
Selenium RC nodes
36
Hudson master(selenium hub)
Hudson slaves
![Page 37: Continuous Integration in the Cloud with Hudson · Continuous Integration in the Cloud with Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers. Rise of](https://reader036.fdocuments.net/reader036/viewer/2022071217/604c38c936786153b219a980/html5/thumbnails/37.jpg)
Why Selenium Grid & Hudson?> Hudson wants a heterogeneous cluster
Selenium wants that, too
> Centralized management> Automatic selenium installation
37