Andrey Adamovich and Luciano Fiandesio - Groovy dev ops in the cloud
description
Transcript of Andrey Adamovich and Luciano Fiandesio - Groovy dev ops in the cloud
Luciano Fiandesio
Bio: Developer, speaker, author, kitchen hacker
Company: Aestas/IT (http://aestasit.com)
E-mail: [email protected]
Linkedin: http://www.linkedin.com/in/lucianofiandesio
••••
04
Andrey Adamovich
Bio: Developer, coach, speaker, author
Company: Aestas/IT (http://aestasit.com)
E-mail: [email protected]
Linkedin: http://www.linkedin.com/in/andreyadamovich
••••
05
What's this presentation about?
Our take on:
DevOps
Software Provisioning
Continuous Integration
Continuous Delivery
••••
06
Technologies
AWS - http://aws.amazon.com
Groovy - http://groovy.codehaus.org
Gradle - http://gradle.org
Jenkins - http://jenkins-ci.org
Puppet - http://puppetlabs.com
•••••
07
C.A.M.S.
Culture: People over processes and tools. Software is made by and
for people.
Automation: Automation is essential for DevOps to gain quick
feedback.
Measurement: DevOps finds a specific path to measurement. Quality
and shared (or at least aligned) incentives are critical.
Sharing: Creates a culture where people share ideas, processes, and
tools.
•
•
•
•
14
Infrastructure as Code
Automate the provisioning and maintenance of servers:
Build from source control
Utilize Open Source tools
Ensure testability
•••
22
Building an automation toolkit
Automation is key
We are JVM hackers
Fragmented ecosystem
Full-stack approach
••••
25
Tooling
Infrastructure connectivity
Infrastructure provisioning
Infrastructure virtualization
Infrastructure testing
••••
26
Sshoogr Example I
remoteSession {
url = 'user2:654321@localhost:2222'
exec 'rm -rf /tmp/*'
exec 'touch /var/lock/my.pid'
remoteFile('/var/my.conf').text = "enabled=true"
}
01.
02.
03.
04.
05.
06.
29
Sshoogr Example II
remoteSession {
scp {
from { localDir "$buildDir/application" }
into { remoteDir '/var/bea/domain/application' }
}
}
01.
02.
03.
04.
05.
06.
30
Sshoogr Example III
def result = exec(command: '/usr/bin/mycmd',
failOnError: false, showOutput: false)
if (result.exitStatus == 1) {
result.output.eachLine { line ->
if (line.contains('WARNING')) {
throw new RuntimeException("Warning!!!")
}
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.31
Why Groovy?
Groovy is perfect choice for scripting
Very mature, concise syntax
Extremely easy to produce DSL
We wrote a book about it!
••••
32
Why Puppet?
More mature than competition
Large community
Readable DSL
No need to learn Ruby ;)
••••
35
Gradle Example I
task uploadModules << {
remoteSession {
exec 'rm -rf /tmp/repo.zip'
scp {
from { localFile "${buildDir}/repo.zip" }
into { remoteDir "/root" }
}
...
01.
02.
03.
04.
05.
06.
07.
08.
50
Gradle Example I
...
exec 'rm -rf /etc/puppet/modules'
exec 'unzip /tmp/repo.zip -d /etc/puppet/modules'
}
}
01.
02.
03.
04.
05.
51
Gradle Example II
task puppetApply(dependsOn: uploadModules) << {
remoteSession {
scp {
from { localFile "${buildDir}/setup.pp" }
into { remoteDir "/tmp" }
}
exec 'puppet apply /tmp/setup.pp'
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.52
In the meanwhile...
We started developing complex Puppet modules
Modules needs proper testing
...on different platforms
•••
54
Do you test, right?
How to test this stuff?
How to reuse a JUnit approach to testing?
We wanted things to be SIMPLE!
•••
55
PUnit
Simple testing tool for verifying remote server state
Uses sshoogr and JUnit
Reuse reporting features of JUnit and Jenkins
As simple as ...
••••
57
PUnit Example I
class DerbyInstallTest
extends BasePuppetIntegrationTest {
@Before
void installDerby() {
apply("include derby")
}
...
}
01.
02.
03.
04.
05.
06.
07.
08.
58
PUnit Example II
@Test
def void ensureDerbyRunning() {
command('service derby status > derbystatus.log')
assertTrue fileText("/root/derbystatus.log")
.contains('Derby')
assertTrue fileText("/root/derbystatus.log")
.contains('is running.')
}
01.
02.
03.
04.
05.
06.
07.
08.
59
PUnit Example III
@Test
def void ensureCanConnect() {
Thread.sleep(10000)
uploadScript()
command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' +
'testDataScript.sql > derbytest.log')
...
01.
02.
03.
04.
05.
06.
07.
60
PUnit Example III
...
// Check if the log of the insert
// operation contains the word ERROR.
assertFalse(
"The script should return at least one error",
fileText("/root/derbytest.log")
.contains('ERROR')
)
...
01.
02.
03.
04.
05.
06.
07.
08.
09.61
PUnit Example III
...
// Check on data that was inserted into a table.
assertTrue(
"The log should contain a SELECT result",
fileText("/root/derbytest.log")
.contains('Grand Ave.')
)
}
01.
02.
03.
04.
05.
06.
07.
08.
62
More examples
session {
tunnel ('127.0.0.1', 8080) { int localPort ->
// Login to Jenkins.
def driver = new HtmlUnitDriver(false)
driver.manage().timeouts().pageLoadTimeout(300, TimeUnit.SECONDS).implicitlyWait(30, TimeUnit.SECONDS)
driver.get("http://127.0.0.1:${localPort}/login")
...
01.
02.
03.04.
05.
06.
07.
08.
09.
64
More examples
...
def input = driver.findElement(By.name('j_username'))
input.sendKeys('john')
input = driver.findElement(By.name('j_password'))
input.sendKeys('123456')
input.submit()
...
01.
02.
03.
04.
05.
06.
07.
65
More examples
...
// Verify login.
def wait = new WebDriverWait(driver, 30)
wait.until ExpectedConditions.presenceOfElementLocated (By.linkText('John Doe'))
...
}
}
01.
02.
03.
04.
05.
06.
07.
66
More examples
session {
tunnel ('127.0.0.1', 80) { int localPort ->
// Initilize repository connection data.
DAVRepositoryFactory.setup()
def url = SVNURL.create('http', null, '127.0.0.1', localPort, 'repos/cafebabe', true)
def repository = SVNRepositoryFactory.create(url)
println "Verifying SVN repository at ${url}"
...
01.
02.
03.04.
05.
06.
07.
08.
09.
67
More examples
...
// Setup credentials.
def authManager = SVNWCUtil.createDefaultAuthenticationManager('joe', '123456')
repository.setAuthenticationManager(authManager)
// Verify repository is at revision 0.
assertEquals 0, repository.getLatestRevision()
...
01.
02.
03.
04.
05.06.
07.
08.
68
More examples
...
// Commit first revision.
ISVNEditor editor = repository.getCommitEditor("Initial commit.", null)
editor.with {
openRoot(-1)
addFile('dummy.txt', null, -1)
applyTextDelta('dummy.txt', null)
def deltaGenerator = new SVNDeltaGenerator()
...
01.
02.
03.
04.
05.
06.
07.
08.
09.69
More examples
...
def checksum = deltaGenerator.sendDelta('dummy.txt', new ByteArrayInputStream("data".getBytes()), editor, true)
closeFile('dummy.txt', checksum)
def commitInfo = closeEdit()
println commitInfo
}
...
01.
02.
03.
04.
05.
06.
07.
70
More examples
...
// Verify repository is at revision 1 now.
assertEquals 1, repository.getLatestRevision()
}
}
01.
02.
03.
04.05.
06.
71
Scalability
How do we test on different OS?
How do we run parallel tests on multiple architectures?
How do we avoid selling our houses?
•••
73
Gramazon Example I
task startInstance(type: StartInstance) {
keyName 'cloud-do'
securityGroup 'cloud-do'
instanceName 'gramazon/cloud-do'
stateFileName 'cloud-do.json'
ami 'ami-6f07e418'
instanceType 't1.micro'
waitForStart true
}
01.
02.
03.
04.
05.
06.
07.
08.
09.78
Gramazon Example II
task terminateInstance(type: TerminateInstance) {
stateFileName 'cloud-do.json'
}
01.
02.
03.
79
The flow
Start instance(s)
Upload manifests
Run tests
Generate report
Terminate instance(s)
1.
2.
3.
4.
5.
85
Conclusions
Reuse your existing Java knowledge
...to build a bridge between DEVs and OPs
Reuse development best practices for OPs
Don't be afraid to try new technologies
Automate!
•••••
89
Technologies to follow
Vagrant - http://www.vagrantup.com/
Docker - https://www.docker.io/
Packer - http://www.packer.io/
Qemu - http://wiki.qemu.org/Main_Page
jclouds - http://jclouds.apache.org/
Cloudbees - http://www.cloudbees.com/
••••••
97
Source code
Sshoogr: https://github.com/aestasit/sshoogr
Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle
PUnit: https://github.com/aestasit/puppet-unit
Gramazon: https://github.com/aestasit/gramazon
Imgr: https://github.com/aestasit/imgr
•••••
100