Testing Distributed Micro Services. Agile Testing Days 2017
-
Upload
carlos-sanchez -
Category
Software
-
view
109 -
download
3
Transcript of Testing Distributed Micro Services. Agile Testing Days 2017
![Page 1: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/1.jpg)
TESTING DISTRIBUTEDMICRO-SERVICES
Carlos Sanchez
@csanchez csanchez.org
Watch online at carlossg.github.io/presentations
![Page 2: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/2.jpg)
ABOUT MESenior So�ware Engineer @ CloudBees
Author of Jenkins Kubernetes plugin
Long time OSS contributor at Apache Maven, Eclipse,Puppet,…
Google Cloud Platform "Expert"
![Page 3: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/3.jpg)
DOCKER DOCKER DOCKER
![Page 4: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/4.jpg)
![Page 5: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/5.jpg)
OUR USE CASE
Scaling JenkinsYour mileage may vary
![Page 6: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/6.jpg)
![Page 7: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/7.jpg)
![Page 8: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/8.jpg)
![Page 9: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/9.jpg)
A 2000 JENKINS MASTERS CLUSTER3 Mesos masters (m3.xlarge: 4 vCPU, 15GB, 2x40 SSD)317 Mesos slaves (c3.2xlarge, m3.xlarge, m4.4xlarge)7 Mesos slaves dedicated to ElasticSearch: (c3.8xlarge: 32vCPU, 60GB)
12.5 TB - 3748 CPU
Running 2000 masters and ~8000 concurrent jobs
![Page 10: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/10.jpg)
ARCHITECTURE
![Page 11: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/11.jpg)
![Page 12: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/12.jpg)
Isolated Jenkins masters
Isolated build agents and jobs
Memory and CPU limits
![Page 13: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/13.jpg)
![Page 14: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/14.jpg)
![Page 15: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/15.jpg)
CLUSTER SCHEDULINGDistribute tasks across a cluster of hosts
HA and fault tolerant
With Docker support of course
![Page 16: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/16.jpg)
INFRASTRUCTURERunning in public cloud, private cloud, VMs or bare metal
![Page 17: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/17.jpg)
APACHE MESOS & MESOSPHEREMARATHON
A distributed systems kernel
![Page 18: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/18.jpg)
ALTERNATIVES
Docker Swarm / Kubernetes
![Page 19: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/19.jpg)
"UNIT" TESTINGDOCKER IMAGES
![Page 20: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/20.jpg)
THE DOCKERFILEA lot like a shell script
RUN comands
COPY files
...
![Page 21: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/21.jpg)
DOCKERFILEFROM openjdk:8-jdk
RUN apt-get update && apt-get install -y git curl && rm -rf /var/lib/apt/lists/*
ARG JENKINS_VERSION ENV JENKINS_VERSION ${JENKINS_VERSION:-2.19.3} ARG JENKINS_SHA=e97670636394092af40cc626f8e07b092105c07b
ARG JENKINS_URL=https://repo.jenkins-ci.org/public/org/jenkins-ci/main/jenkins-war/${JENKINS_VERSION}/jenkins-war-${JENKINS_VERSION}.war
RUN curl -fsSL ${JENKINS_URL} -o /usr/share/jenkins/jenkins.war \ && echo "${JENKINS_SHA} /usr/share/jenkins/jenkins.war" | sha1sum -c -
COPY jenkins-support /usr/local/bin/jenkins-support COPY jenkins.sh /usr/local/bin/jenkins.sh ENTRYPOINT ["/usr/local/bin/jenkins.sh"]
![Page 22: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/22.jpg)
Mocking and stubbing are your friends
![Page 23: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/23.jpg)
BUILDING WITH JENKINS DOCKERPIPELINE
def maven = docker.image('maven:3.3.9-jdk-8');
stage 'Mirror' maven.pull() docker.withRegistry('https://secure-registry/', 'docker-registry-login') {
stage 'Build' maven.inside { sh "mvn -B clean package" }
stage 'Bake Docker image' def pcImg = docker.build( "examplecorp/spring-petclinic:${env.BUILD_TAG}", 'app')
pcImg.push(); }
![Page 24: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/24.jpg)
TESTING WITH JENKINS DOCKERPIPELINE
Build + test + promotion
Promotion = different Docker registries for differentenvironments
devstagingproduction
![Page 25: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/25.jpg)
TESTING WITH JENKINS DOCKERPIPELINE
pcImg = docker.image("examplecorp/spring-petclinic:dev")
stage 'Test' docker.withRegistry('https://dev.docker.example.com/', 'docker-registry-login') {
pcImg.withRun { petclinic -> sh "test -f /var/some_file" } }
stage 'Promote' docker.withRegistry('https://staging.docker.example.com/', 'docker-registry-login') {
pcImg.push() }
![Page 26: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/26.jpg)
DOCKER WORKFLOW PLUGIN DEMOhttps://github.com/jenkinsci/docker-workflow-
plugin/tree/master/demo
![Page 27: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/27.jpg)
USING BATSTesting using shell scripts!
bats tests/install-plugins.bats
Examples fromhttps://github.com/jenkinsci/docker/tree/master/tests
![Page 28: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/28.jpg)
BATS#!/usr/bin/env bats
SUT_IMAGE=bats-jenkins
load 'test_helper/bats-support/load' load 'test_helper/bats-assert/load' load test_helpers
@test "plugins are installed with plugins.sh" { run docker build -t $SUT_IMAGE-plugins $BATS_TEST_DIRNAME/plugins assert_success run bash -c "docker run --rm $SUT_IMAGE-plugins ls -1 \ /var/jenkins_home/plugins" assert_success assert_line 'maven-plugin.jpi' assert_line 'maven-plugin.jpi.pinned' assert_line 'ant.jpi' assert_line 'ant.jpi.pinned' }
![Page 29: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/29.jpg)
BATS@test "test jenkins arguments" { local version=$(grep 'ENV JENKINS_VERSION' Dockerfile | \ sed -e 's/.*:-\(.*\)}/\1/') # need the last line of output assert "${version}" docker run --rm --name $SUT_CONTAINER \ -P $SUT_IMAGE --help --version | tail -n 1 }
![Page 30: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/30.jpg)
ISOLATIONThere is no 100% isolation
![Page 31: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/31.jpg)
MEMORY ISSUES WITH CONTAINERSScheduler needs to account for container memory
requirements and host available memory
Prevent containers for using more memory than allowed
Memory constrains translate to Docker --memory
![Page 32: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/32.jpg)
WHAT DO YOU THINK HAPPENS WHEN?Your container goes over memory quota?
![Page 33: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/33.jpg)
![Page 34: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/34.jpg)
WHAT ABOUT THE JVM?WHAT ABOUT THE CHILD PROCESSES?
![Page 35: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/35.jpg)
END TO END TESTINGOF MULTIPLECONTAINERS
![Page 36: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/36.jpg)
JENKINS PIPELINE WITH MULTIPLE CONTAINERSpcImg = docker.image("examplecorp/spring-petclinic:dev")
stage 'Test' docker.withRegistry('https://dev.docker.example.com/', 'docker-registry-login') {
pcImg.withRun {petclinic -> testImg.inside("--link=${petclinic.id}:petclinic") { wrap([$class: 'Xvnc', takeScreenshot: true, useXauthority: true]) { sh "mvn -B clean test" } } } }
stage 'Promote' docker.withRegistry('https://staging.docker.example.com/', 'docker-registry-login') { pcImg.push() }
![Page 37: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/37.jpg)
USING CONTAINER GROUPS
![Page 38: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/38.jpg)
DOCKER COMPOSERuns multiple containers
Containers can access each other
![Page 39: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/39.jpg)
DOCKER COMPOSEversion: '2'
services:
example: image: acme/example:latest ports: - "5050:5050"
maven: image: maven:3-jdk-8 command: mvn test
![Page 40: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/40.jpg)
JENKINS KUBERNETES PLUGINThe Jenkins job can run in a Kubernetes Pod
(group of containers)
Containers in a Pod can access each other at localhost
![Page 41: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/41.jpg)
KUBERNETES PLUGIN PIPELINESEx. run maven tests against webapp with Selenium, using
pre-made Docker imagespodTemplate(label: 'petclinic', containers: [ containerTemplate( name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'), containerTemplate( name: 'petclinic', image: 'csanchez/petclinic'), containerTemplate( name: 'selenium', image: 'selenium/standalone-firefox') ]) {
node ('petclinic') { stage 'Get a Maven project' git 'https://github.com/jenkinsci/kubernetes-plugin.git' container('maven') { stage 'Build a Maven project' sh 'cd demo/test && mvn -B clean test' } } }
![Page 42: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/42.jpg)
HOW ARE WE DOING IT
![Page 43: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/43.jpg)
HOW ARE WE DOING ITOn each commit and PR
Provisioning of the infrastructure using Terraform
Installation of the cluster scheduler (Mesos & Marathon)
Continuously creating clusters from scratch
![Page 44: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/44.jpg)
TERRAFORMInfrastructure-As-Code
![Page 45: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/45.jpg)
TERRAFORMresource "aws_instance" "worker" { count = 1 instance_type = "m3.large" ami = "ami-xxxxxx" key_name = "tiger-csanchez" security_groups = ["sg-61bc8c18"] subnet_id = "subnet-xxxxxx" associate_public_ip_address = true tags { Name = "tiger-csanchez-worker-1" "cloudbees:pse:cluster" = "tiger-csanchez" "cloudbees:pse:type" = "worker" } root_block_device { volume_size = 50 } }
![Page 46: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/46.jpg)
TERRAFORMState is managedRuns are idempotentterraform apply
Sometimes it is too automaticChanging image id will restart all instances
![Page 47: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/47.jpg)
![Page 48: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/48.jpg)
IF YOU HAVEN'T AUTOMATICALLYDESTROYED SOMETHING BY
MISTAKE,YOU ARE NOT AUTOMATING ENOUGH
![Page 49: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/49.jpg)
HOW ARE WE DOING ITIn AWS and OpenStack
5 different combinations
More combinations on demand
![Page 50: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/50.jpg)
HOW ARE WE DOING ITA�er creation we launch acceptance tests
Some python scripts
Some Selenium tests
Clusters get destroyed at the end
![Page 51: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/51.jpg)
MONITORING IS THE NEW TESTINGWe gather from the cluster:
logsconfigurationoutputs
Attached to the build to diagnose errors in CI
but also used by our customers to send us information
![Page 52: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/52.jpg)
HOW ARE WE DOING ITFeedback is published to Github PR
![Page 54: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/54.jpg)
CANARY DEPLOYMENTS
![Page 55: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/55.jpg)
SCALINGNew and interesting problems
![Page 56: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/56.jpg)
AWSResource limits: VPCs, S3 snapshots, some instance sizes
Rate limits: affect the whole account
Always use different accounts for testing/production andpossibly different teams
Retrying is your friend, but with exponential backoff
![Page 57: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/57.jpg)
EMBRACE FAILURE!
![Page 58: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/58.jpg)
OPENSTACKCustom flavors
Custom images
Different CLI commands
There are not two OpenStack installations that are the same
![Page 59: Testing Distributed Micro Services. Agile Testing Days 2017](https://reader033.fdocuments.net/reader033/viewer/2022052606/58e544c81a28ab3a468b4f8f/html5/thumbnails/59.jpg)
DANKE!RATE THIS SESSION IN
AGILETESTINGDAYS.COMcsanchez.org
csanchez
carlossg