Explore the State of Open Source Performance Testing in Continuous Delivery Pipelines with...

World®’16

ExploretheStateofOpenSourcePerformanceTestinginContinuousDeliveryPipelineswithBlazeMeter ChiefScientistAndreyPokhilkoAndreyPokhilko - ChiefScientist,BlazeMeter - CATechnologies

DO5X54E

DEVOPS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

ContinuousIntegrationhasrapidlybecomethedefactostandardinsoftwaredevelopmentprocesses.Whilefunctionalandunittestinghaveprovedrelativelyeasytointegrateintotheseprocesses,performancetestinghastypicallyraisedmorechallenges.Inthistalk,AndreywillexaminekeyapproachestointegratingperformancetestingintoContinuousDeliveryworkflows,includinghowtoovercomethekeychallengesencounteredineachmethod.

Ultimately,atruecontinuousdeliveryprocessreliesupontheactualizationoffullyautomated,configurationdrivenandversion-basedworkflows.YouwillgetanunderstandingonhowtoevolveyourCIsystemtofacilitatethis.

AndreyPokhilkoCATechnologiesChiefScientist,BlazeMeter


ImportanceofCIandCD

§ Machinetimecostsnothing,humantimepriceless

§ De-factowinningpractice

§ MostadvancedteamsgowithCD

§ Alotofteamsarestillinprocessofadoptingit


Agenda

THINGSWEPUTINTOCI

BESTPRACTICE(FORNOW)

CHALLENGESOFTESTINGINCI

OVERCOMINGCHALLENGES

1

2

3

4

JENKINSFORPERFORMANCETESTING

SOMEUSEFULPLUGINS

5

6


Agenda

THINGSWEPUTINTOCI




1

2

3

4


SOMEUSEFULPLUGINS

5

6


TriggeringandPreparations

§ Whentodothejob

§ VCScheckout+dependenciescheckout

§ Buildingproject(compilingetc.)

§ Putresultingpackagesintorepos


QualityControl

1. Staticcodeanalysis

2. Unittests

3. Functionaltests

4. Performancetests

needs deployment


Deployment

§ ToQAenvironment(forfurthermanualtests)

§ Tostagingenvironment

§ Toproductionenvironment


Agenda

THINGSWEPUTINTOCI




1

2

3

4


SOMEUSEFULPLUGINS

5

6


OneWayRoad(Driverless)

VCSCheckout+

DependenciesBuild Unit

Tests

FunctionalTests

PerfromanceTests

Deployment

Filter Filter Filter Filter


VCS-DrivenPipeline

§ NaturalevolutionofCIsystems

§ Branchingandpullrequests

§ Jenkins2.0pipelines

§ TaurusToolaspartofthisapproach


Agenda

THINGSWEPUTINTOCI




1

2

3

4


SOMEUSEFULPLUGINS

5

6


Challenge#1:TestEnvironment

§ Applicationsarecomplex§ Lotsofdependencies§ Third-partysystems

Databases

Microservices

Third-parties


Challenge#2:TimeConsuming

§ Preparations§ Alotoffunctionaltests§ Performancetestsarenaturallylong


Challenge#3:DebuggingCIJobs

§ Evolvingtestcomplexity§ Debuggingandtroubleshooting§ Buildhistoryisavalue


Challenge#4:ResultsAnalysis

§ Richreportingneeded§ JenkinsUIhasitslimits§ AutomateddecisionmakingforCD§ Collaborationfornon-techpeople

wut???


Agenda

THINGSWEPUTINTOCI




1

2

3

4


SOMEUSEFULPLUGINS

5

6


Overcome:TestEnvironment

§ Containershelp(youhavetousecontainers:)§ Youdon’tneedtobeasrealistic

≈


Overcome:TimeConsuming

§ Preparewhatyoucanupfront- inthenight§ Reusewhatyoucan§ Shorttestscanrevealalotofthings§ Don’ttrytoputspikeandendurancetestsintoCI§ Parallelizetests,Jenkins2.0orTaurushelps


Overcome:DebuggingandImproving

§ TaurusTool,thehandheldpieceofCI


Overcome:DebuggingandImproving

LoadTestJob

LocalDebugging


Overcome:Reporting&ResultConsumption

§ Purpose-builtserviceswithintegrations§ Overview&StatusinJenkins


Overcome:DecisionMaking

§ Deploytostagingisgreatwin§ Havepass/failcriteria,includingAPMinfo§ It’soktohavedeployasseparatejob(forUAT,toavoid

disruptions)


Agenda

THINGSWEPUTINTOCI




1

2

3

4


SOMEUSEFULPLUGINS

5

6


WhyUseJenkinsforTesting?

§ Multi-stepprocess§ Longprocess§ Repetitiveprocess


StartSimple

LoadTest1

LoadTest2

LoadTest3

Center of Excellence

Developer

Operations

QA

Maintain Jobs


BecomeContinuous

§ Periodicjobshavemanyapplications§ Havevalidfailurecriteria(thresholds)§ Usenon-blockingdownstreamjobs


PursueContinuousDelivery

FastFeedbackDeveloper/

DevOps

Build+xUnit

FunctionalTests

LoadTests

Deploy Staging

SlowerFeedback


Agenda

THINGSWEPUTINTOCI




1

2

3

4


SOMEUSEFULPLUGINS

5

6


PlotPlugin


PerformancePublisherPlugin


BlazeMeterPlugin


TaurusTool

§ Whynotplugin,butcommand-linetool§ UniversalplugfortestsinsideJenkins§ TaurusdesignedtoworkwithotherJenkinsplugins

gettaurus.org


FinalSummary

WehavetotestinsideCI

Therearewaystostartitsimple

Therearetools&pluginstohelp

Jenkins2.0pipelinesFTW!


Questions?


RecommendedSessions

SESSION# TITLE DATE/TIME

DO5T57SCaseStudy:HowIMovedTheNFLfromWaterfalltoAgilein4Months 11/16/2016at03:45pm

DO5X55SCaseStudy:WalmartBrasil—SeasonalPeakTrafficReadiness 11/16/2016at04:30am

DO5X56SNext-GenerationPerformanceTestingwithBlazeMeter,ServiceVirtualizationandTestDataManagement 11/17/2016at04:30pm


MustSeeDemos

ModernizeAppDeliveryIntegratedCDTheater5- DOV501P

DeliverTestDataFasterTestDataManagerTheater5- DOV511P

DeliverBetterAppsServiceVirtualizationTheater5- DOV507P

OrchestrateYourReleaseReleaseAutomationTheater5- DOV513P


Thankyou.

Stayconnectedatcommunities.ca.com

Explore the State of Open Source Performance Testing in Continuous Delivery Pipelines with...

Technology

Transcript of Explore the State of Open Source Performance Testing in Continuous Delivery Pipelines with...