Load Tests 101! Why and where to start?
-
Upload
aurelien-deleusiere -
Category
Software
-
view
93 -
download
2
Transcript of Load Tests 101! Why and where to start?
(RE)LOAD TESTS 101
WHY AND WHERE TO START?
AURÉLIEN DELEUSIÈRETHE ONLINE COLDFUSION MEETUP
(c)2016 A. Deleusière
WHO AM I?
• Software engineer specializing in web technologies, consulting, development, architecture and troubleshooting
• Founder of Webellian based in France and Poland
• ColdFusion since 1996, Java, PHP (mainly for CMS), AngularJs, ExtJS…
(c)2016 A. Deleusière
WHY LOAD TESTING?
Load Testing
Performance
Resources
Scalability
Robustness
(c)2016 A. Deleusière
WHERE TO START?
• Depends on the architecture!
• Two approaches:
• Define the limit of a system in the absolute
• Validate the capability of a system to resist to an estimated traffic, anticipate spikes link to special events
(c)2016 A. Deleusière
SCENARIO “CALIBRATION”
• have a picture of the “background noise” of the scope tested
• have the best response time to calibrate results analysis
(c)2016 A. Deleusière
SCENARIO “HEALTHINESS”
• have a picture of the profile of resources consumption (JVM Memory and Garbage Collection, System CPU, I/O)
• validate the healthiness of the component before Stress Scenario
(c)2016 A. Deleusière
SCENARIO “STRESS”
• determine the maximum load capacity (MAX) where the target conditions are reach in term of response time and stability
• determine acceptable response time variation (in percentile)MAX
(c)2016 A. Deleusière
SCENARIO “ENDURANCE”
• validate the maximum capacity determined during Stress Scenario
• validate the healthiness of the component under target load
(c)2016 A. Deleusière
SCENARIO “BURST”
• validate the capacity of the component to support a sudden high pressure
• this case of sudden high pressure can happen when you put a server back in the pool, or when a special campaign is starting
(c)2016 A. Deleusière
ABOUT GATLING
• Very high performance testing Engine
• 1 thread ≠ 1 request
• Distributed multi-nodes injectors
• Scala descripted scenarios (≈ DSL)
(c)2016 A. Deleusière
ABOUT GATLING
Question: How much performant?
How many requests per second with a single injector (16 vcpu on AWS)?
(c)2016 A. Deleusière
A SIMPLE EXAMPLE
• Produce a chain “ABCDEFGHIJKLMNOPQRSTUVWXYZ”
• By concatenating ASCII characters from 65 to 90
• Tests done locally:
• MacBook Pro running Virtual Box guest Ubuntu
• ColdFusion 2016u3 (2016.0.03.300466) Developper Edition
• Lucee 5.0.0.254
(c)2016 A. Deleusière
REAL EXAMPLE: A QUOTATION SERVICE
• Service Layer is composed by several Component (S1, S2
…) behind a Load Balancer (LB) in a stateless fashion.
• The Service Layer is behind a Gateway in charge of Authentication, Transformation, Routing and Logging that is also composed by several instances (G1, G2) behind a Load Balancer.
• No database, configuration in memory from configuration (XML, DSL) files.
• What to be tested?...
Serv
ices
LB
Gate
way G3 G2
LB
TRAFFIC
Datacenter 1 Datacenter 2
S1 S3 S5 S7 S2 S4 S6 S8
G1 G4
(c)2016 A. Deleusière
Agents
REAL EXAMPLE: A QUOTATION SERVICE
• Isolate a representative architecture to be tested…
• …in an environment identical to production
• Install the load tests Agent as close as possible to the component to be tested
• Install tools to monitor the resources (JVM memory details, GC, System CPU) like Fusion Reactor or Mission Control
• But you can also do test as a “black box”
Services S1 S2
LB
Gateway G1 G2
LB
TRAFFIC
A1 A2
(c)2016 A. Deleusière
FUSION REACTOR
(c)2016 A. Deleusière
MISSION CONTROL
(c)2016 A. Deleusière
REAL EXAMPLE: A QUOTATION SERVICE
The Load Tests are run against one single component to measure the intrinsic capability of one single instance.
Main objectives:• Measure the maximum capacity of one single component
• Measure the response time at component level
Services S1 S2
LB
Gateway G1 G2
LB
TRAFFIC
A
(c)2016 A. Deleusière
REAL EXAMPLE: A QUOTATION SERVICE
The Load Tests are run against a couple of component behind the Load Balancer.
Main objectives:• Determine the capacity factor between 1 instance and 2
instance that is generally is not linear
• Measure the response time to evaluate the latency introduced by the network devices
Services S1 S2
LB
Gateway G1 G2
LB
TRAFFIC
B
(c)2016 A. Deleusière
REAL EXAMPLE: A QUOTATION SERVICE
The Load Tests are run against the full Service layer through one single Gateway instance.
Main objectives:
• Measure the capacity of one single Gateway instance
• Measure the response time to evaluate the latency introduced by one Gateway instance
Services S1 S2
LB
Gateway G1 G2
LB
TRAFFIC
C
(c)2016 A. Deleusière
REAL EXAMPLE: A QUOTATION SERVICE
The Load Tests are run against the full stack.
Main objectives:• measure the response time with the full stack to validate the
order of magnitude for SLA
If possible validate this result with the full stack.
Services S1 S2
LB
Gateway G1 G2
LB
TRAFFIC
D
(c)2016 A. Deleusière
INITIAL TEST CAMPAIGN
(c)2016 A. Deleusière
INITIAL TEST CAMPAIGN
(c)2016 A. Deleusière
INITIAL TEST CAMPAIGN
(c)2016 A. Deleusière
INITIAL TEST CAMPAIGN
(c)2016 A. Deleusière
INITIAL TEST CAMPAIGN
(c)2016 A. Deleusière
CHRISTMAS 2016 SPIKE CAMPAIGN
(c)2016 A. Deleusière
1. GATHER EXISTING DATA
Experience from 2015:
• brutal ramp up at 6:00am from almost nothing to 2000 rq/min;
• peak around 7:00am around 1850 rq/min;
• plateau between 9:00am and 10:30am around 1500 rq/min;
• then 30% to 40% of sales more than usual during the next weeks.
Let’s focus on the 2000 rq/min
ILLUSTRATIVE
(c)2016 A. Deleusière
2. CHALLENGE BUSINESS EXPECTATIONS
Based on the current trends, the business expects an increase of 30%.
Let’s challenge our system with:• 2600 rq/min• A brutal ramp up from 0
(c)2016 A. Deleusière
3. FORECAST YOUR TRAFFIC
3. 2600 rq / min => 44 rq/sWrong! You have to correct the linear projection
“Feel” the volatility of what is your case.Here for a starting spike it should quite stable => 2x
Let’s challenge our system with:• 88 rq/s• A brutal ramp up from 0
(c)2016 A. Deleusière
4. SIZE YOUR SYSTEM
Original system design
Serv
ices
LB
Gate
way G3 G2
LB
TRAFFIC
Datacenter 1 Datacenter 2
S1 S3 S5 S7 S2 S4 S6 S8
G1 G4
(c)2016 A. Deleusière
SERVICE CAPABILITY
(c)2016 A. Deleusière
SERVICE CAPABILITY
(c)2016 A. Deleusière
SERVICE CAPABILITY
Service 80 rq/s per server
(c)2016 A. Deleusière
GATEWAY CAPABILITY
(c)2016 A. Deleusière
GATEWAY CAPABILITY
(c)2016 A. Deleusière
GATEWAY CAPABILITY
(c)2016 A. Deleusière
GATEWAY CAPABILITY
Gateway 25 rq/s per server
(c)2016 A. Deleusière
CHRISTMAS 2016 SPIKE CAMPAIGN
4. Size your system
Target 88 rq/s
Service: 80 rq/s x 8 = 640 rq/s OKGateway: 25 rq/s x 4 = 100 rq/s OK
But….you have to expect a disaster!Service: 80 rq/s x 4 = 320 rq/s OKGateway: 25 rq/s x 2 = 50 rq/s NOT OK!
Required action: add 4 new nodes in Gateway layer.
Serv
ices
LB
Gate
way G3
LB
TRAFFIC
Datacenter 1
S1 S3 S5 S7
G1 G2
Datacenter 2
S2 S4 S6 S8
G4
(c)2016 A. Deleusière
1. Gather existing data2. Challenge business expectations3. Forecast your traffic4. Size your system5. Seat back in comfort on D-Day and bring the popcorns
(c)2016 A. Deleusière
CHRISTMAS 2016 SPIKE CAMPAIGN
• Measured results
(c)2016 A. Deleusière
CHRISTMAS 2016 SPIKE CAMPAIGN
(c)2016 A. Deleusière
CHRISTMAS 2016 SPIKE CAMPAIGN
Estimated: 2600
(c)2016 A. Deleusière
CHRISTMAS 2016 SPIKE CAMPAIGN
Estimated: 88
(c)2016 A. Deleusière
WHEN LOAD TESTING?
• Usually done on Major version releases
• Often by a specialist consultant
• Should be integrated into your automated tests
(c)2016 A. Deleusière
SPECIAL THANKS
• Mohamadou Mballo, a genius developer that just felt in love with CFML
• Stéphane Landelle father of Gatling for his support and the goodies to give away
• Yamina, my supportive girlfriend, that would be upset if she’s not listed on this page…
…and all of you making CFML so special…
(c)2016 A. Deleusière
You liked it? Tweet it!@adeleusiere