Post on 22-Feb-2017
© naked ALM – Martin Hinshelwood 2013
Martin Hinshelwood | @MrHinshConsultant | MVP | Ranger | Trainer | Coachmartin@nkdagility.com | http://nkdagility.com/blog
Load Testing with Visual Studio Team Services
Scrum?
Join the #DevOps conversation #NDCOlso with @MrHinsh
Join the #DevOps conversation #NDCOlso with @MrHinsh
Join the #DevOps conversation #NDCOlso with @MrHinsh
Join the #DevOps conversation #NDCOlso with @MrHinsh
Professional Scrum Teams
Mechanical /
Amateur / Flaccid Scrum
Values and
Principles
Technical Excellen
ce
The Scrum Guide: http://www.scrumguides.org/
FLOW OFCUSTOMER
VALUE
TEAMAUTONOMY
& ENTERPRISEALIGNMENT
BACKLOG refined with LEARNING
EVIDENCEgathered in
PRODUCTION
MANAGED TECHNICAL
DEBT
PRODUCTION FIRST MINDSET
INFRASTRUCTUREis a FLEXIBLE
RESOURCE
Areas for improvement or Goals to achieve
Join the #DevOps conversation #NDCOlso with @MrHinsh
DevOps?
Join the #DevOps conversation #NDCOlso with @MrHinsh
Join the #DevOps conversation #NDCOlso with @MrHinsh
Join the #DevOps conversation #NDCOlso with @MrHinsh
Join the #DevOps conversation #NDCOlso with @MrHinsh
P R A C T I C E SAutomated TestingContinuous IntegrationContinuous DeploymentRelease Management
P R A C T I C E SUsage MonitoringTelemetry CollectionTesting in ProductionStakeholder Feedback
P R A C T I C E STesting in ProductionUsage MonitoringUser TelemetryStakeholder feedbackFeature flags
P R A C T I C E SCode ReviewsAutomated TestingContinuous Measurement
P R A C T I C E SApplication Performance ManagementInfrastructure as CodeContinuous DeliveryRelease Management Configuration Management Automated Recovery
P R A C T I C E SApplication Performance ManagementInfrastructure as CodeContinuous DeploymentRelease Management Configuration Management Automated Recovery
P R A C T I C E SEnterprise AgileContinuous IntegrationContinuous DeploymentRelease Management
FLOW OFCUSTOMER
VALUE
TEAMAUTONOMY
& ENTERPRISEALIGNMENT
BACKLOG refined with LEARNING
EVIDENCEgathered in
PRODUCTION
MANAGED TECHNICAL
DEBT
PRODUCTION FIRST MINDSET
INFRASTRUCTUREis a FLEXIBLE
RESOURCE
DevOps habits and practices
Join the #DevOps conversation #NDCOlso with @MrHinsh
Testing?
What testing do you do?
Functional Tests
• User Story Tests• Prototypes• Simulations
Exploratory Testing
• Usability Testing• User Acceptance
Testing• Alpha/Beta
Developer Tests
• Unit Tests• Component Tests
Performance & Load Testing
Security Testing-ility Testing
Supp
orti
ng t
he T
eam
Technology Facing
Critique Product
Business Facing
Automated
Automated& Manual
Automated & Tools
Manual
Join the #DevOps conversation #NDCOlso with @MrHinsh
Join the #DevOps conversation #NDCOlso with @MrHinsh
Don’t leave load testing until last!
This demonstration will cover:
• Creating a basic cloud load test• Import Archive• Custom load test in Visual Studio
DEMO: URL Cloud Load Test
That must cost a LOT!
• 20k: FREE• 2m: $8000 (64k NOK)• 2m-10m: $16000 (130k NOK)
So for 2 minutes of 200 simultaneous users per build for 10 deployments per day at 4000 VMU’s, it would be FREE
Before it costs any money you can load test 50 deployments at 200 simultaneous users for 2 minutes.
Or you could ramp up to 1000 simultaneous users.
That must be expensive?Examples Cloud-based Load Testing*
* Requires Visual Studio Enterprise- Services purchased via Microsoft Azure based on actual usage- Additional services are accessible to all users on the account and shared
among them- Build time is actual computing time required to run the build, and
does not include queuing time (if any)- Load test runs specify the number of virtual users and duration in
minutesVirtual users x minutes = virtual user minutes (VUM)
FREE: 20,000 virtual user minutes per monthTHEREAFTER:$0.004/virtual user minute for 20,001-2M VUM$0.002/VUM for 2,000,001-10M VUM$0.001/VUM for usage above 10M VUM/month
Professional DevOps ?
Continuous Delivery
Continuous Integration
Release Management
Streamline and automate the workflow between development and IT Ops and deliver higher quality software more frequently with less risk.
Continuous Quality
Enabling continuous value deliveryDeveloper Version Control Build and
Unit TestingAutomated Acceptance
Testing
Automated Performance
TestingRelease
Check-in Trigger
Feedback
Approval
Feedback
Feedback
Trigger
Trigger
Continuous integration (CI)Developer Version Control Build and
Unit TestingAutomated Acceptance
Testing
Automated Performance
TestingRelease
Check-in Trigger
Feedback
Approval
Feedback
Feedback
Trigger
Trigger
Continuous integration (CI)Developer Version Control Build and
Unit TestingAutomated Acceptance
Testing
Automated Performance
TestingRelease
Check-in Trigger
Feedback
Approval
Feedback
Feedback
Trigger
Trigger
Continuous integration (CI)Developer Version Control Build and
Unit TestingAutomated Acceptance
Testing
Automated Performance
TestingRelease
Check-in Trigger
Feedback
Approval
Feedback
Feedback
Trigger
Trigger
This demonstration will cover:
• Integrating into your DevOps pipeline• https://www.visualstudio.com/docs/integrate/ap
i/load-test/overview
DEMO: Cloud Load Test in VSTS
Join the #DevOps conversation #NDCOlso with @MrHinsh
© naked ALM – Martin Hinshelwood 2013
Professional Teams
Professional Scrum Professional DevOps
Join the #DevOps conversation #NDCOlso with @MrHinsh
© naked ALM – Martin Hinshelwood 2013
Take-Away for Cloud Load Testing
• Cloud-load testing: rich reports in the web• Running Apache JMeter based load tests in the cloud – how to• Creating Load Test Plugins for Cloud-based Load Test• Scheduling Load Test Execution• Performance Testing
Join the #DevOps conversation #NDCOlso with @MrHinsh
© naked ALM – Martin Hinshelwood 201341© 1993-2015 Scrum.org, All Rights Reserved
Suggested Reading
Professional ALM with Visual Studio 2013 The Phoenix Project Software in 30 Days
Join the #DevOps conversation #NDCOlso with @MrHinsh
© naked ALM – Martin Hinshelwood 2013
Suggested Reading
Join the #DevOps conversation #NDCOlso with @MrHinsh
Martin Hinshelwood | @MrHinshConsultant | MVP | Ranger | Trainer | Coachmartin@nkdagility.com | http://nkdagility.com/blog
ThanksLoad Testing with Visual Studio Team Services