Lessons from DevOps: Taking DevOps practices into your AppSec Life
Best Practices for DevOps in Mobile App Testing
-
Upload
bitbar -
Category
Technology
-
view
2.056 -
download
0
Transcript of Best Practices for DevOps in Mobile App Testing
3 March 2016 – 12pm GMT
Ville-Veikko HelppiHead of Demand [email protected]
Best Practices for DevOps in Mobile App Testing
W E B I N A R
Agenda• Deconstructing ‘Mobile DevOps’
• Mobile Product Development Life-Cycle & Used Tools
• The DevOps Approach in Mobile App Development & Testing
• Best Practices for DevOps in Mobile App Testing
• Q&A
http://testdroid.com
DECONSTRUCTING “MOBILE DEVOPS”
http://testdroid.com
Mobile DevOps
DevOps = Practices that emphasize the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. --Wikipedia
• Development focus on building the actual product and using relevant tools
• QA doing continuous testing on robustness, functionality, performance, and preferably use test automation
• Operations maintain the continuous build/integration/deployment/delivery environments, and do releases
3 Things in Mobile DevOps
• 3 DevOps Principles1. Develop & Test Against Real Environment
2. Deploy Frequently
3. Continuously Validate Quality Characteristics
• 3 Business-Critical DevOps Categories1. Continuous Integration & Continuous Delivery
2. App and Infrastructure (e.g. Back-End) Testing & Monitoring
3. Entire Dev&Test Process and Mobile App Delivery
http://testdroid.com
DevOps Strategy & Typical Goals
• Maximize the Delivered Value• Customers, Internally between BUs and Teams
• Maximize the Efficiency and Productivity• Improved Process, The ‘Right’ Tools & Methods, Faster T2M
• Maximize the Quality and Robustness of Apps• Better Quality with Equal Effort, Investments in Quality
• Maximize the Support, Service & The Delivery• Alignment and Collaboration between Teams
The highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Agile Manifesto – Principle #1“ “http://testdroid.com
How DevOps Can Make A Difference
• Based on Testdroid Customer Survey 2015• Things That Go Wrong with App Testing
1. Too Long Dev-Test-Feedback Cycles
2. Bug Catching/Reporting Too Slow to Developers
3. Non-Functional or Not Well Utilized Test Automation
4. Use of Emulators - Setting Up & Maintaining Real Devices
5. Access to Relevant Mobile Platform / OS version / Devices
6. Use of Manual Testing – The Weakest Link in Agile and Devops Process
7. No Geographically Relevant Testbed – Devices, Networks, Back-End Integration
http://testdroid.com
Where DevOps Make A Difference
• Devices / Hardware- Handling of new devices and other new HW used in the infrastructure
• Infrastructure- Ready-to-Go infrastructure means no delays for development, testing and result-
driven approach speeds things up & saves time and money
• Support- SW devs, SW devs in test (SDET), QA engineers – all need different type of support
when using the infrastructure and DevOps approach can help a lot
• Operations- Maintaining and Upgrading infrastructure, devices, networks, additional SW
• Tools Integration - Support for multiple test tools gives the users/developers freedom of choice. No
vendor, tech, tools lock-ins!http://testdroid.com
MOBILE DEVOPS & DEVELOPMENT- LIFECYCLE & TOOLS
http://testdroid.com
DevOps Challenges *• Building the DevOps culture across organization• Overcoming the Dev vs. Ops mentality• Moving from legacy (and sometimes on-premise setup)
to cloud based tools and environment• Broader scope than before: SW and HW for mobile dev• Learning new skills – DevOps practice requires change
and collaboration across teams• Mobile Challenges:
• Fragmentation, Tools and Platforms, Not Enough Real Devices• Less Worrying Challenges on Mobile:
• Cross-platform, Using Cloud Efficiently
* = Based on Testdroid Customer Survey 2015http://testdroid.com
DevOps ‘Toolchain’
• Code – Code Development and Code Reviews, Continuous Integration tools
• Build – Version control tools, SCM, code merging, Build status
• Test – Test and results determine performance
• Package – Artifact repository, Application pre-deployment staging
• Release – Change management, Release approvals, release automation
• Configure – Infrastructure configuration and management, Infrastructure as Code tools
• Monitor – App perf. monitoring, UX
DevOps ‘Toolchain’
Code & Build
Continuous Integration, Continuous Development
Package, Release, Configure & Monitoring
Continuous Delivery, Continuous Deployment/Release
GOAL: End-to-End Transparency, Traceability, Seamless Workflow, Continuous Everything!
2 Valid Options for Mobile Testing: Cloud or On-
Premisehttp://testdroid.com
DevOps – Testing in the ‘Cloud’
1
2
3
DevOps – Testing On-Premise
CONTINUOUSINTEGRATION
http://testdroid.com
Benefits of Test Automation• App Compatibility & Global coverage
• Reusability• Speed & Efficiency
• Test efficiency• Test effectiveness• Time-to-market
http://testdroid.com
DevOps Must Rely on Automation
• DevOps is more than just a tool or a process change; it inherently requires an organizational culture shift
• Agile methodologies back up this thinking• Increase focus on test automation and CI methods• Best practices and continuous iteration
TimeResources
Infrastructure
TrainingTools
http://testdroid.com
THE DEVOPS APPROACH IN MOBILE APP TESTING
http://testdroid.com
Testing Philosophy
http://testdroid.com
Manual AutomationTesting Philosophy
http://testdroid.com
Smaller coverage, More money burnt & time wasted, Error-prone
Manual AutomationLarge coverage,
quickly completed,
Less money & time wasted, Exact results.
Testing Philosophy
Top Requirement for Testing
• Real Devices and Real Browsers, as emulators cannot help you to test...
• User Experience • Usability• Hardware• Software• Infrastructure
0 % = the percentage of your app users that use emulator to run your app!
http://testdroid.com
Devices & Daily Tasks for DevOps
• In order to maintain mobile device farm, environment and all aspects of SW & HW the following daily check-ups must be done:• Are all devices connected and operating?• Are all devices WiFi connected?• Do all devices have an active USB connection?• Are battery levels adequate (at least 50%) for every device?• Are there any pop-ups, system notification or other system
dialogs open on the screen?• Are screens unlocked?
http://testdroid.com
Troubleshooting Ain’t The Same
http://testdroid.com
Automation vs Manual
http://testdroid.com
Test Breakdown
FAILED FAILED
http://testdroid.com
Mobile Test AutomationAutomatic test exercisers Record and PlaybackHand written test scripts
Benefits:Accurate, specific to your testing needs, plenty of options with frameworks, tools
Fast to create, accurate, not as sensitive to human-errors as hand-written tests, tools avail’ty
Fastest & extremely automated, excellent for smoke testing/quick testing, availability
Tradeoffs:Takes a lot of time, ties resources to write test cases/scripts, error-prone (humans)
Compelling Recorder + Playback tools available for only few test automation frameworks
Not accurate as real test cases
http://testdroid.com
Why Should DevOps Be Interested in …
Robotium uiautomator
Espresso Appium Calabash
Android Yes Yes Yes Yes Yes
iOS No No No Yes Yes
Mobile web Yes (Android)
Limited to x.y clicks
No Yes (Android & iOS)
Yes (Android)
Scripting Language
Java Java Java Almost any Ruby
Test creation tools
Testdroid Recorder
UI Automator viewer
Hierarchy Viewer
Appium.app CLI
Supported API levels
All 16 => 8, 10, 15- All All
Community Contributors Google Google Active Pretty quiet
http://testdroid.com
BEST PRACTICES FOR DEVOPS IN MOBILE APP TESTING
http://testdroid.com
Automate (Everything)• When it comes to mobile app testing, the more
you can automate repetitive things, the better• Not always possible but leave the last
‘exploratory testing’ for the final phase• Smoke, Unit, All sorts of Performance,
Regression testing should be automated• Run your test patterns against EVERY build• Start small but make sure you can scale-up
http://testdroid.com
Ensure Workflow is Seamless
• Only use tools, frameworks and software that is fully compatible across used toolchain/workflow
• Ensure all processes, results, tools are entirely transparent and available for use across teams
• This has a significant impact on productivity and efficiency
• Open source is always highly recommended
http://testdroid.com
Rely Only On Real & Quality HW
• Mobile device farms require lots of hardware – this is the correct place to invest in quality HW
• Make sure the mobile device coverage is adequate – globally
• Execute the upgrade and support policies relentlessly
http://testdroid.com
“The Right” Testing Framework
• Variety of choices when it comes mobile test automation frameworks, ask the right questions:• App, Game, Web? Native, Hybrid?
• Mobile Back-End Integrated?
• Who needs to write, read and interpret tests?• Does test run results provide you what you
organization needs / is looking for?
http://testdroid.com
THANK YOU!More information about frameworks,
devices and testing at www.testdroid.com
http://testdroid.com