Don’t Be Another Statistic! Develop a Long-Term Test Automation Strategy

T21 Test Automation 5/5/16 15:00

Don't Be Another Statistic! Develop a Long-term Test Automation Strategy

Presented by:

Lee Barnes

Utopia

Brought to you by:

350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.stareast.techwell.com/

Lee Barnes Utopia

Founder and CTO of Utopia Solutions Lee Barnes is responsible for the firm's delivery of software quality solutions which include process improvement, performance management, test automation, and mobile quality. Lee has more than twenty years of experience in the software quality assurance and testing field. He has successfully implemented test automation and performance testing solutions in hundreds of environments across a wide array of industries. A recognized leader in his field, Lee speaks regularly on related topics.

© Utopia Solutions

Don’t Be Another Statistic! Develop a Strategy for Long-term Test Automation Success

Lee Barnes, CTO

Utopia Solutions

© Utopia Solutions

Agenda

• What is the end goal of test automation?

• Why do we need a strategy?

• How do we know if we’re ready to succeed?

• What should a strategy look like?

Questions to Answer Today…

© Utopia Solutions

SECTION 1

What is Our End Goal?

© Utopia Solutions 4

Common Test Automation Goals

Test Coverage

Resource Value

Test Cycle Time


Test Automation Success Criteria

• Issue detection and recovery

• Accurate verification

• Unattended execution Reliable

• Minimum sensitivity to application and test case changes

• Test case definition separate from automation code

Maintainable

• Test coverage expanded efficiently

• Automated test cases created by non-automation resources

Scalable

© Utopia Solutions

SECTION 2

Why Do We Need a Strategy?


Why Test Automation Fails

Organizational

• Lack of support

• Unrealistic expectations

• Disjointed org structure

• Missing skill sets

Process

• Immature testing process

• Segregated processes

• Lack of test environment / data control

Technical

• Poor tool support for AUT

• Under or over design automation approach

© Utopia Solutions

SECTION 3

How Do We Know if Were Ready?


Are We Ready?


Test Automation Readiness Assessment

Org

• Structure

• Support

• Goals and Expectations

• Resources and Skills

Process

• Test Maturity

• Test Assets

• Process Integration

• Test Environment / Data

Tech • AUT Technology

• AUT / Tool Integration

Assessment Areas

© Utopia Solutions

Organizational Questions

• Expectations

– What are our goals for test automation?

– Do we understand the effort and cost to implement and maintain test

automation

– Do we understand the required skill sets for successful test automation?

• Structure

– How is the testing function organized?

– Where are the automation resources relative to everyone else?

• Resources

– Are our automation engineers allowed to focus on automation?

– Do our automation engineers have the appropriate skills?


Common Assessment Findings – Organization

Distributed Testing /

Automation Efforts

App

App

App

App • Uneven automation

resource demand

• Inconsistent approach

• Duplicated efforts

• Restricted skill growth and knowledge sharing

Leads to…

© Utopia Solutions

Process Questions

• Test Process

– What are the testing activities that can benefit from automation?

– Do we have testing assets that can promote automation?

• Development Process

– What type of development process is used for the AUT?

– Are the development and testing processes integrated?

• Test Environment

– What environments do we have available for test automation?

– Do we have control over the test data in these environments?


Common Assessment Findings – Process

Analyze Design Build

Test Deploy

No ability to identify impact of app changes on automated test suite

Unexpected / Unpredictable Maintenance


Process

Analyze Design Build Test Deploy

Capture potential impacts as early as possible to enable proactive maintenance

Capture App Changes

Analyze Impact

Maintain Execute

Automation Activities

Application SDLC


Common Assessment Findings - Environment

Shared Test Environment

Uncontrolled Test Data

Integration

Functional

Performance

Integration Test

• Schedule conflicts

• Unreliable execution

• Unnecessary “hardening” of the test suite

Leads to… Automation

© Utopia Solutions

Technical Questions

• Applications Under Test

– Do we know which AUTs should be targeted for test automation?

– Do we know the technologies of the AUTs in scope for test automation?

– Where are the AUTs in their application lifecycle?

• Test Automation Tool

– Do we have access to an automation tool (or tools)?

– Does our automation tool appropriately support our AUTs?

– If we don’t have a tool, do we understand how to select one?

© Utopia Solutions

SECTION 5

What Does a Strategy Look Like?


Mobile Test Strategy Topics

Test Automation

Strategy

Process & Organization

Environment

Technical Resources

Scope & Roadmap

© Utopia Solutions

Ideal Scope Attributes

Testing Support Activity

• Environment / data setup

• Tedious output verification

Test Coverage

• Focused on high frequency test testing activity

• NOT focused on automating manual test scripts verbatim

• Test cases distilled into levels

AUTs

• Supported completely by test tool

• Multiple AUTs from same development framework / technology

• Frequent release cycles

• Large regression effort

© Utopia Solutions

Ideal Organizational Attributes

Structure

• Centralized test automation function

• Automation roles are defined

Goals and Expectations

• Automation is an investment

• 100% automation is not likely

• Automation requires specialized skills

• Automation will free up resources to work on high-value tasks

Support

• Executive management has visibility to automation

• Automation has a dedicated budget

• Automation is represented appropriately on project plans

Resources and Skills

• Resources are dedicated to automation

• Resources are skilled in automation as a process AND technology

© Utopia Solutions

Ideal Process and Environment Attributes

Test Process

• Test objectives are clear and measurable

• Test cases are defined and documented

• Expected results are predictable

• Basic metrics for testing and automation

SDLC Integration

• AUT in a frequent release state with visibility to future roadmap

• Visibility to system changes as early as possible in release cycle

Test Environment

• Test environment dedicated to automation

• Test data controlled


Automation Strategy – Roadmap

• Begin with required organizational and process changes

• Overlap with base framework and minimal coverage

• Enhance and add test coverage over small iterations

Create Implementation Timeline

Phase n Phase 1

Continuous Process & Organizational Improvements

Base Framework

Initial Coverage

AUT 1

Framework Enhancements

Expand Coverage

Initial Coverage

Phase 2

AUT 2

Expand Coverage

Expand Coverage

Initial Coverage

Phase 3

AUT 3


Test Results

Test Automation Approach

Test Automation Framework

Automated Test Cases

Test Definition

Test Execution

Non-technical / Test Analysts & Subject Matter Experts

Technical / Automation Engineers


Regression Test Automation and Agile Cycles

App Dev

Cycle (Sprints)

User Stories

Application

Automation Requirements

Automation Dev Cycle

Automated Test Suite

Development and automation cycles NOT necessarily in

sync


Key Summary Points

• Test automation is NOT a tool!

• Understand your readiness to succeed from these perspectives…

• Organization

• Process & Environment

• Technical

• Develop a strategy to address readiness gaps and implement in small phases


Questions…

…. and Answers!

Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: [email protected] Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com

mailto:[email protected]

Don’t Be Another Statistic! Develop a Long-Term Test Automation Strategy

Software

Transcript of Don’t Be Another Statistic! Develop a Long-Term Test Automation Strategy