Service Overview Business Assurance · management/new-model-testing-new-test-process-tool/ An...

Intelligent Definition and Assurance 12-Jun-17

© 2017 Gerrard Consulting Limited 1

New Model Testing:A New Test Process and Tool

@pau

l_ge

rrard

Paul [email protected]

gerrardconsulting.com

“Exploratory Testing

and bots for navigation

and insight”

My Goals

• Can the best of structured/exploratory testing

approach be merged to create a better

process?

• Share some R & D with you

• Get you thinking about your test process and

opportunities to improve

• Demo a tool I’m working on

• Invite collaboration.

Intelligent Definition and Assurance Slide 2



Testing Styles, Approaches

“Structured/Waterfall/

Staged” Testing• Systematic

• Transparent, Documented

• Reviewable

• Auditable

• Repeatable, measurable

• Automatable

• Inflexible, not responsive

• Obsolescent/inaccurate documentation

• Prone to biases, inattention

• Outdated process

• Expensive, Inefficient

• Unimaginative, boring

“Exploratory” Testing

• Agile (with a small ‘a’)

• Improvised, imaginative

• Flexible and responsive to change

• Faster, cheaper

• More effective

• Personally enjoyable

• Not repeatable

• Not easily automated

• Little or no documentation

• Hard to manage

• Hard to scale

• Opaque

• Not auditable, measurable.

Intelligent Definition and Assurance Slide 3Some of these are perceptions – which vary

An Obvious Question

Is it feasible to combine the best of

structured and exploratory testing

and create a new test approach?



Different strokes for different folks

• I’ve never believed that these styles are ‘opposites’, rather they differ in emphasis

• Like the Agile manifesto, it’s more a set of values/preferences that drive behaviour:

– Differences in scale and timing

– Planned v improvisational

– Heavy v light documentation

– Process and governance v agility and freedom

– Etc. etc.

• There are many approaches, not just two.


Profiles of the three patterns

Characteristic Structured Agile Continuous

Structure Managed Autonomous Production Cell

Pace/cadence Business decision Team decision Feedback

Leadership Project Managed Guided Research Line Managed

Definition Fixed spec Dynamic spec Live Specs

Testing Scripted Exploratory Automated

Auto. Test Retrospective Developer led Pervasive

Measurement Pervasive Avoided Analytics

Governance Bureaucratic Trust-based Insight-Driven




Forget Logistics(for the time being)

Document or not?

Automated or manual?

Agile v waterfall?

This business or that business?

This technology v that technology?

ALL Testing is

Exploratory

We explore sources of knowledge ...

... to build test models ...

... that inform our testing.



Judgement, exploring and testing

Testing(the system)

Our model(s) are adequate

Our model(s) are not adequate

Exploring (sources)

Judgement

Createsmodels

Usesmodels

We explore sources of knowledge to build test models that inform our testing

BTW – Do Developers think/explore the same way? I think so.

New Model Testing

My BBC talk: http://www.bbc.co.uk/academy/technology/article/art2015052211302939829 page paper: http://dev.sp.qa/download/newModel

http://www.bbc.co.uk/academy/technology/article/art20150522113029398

http://dev.sp.qa/download/newModel



The New Model

• Makes no assumption about logistics, context

• It is not a process model with entry/exit criteria, procedures, deliverables etc.

• All models are wrong, but I believe this is useful

– An attempt to understand our thought processes

– Our brains can work on several processes simultaneously

– Can help us better understand, information flows, feedback and review processes, automation etc.

– Focuses attention on (test) model-making.


Will Robots Replace

Testers?

Some research

There is a paper at:

https://tkbase.com/resources/viewResource/14

https://tkbase.com/resources/viewResource/14



A recent study*…

• Over the next two decades, 47% of jobs in the US may be under threat

• It ranks 702 occupations in order of their probability of computerisation– Telemarketers: 99% likely

– Recreational therapists: 0.28% likely

– Computer programmers: 48% likely

• Something significant is going on out there

• If programmers have a 50/50 chance of being replaced by robots, we should think seriously on how the same might happen to testers.

* “The future of employment: how susceptible are jobs to computerisation?”

http://www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdfIntelligent Definition and Assurance Slide 13

Some systems-related occupations


Occupation Rank (out of

702)

Probability of

Computeris-

ationComputer and Information Research Scientists 69 1.5%

Network and Computer Systems Administrators 109 3.0%

Computer and Information Systems Managers 118 3.5%

Information Security Analysts, Web Developers,

and Computer Network Architects

208 21%

Computer Occupations, All Other 212 22%

Computer Programmers 293 48%

Computer Support Specialists 359 65%

Computer Operators 428 78%

Inspectors, Testers, Sorters, Samplers and

Weighers

670 98%



Test Automation =

Mechanical Tools

What we REALLY need are

THINKING TOOLS

The term Test Automation misleads

• It misleads as a label because the whole of

testing cannot be automated

• The label is bad, but the scope of Test

Automation is what I call ‘Applying’ in the

New Model




Testers need Thinking Tools

• There are ten testing activities in the New Model– Test automation tools only support one: ‘Applying’

• The remaining nine activities (information gathering, analysis, modelling, challenging, test design and so on) are not well supported

• All require some level of thinking and skills

• Checking is possible when a system and its purpose are well understood and trusted

• Test automation tools are simple in principle…… compared to the rest of the test process.


Four quadrant model of intelligent

test tools

Ability to Investigate

Ab

ility

to

Cap

ture

Kn

ow

led

ge

• Text editors, Screen Shots

Models, visualisations, relationships, transformations

• Note Takers

• Mind Maps

• UML/Case Tools

Co

ntro

l, imagin

ation

, discern

men

t, foresight

• Pencil and paper, sketching tools





TERMINATOR

TESTERNot Yet!

Cervaya™

Tool Architecture

A nine month bot journey

(But it’s been a twenty year testing

journey so far)



The vision thing

• I want a bot partner/pair that supports exploratory testing

• You know my view (model)of testing already

• Can we use the explore v test paradigm in a bot that allows you to:– Explore, take notes and model

– Record ideas, risks, tests, outcomes and bugs

– Generate reports and documentation as a consequence, rather than requiring you write them

• Codename: Cervaya (cervaya.com is a holding page for now)


Schematic


Schema Server

Schema Manager(web site)

Schema Repository

(web service)

Cervaya Bot Client

Robot Engine

Cervaya Server

Actions(web services)

Application Web Site

Speech Recognition

Interface

Command Line

Interface

Administer Schemas through the Web

DownloadSchema Perform Robot

Actions through services

Application Reporting, Monitoring, Control,

Management

Many App Servers – one (or many) per Schema

robotschemascervaya



The target user is…

• (Not really a freestyle exploratory tester working in a start-up or small product company)

• Target users are testers working in:– Regulated, high-integrity, safety critical

– High documentation, high accountability

– Environments where testers are constrained by their processes

– Testers struggling to apply ‘trad’ methods in an Agile, Digital, DevOps environment

• Why can’t we dictate a detailed test plan for a bot to document, analyse, visualise?


Bot Schema – State Model+

Slide 24Intelligent Definition and Assurance



Simplified Bot Schema


Session

Notes

Observation

Navigation

Modelling

Home

Always start here –ready to commence the

session

In session – everything is done inside a session

Places, features, forms and fields comprise the

model

Notes and observations are related to places,

features, forms or fields

Notes can be ideas, concerns, risks, questions etc.

Test

Tests focus on forms and fields (but also end-to-

end scenarios)

The location hierarchy

• Explorers create maps (or should do!)

• The system model/map is a hierarchy of locations as follows:

• Application

– Places

• Features

– Forms

» Fields

• All other content is located with respect to some level and location in the system model.




Demo

https://huddle.eurostarsoftwaretesting.com/resource/test-

management/new-model-testing-new-test-process-tool/

An early, partial prototype

31:10 to 43:26

Demo agenda

• Cervaya– apps, versions and charters

• Speechbot

• Thinkbot

• Cervaya– Explorer

– Session Status/Log

– Creating and running tests

– Viewing the model

• D3 and system maps/data visualisations.

https://huddle.eurostarsoftwaretesting.com/resource/test-management/new-model-testing-new-test-process-tool/



A New Test Process

Exploration support

• Frustration: – testers are unimaginative, working by-rote

– constant pressure to cut costs

• Productivity of exploratory testing is proven

• Testers want to explore, but the need for control and documentation constrains them

• Testers needs tools that can capture plans and tester activity in real-time

• The next generation will be led by tools that support the exploration of sources of knowledge

• These tools might use a “Surveying” metaphor.




Staged Process

System Testing Re-Test

Big, up-front Test Planning

RegressionTest

Development

Automation

From Staged to Continuous

Scoping exploration sessions

Interactive Testing Sessions

Clarifications

Continuous Integration

ChangesRefinements

Continuous Dev, Test and Delivery

Bot Supported

Bot Supported

Test Process

• Cervaya can be used to create test plans

– Initially based on documentation

– Evolving plan based on chartered sessions

• Testing in the small – chartered testing

• Testing in the large – Cervaya has the building

blocks for end to end tests

• Could it generate the documentation

required in regulated industries? Why not?




Speech recognition, text to speech

• Right now, I’m using:

– Google Speech Recognition (a slow web service)

– pyttsx a Python Text to Speech module (depends on

pywin32-extensions for Windows)

• I’m looking at Amazon Alexa as an alternative

– https://www.youtube.com/watch?v=hPXS7rC1PWo

– Alexa is the web service, Echo Dot is the latest device

that captures and synthesises voice

– It promises to be a lot faster… so I’ve ordered one.


Real-Time test management

• Once you have a shared system model:

– The activity of the team can be visualised in real-time

– Tester activity: visible – testers liaise when necessary

– Test paths through the system map can be traced

• Developers can also see progress and requests

for more information in context:

• Cervaya has an integration with Slack:

– Model changes, additions, queries, test failures can be

posted to Slack/ChatOps service.


https://www.youtube.com/watch?v=hPXS7rC1PWo



From Staged to Continuous Testing

• Waterfall projects create, execute tests in stages

• Shifted-left testers can:

– Model features as they are created by the developers

– Create accurate tests a low level as the functionality emerges over time

– End to end tests use low-level tests as building blocks

• Tests and test run histories are manage against versions (and platforms/environments soon)

• The aim: to export tests as soon as practical to be automated to provide continuous feedback.


The best of both worlds?

“Structured/Waterfall/

Staged” Testing

• Systematic

• Transparent, Documented

• Reviewable

• Auditable

• Repeatable, measurable

• Automatable

“Exploratory” Testing

• Agile (with a small ‘a’)

• Improvised, imaginative

• Flexible and responsive to

change

• Faster, cheaper

• More effective

• Personally enjoyable


We need to change the testing metaphor



Surveying and testing – the new

process?• The “tester as surveyor” affects the relationship

of testing to development

• A new style of testing process emerges:

– No more documentation created in a knowledge vacuum

– Iterative, incremental knowledge capture closely aligned with the emergent delivery of functionality

• Could this be an Agile test process at last?

• It fits the increasingly popular Continuous Delivery, DevOps development approaches.


Opportunities - 1

• Given a model of the SUT, it is viable to export a ‘template’ automated script

• Maintaining the link between Cervaya and a test execution engine means:

– Can run ‘manual’ tests using a tool

– Can generate all-pairs, all combo tests or other tests that are tagged

– See results mapped to the model – test visualisations

– Bots can provide navigation support, (some) checks and balances and feedback.




Opportunities - 2

• The interactions between tester, Cervaya, test automation and the SUT means:– Can record interactions in a richer way

– Capture patterns of success and failure

– Handoff the data to Machine Learning tools

– The bot can provide hints and reminders

• Ultimately, ML tools might be able to:– recognise UI elements

– construct models

– generate and execute tests for you

• I predict – less than 5 years away.


I’m looking for

Collaboration

Want to know more?

Email [email protected]

mailto:[email protected]



Q & maybe some A

System Surveying

• A System Survey incorporates features, and captures the architecture of the system from a user perspective– Testers pair with developers and survey features

– The knowledge required to build systems emerges over time

– So does the design of the system

– So should the models and documentation produced by testers

• Surveys that evolve the System Model/Map are shared

• The basis of component and system tests is the System Model itself

• No need for extensive scripts or test procedures!

• The information required for scripting is in the model.




A process that suits automation

• Test process comprises a sequence of parallel actions– Sequence: survey, model refinement then testing

– Parallel: small subsets of functionality selected for surveys

– These processes are both iterative and incremental as learning proceeds

• Scalable: if you survey it, you can test it

• Automate: What you can survey and test, you can probably automate

• “Humans make the early maps; tools will follow the trails we make.”

• We don’t need Machine Learning to do this:– Simple tools make suggestions that better inform and enrich

exploration and testing.


All exploration and testing is

driven by charters

• Charters represent the low-level plans for all

sessions

• Charters can be reused for multiple sessions,

if required




Services

• Can Cervaya handle web services as objects

to test?

• In principle yes – use your imagination:

– Map the form/field information to service/fields

– Service tests map directly to form tests

• But we need to create an option that doesn’t

require you to use your imagination.


Sessions and logging

• Sessions can be as short or as long as you like

• Testers typically work in 60-120 minute bursts

– 90 minutes is typical

• Every action in a session is logged against the

session (locations selected, created, notes

taken, test created and run)

• The activity of all testers is logged and can be

scrutinised for audit purposes.




Running a Test

• When you run a test

– You are presented with the test details (navigation, input values and expected outcomes)

– You can log a test status, outcomes if different from expected and an interpretation

– You can also log a screen shot or other file

• The test run log record the detail above and tags the run with the app version/build defined in the charter

• You can (if required) run the same test multiple times in the same session and against the same version.


Challenges, opportunities

• Is it feasible to generate

a high or low level test

plan from Surveyor?

• Environments/platforms

• Integrations

– Chatbot – Slack etc.

– Internal IM

– Stories

– Test automation a la

BDD – both ways

• Visualisations

– Coverage

– Reporting

– Impact analysis

• Other benefits

– Instant replays

– Heat map, floor plan

– Tester paths

– Real time notifications

– Real time test(er) management




New Model Testing:A New Test Process and Tool

@pau

l_ge

rrard

Paul [email protected]

gerrardconsulting.com

Service Overview Business Assurance · management/new-model-testing-new-test-process-tool/ An...

Documents

Transcript of Service Overview Business Assurance · management/new-model-testing-new-test-process-tool/ An...