Ha(API)y testing in the hybrid-cloud & beyond

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Ha(API)y testing in the hybrid-cloud & beyondHybrid Automation (API) Testing

Jonathon Wright / “May the forth be with you”, 2014

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2

non-Ha(API)y testing

Currently we have over 5,000 end

users of HP ALM across the company

In a single ALM project we have 19,646 tests

scheduled for manual execution

Part of the current initiative we have

targeted 2,700 applications across the

company for automation

The current process is that we walk through

each test first before it can be automated


Each wave delivers automation for 250

projects across 100s of applications

Once delivered automation executes against

each build / release and can take over a week

non-Ha(API)y testing

The current initiative touches over

10,000 resources & stakeholders

All have reporting requirements ranging from

executive board level to individual teams

practicing agile and traditional test

methodologies


Adopting Global Agile Portfolio Management

Board of Directors

Exec Board

VP (Portfolio)

VP-1 (Domains)

VP-2 (Apps)

VP-3 (Projects)

VP-4 (Teams)

CIO

ITPO

ITDO

ITAO

ITPM

ITTM

CIO-1

ITPO-1

ITDO-1

ITAO-1

ITPM-1

ITTM-1

CIO-2

ITPO-2

ITDO-2

ITAO-2

ITPM-2

ITTM-2

CIO-3

ITPO-3

ITDO-3

ITAO-3

ITPM-3

ITTM-3

CIO-4

ITPO-4

ITDO-4

ITAO-4

ITPM-4

ITTM-4

Tier 0 - Programme – Executive Level (x4)

Tier 1 - Workstreams – Portfolio Level (x8)

Tier 2 - Domains – Solution Level (x330)

Tier 3 - Sub-Domains – Application Level (x2,738)

Tier 4 - Project-ID – Project Level (x7,324)

Tier 5 - Staff-ID – Team Level (x10,000+)


Adopting Ha(API)y testing

83%

6%

11%Automation (API)

Unsuitable

Unsupported

41%

33%

6%

20%Automation (UI)

No UI

Unsuitable

Unsupported

Native Support x2

Testing B(UI)ilds Ha(API)y Testing


TESTING

AS A SERVICE(TaaS)

INFRASTRUCTURE

AS A SERVICE(IaaS)

SOFTWARE

AS A SERVICE(SaaS)

REAL-TIME VISIBILITY

AND CONTROL

INSTANT SCALABILITYSCALE UP AND DOWN

HIGH AVAILABILITY

CMMI

PLATFORM

AS A SERVICE(PaaS)

ISO 29119PAY AS YOU USE(PAYU)

SLA’s

Adopting Cloud - Challenges & Opportunities

“We’ve all heard the claims that cloud computing will, without any up-front investment, provide instant

scalability, flexibility, and availability for testing-on-demand. But how well does this work in practice?”


Adopting Cloud – Testing Challenges (UI)

Focus shift right

from physical (UI)

Identity

Services

Connectivity

(HTTPS)

Workflows

(Logic)

DOM

(HTML5)

BIG Data

(Database)

External

Services

Workflows

(Logic)

• OCR / Image

• Object Models

• XPATH

• Regular

Expressions

• Descriptive

Programming

• Fuzzy Logic

• Artificial

Intelligence

• Social

Intelligence


Adopting Cloud – Testing Opportunities (API)

Focus shift right from physical (UI)

to logical (API) / messaging layer /

database layer

API

(SOA)

AJAX

CORBA

AMF

HTTP

SOAP

REST

Database(s)

ODBC

Ole DB

ADO .NET

JDBC

Java

Native

Messaging

ESB

AMQP

MSMQ

MQ

JMS

Tibco


Cloud Discovery – Learn physical endpoint(s)

SOA

Traditional functional testing focused on the physical (UI) endpoint

Traditional non-functional testing focused on the messaging layer endpoint(s)

Messaging Layer API SOA DBAPI

Physical Server Endpoint

Physical Client Endpoint

Functional Non-Functional

UI

Ha(API)y


Cloud Discovery – Model logical endpoint(s)

Endpoint discovery tool to map Cloud(s)

“Globally, mobile phone users generate 5 petabytes of mobile data”

petabytes[1pb equals 1,073,741,824mb]

Source – Harvard School of MT Lab (2014)

• Should we be adapting Network Fabric

Models to map our Private Cloud(s)?

• Imagine been asked as your first task to

index 14,000 database endpoints?


Hybrid Cloud demands Hybrid Testing

Manu’mation“Manu’mation is an approach to automated testing that leverages the manual effort

involved in traditional testing to encapsulate both user and system under test interactions”Source – Test Automation Patterns - Wikispace (2014)

Source - http://en.wikipedia.org/wiki/Hybrid_testing

Source - Experiences of Test Automation – Book (2012)

‘First Day’ Testing“The key to ‘first day’ test automation is to create a dynamic test repository cube based on

a combination of both fuzzy logic and descriptive languages (DSLs) to support the design

and development of tests before the actual delivery of the Solution Under Test (SUT)”

http://en.wikipedia.org/wiki/Hybrid_testing


Manu'mation

Adopting the ‘Test First Delivery’ (TFD) approach

Manu’mation

• Execute the manual tests via

HP Sprinter

• Capture test assets including

associated injected data set

• ‘Export to UFT’ as test artefacts that

can be extracted in meta data (XML)

Learn physical endpoint(s)

• Define physical endpoint(s)

• Learn (record) physical endpoint(s)

• Simulate (playback) virtual endpoint(s)

• Define (checkpoints) verification points

• Validate (replay) virtual endpoint(s)

Ha(API)y Testing

• Leverage HP UFT API Testing

• Enable Service Virtualisation

• Execute tests in minutes not hours

• Accurately report test results back into

HP ALM against each build / release

“Manu’mation is an approach to automated testing that leverages the manual effort involved

in traditional testing to encapsulate both user and system under test interactions”

Source – Test Automation Patterns - Wikispace (2014)

http://testautomationpatterns.wikispaces.com/MANU%27MATION

http://testautomationpatterns.wikispaces.com/MANU'MATION


Manu'mation powered by HP Sprinter

‘Export to UFT’ plus associated data injection

set

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. http://msdn.microsoft.com/en-us/bb906065.aspx

Cloud Testing – Testing Challenges

Focus shift from the private cloud network accessible with direct

access via the messaging layer

“A Message Bus is a combination of a common data model, a common command set,

and a messaging infrastructure to allow different systems to communicate

through a shared set of interfaces”

“An enterprise service bus (ESB) is a software architecture model used for designing and

implementing the interaction and communication between mutually interacting software

applications in Service Oriented Architecture (SOA)”

To secure (https) public cloud with limited access to the enterprise

(internet) service bus


Cloud Testing – Learn physical endpoint(s)HP Service Virtualisation

• Learns from the real

endpoint(s)

• Simulate response using

recorded or user-defined

response

• Emulate endpoint(s)

performance


Application

Server

Rules

Engine

MQ API

Data

Warehouse

Cloud Testing – Define virtual endpoint(s)

Application

Server

Rules

Engine

MQ

Discover

Data

Warehouse

XML XML

APIMQXML

Test Execution (API)

Service Virtualization

Learn physical endpoint(s)

Define virtual endpoint(s)

Stub component(s)

Shim method(s)

Stub database(s)

API


Embracing Hybrid Cloud – Testing Challenges

Introducing emerging complex

cloud ecosystems

(commercial or community)

• Third-party public / private cloud endpoint(s)

– Access to live public endpoints for testing?

(test.paypal.com)

– Access to private endpoints for testing?

(VPN tunneling)

• Security challenges around cloud platforms

• Big Data (masking / protection / regulations)

• Cloud 2.0 (moving away from legacy cloud)

– BlackOps “data-in, data-out’ (DIDO) model


PublicVPN Tunnelling

Private

SUT SUTSUT

Community

Clouds

Private

Clouds

Public

Clouds

VPN Endpoint VPN Endpoint

Hybrid Cloud


Hybrid Cloud

Public

Clouds

Private

Clouds

Hybrid Cloud

VPN Tunnelling

Public

VPN Endpoint

Private

VPN Endpoint

SUT SUTSUT


Syn

c

Systemic Failure

(DNS or Network)

Hybrid Cloud – Testing Challenges

How do you provide realistic DR / resilience testing in the public cloud?

“Don’t assume the ecosystems (commercial or community) and

other fulfilment systems to be tested in anything but isolation.” - Julie Gardiner (2012)


‘First Day’ Testing

Adopting the ‘Test First Delivery’ (TFD) approach

TFD >= ATDD/TDD/BDD

• Create test definition(s)

• Model the sequence of test event(s)

• Build the Control Flow within UFT

• Define (checkpoints) verification

points

Define virtual endpoint(s)

• Define virtual endpoint(s) in SV

• Define (mock) the rule(s) in SV

• Generate (emulate) data-driven in SV

• Simulate (playback) virtual endpoint(s)

• Validate (replay) virtual endpoint(s)

Ha(API)y Testing

• Execute UFT BPT components

• Enable Service Virtualisation

• Execute tests without dependency of

the Solution Under Test (SUT)

• Leverage SV with HP LR

“The key to ‘first day’ test automation is to create a dynamic test repository cube based on a

combination of both fuzzy logic and descriptive languages to support the design and

development of tests before the actual delivery of the Solution Under Test (SUT)”

Source – Experiences of Test Automation (2012)

http://www.dorothygraham.co.uk/automationExperiences

http://www.dorothygraham.co.uk/automationExperiences


‘First Day’ - API Testing – Test Designer

Drag-and-drop test

elements to the canvas

DBDB

SOA

DBDB

Enterprise

DBDB

Middleware

DBDBSystem

DBDB

Flow Control

DBDBCustom


‘First Day’ powered by HP UFT (API Testing)

vs. PCAP packet capture (promiscuous mode)Define virtual endpoint(s) Learn physical endpoint(s)


Endpoint(s) Service Modelling

‘First Day’ powered by HP Service Virtualization

Endpoint(s) Performance Modelling Endpoint(s) Data Modelling


Leveraging the ‘First Day’ approach of

‘Test First Delivery’ (TFD) utilising

HP Service Virtualisation & UFT (API)

Enabled us to execute all of the tests without

the dependency of the Solution Under Test

(SUT) from the very ‘first day’ in Sprint 0.

Ha(API)y testing

Adopting the ‘Manu’mation’ approach of

‘Test First Delivery’ (TFD) utilising HP Sprinter

Enabled us to capture 19,646 tests over 4 major

releases across 12 sprints.


select <IaaS>

select <PaaS>

select <SaaS>

select <TaaS>

select <TaaaS>

Cloud Recipe for Solution Under Test

Cloud Recipe (IFTTT)

Cloud 2.0 – Moving beyond the legacy cloud

“IBM are working with other cloud suppliers to establish the standards for cloud service provider compliance.”

- Murat Gunenc, IBM Cloud Representative @Rhodes W1 (2011)

CloudSystem Automation

YourTestCloud.com

“Heterogeneous test cloud that are open, flexible and extensible”

Testing Cloud

If <Solution Under Test>

Then <Build Cloud 2.0>

Testing Cloud


Final Thoughts

Thank you

“The best aspects of these proven approaches demonstrate

how they have evolved over the past decade, and this echoes

some of the changes towards more lean and agile business

methodologies. They are in a constant state of evolution –

just as the underpinning technology evolves over time.”

My section in the ‘Experiences in Test Automation’ book

(Dorothy Graham & Mark Fewster) 2012.

Ha(API)y testing in the hybrid-cloud & beyond

Technology

Transcript of Ha(API)y testing in the hybrid-cloud & beyond