Automade

Test Automation

Data Vault and Data Warehouse Automation

9th of December

Stefaan De Vos

Test Automation

Challenges

• Lead times of BI / Analytical projects are ever decreasing• Tools

• Automation (Wherescape...)

• Virtualization (Denodo, CIS...)

• Appliances (Netezza, DATAllegro...)

• Methodologies

• Data Vault

• Agile DWH

BI went Agile, but Testing didn’t

Challenges

• Data• Larger data volumes

• IoT

• Unstructured data / poor data quality• Social networks

• No relevant test data sets available

• The number of possible test cases are near infinite

Data Volume & Quality

Challenges

Review

requirements

Review

requirements

Define testing

Strategy

Define testing

StrategyPrepare Test DataPrepare Test DataEntry / Exit criteria

Design Test Cases &

Scripts

Design Test Cases &

Scripts

Configure the test

environment

Configure the test

environment

Prepare & Design

Agree on Entry / Exit

test criteria

Agree on Entry / Exit

test criteria

Integration

testing

Integration

testing

Performance testing Performance testing

Acceptance testing Acceptance testing

- Basic testing

- DI jobs accessible

- Reports accessible

- Cleansed

- Complete

- Correct

- Integrated

- Valid reports

- Relevant data

- Available data

- Consistent data

- Accessible

- DI & BI integration

- Full test cycles

- Check NFRs

- Scalable

- Check SLAs

- Peak user

- Peak loads

- Functional

- User

- Production

- SME

- End user

Construct

Defect metrics

review

Defect metrics

review

Performance

statistics

Performance

statisticsLessons learntLessons learnt

Process & Quality Improvement

Accept

Data Completeness

testing

Data Completeness

testing

BI & Analytical

Testing

BI & Analytical

Testing Smoke test

Unit test

Smoke test

Unit test

BI Testing Lifecycle

Challenges

** Regulatory compliance might require to use the v-model, e.g. validated environments.

Functional

analysis

Requirements

functional

Non-functional

Technical

Design

ConstructionUnit

Smoke Test

Data

Completeness

BI &

Analytical

Integration

testing

Performance

User

ProductionAcceptanceValidation

Verification

BI Testing V-Model

Challenge

• Lots of data

• Lots of testing to be done

• Little time to do it

Summary

Test Automation

Complete DWH Testing

• Complete testing is a must

• It requires:• Testing methodology

• Right project culture/mindset and organization

• Tools

Our View

Complete DWH Testing

Database

integrity

checking.

Risk based

testing

Effective defect

management

and

collaboration

End to End

Performance

testing

Adherence to

compliance and

regulatory

standards

and…AUTOMATE

Critical Success Factors

Test AutomationLeverage social development principles to deepen functionalities

Testers

- functional testing

- regression testing

- result analysis

Developers / DBAs

- unit testing

- result analysis

Data Analysts

- review, analyze

data

- verify mapping

failures

Operations teams

- monitoring

- result analysisCo

lla

bo

rati

on

/ W

ork

flo

wTe

st M

an

ag

em

en

t

Rational Quality Manager JIRA Team Foundation ServerQuality Center

Test Automation

What to Automate?

Complex

Functional

SQL

Validation

Reconciliation

Test Automation

Basic functionality

• Auto detection of anomalies (or at least prior to being detected by a user)

• Targeted regression testing for planned changes

• Data error identification errors by comparing (huge) data result sets

• Data error detection via a rules engine

• In between data layer reconciliation and auditing

• Test data generation

Additional functionality

• No programming or coding

• Heterogeneous connectivity

• Collaboration and workflow capabilities

• Visually attractive development and monitoring environment

• Intuitive reports & dashboards

Functionality

Test Automation

• Shorten regression cycles

• Save report developers time

• Test the same data set in less time

• Test more

• Faster deployment of defect resolution cycle

• Faster deployment of enhancements

• Less cumbersome upgrades/migrations

• Enabler for• Implementing continuous testing• operationalization of testing

Benefits

Test Automation

Vendors - tools

• ICEDQ

• RTTS -QuerySurge

Unconnected ETL

Source Data Layer Target Data Layer

ETL - ELT

1 3

2

Load test data

Execute Job externally

Extract result

4 Compare & report

• Validation of the business rules implemented in ETL processes are assessed by comparing the results against a ground truth.

• The ETL processes are executed separated from the test automation tool

• Less adequate for multi-step ETL processes.

Approach

Test Automation

Vendors - tools

• Zuzena

Connected ETL

• The business rules present in the ETL tool are analyzed and the test results are assessed against the anticipated results.

• The test automation tool executes the ETL processes.

Approach

Source Data Layer Target Data Layer

2 4Load test data Extract result

5 Compare & report

1

Analyze logic

3

Run job

Test Automation

Vendors - tools

• Report Valid8tor (BO)

• 360Bind (BO)

• Integrity Manager (MSTR)

• Report Validator - BSP Software (Cognos)

Report Integrity validator

• Parallel testing of 2 live systems

• Comparing against a (historical) ground truth

• Comparing against known good baselines

Approach

Reports

Reporting data Layer

Load test data Scrape data

4 Compare & report

1 3

Run report

2

Test automation

QuerySurge

• Integrates with HP QC / IBM RQM / MSFT TFS

• Provides collaboration features

pulls data from data sources

pulls data from target data store

compares data quickly

generates reports, audit trails

reports

SQL

Design Tests

Scheduling

Reporting

Run

Dashboard

Wizards

Data Health

Dashboard

• a SQL execution and data comparison tool working against heterogeneous datasources.

• No programming needed

Automade introduction

• Automade• Provides an agile answer to the ever-increasing information appetite

• By automating the mind-numbing aspects of constructing, maintaining and testing of data warehouses.

• Automade is a spinoff of MindThegap and has established partnerships with

Test automation

Thank you for listening,

Any questions?

Feel free to send questions & feedback to stefaan.devos@automade.be