ETL Testing Strategy - Migration Projects

By – Nishikant Mathapati

Index

ETL Process

Test Strategy

Challenges / Risks

Solution for Risk

Benefits

2 ETL Testing Strategy - Migration

Project

ETL Process

The ETL process is being used to perform migration of data from

various legacy source systems to the target system.

Different ETL stages are -

3 ETL Testing Strategy - Migration

Project

Source Extracted

Data – Delta Stage 1

Transformed Data - Delta

stage 2

Data Loaded to

Target tables

ETL Process Contd…

Sample architecture

4 ETL Testing Strategy - Migration

Project

ETL Process Contd…

ETL process can be implemented using various tools like Cognos,

Informatica, Control-M etc.

Required fact and dimension tables will be created, mock / existing

production history like data will be acquired from business for testing

purpose.

Data cleansing will be done on data provided by business.

Workflows will be designed which will perform all requisite operations

on data and then data will be loaded to target system.

5 ETL Testing Strategy - Migration

Project

Test Strategy

Thorough testing for Migration process is important

The ETL process can be effectively validated using checks as

specified on next slide

6 ETL Testing Strategy - Migration

Project

Test Strategy Contd…

7 ETL Testing Strategy - Migration

Project

Challenges / Risks faced

Common issues which could be faced in every ETL project implementations are –

1. Workflow failures with reasons such as

- Database drivers not found

- Unique constraint violation

- Not enough disk space for delta stage tables

2. Data rejections due to reasons such as

- Data cleansing not done for fact tables

- Production look a like data not provided by business

- Revised data not uploaded to facts tables

8 ETL Testing Strategy - Migration

Project

Solutions

Liaise with development team and create a checklist for all

common failures which are repeated and can be avoided

Include this list as part of entry criteria to enter in to any testing

phase

Smoke test any new build delivered considering the prepared

checklist

E.g. – Run the workflow which will extract and run very large

amount of data in all delta stages and in target system

9 ETL Testing Strategy - Migration

Project

Benefits

A good confidence can be established on to the data tested

By avoiding common failures one can spend more time in

development and testing related activities

Data quality will be improved

A common practice can be implemented in similar projects where

Data Migration / Integration is implemented through ETL process

10 ETL Testing Strategy - Migration

Project

Q & A

Thank you all !!!

Questions Please

11 ETL Testing Strategy - Migration

Project