built by

Introducing the newQuery Wizards

We just made data testing REALLY EASY! 

No programming needed

Automate your Data Warehouse & Big Data Testing and Reap the Benefits

*available for download on August 3, 2015

built by

QuerySurge ™

About FACTS

Founded: 1996

Locations: New York (HQ), Atlanta, Philadelphia, Phoenix

Strategic Partners:IBM, Microsoft, HP, Oracle, Teradata, HortonWorks, Cloudera, Amazon

Software:

QuerySurge

RTTS is the leading provider of software & data quality for critical business systems

“70% of enterprises have either deployed or are planning to deploy big data projects and programs this year”

– analyst firm IDG

“46% of companies cite data quality as a barrier for adopting Business Intelligence products.”

- InformationWeek

“Poor data quality is a primary reason for 40% of all business initiatives failing to achieve their targeted benefits.”

- analyst firm Gartner

Data Quality Issues

built by

QuerySurge ™

2 Prevalent Data Testing Strategies

built by

QuerySurge ™

1) Stare & Compare

2) Minus Queries

Data Testing Strategy #1: Stare & Compare

built by

QuerySurge ™

• Review Mapping Document (business rules, data flow mapping, data movement requirements) • Write Tests in SQL editor• Execute 2 Tests: 1 at Source & 1 at Target • Dump results to 2 Excel files• Compare results by eye (Stare & Compare)

Difficulty with Stare & Compare:Impossible to visually compare millions/billions of data sets visually.

Example: Current QuerySurge customer has:

• a single test with 100 million rows • 200 columns • = 20 billion data sets • the client has > 7,000 total tests

Data Tester’s Current Process

built by

QuerySurge ™

MINUS QUERIES subtract one result set from another result set to show difference Comment: MINUS QUERIES need to be executed 2x (Source MINUS Target; Target MINUS Source)

Result sets may not be accurate when dealing with duplicate rows of data

No historical data from past testing – audit and regulatory issues

Processing of minus queries puts pressure on the servers

Double execution means 2x testing time and resource utilization

Potential for false positives (bad data could exist on both sides of an ETL leg)

Data Testing Strategy #2: Minus Queries

Minus Query #1: Table_1 MINUS Table_2

Minus Query #2: Table_2 MINUS Table_1

Result Set #1

Result Set #2

ISSUES with MINUS QUERIES

Write 2 MINUS queries in SQL editor

Execute MINUS queries 2x

Data Testing Strategies

built by

QuerySurge ™

a fundamental issue with the 2 Strategies:

Assumption that all team members understand and can write SQL code

What is QuerySurge ™?

the collaborative Data Testing solution

that finds bad data & provides a holistic view

of your data’s healthData Testing

built by

Improve the Health of your Data

• Reduce your costs & risks

• Improve your data quality

• Accelerate your testing cycles

• Share information with your team

with QuerySurge ™ you can:

built by

QuerySurge ™

• Provides huge ROI (i.e. 1,300%)*

*based on client’s calculation of Return on Investment

the QuerySurge advantage

built by

QuerySurge ™

Automate the entire testing cycle Automate kickoff, tests, comparison, auto-emailed results

Create Tests easily with no SQL programming ensures minimal time & effort to create tests / obtain results

Test across different platforms data warehouse, Hadoop, NoSQL, database, flat file, XML

Collaborate with team Data Health dashboard, shared tests & auto-emailed reports

Verify more data & do it quickly verifies up to 100% of all data up to 1,000 x faster

Integrate for Continuous Delivery Integrates with most Build, ETL & QA management software

Finding Bad Data

SQL

HQL

SQL

HQL

SQL

SQL

QS pulls data from data sources QS pulls data from target data store QS compares data quickly QS generates reports, audit trails

How?

Reports, Data Health Dashboard

built by

QuerySurge ™

Source Data

Target Data

QuerySurge™ Architecture

Web-based…

Installs on...

Linux

Connects to…

…or any other JDBC compliant data source

built by

QuerySurge ™

QuerySurgeController

QuerySurgeServer

QuerySurgeAgents

Flat Files

Collaboration

Testers - functional testing - regression testing- result analysis

Developers / DBAs- unit testing- result analysis

Data Analysts- review, analyze data - verify mapping failures

Operations teams - monitoring- result analysis

Managers- oversight- result analysis

Share information on the health of your data

built by

QuerySurge ™

built by

QuerySurge ™

built by

QuerySurge ™

From a recent poll1 of: • Big Data Experts• Data Warehouse Architects• Solution Architects• ETL Architects

Recent Survey: Data Experts

Consensus Answer: 80% of data columns have no transformation at all

Our Question: What % of columns in your Data Warehouse have no transformations at all?

1Poll conducted by RTTS on targeted LinkedIn groups

Why is this important?

Fast and Easy. No programming needed.

built by

QuerySurge ™

QuerySurge™ Modules

Compare by Table, Column & Row

• Perform 80% of all data tests

• Automatically generates SQL code

• Opens up testing to novice & non-technical team members

• Speeds up testing for skilled SQL coders

• provides a huge Return-On-Investment

built by

QuerySurge ™

QuerySurge™ Modules

3 Types of Data Comparison Wizards:

The Query Wizards also provide you with automated features for:o filtering (‘Where’ clause) ando sorting (‘Order By’ clause)

Column-Level Comparison:This is great for Big Data stores and Data Warehouses where tables will have some columns containing transformations and some columns with no transformations. Many tables and columns can be compared simultaneously and quickly.Table-Level Comparison:This comparator is great for Data Migrations and Database Upgrades with no transformations at all. Many tables can be compared simultaneously and quickly.Row Count Comparison:Great for all - Big Data stores, Data Warehouses, Data Migrations and Database Upgrades. Many tables and rows can be compared simultaneously and quickly.

Uses: Tests the columns that have no transformations,

which means it tests approximately 80% of your data store without you writing any SQL code

Tests: Big data, data warehouses

Value added: novice or non-technical: no coding needed,

productive immediatelyexperienced user: saves time

built by

QuerySurge ™

SQL

SQL

HQL

SQL

SQLHQLSQLSQL

SQL

SQL

SQL

SQL

HQLSQL

SQL

HQL

SQL

SQLSQL

built by

QuerySurge ™

pick Source & Target

pick Comparison Type

pick Tables & Columns

Auto-generated SQLAuto-generated SQL

pick Filter (‘where’) , S

ort (‘Order By’)

Uses: Verifies data loads when no transformation occurs

Tests: data migrations, upgrades

Value added: novice or non-technical: no coding neededexperienced user: saves time

SQLSQLHQL

SQL SQL

SQL

HQLSQL

SQL

built by

QuerySurge ™

Use: Verify that the amount of rows from the source match the amount from the target

Tests: Big data, data warehouse, data migration, database upgrades, data interfaces

Value added:novice: no coding neededexperienced user: saves time

built by

QuerySurge ™

SQLHQL

SQLSQL SQL

SQL

SQLSQL

SQL

HQL

HQL

HQL

_________Total

built by

QuerySurge ™

all QuerySurge™ Modules

Design Library

SchedulingDeep-Dive Reporting

Run Dashboard

Query Wizards

Data Health Dashboard

Design Library• Create custom Query Pairs (source & target SQLs)• Great for team members skilled with SQL

QuerySurge™ Modules

Scheduling Build groups of Query Pairs Schedule Test Runs for:

• immediately• at a specific date/time• automatically after build or

ETL process

built by

QuerySurge ™

Deep-Dive Reporting Examine and automatically

email test results

Run Dashboard View real-time execution Analyze real-time results

QuerySurge™ Modules

built by

QuerySurge ™

QuerySurge Test Management Connectors

built by

QuerySurge ™

Drive QuerySurge execution from your Test Management Solution

Outcome results (Pass/Fail/etc.) are returned from QuerySurge to your Test Management Solution

Results are linked in your Test Management Solution so that you can click directly into detailed QuerySurge results

• HP ALM (Quality Center)

• Microsoft Team Foundation Server

• IBM Rational Quality Manager

Integration with leading Test Management Solutions 

Licensing

License Types: Full User Participant User (i.e. read-only)

built by

QuerySurge ™

License Model for:

Perpetual own QuerySurgeTM, pay annual maintenance

Subscription use QuerySurgeTM for set period (12 months)

QuerySurge™

05/03/2023 27built by

QuerySurge ™

Training CoursesData Warehouse Testing• Data Warehouse & ETL Testing Fundamentals (1 day)• Fundamentals of QuerySurge (1 day)• Introduction to SQL for QuerySurge (1 day)• Advanced SQL techniques for QuerySurge (1 day)

Big Data Testing• Big Data And ETL Testing Fundamentals• Introduction To Big Data Testing Using Hive And HQL

ConsultingRTTS, the software quality experts (and developer of QuerySurge), provides consulting solutions to the challenges of Big Data & Data Warehouse / ETL Testing

• Jumpstart 2-week program – combines training courses, mentoring, consulting

• Staff Augmentation – add additional RTTS resources to your team

• Outsourcing - RTTS can perform all testing, including planning, design, execution

built by

QuerySurge ™

Support• Live Chat through QuerySurge or web site

• Email support through Zendesk

• Yammer Network

• Webex sessions

• Phone support

(1) Trial in the Cloud of QuerySurgeTM, including self-learning tutorial that works with sample data for 3 days

(2) Downloaded Trial of QuerySurgeTM, including self-learning tutorial with sample data or your data for 15 days

(3) Proof of Concept of QuerySurgeTM includes our team of experts assisting you for 30 days

for more information on (1), (2) and (3),

Go to querysurge.com/compare-trial-options

TRIAL IN THE CLOUD

built by

QuerySurge ™

Free TrialsQuerySurge™

Proof of

Concept

built by

QuerySurge ™

QuerySurge

For more on the Query Wizards, go to querysurge.com/querysurge-query-wizards