Data Warehouse Architecture

-Processes-

Overview

• Architecture – a technical blueprint stage

• Must support 3 major driving forces:– Populating the warehouse

• Data extraction, cleaning and loading

– Day-to-day management of the warehouse• Large volumes of data, create/delete summaries

– The ability to cope with requirement evolution• Cope with future changes with query profiles

Typical Process Flow

• Extract and load the data

• Clean and transform data into a form that provides good query performance

• Backup and archive data

• Manage queries, and direct them to appropriate data sources

Extract & Load Process

• Extract– Takes data from source systems and make it available to the

data warehouse

• Load– Takes extracted data and loads it into the data warehouse

• Data in operational systems is held in a from suitable for that system

• Before loading the data into the DW, information content must be reconstructed

• Data must become value added business information– Extract & load process must take data and add context and

meaning

Issues with ELP

• When to start extracting the data, run transformation and consistency checks and so on?– A controlling mechanism is essential to fire each

module when appropriate

• When to extract?– Data must be in consistent– Start extracting data from data sources when it

represents the same snapshot of time as all other data sources

• Eg. Customer database

Issues…

• Loading the data– Extracted data are loaded into temporary data store

to perform clean up and check for consistency– Do not execute consistency checks until all the data

sources have been loaded into the temporary data store

• Eg. Customer canceling subscriptions

– Error recovery must be an integral part of the design– The effort required to clean up the source systems

increases exponentially with the number of overlapping data sources

Issues…

• Copy Management tools and clean up– Eg. IBM’s Information Warehouse Framework

• Data Refresher & Data Hub

– Most copy management tools do not have the capability of performing consistency check directly (user must write the logic & code it)

– Make cost-benefit analysis before purchasing copy management tool

• If source systems do not overlap, then consistency checks are very simple

Clean and Transform Data

• Steps involved are:– Clean and transform the loaded data into a

structure that speeds up queries– Partition the data to speed up queries,

optimize hardware performance and simplify the DW management

– Create aggregations to speed up the common queries

Clean and Transform Data…

• Data needs to be cleaned and checked in the following ways:– Make sure data is consistent with itself– Make sure data is consistent with other data within the same

source– Make sure data is consistent with data in the other source

systems– Make sure data is consistent with the information already in the

DW

• Once data is cleaned, convert source data into a structure that is designed to balance query performance and operational cost– The structure must be suitable for long term storage

Backup & archive

• Regular backup is essential to recover data from loss

• Archiving– Older data is removed from the system in a format

that allows it to be quickly restored if required– Issue

• As DW evolves, all information may change• Hence to ensure that a restored archive is valid, it becomes

necessary to extract all related data and structures as well

Query Management Process

• It is a system process – Manages the queries – Speeds them up by directing queries to the

most effective data source– Ensure that all system resources are used

effectively– Monitor query profiles – manage which

aggregations to generate– This process operates at all times