Informix physical database design for data warehousing

0

Physical Database design for Data Warehousing

Keshava Murthy,

Architect, IBM Informix Development

1Source: Forrester

Query Tools

Analytics

BPS Apps

BI Apps

LOB apps

Databases

Other transactional data sources

I/O & data loading

Query processing

DBMS & Storage mgmt

Enterprise Data Warehouse

4

Data Warehouse Schema and Queries.

• Characterized by:

– “Star” or “snowflake” schema:

– Complex, ad hoc queries that typically

• Look for trends, exceptions to make actionable business decisions

• Touch large subset of the database (unlike OLTP)

• Involve aggregation functions (e.g., COUNT, SUM, AVG,…)

City

Region

Store

SALES

Product

Period

Brand

Month

Quarter

Category

Dimensions

Fact Table

5

Store Sales ER-Diagram from TPC-DS300GB database

287,997,024

20

73,049

1,920,800

1000

204,000

1,000,000

402

86,400

7200

2,000,000

6

select s_store_name, s_store_id,

sum(case when (d_day_name='Sunday') then ss_sales_price else null end) sun_sales,

sum(case when (d_day_name='Monday') then ss_sales_price else null end) mon_sales,

sum(case when (d_day_name='Tuesday') then ss_sales_price else null end) tue_sales,

sum(case when (d_day_name='Wednesday') then ss_sales_price else null end) wed_sales,

sum(case when (d_day_name='Thursday') then ss_sales_price else null end) thu_sales,

sum(case when (d_day_name='Friday') then ss_sales_price else null end) fri_sales,

sum(case when (d_day_name='Saturday') then ss_sales_price else null end) sat_sales

from store_sales, store, date_dim

where d_date_sk = ss_sold_date_sk and

s_store_sk = ss_store_sk and

s_gmt_offset = -5 and

d_year = 2002

group by s_store_name, s_store_id

order by s_store_name, s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales

Aggregates

Fact table

Dimension tables.

Equijoins between primary(dimension) and

foreign keys(fact)

Predicates on the dimension tables

Grouping and ordering.

7

select first 100 i_item_id,

avg(ss_quantity) agg1,

avg(ss_list_price) agg2,

avg(ss_coupon_amt) agg3,

avg(ss_sales_price) agg4

from store_sales, customer_demographics, date_dim, item, promotion

where ss_sold_date_sk = d_date_sk and

ss_item_sk = i_item_sk and

ss_cdemo_sk = cd_demo_sk and

ss_promo_sk = p_promo_sk and

cd_gender = 'F' and

cd_marital_status = 'M' and

cd_education_status = 'College' and

(p_channel_email = 'N' or p_channel_event = 'N') and

d_year = 2001

group by i_item_id

order by i_item_id;

Aggregates

Fact tableDimension tables.

Equijoins between primary(dimension) and

foreign keys(fact)

Predicates on the dimension tables

Grouping and ordering.

8

1) dwa_ds.store_sales: INDEX PATH (SKIP SCAN)

(1) Index Name: martinfu.sold_date_store_salesIndex Keys (Detached): ss_sold_date_sk (Parallel, fragments: ALL)Lower Index Filter: dwa_ds.store_sales.ss_sold_date_sk = stream from dwa_ds.date_dim.d_date_sk

2) dwa_ds.customer_demographics: SEQUENTIAL SCAN

Filters:Table Scan Filters: ((dwa_ds.customer_demographics.cd_education_status = 'College' AND dwa_ds.customer_demographics.cd_marital_status = 'M' ) AND dwa_ds.customer_demographics.cd_gender = 'F' )

DYNAMIC HASH JOINDynamic Hash Filters: dwa_ds.store_sales.ss_cdemo_sk = dwa_ds.customer_demographics.cd_demo_sk

3) dwa_ds.promotion: SEQUENTIAL SCAN

Filters:Table Scan Filters: (dwa_ds.promotion.p_channel_event = 'N' OR dwa_ds.promotion.p_channel_email = 'N' )

DYNAMIC HASH JOINDynamic Hash Filters: dwa_ds.store_sales.ss_promo_sk = dwa_ds.promotion.p_promo_sk

4) dwa_ds.item: SEQUENTIAL SCAN

DYNAMIC HASH JOINDynamic Hash Filters: dwa_ds.store_sales.ss_item_sk = dwa_ds.item.i_item_sk

5) dwa_ds.date_dim: SEQUENTIAL SCAN

Filters:Table Scan Filters: dwa_ds.date_dim.d_year = 2001

DYNAMIC HASH JOIN (Index Push Down Key: dwa_ds.date_dim.d_date_sk to dwa_ds.store_sales)Dynamic Hash Filters: dwa_ds.store_sales.ss_sold_date_sk = dwa_ds.date_dim.d_date_sk

9

Store_sales

customer_demographics

item

date_dim

promotion

287,997,024

300GB database

1,920,800

204,000

73,049

1000

(p_channel_email = 'N' or p_channel_event = 'N')

d_year = 2001

cd_marital_status = 'M' and

cd_education_status = 'College'

10

Store_sales

customer_demographics (1.9M)

Item (200K)

Promotion (1000)

287,997,024

100GB database

date_dim (73K)

Skip Scan

Sequential Scan

Hash Join:

ss_cdemo_sk = cd_demo_sk

Hash Join: ss_promo_sk = p_promo_sk

Hash Join: ss_item_sk = i_item_sk

Hash Join ss_sold_date_sk = d_date_sk

(p_channel_event = 'N' OR p_channel_email = 'N' )

Index Push Down Key: (d_year = 2001)d_date_sk to dwa_ds.store_sales

((cd_education_status = 'College'

AND cd_marital_status = 'M' ) AND cd_gender = 'F' )

Sequential Scan

Sequential Scan

Sequential Scan

Index scan Sort rowids

Probe

Build

Build

Build

Build

Probe

Probe

Probe

11

Store_sales

customer_demographics (1.9M)

Item (200K)

Promotion (1000)

Start Fact table scan

300GB database

date_dim (73K)

Skip

Scan

Sequential Scan

Hash Join:

ss_cdemo_sk = cd_demo_sk

Hash Join: ss_promo_sk = p_promo_sk

Hash Join: ss_item_sk = i_item_sk

Hash Join ss_sold_date_sk = d_date_sk

(p_channel_event = 'N' OR p_channel_email = 'N' )

Index Push Down Key: (d_year = 2001)d_date_sk to dwa_ds.store_sales

((cd_education_status = 'College'

AND cd_marital_status = 'M' ) AND cd_gender = 'F' )

Sequential Scan

Sequential Scan

Sequential Scan

FactIndex scan

Sort rowids

Probe

Build

Build

Build

Build

Probe

Probe

Probe

Open

Open

Open

Open

Open

next

next

next

next

next

12

Intersection

of rowids

13

Tasks for database design

• Logical design

• Physical design

• Hardware configuration

• Informix configuration

• Feedbacks, changing needs.

14

• Reduce IO

• Improve IO throughput

• Improve CPU and network

• Improves administration efficiency

• Do more with less

Why is physical database design necessary?

Performance. Performance. Performance.

15

1. Data type selection

2. Indexes

3. Summary tables

4. Compression

5. Memory allocation

6. Table partitioning

Tasks of physical database design

16

• Numeric is faster than character

• BIGINT, BIGSERIAL is faster than int8, serial8

• Fixed char is faster than varchar, lvarchar

• All the character types exploit light scan

• Larger types means larger indices

• Date-time-columns use integer in fact table..

• (RDS – storing the DOMAINS for publishing)

Data type selection

18

create table "dwa_ds".web_sales(

ws_sold_date_sk integer,ws_sold_time_sk integer,ws_ship_date_sk integer,ws_bill_customer_sk integer,ws_bill_cdemo_sk integer,ws_bill_hdemo_sk integer,ws_bill_addr_sk integer,ws_ship_customer_sk integer,ws_ship_cdemo_sk integer,.....ws_quantity integer,ws_wholesale_cost decimal(7,2),ws_list_price decimal(7,2),ws_sales_price decimal(7,2),ws_net_paid decimal(7,2),ws_net_paid_inc_tax decimal(7,2),ws_net_paid_inc_ship decimal(7,2),ws_net_paid_inc_ship_tax decimal(7,2),ws_net_profit decimal(7,2),primary key (ws_item_sk,ws_order_number) disabled

);

19

Select d_month,i_category,sum(ws_net_paid) as total_sum,substr(i_category,2) as lochierarchy

fromweb_sales

,date_dim,item

whered_year = 2002

and d_date_sk = ws_sold_date_skand i_item_sk = ws_item_skgroup by i_category,i_classorder by d_month, i_category;

20

1. Very few indices

2. Leaner indices – Informix can combine when necessary

3. Keep load, insert performance in mind

1. When possible disable

2. Attach/detach performance with indexes

4. Scan – Scan – Scan

5. Provide better options for Optimizer

6. Improve scan for tables – more for dimension than facts

7. Exploit multi-index scans

8. Star join can work with indexes or without indexes.

Index Design

21

1. Handle the “canned” queries by pre-computing the answers

2. Saves system resources for handling complex ad-hoc queries

3. Can create multiple levels of summaries

-- sales by region, by part, by (day, week, month)

4. Need time to create the summary tables ahead of time

Summary tables

Fact table > billons of rows

Summary level 1 (day)

Summary level 2 (week)

Summary level 3 (week)

22

1. Very few indices

2. Leaner indices – Informix can combine when necessary

3. Keep load, insert performance in mind

1. When possible disable

2. Attach/detach performance with indexes

4. Scan – Scan – Scan

5. Provide better options for Optimizer

6. Improve scan for tables – more for dimension than facts

7. Exploit multi-index scans

8. Star join can work with indexes or without indexes.

Traditional Compression

23

Compression On Data Page With Multiple Rows

compress repack

Uncompressed Compressed Compressed

shrink

Multiple Compressed

Pages

Empty Data Pages

Animated

Slide

• To use repack and shrink, the data need not be compressed

• repack and shrink can be done independently

• Shrink does not do an implicit repack

24

1. IDS can configure page sizes that are multiples of the

2. default page size

3. Depending on OS can be 2k, 4k, 8k, or 16k

4. Each page size will necessitate its own buffer pool

5. Can be useful if OS supports larger page sizes for buffer transfers.

6. Ensure adequate memory exists

Pagesize Considerations

25

• Motivation

–Performance, Performance, Performance

–Performance of the queries

–Performance of Warehouse tasks

• Extract jobs

• Load performance

• Transformation

• Summary data creation

26

• Starting point

– Logical design

• Observations

– Presence or absence of indices and constraints

• What are your expected maintenance tasks

– Daily tasks

– Weekly tasks

– Monthly tasks

– Quarterly tasks

– Yearly tasks

27

Topics for the day

• Create table:fact an dimension

• Create Index: Deciding to which indices to create and their keys and attributes

• Attach/detach

• Drop/Truncate

• SELECT, UPDATE, DELETE

• Update Statistics

• Distribution

28

Features• External table

• Options for loading/unloading

• Fragment level stats

• Fragmentation – interval/etc

• PDQ

• Interplay with configuration

• Keeping things ONLINE

29

Fragmentation = Partitioning

• In Informix, they are synonymous

• Informix was the first to implement this feature. Industry is using partitioning for some time now

• Within Informix syntax, we’re making fragmentation and partitioning synonymous.

• Informix Fragmentation has nothing to do with disk fragmentation.

30

Fragmentation = Partitioning

• In Informix, they are synonymous

• Informix was the first to implement this feature. Industry is using partitioning for some time now

• Within Informix syntax, we’re making fragmentation and partitioning synonymous.

• Informix Fragmentation has nothing to do with disk fragmentation.

31

Situation

1. OLTP– Data volume

– performance

2. Data Warehousing– Data Volume

– Performance

– Data Management

3. Mixed workload– Do (1) and (2) together

32

Problem areas and Opportunities

• Capacity

• Storage IO throughput

• Performance

• Availability

• Storage management

• Reliability

• Time cyclic data management

• Security

33

Capacity• By default, a table gets one partition

• Limited number of pages in each partition

• Capacity is limited by the number of pages.

– 2KB pagesize on most platforms.

– 4KB pagesize on Windows and AIX

• Increase the capacity by creating dbspace with up to 16KB pagesize.

• Compress the table table to reclaim space

• Consider fragmenting the table.

34

Storage IO throughput• Warehouse and complex queries and DDL

operations depend on IO throughput significantly.

• By creating multiple fragments in multiple dbspaces, you give Informix opportunity to parallelize operations

35

Performance• Enable parallel operations for queries

– SELECT

– INSERT

• Exploit fragment elimination (aka partition pruning)

• Improve create index timings

• Improve update statistics operation

36

Availability• Fragments distributed in multiple disks or

volumes are more tolerant of failure

• If a disk failed, you can skip over those fragments via DATASKIP

37

Storage management• Manage the data distribution on different storage

disks or storage managers

• Match disk speeds to different quality of service

– Recent data on faster/expensive storage

38

Time cyclic data management• The table would maintain specific period of data.

– Last 25 hours

– Last 3 months or 13 months

– Last 7 years

• At every interval do the three operations

– Detach the data no longer needed

– Attach the new data or make room for it.

– Modify the expressions to represent the new window

39

4Q2008 1Q2009 2Q2009 3Q2009 4Q2009 1Q2010 2Q2010

working window

Time cyclic data management

40

Security• Grant or revoke access per fragment

– Grant and revoke is done on the tables

• Example

– Table is fragmented on states

– Table is fragmented on departments

– Table is fragmented on date or datetime

41

Fragmentation Overview

42

DBSPACE

CHUNK CHUNK CHUNK

Extent Extent

Extent

CHUNK

Pages

Pages

Pages

Partition

Extent

43

Customer_table Partition

idx_cust_id

Customer_table

Partition

Paritition

Storesales_table

Idx_store_id

Partition

Partition

Tables, Indices and Partitions

44

Fragment1 Fragment2 Fragment3 Fragment4

Fragmentation by Round Robin

Fragmentation by Expression

September October November December

45


CREATE TABLE customer (id int, state char(2))

FRAGMENT BY ROUND ROBIN

in dbspace1, dbspace2, dbspace3;

CREATE TABLE customer (id int, state char(2))

FRAGMENT BY ROUND ROBIN

PARTITION part1 IN dbspace1,

PARTITION part2 IN dbspace1,

PARTITION part3 IN dbspace2;

part1 part2 part3

46


• Predictable distribution of data and use of storage.

• Create indices with either round robin or expression strategy

• No fragment elimination possible

• Use expression strategy for indices to get benefit of elimination

• Index expression depends on the access pattern.

• Blobs avoid extra writes in RR strategy.

47

Fragmentation by expressionCREATE TABLE customer (id int, state char (2), zipcode decimal(5,0))

FRAGMENT BY EXPRESSION

(state = “CA“) in dbspace1,

(state = “KS") in dbspace2,

(state = “OR") in dbspace3,

(state = “NV") in dbspace4;

CREATE TABLE customer (id int, state char (2))


PARTITION part1 (state = “CA") in dbspace1,

PARTITION part2 (state = “KS") in dbspace1,

PARTITION part3 (state = “OR") in dbspace1,

PARTITION part4 (state = “NV") in dbspace1;

CA KS OR NV

48

Fragmentation by expression

CREATE TABLE customer (id int, state char (2), zipcode

decimal(5,0))


partition partca93 (state = “CA“ and zipcode < 93000) in dbspace1,

partition partcagt93 (state = “CA“ and zipcode >= 93000) in dbspace5,

PARTITION part2 (state = “KS") in dbspace2,

PARTITION part3 (state = “OR") in dbspace2,

PARTITION part4 (state = “NV") in dbspace3;

KS OR NVCA

< 93000

CA

>= 93000

49

Fragmentation by expression

CA KS OR NV

• Destination of the row depends on the row data

• Design of expressions requires understanding and estimation of data sets

• Expressions provide a flexible mechanism

• With flexibility comes complexity

• Can make fragment elimination complex

50

jan_partition

feb_partition

march_partition

april_partition

may_partition

june-partition

july_partition

august_partition

september_partition

october_partition

november_partition

1/1/2010 to 1-31-2010

2/1/2010 to 2-28-2010

3/1/2010 to 3-31-2010

4/1/2010 to 4-30-2010

5/1/2010 to 5-31-2010

9/1/2010 to 9-30-2010

8/1/2010 to 8-31-2010

7/1/2010 to 7-31-2010

6/1/2010 to 6-30-2010

10/1/2010 to 10-31-2010

11/1/2010 to 11-30-2010

Orders Table for 2010

51

Attached Index on a Fragmented Table

• Large table DSS or OLTP environment.

• Attractive index parallel scans.

• Attractive index fragment elimination and smaller btrees.

• Attractive scans on data pages in parallel.

• Balanced I/O for indexes and data pages.

52

Detached Fragmented Index on a Non-fragmented Table

• OLTP environment with high index hits vs. data page hits (key only reads).

• Attractive index scans in parallel

• Attractive index lookups with fragment elimination and smaller btrees.

• Unattractive scans on data pages in series.

53

Detached Index on a Fragmented Table

• DSS environment with some selective queries.


• Unattractive index read in series.

54

Detached Fragmented Index on a Fragmented Table

• Mixed OLTP and DSS environments with data fragmented for DSS and index fragmented of OLTP or Selective queries and non-selective queries on different columns in a DSS environment.

• Attractive index parallel scans.

• Attractive index fragment elimination and smaller btrees.


• Balanced I/O for indexes and data pages.

55

Three things important in determining

your strategy and expressions

Workload. Workload. Workload.

56

Create table history (

Docid char(14),

..

) fragment by expression

Partition p1 (docid >= “abcdef20112001” and docid<(babcde20111000) in dbs1,

Partition p2(docid >= “babcde20111000” and ..

Docid: abcdef20110001

57

Factors to consider

• Query Performance

– What kind of queries?

– Data Distribution

– Which applications use this table?

• Storage Management

– How do you handle data growth?

– What’s the table management strategy when dataset grows

58

Fragmentation Objectivesscan threads *

fragments

ParallelismFragments are accessed in

parallel, decreasing scan or insert

time.

Fragment EliminationUnneeded fragments are

eliminated, decreasing scan or

insert time, and reducing disk

contention.

scan threads *

fragments

scan threads *

fragments

Fragment Elimination &

ParallelismBoth goals are achieved.

XX XX

XX

59

OLTP characteristics:

• high volume of short transactions

• each transaction accesses a few rows

• index access method is used.

For this environment:

• Fragmentation by round

robin or expression strategy

• Reduce index access times using fine grain expressions

on the index fragment.

– Fragment recent data by day and older data by week or month.

60

Data Warehousing characteristics

• low volume of long running queries

• queries access most of the rows in each fragment

• very few indexes are generally required

• preferred access method is sequential scan

• preferred join method is the hash join

For this environment:

• fragment elimination

• parallel scan the needed fragments

61

• Multi-threaded Dynamic Scalable Architecture (DSA)– Scalability and Performance

– Optimal usage of hardware and OS resources

• DSS Parameters to optimize memory– DSS queries

– Efficient hash joins

• Parallel Data Query for parallel operations– Light scans, extensive

– calculations, sorts, multiple joins

– Ideal for DSS queries and batch operations

• Data Compression

• Time cyclic data mgmt– Fragment elimination, fragment

attach and detach

– Data/index distribution schemas

– Improve large data volume manageability

– Increase performance by maximizing I/O throughput

• Configurable Page Size

– On disk and in memory

– Additional performance gains

• Large Chunks support

– Allows IDS instances to handle large volumes

• Quick Sequential Scans

– Essential for table scans common to DSS environments 17

Top IDS features utilized for building warehouse

Source:

62

• Multi-threaded Dynamic Scalable Architecture (DSA)– Scalability and Performance


• DSS Parameters to optimize memory– DSS queries


• Parallel Data Query for parallel operations– Light scans, extensive




• Time cyclic data mgmt– Fragment elimination, fragment

attach and detach










– Essential for table scans common to DSS environments 17


Source:

Fragmentation Features

63

Fragmentation Objectives

scan threads *

fragments

ParallelismFragments are accessed in

parallel, decreasing scan or insert

time.

Fragment EliminationUnneeded fragments are

eliminated, decreasing scan or

insert time, and reducing disk

contention.

scan threads *

fragments

scan threads *

fragments

Fragment Elimination &

ParallelismBoth goals are achieved.

XX XX

XX

64

Typical Query with Non-PDQ vs. PDQ

Scan

Join

Sort

Send to client

Scan Scan

Sort

Scan

Join

Sort

Send to client

65

PDQ/Fragmentation

• Consider fragmenting any large table in a dbspace that is getting a lot of IO activity

• Consider fragmenting any large table if scans must be done against the table

• Do not put multiple fragments of the same table on the same physical device– Be aware of the I/O throughput of your storage

• Avoid using round robin fragmentation for indexes.

• Do not over-fragment.

– The cost of managing fragmentation can outweigh the benefits when there are excessive fragments.

66

PDQ Configuration

• MAX_PDQPRIORITY

– Set highest percentage of PDQ resources that a single client can use

• b

– Max number of DSS queries that can be run together

• DS_TOTAL_MEMORY

– Total memory reserved for PDQ

• DS_MAX_SCANS

– Max number of parallel scans allowed. Leave at default (1048567)

67

PDQ Configuration

• If the site is primary a DSS system, then it is recommended that most of the allocated memory be in the virtual buffers and that DS_TOTAL_MEMORY be very large

• PDQ can be used in smaller memory environments by setting PDQ_PRIORITY to 1 so that parallel scans can be done.

68

PDQ Configuration

• onmode can be used to dynamically change PDQ parameters

– onmode –M (DS_TOTAL_MEMORY)

– onmode –Q (DS_MAX_QUERIES)

– onmode –D (MAX_PDQPRIORITY)

– onmode –S (DS_MAX_SCANS)

69

Design for Time Cyclic data mgmtcreate table mytrans(

custid integer,

proc_date date, store_loc char(12)

….

) fragment by expression

......

(proc_date < DATE ('01/01/2009' ) ) in fe_auth_log20081231,

(MONTH(proc_date) = 1 ) in frag2009Jan ,

(MONTH(proc_date) = 2 ) in frag2009Feb,….

(MONTH(proc_date) = 10 and proc_date < DATE ('10/26/2009' ) ) in frag2009Oct ,

(proc_date = DATE ('10/26/2009' ) ) in frag20091026 ,

(proc_date = DATE ('10/27/2009' ) ) in frag20091027,






;

70

Can eliminateCan eliminateCan eliminateEquality expression

Cannot eliminateCannot eliminateCan eliminateRange expression

Nonoverlapping

Multiple column key

Overlapping on a

single column key

Nonoverlapping

Single fragment key

Type of filter (WHERE clause)

Fragment elimination

71

create table f1(a int, b varchar(32)) fragment by expression

partition p1 (a = 1) in rootdbs,




partition p5 (a = 5) in rootdbs;

insert into f1 select 1, 'oneworld' from systables;

insert into f1 select 2, 'twoworld' from systables;

insert into f1 select 3, 'threeworld' from systables;

insert into f1 select 4, 'fourworld' from systables;

insert into f1 select 5, 'fiveworld' from systables;

insert into f1 select * from f1;

create index f1ab on f1(a,b);

create index f1b on f1(b);

update statistics high for table f1;

72

QUERY: (OPTIMIZATION TIMESTAMP: 9-29-2010 00:56:12)

------

select * from t1 where a=1

Estimated Cost: 5

Estimated # of Rows Returned: 136

1) keshav.t1: INDEX PATH

(1) Index Name: keshav.t1abIndex Keys: a b (Serial, fragments: 0)

Fragments Scanned: (0) p1 in rootdbsLower Index Filter: keshav.t1.a = 1

73

QUERY: (OPTIMIZATION TIMESTAMP: 9-29-2010 01:06:54)

------

select * from t1 where a <3

Estimated Cost: 11


1) keshav.t1: SEQUENTIAL SCAN (Serial, fragments:

0, 1)

Fragments Scanned: (0) p1 in rootdbs, (1) p2 in

rootdbs

Filters: keshav.t1.a < 3

74

select count(*) from t1 where a = 3 and b =

'threeworld'

Estimated Cost: 1



(1) Index Name: keshav.t1ab

Index Keys: a b (Serial, fragments: 2)

Fragments Scanned: (2) p3 in rootdbs

Lower Index Filter: (keshav.t1.b =

'threeworld' AND keshav.t1.a = 3 )

75

select count(*) from t1

where a = a+1 and b = 'threeworld'

Estimated Cost: 11



Filters: keshav.t1.a = keshav.t1.a + 1

(1) Index Name: keshav.t1b

Index Keys: b (Serial, fragments: ALL)

Lower Index Filter: keshav.t1.b = 'threeworld'

76

Managing Change• You have a round robin strategy, but running out of

space

• You have a round robin strategy, but need to modify it into expression expression strategy

• Time Cyclic changes aka Roll-on Roll-off

– Daily, weekly, monthly, yearly

– Hourly anyone?

– I’m running out of space in a fragment!

– I have uneven distribution of data

– I can’t have any down-time!

77

Managing Change

• You have a round robin strategy, but running out of space

– Analyze how frequently you’re adding new fragments

– Consider moving to higher pagesize (2K to 16K)

– All fragments of the table have to be in same pagesize.

– But, the indices can be in a different pagesize than table. But, all fragments in an index use same pgsize

– On 11.50.xC4 or above, you have another option!• You can compress the table data

– Consider moving to higher pagesize (2K to 16K) for further capacity

78

Managing Change• You have a round robin strategy, but

need to modify it into expression expression strategy

–You need exclusive access to the table

–You need to schedule the down time.

–ALTER FRAGMENT… INIT

79

Managing Change


4Q2008 1Q2009 2Q2009 3Q2009 4Q2009 1Q2010

Current working set

Older rolled-off data

New data waiting to

roll on

2Q2010

80

Managing Change


Q1 Q2 Q3 Q4

Q1 Q2

81

Managing Change• Time Cyclic changes aka Roll-on Roll-off

– Daily, weekly, monthly, yearly

– Consider the data distribution and the queries on it.

– Fine granularity can help speed up analysis

– Consider hybridcreate table t(a int, b date) fragment by expression

partition p1 (month(b) = 10) in dbs,

partition p2 (month(b) = 11) in dbs,

partition p3 (b >= date("12/01/2009") and b <= date("12/15/2009")) in

dbs,

partition p4 (b >= date("12/16/2009") and b <= date("12/31/2009")) in

dbs;

– Test your strategies to ensure fragment elimination.

82

Source: Forrester

Query

Tools

Analytics

BPS

Apps

BI

Apps

LOB

apps

Databases

Other

transactional

data sources

I/O & data loading

Query processing

DBMS & Storage mgmt

Interconnect & Networking

Enterprise Data Warehouse Platform

83

Source: Forrester

Query

Tools

Analytics

BPS

Apps

BI

Apps

LOB

apps

Databases

Other

transactional

data sources

I/O & data loading

Query processing

DBMS & Storage mgmt

Interconnect & Networking

Enterprise Data Warehouse Platform

Data Loading

External Table

Online

attach/detach

Data & storage

management

Deep Compession

Interval and List

Fragmentation

Online attach/detach

Fragment level stats

Storage provisioning

Table defragmenter

Query Processing

MERGE

Hierarchical Queries

Light Scans

Multi-Index Scan

Skip Scan

Bitmap Technology

Star and Snowflake

join optimization

Implicit PDQ

Tooling

Microstrategy

Cognos

Pentaho

Jaspersoft

SQW

84

• Multi-threaded Dynamic Scalable Architecture (DSA)

– Scalability and Performance


• DSS Parameters to optimize memory

– DSS queries


• Parallel Data Query for parallel operations

– Light scans, extensive




• Time cyclic data mgmt

– Fragment elimination, fragment attach and detach










– Essential for table scans common to DSS environments


85

New fragmentation Strategies in Informix v11.70

• List Fragmentation

–Similar to expression based fragmentation

–Syntax compatibility

• Interval Fragmentation

–Like expression, but policy based

– Improves availability of the system

86

Time Cyclic Data management

• Time-cyclic data management (roll-on, roll-off)

• Attach the new fragment

• Detach the fragment no longer needed

• Update the statistics (low, medium/high) to keep everything up to date.

fieldfieldfieldfieldfieldfieldfield

fieldfieldfieldfieldfieldfieldfield field

fieldfieldfieldfieldfieldfield





JanJan FebFeb MarMar AprApr

May 09May 09

Dec 08Dec 08

enables storing data over time

87








JanJan FebFeb MarMar AprApr

May 09May 09

Dec 08Dec 08

• ATTACH, DETACH and rest of ALTERs require exclusive access– Planned Downtime

• These can be scripted, but still need to lock out the users– Informix 11.50.xC6 has DDL_FORCE_EXEC to lock out the users

• Expression strategy gives you flexibility, but elimination can be tricky.

Time Cyclic Data management

88

Fragment by Expressioncreate table orders

(order_num int,order_date date,customer_num integer not null,ship_instruct char(40),backlog char(1),po_num char(10),ship_date date,ship_weight decimal(8,2),ship_charge money(6),paid_date date ) partition by expression

partition prv_partition (order_date < date(’01-01-2010’)) in mydbs,partition jan_partition (order_date >= date(’01-01-2010’) and

order_date < date(’02-01-2010’) in mydbs,partition feb_partition (order_date >= date(’02-01-2010’) and

order_date < date(’03-01-2010’) in mydbs,partition mar_partition (order_date >= date(’03-01-2010’) and

order_date < date(’04-01-2010’) in mydbs,partition apr_partition (order_date >= date(’04-01-2010’) and

order_date < date(’05-01-2010’) in mydbs,…

89

Fragment by Interval

Interval Value

Initial Partition

Partition Key

dbspaces

create table orders(order_num int,order_date date,customer_num integer not null,ship_instruct char(40),backlog char(1),po_num char(10),ship_date date,ship_weight decimal(8,2),ship_charge money(6),paid_date date )

partition by range(order_date) interval(1 units month)store in (dbs1, dbs2)partition prv_partition values < date(’01-01-2010’) in dbs3;

90

Interval Fragmentation

• Fragments data based on an interval value

– E.g. fragment for every month or every million customer records

• Tables have an initial set of fragments defined by a range expression

• When a row is inserted that does not fit in the initial range fragments, IDS will automatically create fragment to hold the row (no DBA intervention)

• No X-lock is required for fragment addition

• All the benefits of fragment by expression

91

ONLINE attach, detach

• ATTACH

– Load the data into a staging table, create the indices exactly as you have in your target table.

– Then simply attach the table as a fragment into another table.

• DETACH

– Identify the partition you want to detach

– Simply detach the partition with ONLINE keyword to avoid attemps to get exclusive access

92

Attach Example

ALTER FRAGMENT ONLINE ON TABLE “sales”.orders

ATTACH december_orders_table as PARTITION december_partition

values < 01-01-2011;

93

Attach Example

94

Attaching online

95

December_orders_table

Table to attach

orders

query1 query2

Issue ALTER ATTACH ONLINE

query1query2

Query1 and Query2 continue and won’t access the new partition

Attach

Modify the dictionary entry to indicate online attach

is in progress. Other sessions can read the list but

cannot modify.

query1query2

query3

New queries will work on the table and won’t

consider the table fragment for the queries.

ONLINE ATTACH operation is complete.

Table is fully available for queries

query3

query4

Get exclusive access to the partion list (in the

dictionary) .The dictionary entry gets modified

and new dictionary entries for new queries from

here on

New queries will work on the table and willconsider the new table fragment .Attaching online

96

ONLINE operations• ATTACH a fragment• DETACH a fragment• MODIFY transition value• Automatic ADDing of new fragments on insert

or update• tasks eliminated by interval fragmentation

– Scheduling downtime to get exclusive access for ADD, ATTACH, DETACH

– Defining proper expressions to ensure fragment elimination

– Running of update statistics manually after ALTER operations

– Time taken to collect statistics is reduced as well.

97

UPDATE STATISTICS during ATTACH, DETACH

• Automatically kick-off update statistics refresh in the background –need to enable fragment level statistics

• tasks eliminated by interval fragmentation–Running of update statistics manually

after ALTER operations–Time taken to collect statistics is

reduced as well.

98

List fragmentation

CREATE TABLE customer

(id SERIAL, fname CHAR(32), lname CHAR(32), state CHAR(2), phone CHAR(12))

FRAGMENT BY LIST (state)

PARTITION p0 VALUES ("KS", "IL", "IN") IN dbs0,

PARTITION p1 VALUES ("CA", "OR", "NV") IN dbs1,

PARTITION p2 VALUES ("NY", "MN") IN dbs2,

PARTITION p3 VALUES (NULL) IN dbs3,

PARTITION p4 REMAINDER IN dbs3;

99

Smarter Statistics Collection

100

Fragment Level Statistics (FLS)

• Generate and store column distribution at fragment level

• Fragment level stats are combined to form column distribution

• System monitors UDI (Update/Delete/Insert) activities on each fragment

• Stats are refreshed only for frequently updated fragments

• Fragment level distribution is used to re-calculate column distribution

• No need to re-generate stats across entire table

101

Generating Table Level Statistics

• Distribution created for entire column dataset from all fragments.

• Stored in sysdistrib with (tabid,colno) combination.

• Dbschema utility can decodes and display encoded distribution.

• Optimizer uses in-memory distribution representation for query optimization.

DataDistribution

CacheFeed

Sorted

Data

Feed

Column

Data

Store

Encoded

Distribution

Decode

Distribution

BinGenerator& Encoder

SORT

SysdistribCatalogtable

Frag1

Frag2

Fragn

102

Generating Fragment Level StatisticsData

Distribution Cache

Feed

Sorted

Data

Feed

Column

Data

Store

Encoded

MinibinsDecode

Distribution

Mini-BinGenerator& Encoder

SORT

SysfragdistCatalogTable

Frag1

Frag2

Fragn

SORT

SORT



Mini-Bin Merger & Bin

Encoder

SysdistribCatalogTable

SORT

Feed

decode

Minibins

Store

Encoded

Distribution

103

STATLEVEL property

STATLEVEL defines the granularity or level of statistics created for the table.

Can be set using CREATE or ALTER TABLE.

STATLEVEL [TABLE | FRAGMENT | AUTO] are the allowed values for STATLEVEL.

TABLE – entire table dataset is read and table level statistics are stored in sysdistrib catalog.

FRAGMENT – dataset of each fragment is read an fragment level statistics are stored in new sysfragdist catalog. This option is only allowed for fragmented tables.

AUTO – System determines when update statistics is run if TABLE or FRAGMENT level statistics should be created.

104

UPDATE STATISTICS extensions

• UPDATE STATISTICS [AUTO | FORCE];

• UPDATE STATISTICS HIGH FOR TABLE [AUTO | FORCE];

• UPDATE STATISTICS MEDIUM FOR TABLE tab1 SAMPLING SIZE 0.8 RESOLUTION 1.0 [AUTO | FORCE ];

• Mode specified in UPDATE STATISTICS statement overrides the AUTO_STAT_MODE session setting. Session setting overrides the ONCONFIG'sAUTO_STAT_MODE parameter.

105

UPDATE STATISTICS extensions

• New metadata columns - nupdates, ndeletes and ninserts –in sysdistrib and sysfragdist store the corresponding counter values from partition page at the time of statistics generation. These columns will be used by consecutive update statistics run for evaluating if statistics are stale or reusable.

• Statistics evaluation is done at fragment level for tables with fragment level statistics and at table level for the rest.

• Statistics created by MEDIUM or HIGH mode (column distributions) is evaluated.

• The LOW statistics is saved at the fragment level as well and is aggregated to collect global statistics

106

Alter Fragment Attach/Detach• Automatic background refreshing of column statistics after

executing ALTER FRAGMENT ATTACH/DETACH on a table with fragmented statistics.

• Refreshing of statistics begins after the ALTER has been committed.

• For ATTACH operation, fragmented statistics of the new fragment is built and table level statistics is rebuilt from allfragmented statistics. Any existing fragments with out of date column statistics will be rebuilt at this time too.

• For DETACH operation, table level statistics of the resulting tables are rebuilt from the fragmented statistics.

• The background task that refreshes statistics is “refreshstats”and will print errors in online.log if any are encountered.

107

Not yetYesYesYesCreate index ONLINE

YesYesYesYesFLS

YesNoNoNoStorage Provisioning

YesYesYesYesSmarter Stats

Yes -- MODIFY transition value

NoNoNoMODIFY ONLINE

YesNoNoNoDETACH ONLINE

YesNoNoNoATTACH ONLINE

YesYesYesNoEquality Expression

YesYesYesNoRange Expression

YesYesYesYesParallelism

IntervalExpression

ListRound Robin

1088/26/2011Session TPM3 108

Informix physical database design for data warehousing

Technology

Transcript of Informix physical database design for data warehousing