Customer Cubing Services Education Session

90
© 2010 IBM Corporation Business Analytics Cubing Services Cubing Services Development 29 April 2010

Transcript of Customer Cubing Services Education Session

Page 1: Customer Cubing Services Education Session

© 2010 IBM Corporation

Business Analytics

Cubing Services

Cubing Services Development

29 April 2010

Page 2: Customer Cubing Services Education Session

© 2010 IBM Corporation2 Footer Field

Business Analytics

Disclaimer

The information on any new products in this presentation is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information on new products is for informational purposes only and may not be incorporated into any contract. The information on new products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. The development, release, and timing of any features or functionality described for our products remains at our sole discretion.

Page 3: Customer Cubing Services Education Session

© 2010 IBM Corporation3

Business Analytics

Agenda

IBM OLAP portfolio – how Cubing Services fits in

Cubing Services architecture and clients

Lifecycle

Best Practices

New features in 9.7 – dimensional security, virtual cubes

Performance and scalability proof points

Page 4: Customer Cubing Services Education Session

© 2010 IBM Corporation4

Business Analytics

OLAP portfolio from IBM – complete coverage

OLA

P

Read/write user communities 64 bit in-memory fast performance Larger highly dynamic data sets -------------------------------------------- Dynamic changing dimensions and hierarchy,

what-if scenarios, data contribution Common dimensions shared across multi-

cube models Budgeting and planning Personal and corporate data sources

Large read only user communities Moderate ‘focused area’ data sets Pre-aggregated static fast results -------------------------------------------- Off-line portable/ partitioned storage Rapid startup for advanced business

user self service modeling Automatic time series analysis &

trending Point in time data Personal and corporate data sources

Large user communities Large data sets Planned performance/optimization facilities -------------------------------------------- Optimized ROLAP Zero Latency High volume concurrency Largest Data Volumes Centralized IT management of information

IBM Cognos PowerCube

Optimized for broad, general purpose BI usage

IBM Cognos TM1

Optimized for write back and high volatility applications

IBM Infosphere Warehouse Cubing Services

Optimized for very large datasets with very large dimensions

Specialized OLAP Modeling Tools

Turbo IntegratorMO

DEL

ING

Transformer Design Studio

USE

R

Complete, Consistent View of Information

IBM Cognos BI / FPM

Page 5: Customer Cubing Services Education Session

© 2010 IBM Corporation5

Business Analytics

InfoSphere Warehouse Cubing Services

InfoSphereWarehouse

Warehouse Modeling

Cubing Services

Cognos 8Excel & 3rd

party toolsIBM tools

Large enterprise IT deploymentsPrimary OLAP Use:

Ideal for:• Very large data sets with very

large dimensions – SKU level data

• Enterprise rollouts requiring near real time data

Primary owner - IT departments

Analytics & reporting forbanking, retail & insurance

Because of its unique ability to:• Accelerate OLAP queries

optimized on InfoSphere Warehouse

• Integrated IT tooling

Page 6: Customer Cubing Services Education Session

© 2010 IBM Corporation6

Business Analytics

Key capabilities - InfoSphere Warehouse Cubing Services

Lower cost of ownership bringing design, query and

performance together

Integrated IT tooling

Support many users with accelerated OLAP

queries

Integrated administration for warehouse and cubes

Unlimited scalability

Supporting a large number of deep dimensions with

SKU level data64-bit high performance caches for very large data

volumes and queries

Near real time data access with high speed

performance

High performance ROLAP solution

Cognos 8 front end clients and Microsoft Excel via Pivot table servicesIndustry standard ODBO & XMLA APIs enable 3rd party applications

High performance, scalable cubing engine

Integrated design tooling with Design Studio &

Rational Data Architect

SQL access optimized for InfoSphere Warehouse

The Optimization Advisor accelerates all SQL-based queries in the

warehouse

Increase performance & scalability using

optimization features of InfoSphere Warehouse

Page 7: Customer Cubing Services Education Session

© 2010 IBM Corporation7

Business Analytics

InfoSphere Cubing Services architecture

CubingServices

InfoSphereWarehouse

MQT’s, MDC’s

Excel

AlphabloxMDXSQL

Web Administration Console

Eclipse Design StudioOLAP Metadata Optimization

AdvisorMDX

Calculation Builder

Cube Manager

Server Management

64-bit in-memory cachingNo data duplication

OD

BONative API

XMLA

BI ToolsCube Security Manager

Cognos

Page 8: Customer Cubing Services Education Session

© 2010 IBM Corporation8

Business Analytics

Excel 2003, 2007

• Pervasive BI tool

• Real-time, live connection to warehouse data via Excel

• Can save point-in-time data, and refresh at any time

• Full use of pivot table services for analysis

Page 9: Customer Cubing Services Education Session

© 2010 IBM Corporation9

Business Analytics

Cognos 8.4 SP2 and later

Page 10: Customer Cubing Services Education Session

© 2010 IBM Corporation10

Business Analytics

Cubing Service Lifecycle

Page 11: Customer Cubing Services Education Session

© 2010 IBM Corporation11

Business Analytics

Run

Design

Optimize

Deploy

AnalyzeCube Server

Administration Console

Design Studio

Cubing Services Advisor

DB2 Design Advisor

Cubing Services Lifecycle – Stages 1 through 5

Page 12: Customer Cubing Services Education Session

© 2010 IBM Corporation12

Business Analytics

Lifecycle Stage 1 – DesignCreating a Cube

Design

• Goal: Define multidimensional metadata that describes the information that is stored in relational tables

Cube Model and Cube Measures Dimensions Hierarchies Levels Members

Star Schema• fact table• dimension tables• outboard tables

Page 13: Customer Cubing Services Education Session

© 2010 IBM Corporation13

Business Analytics

Lifecycle Stage 1 - DesignDesign Studio

DB2 LUW

OLAP Metadata

DB2

BI Designer

Export Import

Deploy for runtime

CreateEditDropAnalyze

Reverse engineerschema

Metadata files

Page 14: Customer Cubing Services Education Session

© 2010 IBM Corporation14

Business Analytics

Page 15: Customer Cubing Services Education Session

© 2010 IBM Corporation15

Business Analytics

Lifecycle Stage 2 – Optimize

Cube server can generate one or more SQL queries to calculate result set for each MDX query

Queries that aggregate large volumes of data can be time consuming

MQT = Materialized Query Table; calculate aggregations up front

DB2 Optimizer can access MQT instead of performing the aggregations over detail data when the query is processed

MQTs are defined like views, physical like tables and transparent to applications like indexes

Optimize

MDX

SQL Data

Result set

MQT

Cube Server

DB2

Page 16: Customer Cubing Services Education Session

© 2010 IBM Corporation16

Business Analytics

Lifecycle Stage 2 – Optimize Cube Optimization: Overview

DB2

MQTs

OLAP Metadata

Optimization Advisor

Model-driven MQT generation

BI Designer

Advise Advise Admin Console

DBA

Design Studio

Page 17: Customer Cubing Services Education Session

© 2010 IBM Corporation17

Business Analytics

Lifecycle Stage 2 – Optimize Cube Optimization: Overview (2)Model driven advisor recommends MQTs for cubes in a cube model

Recommendation is based on metadata, database statistics and data sampling.

Time and resource constraints can be applied to recommendations.

Advisor recommends MQTs that are optimized for Cubing Services.

The advisor produces SQL scripts that create and refresh MQTs

Integrate refresh script in SQW process flows that update the data warehouse tables.

Page 18: Customer Cubing Services Education Session

© 2010 IBM Corporation18

Business Analytics

Lifecycle Stage 3 - Deployment Admin Console

Deploy

DB2OLAP Metadata

MQTs

Cube Server

Admin Console

OLAP metadata

Import

Metadata files

Admin Console

DBA

Page 19: Customer Cubing Services Education Session

© 2010 IBM Corporation19

Business Analytics

Lifecycle Stage 3 - Deployment

A cube is accessible by OLAP clients if it is running on a cube server. Use the Cubing Services administration features in Administration Console to:

– Import OLAP metadata (that was previously exported via Design Studio) into the metadata database for test/production deployments

– Map metadata to relational data source

– Create and configure cube server

– Deploy cubes to cube server

– Configure cube runtime tuning parameters

– Define cube security

– Control cube server and cube runtime states (start, stop, …)

Page 20: Customer Cubing Services Education Session

© 2010 IBM Corporation20

Business Analytics

Page 21: Customer Cubing Services Education Session

© 2010 IBM Corporation21

Business Analytics

Lifecycle Stage 4 - Runtime

Cube server and

cube administrationCube Server

Alphablox Excel

DB2

MQTs

OLAP Metadata

Optimization Advisor

MDX

Admin Console

Other BI tools

Run

Page 22: Customer Cubing Services Education Session

© 2010 IBM Corporation22

Business Analytics

Lifecycle Stage 4 - Runtime Starting a cube

Cube Server

DB2

Dim MemberCache

SQL

Data cache

SQL (seed query)

Page 23: Customer Cubing Services Education Session

© 2010 IBM Corporation23

Business Analytics

Cube Server

DB2

MDX calculation engine

Data cache

SQL

Dim MemberCacheResult set

MDX query

Lifecycle Stage 4 - Runtime Query Processing

• Simple, multi-pass SQL

Page 24: Customer Cubing Services Education Session

© 2010 IBM Corporation24

Business Analytics

Performance Factors

Cubing Services is fundamentally a ROLAP-style solution– Depends on underlying database performance for some aspects of its performance

characteristics– Leverages MQTs, MDC, and other relational database optimizations– Includes additional caching layers for performance

Page 25: Customer Cubing Services Education Session

© 2010 IBM Corporation25

Business Analytics

Tune configuration

Lifecycle Stage 5 - Analyze/Optimize

DB2

DB2DesignAdvisor

Cube Server

Log filesDBA

Create MQTs, MDC,

Indexes

Analyze

DB2W Admin Console

Page 26: Customer Cubing Services Education Session

© 2010 IBM Corporation26

Business Analytics

Best Practices

Page 27: Customer Cubing Services Education Session

© 2010 IBM Corporation27

Business Analytics

Cube Design: Best Practices for Facts

Star schema design

–Good performance starts with good relational modeling

–Define referential integrity (even if they are only informational constraints)

Schemas with multiple fact tables

–Create a view joining the fact tables and include the view as the facts object in the cube

Express measure calculations in SQL in the measure definition in a cube

–This allows Cubing Services to push down measure calculations to DB2 and exploit all of the optimizations provided by DB2

Design

Page 28: Customer Cubing Services Education Session

© 2010 IBM Corporation28

Business Analytics

Cube Design: Best Practices for Dimensions

Avoid large, flat dimensions

– Explore the creation of artificial levels in the hierarchy if it is too ‘flat’ (for example if dimension members have thousands of children)

When specifying level keys:

– Use single column level keys (or as few columns as necessary)

– Use integer values for level keys

Design

Page 29: Customer Cubing Services Education Session

© 2010 IBM Corporation29

Business Analytics

Cube Design: Best Practices for Dimensions

Number of dimensions to define in a cube– No explicit limit, but a large number of dimensions (more than 12) can have the following

negative performance impacts: • Increased memory usage - Memory storage cost of tuples increases proportionately

since a tuple consists of a member from each dimension• SQL query processing time - SQL generated by Cube Server will be longer and more

complex and have a lower probability that DB2 will use an MQT to respond to the SQL query

Page 30: Customer Cubing Services Education Session

© 2010 IBM Corporation30

Business Analytics

Number of members you can have in a dimension or cube– Theoretical limit is the maximum integer value supported by the JVM– In reality, there is a trade off between two approaches:

• Single large cube• - Reduced memory footprint• --- All queries share a single instance of the cube members• - Cache management• --- Caching a single large instance may not be practicable• Multiple smaller cubes (preferable)• - Caching and manageability is easier• - Does require an increase in memory footprint• --- Cube members are duplicated

Cube Design: Best Practices for Dimensions

Design

Page 31: Customer Cubing Services Education Session

© 2010 IBM Corporation31

Business Analytics

Deployment: Best Practices

Cube Server configuration

Uses two consecutive ports (firewall)

Limit number of cubes based on query load, cube sizeCube configuration

Adjust cache strategy and size as necessary

Enable/disable automatic member cache refreshSecurity configuration

Role based; enabled by default

Configurable for each cube; restricts query access

Authentication performed by DB2

Authorization determined by cube server

Deploy

Page 32: Customer Cubing Services Education Session

© 2010 IBM Corporation32

Business Analytics

Runtime: Best Practices – Memory UsageRun

Factors that affect memory usage

– Number of running cubes and their cache configuration

• Member cache (static default, dynamic)

– Memory usage ~ 1.1KB per member• Data cache

– Memory usage ~ 1KB per member– Allocate 10 % of metadata cache size

– Query processing

• MDX query workload (e.g. tuple count)

• Number of concurrently processed queries Keep total cube server memory footprint below 10 GB to avoid memory management overhead of Java.

Run very large cubes on separate cube servers for better scalability.

Run cubes that are accessed by many users concurrently on multiple cube servers.

Page 33: Customer Cubing Services Education Session

© 2010 IBM Corporation33

Business Analytics

Runtime: Best Practices – Query Performance Factors that affect query performance

MDX query type and complexity; ‘costly’ measure definitions

Avoid ‘unnecessary’ measures that result in expensive database calculations (e.g. count(distinct), sum(distinct))

Non-Empty processing requires two-passes; use only with sparse data sets

Result set size

Intermediate result set

Final result set

Number of concurrent users

Available system resources (e.g. heap memory)

Data cache state (cold vs. warm)

Cold cache performance depends on database performance – database

tuning (such as MQTs, indexes, partitioning) is crucial

Define seed query to pre-populate cache upon cube startup

Run

Page 34: Customer Cubing Services Education Session

© 2010 IBM Corporation34

Business Analytics

Analyze and Optimize: Best Practices

Cube server creates six log files (activity, exception, trace, MDX, performance, SQL)

Enable MDX log, SQL log and performance log to analyze query performance

MDX log captures processed queries

SQL log captures generated queries

Performance log captures cache hits and misses, the MDX query elapsed time and the SQL query elapsed time

Higher cache misses indicate too small cache size relative to the amount of data being fetched by the queries. Consider increasing the data cache size and use of a seed query.

High SQL elapsed times could indicate that appropriate indexes and/or MQTs have not been created or are not being used effectively by the DB2 optimizer.

Use SQL queries as an input workload to the DB2 Design Advisor. The tool will recommend additional MQTs, indexes, MDC, and other optimization objects to further optimize DB2 for this Cubing Services workload.

Analyze

Page 35: Customer Cubing Services Education Session

© 2010 IBM Corporation35

Business Analytics

New in Cubing Services 9.7Faster performance, richer functionality, easier to use

Faster performance

Virtual cubes– Enables fast performance in a low latency, real-time warehouse– Facilitates better data management

Dimensional security– Fine grain access control down to the dimension members

Remote server management

Page 36: Customer Cubing Services Education Session

© 2010 IBM Corporation36 Footer Field

Business Analytics

Faster cube load time than 9.5

• Cubes load on average nearly twice as fast in 9.7

• Uses less memory during cube load

Page 37: Customer Cubing Services Education Session

© 2010 IBM Corporation37 Footer Field

Business Analytics

Query response times improve

Results:

• Cold cache response times on average 9.7.1 is 3.5 times faster than 9.5.2

• Warm cache shows similar improvement

• Disk backed data cache leaves more memory for query execution.

Benefits:

• Faster cold cache response times makes real-time OLAP easier since cache refreshes are less impactful to user experience 9.5.1 9.5.2 9.7 9.7.1

Page 38: Customer Cubing Services Education Session

© 2010 IBM Corporation38 Footer Field

Business Analytics

Combines cubes with nearly identical dimensions

Virtual cube used as source for another virtual cube

Combines cubes with common Time dimension

Virtual cubes combine two

cubes

Virtual cubes overview

Sales

Store SalesWeb Sales

Inventory Sales

Inventory

Page 39: Customer Cubing Services Education Session

© 2010 IBM Corporation39 Footer Field

Business Analytics

• A virtual cube is a logical cube that is defined in terms of exactly two existing cubes – “real” or “virtual”• Cubes can be merged in pairs to any depth

• Virtual cubes combine two cubes that:• Share a dimension which is the basis for merging them to

form a virtual cube

• Can belong to different cube models

• Reside in the same database

• Are assigned to the same cube server

Definition

Page 40: Customer Cubing Services Education Session

© 2010 IBM Corporation40 Footer Field

Business Analytics

Primary usage scenarios

• Improve latency by creating historic and delta cubes

• Currency transformations by using currency lookup cube

• Combine data with unlike dimensions or data split by region

Page 41: Customer Cubing Services Education Session

© 2010 IBM Corporation41 Footer Field

Business Analytics

Delta cubes improve performance and lower latency

Time

AllSales

HistoricSales

Current Month Sales

Current Month

Original CubeAll Sales cube

• Small CurrentMonthSales cube is updated daily, hourly, or more frequently

• Large HistoricSales only updated once a month when roll in data from CurrentMonthSales

• Technical benefits:

• Faster MQT updates

• Cache remains warm for HistoricSales

Page 42: Customer Cubing Services Education Session

© 2010 IBM Corporation42 Footer Field

Business Analytics

Performance improvement example

100 times faster* average query response time against fact data updated incrementally using delta cube

* Results will vary based on ratio of data between the small delta cube and the large historic cube.

Page 43: Customer Cubing Services Education Session

© 2010 IBM Corporation43 Footer Field

Business Analytics

Currency conversion scenario

Store Sales Web SalesCurrencyConversion

Store Sales(multiple currencies)

Web Sales(multiple currencies)

Transform sales data so user can see in variety of currencies

Page 44: Customer Cubing Services Education Session

© 2010 IBM Corporation44 Footer Field

Business Analytics

Combines cubes with nearly identical dimensions

Virtual cube used as source for another virtual cube

Combines cubes with common Time dimension

Combine data with unlike dimensions or data split by region

Sales

Store SalesWeb Sales

Inventory Sales

Inventory

Page 45: Customer Cubing Services Education Session

© 2010 IBM Corporation45 Footer Field

Business Analytics

Merging cubes - Dimension Merging

Sales Cube Inv Cube Virtual Cube

Product Product Product

Time Time Time

Inventory Inventory

Store*

Merge dimensions with identical names

Unmatched dimensions added to virtual cube

User can hide unwanted dimensions

Users can merge dimensions with different names by changing the names to match. Details follow.

* The dimension is hidden for virtual cube

Page 46: Customer Cubing Services Education Session

© 2010 IBM Corporation46 Footer Field

Business Analytics

Merging cubes - Level Merging

Sales East Sales West All SalesCountry Country CountryState Province State ProvinceCity City CityNeighborhood Neighborhood

Sales East Sales West All SalesCountry Country CountryRegion Province Region ProvinceState State

Mismatched levels produce meaningless

results

For common dimensions, levels are merged starting from top of hierarchy

Unmatched names are

concatenated

User can’t change level names

Page 47: Customer Cubing Services Education Session

© 2010 IBM Corporation47 Footer Field

Business Analytics

Member Merging

All cube members will be in the virtual cubeCube A and Cube B have

shared dimension [Measures]

Cube A Cube B Virtual Cube

[Store Sales] [Store Sales] [Store Sales]

[Unit Sales East] [Unit Sales East]

[Unit Sales West] [Unit Sales West]

Merge members with identical

names

Unmatched members added to

virtual cube

Virtual Cube

CubeA CubeB

Page 48: Customer Cubing Services Education Session

© 2010 IBM Corporation48 Footer Field

Business Analytics

Merge operators

• Cubes are merged based on operator

• +, -, /, *, Max, Min

• Examples:

• Merge using add operation "+“• select from [Virtual Cube] =

• select from [CubeA] + select from [CubeB]

• Merge using MAX operator• select from [Virtual Cube] =

• MAX(select from [CubeA], select from [CubeB])

Virtual Cube

CubeA CubeB

Page 49: Customer Cubing Services Education Session

© 2010 IBM Corporation49 Footer Field

Business Analytics

Dimensional security with virtual cubes

• Dimensional security for a virtual cube is inherited from its cubes

• There are no specific security settings for virtual cube

• Users will have to set security on the base cubes

Virtual Cube

CubeA CubeB

Set dimensional security here Set dimensional

security here

Intersection of security settings from CubeA

and CubeB

Page 50: Customer Cubing Services Education Session

© 2010 IBM Corporation50 Footer Field

Business Analytics

Dimensional security

Store 1

CA

USA

Without dimensional security

Store hierarchy

San Jose

Store 1

WA

Seattle

Store 2

With dimensional security

USA

San Jose

Store hierarchy

CA

Seattle

WA

Store 2

• Fine-grained access control to OLAP metadata• Allows you to limit the members of a dimension that a role can

access

• By default, allows access to all members

Page 51: Customer Cubing Services Education Session

© 2010 IBM Corporation51 Footer Field

Business Analytics

Security model concepts

• ResourceAn OLAP object on which security is defined, e.g. cube, dimension/hierarchy (including Measures dimension)

• RoleA group of users that share the same security privileges

• PolicyA privilege or rule for accessing a resource, e.g. read access allowed to cube, read access denied to dimension/hierarchy members

• AuthorizationAn association among a resource, role, and policy

Page 52: Customer Cubing Services Education Session

© 2010 IBM Corporation52 Footer Field

Business Analytics

Security model lifecycle1. Create security model in

Design Studio

2. Export security model from Design Studio to XML file

3. Import security model from XML file to Admin Console

4. View security model in Admin Console

5. Start cube server or refresh security

6. Modify security model (if needed)

7. Delete security model from repository

Page 53: Customer Cubing Services Education Session

© 2010 IBM Corporation53 Footer Field

Business Analytics

Security enforcementSingle role/single policy

A user in a single role that is assigned a single policy on a dimension is allowed to access the set of allowed members except the set of denied members

Example:

Allowed Members = {Ascendants([Store].[CA]), Descendants([Store].[CA])}

Denied Members = [Store].[City].MembersUSA

CA

Store1

San Jose

Page 54: Customer Cubing Services Education Session

© 2010 IBM Corporation54

Business Analytics

Security enforcementMultiple roles/multiple policies

A user in multiple roles or a single role with multiple policies is allowed to access union(allowed members) except union(denied members)

Union(denied members)Union(allowed members) DM

DM

AM DM

DM

DM

DMAM = Allowed Member

DM = Denied Member

Page 55: Customer Cubing Services Education Session

© 2010 IBM Corporation55 Footer Field

Business Analytics

Security enforcementHierarchy navigation

Restricted members are skipped in operations like .parent and .children.

Member

Parent Member

Children

Page 56: Customer Cubing Services Education Session

© 2010 IBM Corporation56

Business Analytics

Remote cube server management

Repositorydatabase

WebSphere

Server 3

Server 1

Cube Server 3

Server 2

Cube Server 2

Cube Server 1

Admin Console

Page 57: Customer Cubing Services Education Session

© 2010 IBM Corporation57

Business Analytics

Proof pointsRecent success stories

Telecom customer

Retail customer

IBM integrated supply chain

Page 58: Customer Cubing Services Education Session

© 2010 IBM Corporation58

Business Analytics

Telecom customer overview

Industry: Telecom

InfoSphere Balanced Warehouse like system– Database module - 42 data partitions– Application module - Model 9133-55A, 4 PowerPC_POWER5 processors, 32GB memory

Software– DB2 9.5– InfoSphere Warehouse 9.7.1– Cognos 8.4

Warehouse– Four star schemas– Largest fact table has 1.5 billion rows. Other fact tables have 50 million, 20 million and 8 million rows.– 8-12 dimensions per fact table (most shared)

Cubes– Cube per star schema– Virtual cube merges the four cubes

• 12 dimensions• 25 measures – varying complexity

Cognos reports took 5-7 seconds

Page 59: Customer Cubing Services Education Session

© 2010 IBM Corporation59

Business Analytics

VirtualCube 2

Reactivation

PerformanceDashboard

TossAdjustment

Activation ARPU

VirtualCube 1

Telecom Customer - Cubes

• Assess whether monthly net additions are improving.

• Calculate the average revenue per user (ARPU) to see if it matches industry competitors. This is the total revenue divided by the total subscribers for a particular time period

• Determine if subscriber additions have increased from the same period the previous year.

• Analyze the churn rate for cell phone subscriptions over time to determine whether the company is gaining or losing ground compared to competitors

Page 60: Customer Cubing Services Education Session

© 2010 IBM Corporation60

Business Analytics

Telecom Customer - Details

Tuning– New database with no optimization– Defined primary keys on dimension tables– Defined constraints (not enforced) on star schema joins– Ran Cubing Services Advisor (no data sampling)

• Recommended indexes on fact table foreign keys and dimension level keys• Recommended MQTs

– Built recommended indexes and MQTs

Additional tuning that could be done– Define MDC for fact table– Run Cubing Services Advisor with data sampling to get better recommendations– Run DB2 Design Advisor for additional index/MQT recommendations

Page 61: Customer Cubing Services Education Session

© 2010 IBM Corporation61

Business Analytics

Retail customer overview

Industry: Retail

IBM Smart Analytics System 7600 v9.7 (pSeries and AIX)– Database module - 1 admin node + 2 data nodes (8+1 DPF partitions), Power6, 4-core, 32 GB– Application module - 4-core, 32 GB (actually 8-core 64 GB but only used half)

Software– DB2 9.7.1– InfoSphere Warehouse 9.7.1– Cognos 8.4

Warehouse– One star schema– Fact table has 2 billion rows– Four dimensions

Cubes– One cube

• 4 dimensions• 12 measures – simple aggregations

Cognos reports took 5-7 seconds

Page 62: Customer Cubing Services Education Session

© 2010 IBM Corporation62

Business Analytics

AllTime

Time Category Store

Year

Month

Week

Organization

Category

Department

AllCategories

Region

Store

Area

AllLocations

Retail Customer - Model

Vendor

Vendor

AllVendors

• Which stores generated the highest total sales

• How did sales compare for particular products from the previous year

• Which region was the most profitable

Page 63: Customer Cubing Services Education Session

© 2010 IBM Corporation63

Business Analytics

IBM integrated supply chain

Software:– DB2 9.5.2– InfoSphere Warehouse 9.5.2– Alphablox 9.5.2– AIX 5.3

Warehouse:– 1 large table with dimensions embedded within fact table – over 70 million rows and growing– Offline for maintenance only 1x week for 4 hours– Refreshing cubes takes minutes in most case, up to 20 minutes for largest cube

300 users worldwide

Saved $6 million in first year, and expect to save more than double that in the second year

Page 64: Customer Cubing Services Education Session

© 2010 IBM Corporation64

Business Analytics

Technical resources

Information Center for 9.7 http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jsp

Cubing Services Tutorial http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/topic/com.ibm.dwe.tut.cube.doc/tutcubetutorial.html

Virtual cubes tutorialhttp://publib.boulder.ibm.com/infocenter/db2luw/v9r7/topic/com.ibm.dwe.tut.cube.doc/tutcubetutorial.html

Dimensional security developerWorks articlehttp://www.ibm.com/developerworks/data/library/techarticle/dm-0909securityinfospherecubing/

Virtual cubes developerWorks article http://www.ibm.com/developerworks/data/library/techarticle/dm-0909infospherevirtualcubes/

Page 65: Customer Cubing Services Education Session

© 2010 IBM Corporation65

Business Analytics

Backup

Page 66: Customer Cubing Services Education Session

© 2010 IBM Corporation66

Business Analytics

IBM Cognos PowerCube™ – MOLAP

Business modeling

OLAPRelational Cognos ReportsFlat files

Access to All Data

PowerCube

Trending / Slice & DicePrimary OLAP Use:

Optimized for:• Highest consistent query

performance• Enterprise rollouts such as

internet delivery

Primary owner – Business Analyst or LOB IT

Operational reportingSales trend & ad hoc analysis

Because of its ability to provide:

• Pre-aggregated compressed data that can be disconnected

• Automatic time series analysis, trending, & point in time data

Page 67: Customer Cubing Services Education Session

© 2010 IBM Corporation67

Business Analytics

Key capabilities - Cognos Transformer & PowerCube™

Flexible, fit for purpose, pre-aggregated storage for optimal performance

Partition information by time, level, or security

Build, design, query in disconnected situations*

Incrementally add data hourly, daily, etc

High performance

portable storage

Simple to complex modeling addressing departmental needs

Multiple drill paths, multiple time dimensions

Business rules, calculated members or levels,

custom time functions

Flexible for unique business

needs

With no SQL skills:Merge data from SAP, flat files, other OLAP

dimensions, & Cognos 8

Time series analysis, point in time measures

Enable the businessto model data for their

business problems

Self service business modeling

Series 7 PowerPlay or Cognos 8 front end clients

Page 68: Customer Cubing Services Education Session

© 2010 IBM Corporation68

Business Analytics

What-if scenario modelingPrimary OLAP Use:

Ideal for:• write-back planning

applications in moderate sized communities;

• complex models demanding read/write interactivity

Primary owner – LOB IT

Operational PlanningFinancial analytics &

reporting

IBM Cognos TM1 – in memory MOLAP

OLAPRelational Flat files

Access to All Data

TM1 Servers

TM1 Architect

TM1 clients Cognos 8 clients

Because of its unique:• on demand aggregation and

calculations with 64 bit in- memory processing

Page 69: Customer Cubing Services Education Session

© 2010 IBM Corporation69

Business Analytics

Key capabilities - Cognos TM1

Immediate answers to complex questions on

highly volatile data

Business driven analysis for real time response

Create simulations easily to interactively test

business assumptions

Advanced rules language for complex financial

applications

Enterprise level collaboration

Quickly load & merge large data volumes (intra-day or nightly) from ODBC

/ ODBO & flat files

Data import & scripting engine in a multi-cube

architecture with shared dimensions

Compute hierarchical aggregations & complex calculations on demand

64 bit in-memory processing

Cognos TM1 or Cognos 8 front end clients

Enterprise data contributions and what-if

scenario planning

Easily create and access reports or input templates

deployed via Excel or web

On the fly dimensional hierarchy changes & self service ad-hoc analysis

Page 70: Customer Cubing Services Education Session

© 2010 IBM Corporation70

Business Analytics

Client perspective

Results without dimensional security

Results with dimensional security

The value for Store Sales is not the same for All Store, USA, and CA,

i.e. no visual totals.

Query: select [Measures].[Store Sales] on columns, [Store].members on rows from [sales]

Page 71: Customer Cubing Services Education Session

© 2010 IBM Corporation71

Business Analytics

Working with Virtual Cubes

Users will define virtual cubes using Design Studio.

Administrative tasks for Virtual cubes are performed using the Cubing Services Administration Console

Page 72: Customer Cubing Services Education Session

© 2010 IBM Corporation72

Business Analytics

Create a virtual cube using Design Studio

Cubes

Virtual Cube

Virtual Calculated measures

Virtual Measures

Only renamed virtual dimension will

appear here, not all virtual dimensions

Page 73: Customer Cubing Services Education Session

© 2010 IBM Corporation73

Business Analytics

Create a virtual cube in Design Studio Add virtual cube

Optionally rename dimensions to cause merging of desired ones– Users can hide dimensions so they don’t exist on the virtual cube– Optionally specify a default member for the dimension– Optionally rename members to cause merging them

• Members can have their own merge operators• Members can also be hidden on the virtual cube

– Optionally define calculated members

Export the model

Page 74: Customer Cubing Services Education Session

© 2010 IBM Corporation74

Business Analytics

Use dimension browser in Design Studio to ease design

New Design Studio usability enhancement

– Display hierarchy members directly from the Design Studio

– Support all member types: MDX and SQL calculated members

– Support all hierarchy types

– Support all real and virtual cubes

Use to shorten and simplify design time of real and virtual cubes

Page 75: Customer Cubing Services Education Session

© 2010 IBM Corporation75

Business Analytics

Managing virtual cubes from Admin Console

In Admin Console, virtual cubes belong to a cube model called VIRTUAL. This is where they can be deleted.

Page 76: Customer Cubing Services Education Session

© 2010 IBM Corporation76

Business Analytics

Managing virtual cubes from Admin Console

Starting, deploying to a cube server, stopping, and undeploying virtual cubes operations are similar to real cubes except for a few new dependencies:

– A virtual cube can be started/deployed only after both its direct cubes are started/deployed

– A cube or virtual cube can only be stopped/undeployed after all the virtual cubes that directly use it are stopped/undeployed

Virtual Cube

Page 77: Customer Cubing Services Education Session

© 2010 IBM Corporation77

Business Analytics

Create security model in Design Studio - Cube security

Page 78: Customer Cubing Services Education Session

© 2010 IBM Corporation78

Business Analytics

Create security model in Design StudioDimension security (non-Measures dimension)

Allowed Members and Denied Members must be MDX expressions that evaluate to a set of members

MDX is validated for syntax errors only

Page 79: Customer Cubing Services Education Session

© 2010 IBM Corporation79

Business Analytics

Create security model in Design StudioDimension security (non-Measures dimension) (cont’d)

Page 80: Customer Cubing Services Education Session

© 2010 IBM Corporation80

Business Analytics

Create security model in Design StudioDimension security (Measures dimension)

Page 81: Customer Cubing Services Education Session

© 2010 IBM Corporation81

Business Analytics

Create security model in Design Studio - Roles

Page 82: Customer Cubing Services Education Session

© 2010 IBM Corporation82

Business Analytics

Export security model from Design Studio to XML file

Only security-enabled objects are exported

Only policies and roles that are referenced by an authorization are exported

XML file containing the security model is different from XML file containing the cube model

Page 83: Customer Cubing Services Education Session

© 2010 IBM Corporation83

Business Analytics

Import security model from XML file to Administration Console

1. Select the database that contains the tables that are referenced by the cubes in the XML file

2. Specify the XML file that contains the security metadata

3. Map roles

4. Select collision resolution option - Merge Replace, Merge Ignore, Replace

Page 84: Customer Cubing Services Education Session

© 2010 IBM Corporation84

Business Analytics

Import security model from XML file to Administration Console - Collision resolution options

Old security model

Dimension: Store

Policies: StorePolicy1

Authorizations: (Role1,StorePolicy1), (Role2,StorePolicy1)

Dimension: Product

Policies: ProductPolicy

Authorizations: (Role3,ProductPolicy)

New security model

Dimension: Store

Policies: StorePolicy2, StorePolicy3

Authorizations: (Role1,StorePolicy2), (Role3,StorePolicy3)

Dimension: Measures

Policies: MeasuresPolicy

Authorizations: (Role3,MeasuresPolicy)

Collisions are detected and resolved at the resource level

Example: Merge Replace

Merged security model

Dimension: Store

Policies: StorePolicy2, StorePolicy3

Authorizations: (Role1,StorePolicy2), (Role3,StorePolicy3)

Dimension: Product

Policies: ProductPolicy

Authorizations: (Role3,ProductPolicy)

Dimension: Measures

Policies: MeasuresPolicy

Authorizations: (Role3,MeasuresPolicy)

Added

Kept

Replaced

Accepted

Page 85: Customer Cubing Services Education Session

© 2010 IBM Corporation85

Business Analytics

View security model in Administration Console

View the security metadata associated with a cube and its dimensions by selecting Manage Cube Models <cube model> Edit <cube> View Privileges

Page 86: Customer Cubing Services Education Session

© 2010 IBM Corporation86

Business Analytics

View security model in Administration Console (cont’d)

View the security metadata associated with a role by selecting Manage Roles <role> View Privileges

Page 87: Customer Cubing Services Education Session

© 2010 IBM Corporation87

Business Analytics

Start cube server or refresh security

Hit “Reload Security” button to update the security metadata in a running cube server

Page 88: Customer Cubing Services Education Session

© 2010 IBM Corporation88

Business Analytics

Delete security modelTo remove the security metadata associated with a cube and its dimensions, select Manage Cube Models <cube model> Edit <cube> Delete Cubes and Security

Delete Cubes and Security

Page 89: Customer Cubing Services Education Session

© 2010 IBM Corporation89

Business Analytics

Miscellaneous Design Studio wizards and explorers to manage and view security model

Import wizardImports security model from XML file to database in project or database explorer

Deploy wizardExports security model from database in project explorer to database in database explorer

Create database from reverse engineeringExports security model from database in database explorer to database in project explorer

Database explorer Displays security in the repository database including default cube security

Page 90: Customer Cubing Services Education Session

© 2010 IBM Corporation90

Business Analytics

Thank you!