Oracle Business Intelligence 11g Performance Tuning ... BI 11g... · Oracle Business Intelligence...

Oracle Business Intelligence 11g

Performance Tuning – Customer Success Stories

Antony Heljula December 2013

© Peak Indicators Limited 2

About this Presentation Aggregation using Oracle BI

• This presentation will show how to achieve significant performance gains by using the aggregation capabilities of the Oracle BI Server

• Demonstrating the power of the Aggregate Persistence Wizard

• 2 customer examples will be used

• The methods and techniques discussed can apply to all customers, from small implementations to those involving Exalytics or Exadata


Agenda Performance Tuning – Real Success Stories

• Customer Overview • Proposed Solution • Dashboard Analysis • Tuning Strategy • Aggregate Persistence • Incremental Loading • Summary


Performance Tuning – Real Success Stories Customer Overview


The world's leading provider of global satellite communication services


The UK’s Biggest Outdoor Stores


Customer Overview Oracle BI Specifics (At the beginning)

Oracle BI Version 10.1.3.4 11.1.1.5

No. of Users 100 50

Database Oracle Database EE 10G Oracle Database EE 10G

Data Volumes 450GB (Compressed) 230GB

Platform Windows (OBI) / Solaris (DB) Linux

Dashboards/Reports (approx) 20 / 320 10 / 100

Subject Areas (example) Network, Voice/Data Traffic, Registrations, Faults

Sales, Stock, POs, Budgeting, Deliveries, Customers, Supplier


Customer Overview Observations/Issues

Reports taking up to 27 minutes

Reports were mainly of an “operational” nature, hard to optimise

Many different Time Series calculations used in some reports, going back 2 years

No aggregates deployed yet

Some reports taking up to 2 minutes

Still using OBI 10g, due to resource constraints and other projects taking precedence

Further aggregates on the data warehouse were not created due to project time scales and scope

Data Warehouse on an Oracle DB 10g platform, hardware/software not latest generation


Customer Overview Dashboard Response Times (Avg)

60 sec average

15 min average

Target

Target


Performance Tuning – Real Success Stories Proposed Solution


Proposed Solution Aggregation using Oracle BI

• A solution based on Aggregate Persistence was proposed to both customers: Agile creation of aggregates through the BI Server Rapid build and deployment Elimination of disk/network bottlenecks No dependencies on DW/ETL team to build and

maintain aggregates Flexible approach for maintaining aggregates Maximising scalability and performance Reduced load on the underlying data warehouse

• For Inmarsat, the solution would be prototyped in Oracle BI

11g. RPD and dashboards had to be upgraded as part of the exercise


Proposed Solution Aggregate Persistence

• With minimal effort, OBI will automatically build and populate a series of aggregate tables on a chosen data source

• OBI will then automatically redirect any “summary” reports to use these aggregates

• New aggregates built without downtime

• OBI is 100% available whilst aggregates are being built


Proposed Solution Aggregate Persistence - Exalytics

• In the case of Exalytics, the “aggregate” tables are placed in the TimesTen in-memory database

• TimesTen is located on the Exalytics server

• With data placed in-memory on the BI Server the result is complete elimination of disk and network bottlenecks

• Exalytics-only features include: • Columnar Compression • Exalytics Summary Advisor


Further Notes In-Memory Performance

½ TB RAM, 32 cores Databases for

medium businesses / departmental

1 TB RAM, 40 cores, 2.4TB Flash Middleware consolidation and

performance platform focussed OBI / TimesTen / Essbase / Endeca

4 TB RAM, 22TB Flash, 160 cores, 14 Exadata Storage Cells

Extreme performance for all types of OLTP/DW database workload.

Database Consolidation.

Half/Quarter/Eighth rack options available


Performance Tuning – Real Success Stories Dashboard Analysis


Dashboard Analysis Overview

• A selection of dashboard pages were selected at each customer

• The aim was to build a set of common aggregates that could each be used across multiple dashboard pages. A period of analysis was therefore required to identify the facts/dimensions used across all the dashboards

• The Exalytics “Summary Advisor” would have helped to automate/simplify this process

• Without the Summary Advisor, the process of defining the aggregates had to be done manually

• Intensive process • Approximately 2-3 days of analysis required for 30 reports


Dashboard Analysis Complications

• In order to obtain the facts/dimensions you have to consider several areas of metadata, including: • Analysis columns • Analysis filters • Analysis formulas (e.g CASE or FILTER expressions) • Is Prompted Filters • Dashboard Prompts • Column Selectors


Dashboard Analysis Mock-up of Inmarsat Dashboard

• Dashboards typically “analytical” in nature, showing summaries for network/traffic performance over time

• Dynamically choose to summarise data by Hour/Day/Month • Column selectors allow additional dimensions/facts to be used on reports


Dashboard Analysis Go Outdoors

• Dashboards typically “operational” in nature, providing a snapshot of the current position

• Many metrics covering Sales, Stock, Payments over last 2 years • >10 different Times Series metrics on each report • 10s of Millions of records being scanned


Dashboard Analysis Example Fact/Dimension Matrix

Analyses

Logical Fact Tables

Dimensions

Dimension Hierarchy

Levels


Dashboard Analysis Conclusion

9 aggregates identified that would provide good levels of summarisation and cater for all reports

Summarising to Hourly, Day and Month levels

About 18 dimensions in total used across 9 aggregates

Aggregates were similar in dimensionality to the existing aggregates on the DW, but with some

very granular dimensions removed e.g. Subscriber (IMSI)

No suitable aggregates identified

All reports tended to be at the lowest levels of the 3 biggest dimensions:

Day, Product, Store

The new aggregate tables would not provide enough summary to significantly improve reports

e.g. 36M rows down to 30M rows

A further round of analysis was required – summarising existing data would not provide

sufficient benefit


Performance Tuning – Real Success Stories Tuning for Operational Reports


Tuning for Operational Reports Go Outdoors

• Because the Go Outdoor reports were reporting at the lowest level of detail across the 3 main dimensions, it was impossible to build aggregates that significantly reduced the amount of data

• So how do you achieve the goal of making reports run 20-100x faster?

Day

Product Store

Before

Day

Product Store

After


Tuning for Operational Reports Dimension Elimination

• If you cannot summarise data to a higher level of granularity, then try to see if dimension elimination can help:

Day

Product Store

If we could eliminate the “Time” dimension then data volumes could, in

theory, be reduced by 365x per year


Tuning for Operational Reports

Dimension Elimination • Take this report as an example, the columns on the report suggest that we

should only need to aggregate sales data across 2 dimensions: • Product Dimension (at “Company Division” level) • Stores Dimension (at “Store” level)

Company Division Store


Tuning for Operational Reports

Dimension Elimination • But there are two places which force additional dimension levels

into the query:

This filter is applied across all reports, the “Stock Type” attribute is at the lowest

level of the “Product” dimension

The column formula uses a FILTER() expression based on a dimension

attribute from the “Time” dimension



• This causes two areas of inefficiency as the report forces the use of: 1. The lowest level of the Time dimension 2. The lowest level of the Product dimension

Day

Product Store


HOW DO WE CONVERT?



• We can achieve Dimension Elimination by pushing these calculations and filters down into the RPD

• A new set of metrics were created that combined the common Formula/Filter logic used across the reports:



• The report was now simplified and only involved two dimensions: 1. Column formulas replaced with RPD metrics 2. Redundant filters removed



• End result: • Time dimension now eliminated completely from report • Data summarised up to “Company Division” level of the Product dimension

• Aggregate Persistence would be configured to work of these new dedicated metrics

• A 12x reduction in aggregate data volumes was achieved: • 36M records down to 3M records



Performance Tuning – Real Success Stories Aggregate Persistence


Aggregate Persistence Building the Aggregates

• Now that the aggregates had been defined and the custom calculations created, the Aggregate Persistence Wizard could be invoked to create the aggregate scripts

• NOTE: On an Exalytics machine, the “Summary Advisor” does this automatically for you


Aggregate Persistence Before your begin

• The “Create Aggregates” scripts perform a number of tasks: 1. Create dimension/fact tables 2. Create unique indexes (based on logical keys) 3. Perform full load of data 4. Analyse tables 5. Model new aggregates into the RPD

• Before running them, it is important to verify: • Your RPD is consistent, not locked • Your logical keys in the RPD are properly unique • Your dimension hierarchies are configured with all the required columns • The necessary create privileges are granted on the database schema


• Before running the Aggregate Persistence scripts, run the “Model Checker” to identify any issues with your dimensional models e.g. non-unique keys

• File > Check Models > Complete

Aggregate Persistence

BI Admin Tool “Model Checker”


Aggregate Persistence Model Checker: Reconfigure Dimension Hierarchies

• The Model Checker found dimension hierarchy levels configured in the RPD that were not strictly unique

• This would have caused the Aggregate Persistence to fail when creating unique keys on these columns

There are two “PLAN_NAME” values which are the same. It is not strictly a

Logical Level Key



Model Checker: Reconfigure Dimension Hierarchies

• The fix was simply to move the offending column to a higher level in the dimension hierarchy:



Adding Columns to Dimension Hierarchies

• Aggregate Persistence will only consider dimension attributes which are present in the Dimension Hierarchies

• In the example shown, a number of additional columns were added to the “Product” dimension so that Aggregate Persistence would include them in the aggregate tables

• Your dimension hierarchies will naturally be larger as a result



TimesTen Data Types

• When creating tables in TimesTen, pay very close attention to the data types being used. With TimesTen the NUMBER data type works differently to Oracle and it defaults to a very large precision – your tables will be significantly larger than they need to be

• Recommendations: • For any integer, use one of the TimesTen integer data types (TT_TINYINT, TT_SMALLINT, TT_INTEGER,

TT_BIGINT) • If you need to have decimal places, then specify the exact precision e.g. NUMBER(5,2)

• Read the TimesTen documentation to find out the most appropriate data types for your needs.

For example:

Name Oracle Data Type TT Data Type

---------------------- ------------------ -----------------

PRODUCT_KEY NUMBER TT_SMALLINT

SUPPLIER_KEY NUMBER TT_SMALLINT CUSTOMER_KEY NUMBER TT_INTEGER SALES_VALUE NUMBER NUMBER(5,2) SALES_QTY NUMBER TT_SMALLINT

Failure to use the right data types with TimesTen can make your tables 5x bigger

than they need to be. In one case we reduced a table from 2.4GB

down to 600MB by changing data types


Aggregate Persistence The Results

• 9 aggregates created for Inmarsat • 8 aggregates created for Go Outdoors

Hour / Day / Month aggregates for Inmarsat

“Call Summary” facts

Year AGO and Week sales snapshots for Go Outdoors


Aggregate Persistence Go Outdoors Performance Gain

• 88x average performance gain across 6 dashboards • Some reports reduced down from 25 minutes to less than 20 seconds • Delivered in <10 man-days • Size of Aggregates: 5GB


Aggregate Persistence Inmarsat Performance Gain

• 27x average performance gain across 6 dashboards • Delivered using TimesTen in-memory aggregates (without Compression) • Delivered in <15 man-days • Size of Aggregates: 30GB


Performance Tuning – Real Success Stories Incremental Loading


Incremental Loading Problems with Full Load

• The results of the performance test were well above expectations

• There was one key issue which prevented it from being considered for production use: • Inmarsat: The full load of 9 aggregates took 15 hours • Go Outdoors: The full load of 8 aggregates took 4.5 hours

• A solution for incremental loading had to be found, but the solution had to be based on Aggregate Persistence


Incremental Loading It is possible!

• Incremental load is actually possible with Aggregate Persistence (and Exalytics)

• You can modify the “create aggregate” scripts to implement incremental loading via the BI Server

• Visit our web-site blog: “Incremental Loading Exalytics using Notepad”

• The incremental load scripts can do the following: • Create an aggregate staging table • Populate the staging table with a subset of data extracted from source system • Incrementally update the target aggregate table • Analyze the target aggregate table

http://www.peakindicators.com/index.php/knowledge-base/1134-incrementally-loading-exalytics-using-notepad


Incremental Loading Peak ETA

• To take things a step further, instead of using scripts to incrementally load the aggregates, we developed tool called Peak ETA (Extract Transform Aggregate) • The tool reuses all the code generated by the Aggregate Persistence Wizard and automates the

process of producing all the incremental load scripts

• The key benefits are: In-memory aggregates loaded incrementally and in parallel No downtime (aggregates are fully available whilst being loaded) A GUI console to orchestrate and monitor the loading of aggregates Error Detection with BI Dashboards with alerting capability Automatic code generation Automatically caters for Surrogate Keys RPD Metadata never dropped/recreated

• All this is achieved using standard Exalytics technology and methodology


Incremental Loading Peak ETA

• For more details visit our web-site: • http://www.peakindicators.com/index.php/peak-eta

http://www.peakindicators.com/index.php/peak-eta


Incremental Loading Optimised Incremental Loading – Peak ETA

27x faster 20x faster


Performance Tuning – Real Success Stories Summary


Summary Go Outdoors - Going Forwards

• Aggregate Persistence solution went live December 2012

• Incremental loading running every day

• Extremely reliable – no failures

• Further aggregates continue to be created as required, turnaround time typically 1 man-day

88x faster


Summary Inmarsat - Going Forwards

• Following the performance prototype, an Oracle BI 11g upgrade project was approved

• 11g upgrade completed March 2013 • Upgraded 11g solution making use of Aggregate Persistence

• Inmarsat have recently doubled the number of OBI users

• New dashboards being implemented involving maps and

Oracle Spatial

• Further lines of business being introduced to OBI 11g

• Hopefully in the future a consolidation exercise will see the DW migrated onto their Exadata platform

27x faster


Summary Inmarsat – Satellite Spot Beam Map

NOTE: This map has been produced with mock-up test

data. The shading across the map bears no relevance to actual figures


Summary Aggregate Persistence

• Aggregate Persistence has the potential to become the standard aggregation tool for all customer implementations

• Especially as incremental loading is now possible

• These 2 examples prove Oracle BI 11g can add

significant business value to customers - bringing savings in time, cost and complexity

• We look forward to further enhancements from Oracle!


Further Optimisation! Oracle BI as a Service

• Go Outdoors have migrated their entire DW/BI infrastructure to the Peak Indicators “Oracle BI as a Service” infrastructure

• Powered by Oracle Sun hardware running Oracle VM Server • Managed Services provided by Peak Indicators

• As part of the free migration service, OBIEE upgraded to 11.1.1.7 and

Oracle Database upgraded to 11.2.0.4 • Significant performance improvement for overnight ETL and reporting • Without any tuning, some reports have reduced from 27 minutes down

to 1 minute (due to better hardware and better db optimizer)

• Visit our web-site for more details: • http://www.peakindicators.com/index.php/oracle-bi-as-a-service

http://www.peakindicators.com/index.php/oracle-bi-as-a-service


Peak Performance Performance Tuning Expert Service

• You can achieve the same results!

• Please contact us if you wish to benefit from our “Peak Performance” Expert Service:

[email protected]

mailto:[email protected]

Helping Your Business Intelligence Journey

Oracle Business Intelligence 11g Performance Tuning ... BI 11g... · Oracle Business Intelligence...

Documents

Transcript of Oracle Business Intelligence 11g Performance Tuning ... BI 11g... · Oracle Business Intelligence...