Essbase Statistics DW: How to Automatically Administrate Essbase Using ODI

Essbase Statistics DW: How to automatically administrate

Essbase using ODI

Ricardo Giampaoli

Rodrigo Radtke

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMAbout the Speakers

Giampaoli, Ricardo

• Oracle Ace

• Master in Business Administration and IT management

• EPM Consultant @ Dell

• Essbase/Planning/OBIEE/ODI Certified Specialist

• Blogger @ devepm.com

Radtke, Rodrigo

• Oracle Ace

• Graduated in Computer Engineering

• Software Developer Sr. Advisor at Dell

• ODI, Oracle and Java Certified

• Blogger @ devepm.com

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMWhat we'll learn

• Essbase Performance

• Essbase Caches

• Java Essbase API

• Essbase DW

• ODI Automation

• Results

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMEssbase Performance

• The Essbase performance has a direct relationship with the settings that we apply for each cube that should be based on the design and usage

• The main problem with Essbase statistics is that they are a snapshot of the cube, and we would have a better understanding of the cube if we had the historical information about it

• To have the historical information we need to constantly get and store the statistics somewhere (DW)

• We can analyze the historical information and keep improving the Essbase cubes especially true for BSO, the focus of this presentation

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMEssbase Caches

• Essbase use caches to speed up calculations/retrieves/data load/data extract

Cache Type Description

Index A buffer in memory that holds index pages (.ind files)

Data file A buffer in memory that holds compressed data files (.pag files). Essbase allocates

memory to the data file cache during data load, calculation, and retrieval operations, as

needed. (only when direct I/O is in effect)

Data A buffer in memory that holds uncompressed data blocks. Essbase allocates memory to

the data cache during data load, calculation, and retrieval operations, as needed

Calculator A buffer in memory that Essbase uses to create and track data blocks during calculation

operations

Dynamic

calculator

A buffer in memory that Essbase uses to store all of the blocks needed for a calculation of

a Dynamic Calc member in a dense dimension

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMIndex Cache

• The index Cache is used to hold in memory all used indexes since the application is turned on.

• The maximum index cache size is calculated by multiplying the number of existing Blocks * 112 bytes• A Block is the combination of all store like members from all dense dimensions.

Essbase creates a new Block when a new unique combination of data in the sparse dimensions is inserted/calculated

• The number of existing Blocks can be acquired by:

EAS MAXL ESSCMD JAVA

Right click in a cube edit

properties->Statistics

query database

sample.basic get

dbstats data_block;

select sample basic;

getdbstats;

(IEssCube) cube.getPropertyNames();

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMIndex Cache

• The optimal index cache will vary depending of the design and use of the cube. • If the cube has a lot of historical data

the chances of these index never hit the cache are big

• For that we can define a historical percentage that we can remove from the maximum index cache to find a more realistic cache

• This historical percentage can be obtained by analyzing the outline and see how many members are in use

Years

FY10

FY11

FY12

…

FY17

FY18

FY10 Scenario

Actuals

Q1FCST

Q2FCST

Q3FCST

Q4FCST

Full Index

40%

30%

Recom

mended

Index (

30%)

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMData File Cache

• The data file cache is used to hold compressed data files (essn.pag) in memory and is used only when direct I/O option is in effect.

• It is calculated by summing all essn.pag files, if possible; otherwise, as large as possible

• The size of the essn.pag files can be acquired by:

•EAS MAXL ESSCMD JAVA


properties->Storage

query database

sample.basic list all

file information;

listfiles "" "sample"

"basic";

maxl.execute("query database " +

s_application + "." + s_cube + " list all

file information");

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMData Cache

• The data cache is used hold uncompressed data blocks in memory.

• The recommended size would be 12.5 % of the summing of all essn.pag files (or 0.125 * Data File Cache size)

• Same case as Index cache, just the working blocks are used (no historical)

• The size of the essn.pag files can be acquired by:



properties->Storage

query database

sample.basic list all

file information;

listfiles "" "sample"

"basic";


s_application + "." + s_cube + " list all

file information");

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMCalculator Cache

• A buffer in memory that Essbase uses to create and track data blocks during calculation operations

• Essbase split the sparse dimensions in 2 groups:• Bitmap dimensions: Any sparse dimensions that can be fit inside the bitmap. Each

member combination placed in the bitmap occupies 1 bit of memory, and the entire dimension must fit inside the bitmap.

• Anchoring dimensions: The remaining sparse dimensions that do not fit into the bitmap.

• Because the calculator cache controls the size of the bitmap, the number of sparse dimensions that can fit in the bitmap and anchoring dimensions depends on the size of the calculator cache

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke


• Essbase starts with the first sparse dimension in the outline and fits as many as possible into the bitmap.

• Essbase stops the process when it cannot fit another complete sparse dimension into the bitmap.

• All the remaining sparse dimensions becomes anchoring dimensions

• The number of bitmaps used is determined by the maximum number of dependent parents for any members in the anchoring dimension• The max number of dependent parents is the same as the number of Levels in

a dimension

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPM

BITMAPBITMAP

BITMAPBITMAP

Calculator Cache

• Depending of the cache setting we can have 3 options:

Sparse

Dims

# of

Members

Dependent

Parents

S1 10 NA

S2 10 NA

S3 20 NA

S4 30 NA

S5 200 3

BITMAP

Number of Bitmaps

Dependent Parents + 2

Bitmap Size = (S1*S2*S3*S4)/8

Option Calculator Cache

Single

anchoring,

multiple

bitmaps

(10*10*20*30)/8=7500

3+2=5

7500*5=37500 bytes

Single

anchoring,

single bitmap

(10*10*20*30)/8=7500

1

7500*1=7500 bytes

Multiple

anchoring,

single bitmap

(10*10*20)/8=250

1

250*1=250 bytes

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke


• To calculate the Calculator cache we need to know:• The amount of member in each sparse dimensions

• The amount of levels in the biggest sparse dimension

• The biggest sparse dimensions should be placed in the last place on the outline

• These information can be acquired by:


Amount of members


properties->Dimensions

None None selection.executeQuery(dimension.getName(),

IEssMemberSelection.QUERY_TYPE_DESCENDANTS, 0,

null, null, null);

Levels

Double click in outline->

count the parents

None None IEssDimension dimension = (IEssDimension)

otl.getDimensions().getAt(i);

IEssIterator level = dimension.getLevels();

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMDynamic Calculator Cache

• A buffer in memory that Essbase uses to store all of the blocks needed for calculate a Dynamic Calc member in a dense dimension

• To find the optimal size of this cache we need:• CALCLOCKBLOCK size: it is half the number of expanded blocks that fit into

the data cache

• Expanded Blocks including: Store, Never Share, Dynamic Calc members and dynamic time series members

• The largest expanded block across all databases on the machine.

• The maximum number of expected concurrent users

• Can be analysed by gathering the session info into a table and analyse the patterns but for this presentation is a constant number based in experience

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMDynamic Calculator Cache

• To calculate the Maximum Calculator Cache we need to multiply:• CALCLOCKBLOCK: (Data Cache in bytes (already calculated) / Size of the

expanded Block in bytes) / 2

• The largest expanded Block in bytes on the server

• The maximum number of expected concurrent Users (Constant)

• These information can be acquired by:


Data Cache


properties->Storage

query database

sample.basic list

all file information;

listfiles ""

"sample"

"basic";


s_application + "." + s_cube + " list all file

information");

Amount of members


properties->Dimensions

None None selection.executeQuery(dimension.getNam

e(),

IEssMemberSelection.QUERY_TYPE_DESCEN

DANTS, 0, null, null, null);

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMEssbase Caches Calculation

Cache Type Description

Index number of existing Blocks * 112 bytes = the size of database index

Data file Combined size of all essn.pag files, if possible; otherwise, as large as possible

Data 0.125 * the value of Data File Cache size

Calculator Bitmap size in bytes * Number of bitmaps

Bitmap size in bytes Max((member combinations on the bitmap dimensions / 8), 4)

Number of bitmaps Maximum number of dependent parents in the anchoring

dimension + 2 constant bitmaps

Dynamic

calculator

C * S * U

C is the value of the appropriate CALCLOCKBLOCK setting. (Data Cache in bytes / Size of the

expanded Block in bytes) / 2

S is the size of the largest expanded block across all databases on the machine. Multiply the

number of members in each dense dimension together and multiply by 8 bytes

U is the maximum number of expected concurrent users

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMAutomating using Java Essbase API

• Almost all the information required for this presentation could be exported from Essbase by Maxl/Esscmd commands to a text files

• Java can easily connect and manipulate Essbase through its API• Java can also connect to Oracle Database, run OS commands and more all in one

single code

• Java may be easily deployed to ODI using Procedures and scheduled using ODI Operator

• Combining ODI and Java code creates a powerful and seamlesslyintegration going beyond the database boundaries

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMIE

ssbase

IEssDomain

IEssMaxlSession

IEssOlapServer IEssOlapApplication IEssCube

IEssDimension

IEssCubeOutline IEssMember

IEssMemberSelection

Java Basic

Ora

cle

Data

Sourc

e

Connection PreparedStatement

• Since we are developing one single code we need to connect in Essbase to retrieve the information and load that in the Oracle database

• Essbase API is very similar with what we see in EAS• The structure of the classes are the “same” as we

have in a Essbase server (Server->App->Cube->Otl)

• Make it easy to find out what you are looking

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMEssbase Stats Table

• Essbase has a lot of different stats and information that we can extract from it• Each one has it own amount of columns and metrics

• If we create one table for each kind of structure it will be very trick to maintain and harder to create any generic code

• The best way to extract information is to have just one table where we have the properties in the rows instead of columns• This way we have just one structure for all kind of information, no matter the amount of

columns that returns

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMJava Oracle Connection

• Connecting to an Oracle DB is very simple• We just need the DB URL,

Username and Password

• Since we have a generic table we can have just one prepare statement and use it to insert all kind of Stats

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMJava Essbase Connection

• Connecting to Essbase is also easy the only additional information is that you also need to pass the Provider Service• It may be “Embedded” or by Provider

Service

• Basically we do a sign in to the provider service and then select the olap server that we want to use

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMJava Essbase Get DB Statistics

1. First we get all the cubes from all the apps

2. For each cube we get the its properties which is an array in a Key and a Value format

3. This information is added to the prepare Statement and executed (inserted) into the DB

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMJava Essbase Get File Information

1. In Java we can issue any Maxl command using the “IEssMaxlSession”class

2. The result set contains Columns and rows, similar with what we get in EAS

• We need to Loop through the rows and get the columns that we need

3. This information is added to the prepare Statement and executed (inserted) into the DB

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMJava Essbase Get Outline Information

1. We can get member information using IEssMemberSelection with a custom query or find the member directly in the outline using find member

2. The results contains a set of members that we may loop and analyze its properties• In this case we categorized the results by

storage type

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMJava in ODI

• ODI runs Java code using Java BeanShell Technology• Tip: Create your code in a Java

IDE, test it and copy the “Main” into ODI Procedure

• Other advantage to use ODI is that we can get the connection information from topology and reuse everywhere within the package• Define the connection in the

command on source

• Get the information in the commando on target

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMEssbase DW

• After we gather all the information we needed inside a generic stage table, it is time for us to split, organize and load this information in their historical DW tables

• For this presentation we have 3 different kind of information:• DB Statistics

• File Statistics

• Outline Statistics

• Since their structure are very different we will generate one table for each statistics

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMEssbase DW

• Since we create a generic table to hold all extracted information in rows now we need to PIVOT the data into columns and load it into the historical tables

1. Define the columns to be Pivoted

2. Use an consolidation function on the data column

1. SUM, AVG, MIN, MAX, COUNT…

2. Specify the data to me Pivoted

3. The data MUST be a constant in the “IN” Clouse

3. Data is Pivoted

4. Data is loaded to DW table using Control Append

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMEssbase DW

• With all 3 tables loaded with historical data we can now calculate all the caches we need

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMCalculations

Calc Metric Calculation Result

Index Max Index (TOTAL_BLOCKS * 112) /1024 8.206.282 Kb

Data File Max Data DATA_FILE_TOTAL 3.825.872 Kb

Data Max Data (DATA_FILE_TOTAL * 0.125) 478.234 Kb

Calculator

BITMAP Ceils(exp(sum(ln(MEMBERS))))/838.250.576 b

Anchor Levels + 28

Result BITMAP * Anchor / 1024 298.832 Kb

Dynamic

Calculator

CalcLock((Data Cache * 1024) /

(CEIL(exp(sum(ln(TOTAL-SHARED-LABEL))))*8))/2

598

Max Block Max((CEIL(exp(sum(ln(TOTAL-SHARED-LABEL))))*8)) 409.488 b

Max User Static number = 50 50

Results (CalcLock * Max Block * Max User) /1024 / 1024 11.677 Mb

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMGlue all together in ODI

• To Glue all together in ODI is simple• We may have all the java codes in

steps

• Our 3 pivot queries in 3 insert steps

• Our 5 caches in 5 steps as well creating the analyzes

• And finally a email step to send the information

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMAnalyzes

• We could automatically set the caches using maxl/esscmd commands but this could cause a few problems:• The apps and in some case the Essbase server

needs to be restarted

• Even these settings being what Oracle recommends, each cube has its own characteristics then is too dangerous to automatically set them

• This information greatly helps the Admins to maintain the apps in a more systematic way

Essbase

DW

Stage Area

Analysis

ODI

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPMConclusion

• Automate a task that is hard to keep vigilance (Pro active admin)

• Allow us to track cube growing (preventive hardware actions)

• Identify business seasonality (plan settings ahead based in the past)

• Can be extended for sessions and Essbase statistical tables for deeper analyzes

• Great repository to find abnormal situations

DevEpm.com

@RZGiampaoli

@RodrigoRadtke

@DEVEPM

Ricardo Giampaoli – TeraCorp

Rodrigo Radtke de Souza - Dell

Thank you!

Essbase Statistics DW: How to Automatically Administrate Essbase Using ODI

Data & Analytics

Transcript of Essbase Statistics DW: How to Automatically Administrate Essbase Using ODI