Kumar and Dan

Go to Kscope

How ASO Works

(Based on Your Questions)

Seattle, WA Jun 25, 2014

Steve

And Hybrid

Please envision the standard safe

harbor statement from Oracle here

to protect Steve Job

• A Quick Review of ASO

• A Bit About Memory Management

• Speculation About How Hybrid

Works

• Your Questions

Today’s Session

A Quick Review of ASO

A Bit About Memory Management

Speculation About How Hybrid Works

Your Questions

Today’s Session

Really? You’ve looked inside the computer

and seen an ASO cube?

Who Has Seen an ASO Cube?

Still made and used for field identification of minerals.

Similar guides are available for Trees and Birds.

An ASO Cube You Can Hold in Your Hand

Holes represent metadata presence; notches represent absence.

Note notches for all levels (67510, Abbyville, Kansas and Central).

Query for Check and Adult = ((Check) AND Adult)

Card Used With Sorting-Needle

Cards with holes are pulled up and used for next part of query; cards with notches fall out. Check and adult = ((Check) AND (Adult))

Data Is Queried Using a “Sorting-Needle”

Patented Multi-Processor Sorting-Needle

“You can never be too rich or too thin.”

Wallis Simpson

Disk Speed, CPU Speed and Ram

Both Unix and Windows cache files in memory

Separate (and somewhat redundant) vs.

familiar Data and Index caches

During data load, both input file and output pag,

ind or dat files are cached

Called memory mapped IO

Seen in Windows Resource Manager or on

UNIX in top or nmon:

Memory Management

Memory - Resource Manager

Green is program memory including explicit caches

Dark blue is memory mapped IO

Light blue is memory not being used

top

Memory – Unix: top or nmon

nmon with m (memory) option

Memory – Just for Fun Load = Physical Drive 125 eff threads

Load = SSD Drive 125 eff threads

Memory & CPU – Just for Fun

CPU % = 8,069.7 Cached = 200GB

To see which files are being cached, look at the File

Summary tab of RamMap:

Memory - RamMap

RamMap is part of Microsoft’s SysInternals, and is available for free at:

http://technet.microsoft.com/en-us/sysinternals

… take longer to run sometimes?

Very often, it’s memory-mapped IO

● A copied input file may still be in RAM

● After a calc/query, the dat/pag file is still in RAM

Can be seen in Resource Manager and RamMap

when a file is unzipped and loaded into a cube:

Why Does My Query/Load/Calc…

After Unzip and before load:

Memory – BSO – Unzip and Load

Both zipped and unzipped files remain in

memory

● Unzipped not needed, but remains in memory

About ¼ way through load:

Memory – BSO – Unzip and Load

The .pag files remain in memory even after

they’re written

● Great, for augmentation or querying later

● Might be in data cache if there’s room

About ½ way through load:

Memory – BSO – Unzip and Load

Still room for all .pag files created so far

and both zipped and unzipped input files

About ¾ way through load:

Memory – BSO – Unzip and Load

No more free space

● Unzipped file no longer wholly in memory

● Previous slides showed it at 1,013,624 K

● We didn’t need it again

After load:

Memory – BSO – Unzip and Load

None of unzipped file in memory

Part of ess00001.pag has been forced out

of memory

Running Calc All now will be faster than

after application restart

● Almost all .pag files are in memory

Unix: echo 3 > /proc/sys/vm/drop_caches

Windows: EmptyStandbyList.exe

● From RamMap directory, or

● Interactively, using RamMap:

Memory – How to Clean out Mapped IO

Before: After:

Load ran faster because unzipped was in

memory: 192 seconds

Memory – Clear and Load

Proved by clearing memory mapped IO

rerunning load: 233 seconds 21% longer

41 seconds may not seem long

● SSD’s used – physical would be slower

RamMap at same points in run:

Memory – Clear and Load

RamMap at same points in run (continued):

Memory – Clear and Load

Query Analyzed

Subset of Blocks are Read

Data is Aggregated on-the-fly to a View that will

satisfy the query

Resulting .dat file is written to ASO Cache

and/or disk

Query is processed

.dat file logically discarded

Speculation About How Hybrid Works

ASO Cache size is more critical

● “Pure” ASO query performance not terribly sensitive

to ASO cache size

● Hybrid uses ASO cache and temp tablespace to

create a temporary dat file for every query

● Physical size on disk in hybrid\temp of ess0001.dat

represents the “High-Water” mark of ASO cache size

● Use “High-Water” mark for decisions on cache size

● As always, use the fastest disk (SSD) you have for

the temp tablespace

Speculation About How Hybrid Works

Finally, Your Questions

Seattle, WA Jun 25, 2014

How much of Dan’s

Speculations about Hybrid

were just plain B(&*#@^ ?

Tweeted At: 10 am June 25th 2010

From Twitter

Tell me more

● Can I turn it off?

● Can I change where it goes?

● Can I create the different “new” members on-the-fly?

Renegade Members

How do my queries actually get processed

I want to learn more about how much various

design decisions cost me

Explain Plan

Which Scales better Hybrid or ASO?

I want them to “survive” restructures when

possible

Can I edit them logically instead of by those

weird numbers?

I want help editing my Agg Scripts

Today

Next Release (PS4? – what is a PS4 anyway?)

When does Hybrid make Sense vs ASO

When do I do witch (sic)

ForceCalc vs Fix Parallel

When will I be able to change them without

restarting Essbase?

ASO config settings

When will 64bit ODBC Interface support Data

Export Batch Insert Mode?

What does this question mean?

Why should I care?

64bit ODBC

Such as:

● Actuals updated thru mm/dd/yyy

● Why did you submit that stupid query?

● Have you actually used this data you “had to have”

last week?

I want to send my user’s messages

Restructure Warnings – Long vs Short and

maybe time to complete?

Progress msg or estimate of how much longer

query will take?

Estimates and Warnings

Will I still be able to buy it in the future or only

as part of fill-in-the blank

Where Forth will Essbase Go?