Scaling Data

Copyright © 2003, SAS Institute Inc. All rights reserved.SAS is a registered trademark or trademark of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Other brand and product names are registered trademarks or Trademarks of their respective companies

Scaling SAS® Data Access toOracle® RDBMS

Howard PlemmonsSAS Institute Inc.Andrew HoldsworthOracle Corporation

Scaling

What is Scaling?

Scaling

“To remove the scales of a fish”

“To climb up by means of a scaling ladder”

“To reach the highest point”

Scaling Data

Why Scale to Data

Scaling Data

SAS tools, SAS/ACCESS®

SAS Procedure and Processes

Oracle tools

Oracle Procedures and Processes

Intelligence Value Chain

Intelligence Value Chain Silver into Gold

SAS System 9

SAS V8 vs. SAS System 9

FEATURE SAS V8 SAS System 9

Libname Engine x x

Procedure Interface x x

Fast Load x x

Threaded Interface x

SAS V8 I/O Model

Threaded Interface SAS 9

SAS Procedures proc sort

proc summary

proc dmine

proc reg; proc dmreg

proc means

proc loess; proc dmdb

proc glm

proc robustreg

SAS/ACCESS® Engines

ORACLE

Informix

Sybase

Teradata

Libname and SAS Procedure Controls

dbslice (“where”,”where”,…)

dbsliceparm (ALL,…)

defaults (THREADED_APPS,2)

options sastrace=‘,,t’;

procedure controls – CPU count

Options In Action - DBSLICEPARM

-dbsliceparm none

option dbsliceparm=

libname x oracle user=scott pass=tiger

dbsliceparm=(threaded_apps,2);

proc print data=y.oratab (dbsliceparm=(all,4)); run;

Options In Action - DBSLICE

libname x oracle user=scott pass=tiger;

proc print data=x.oratab (dbslice= (“where x<100”, “where x >= 100”) );

Options In Action – CPUCOUNT, THREADS

CPUCOUNT=

THREADS | NOTHREADS

Process

Libname controls

Procedure controls

Execution

Linear Scalability

Achieved Speedup

Scalability – SAS 9 Threaded speedup in PROC REG

Run on 12-way Unix Box

Scalability – SAS 9 Threaded speedup in PROC SORT

Run on 8-way Unix BoxTests run in memory cache

What Does This Mean - access

393000 Rows

No Threads - baseline

Two Threads (DBSLICE) – 31%

Six Threads (DBSLICEPARM) – 54%

Run on 10-way Unix BoxTests run in memory cache

Scaling Data

Data Volumes

Data ACCESS

Data Organization

Scaling using Oracle - Andrew

Scaling with

The Star Query

Use of Parallelism

Use of the Direct Path

Use of Specialist Indexes

Use of Analytical Functions

Use of Materialized Views

Use of The Oracle9i Optimizer

The Star Query

Product

Geography

Customer

Star Queries The star query is a very common DW

technique. It is highly optimized in Oracle and can be tuned depending on the type of queries. In summary the more known about the query composition the higher level of optimization possible.

Star Query Optimization

The Optimization is 3 step Process1.Apply query predicates to dimension tables to generate

lists of foreign keys into the fact table.

2.Query the fact table using series of single column bit mapped indexes on the foreign keys

3.Having resolved the query within the fact table complete the query by joining back to dimension tables where needed and roll the query up.

Star Queries

– To enable star queries the DBA should do the following1. Build single column bitmapped indexes on each

foreign key in the fact table

2. Build indexes on the dimension tables for query predicates

3. Build indexes on the dimension tables to assist in the join back and roll up process

4. Generate statistics for the schema

5. Set the parameter STAR_TRANSFORMATION_ENABLED=TRUE

Use of Parallelism

Multiple CPUs to execute a single query as well multiple concurrent queries

Execute Table scans, Index probes and scans in parallel

Execute Joins and Sorts in parallel

Execute DML in parallel

Parallelism can be configured manually or automatically

Use of Partitioning

Partitioning was originally designed to allow management of large db objects however by partitioning data performance gains can be made by the following• Partition pruning

• Join optimizations

Partitioning can be done by the following methods• Range e.g. Data or key ranges

• List e.g. Discrete values such as State

• Hash to achieve equal size partitions

Two types of partitioning can be applied

Use of The Direct Path

By pass the conventional transaction layer to insert and copy data within the database

SQL*Loader is user currently by SAS

Other options include• Insert with /*+ append */ hint

• Create Table as Select with NOLOGGING

These constructs can be used to transform vast amounts of data rapidly in parallel

Specialist Indexes

B-Tree Indexes

Bit Mapped Indexes including join indexes

Functional Indexes

Analytical Functions

Oracle has embraced the ANSI OLAP extensions to SQL

These permit faster response times on queries that would require multiple passes of the data with conventional SQL

This allows grouped results and functionality such as moving averages

Materialized Views

Materialized view allow automatic use of summary tables without a user having to re-write the query

Well designed materialized views are small in size and can increase performance by orders of magnitude.

Materialized views are in fact Oracle tables and can use all other features to improve performance

Oracle9i Optimizer

On upgrade of Oracle Releases the Optimizer behavior will change

The Optimizer is tested with over 400,000 SQL Statements

• Where plans change between releases the actual query is ran to test for degradation

• Slower plans are corrected

It is still important to have good representative Statistics

DBMS_STATS package allows parallel generation and migration of schema statistics

Oracle9i Optimizer

Some common Optimizer problems seen with Oracle9i

• Bad or incomplete statistics

• Init.ora parameters influencing optimizer

• SQL written for RBO

Summary

Oracle and SAS provide techniques for scaling to larger databases by optimizing both query performance and fetch performance.

These techniques are simple to adopt and allow huge productivity improvements

We have identified some core technologies here however this is a partial picture of the SAS/Oracle ability.

About the Speakers

Howard Plemmons Andrew HoldsworthSenior Software Manager Director

SAS Institute Inc. Oracle Corp.

SAS Circle 500 Oracle Pkwy,

Cary, NC Redwood Shores, CA94065

Phone:

919-531-7779 650-506-2938

E-mail:

Howard.Plemmons@sas.com Andrew.Holdsworth@oracle.com

Other SUGI Papers/Presentations

•PC File Data Objects Directly from UNIX – 8:00am Tuesday

•SAS/ACCESS and use of Metadata – Rm 619 @ 2:30

•Lessons in Scalability – SAS Presents – 3:20 Tuesday

•Data Warehousing section - performance

Scaling SAS Data ACCESS to ORACLE RDBMS

Scaling Data

Documents

Transcript of Scaling Data

Scaling Cryptographic Techniques by Exploiting Data ...

Scaling HBase for Big Data

Data collection, data processing and scaling

Visualizing Data Using Multidimensional Scaling

Scaling GIS Data in Non-relational Data Stores

Scaling MySQL: Benefits of Automatic Data Distribution

Scaling Face Recognition with Big Data

Scaling Data overview

Scaling up data mining algorithms: review and taxonomy · with this problem: scaling up data mining algorithms and data reduction. However, because data reduction is a data mining

Urban Mobility Scaling: Lessons from 'Little Data'

Measuring and scaling of quantitative data khalid

Auto-scaling Techniques for Elastic Data Stream Processing · Auto-scaling Techniques for Elastic Data Stream Processing Thomas Heinze, ... Auto-scaling Techniques for Elastic Data

Scaling up data science applications

3 Multidimensional Scaling Techniques - University … Multivariate Data Analysis: Chapter 3:– Multidimensional Scaling Techniques 3 Multidimensional Scaling Techniques 3.0 Introduction

Scaling Data in Magento

Scaling Data Science at Airbnb

Scaling Big Data Platform for Big Data Pipeline

Scaling Git with Bitbucket Data Center

Scaling Sensors with Data Synthesis

Scaling up Linked Data