Christo kutrovsky oracle rac solving common scalability problems

Oracle RAC – Solving Common

Scalability ProblemsPresented by: Christo Kutrovsky

© 2015 Pythian - Presentation for Oracle OpenWorld

Who Am I

2

• Oracle ACE

• 15 years in Oracle field

• Joined Pythian 2003

• Part of Pythian Consulting Group

• Special projects

• Performance tuning

• Critical services

• Presenter at: IOUG, RMOUG, UKOUG, OpenWorld

“oracle pinup”

© 2015 Pythian - Presentation for Oracle OpenWorld3 © 2011 Pythian - Confidential

Why Companies Trust Pythian• Recognized Leader:

• Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and SQL Server

• Work with over 150 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments

• Expertise:

• One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 7 Oracle ACEs/ACE Directors.

• Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC.

• Global Reach & Scalability:

• 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response

http://www.upenn.edu/

http://www.upenn.edu/

http://www.mdsinc.com/

http://www.mdsinc.com/


Questions for Audience

5

• How many have RAC today?

• How many have RAC for HA?

• How many have RAC for scalability?

• How many plan to move to RAC?

• How many develop on RAC?

6

Scale UP vs Scale OUT


Scale UP vs Scale OUT

7

• Can have many meanings

• Scale UP

• Get a bigger machine

• Scale OUT

• Get more machines


Database Performance

8

• Has 2 primary components

• CPU – compute capacity

• IO – data transfer capacity

• Disk

• Network

• Cache (RAM)

• Implementation specifics

• Database scalability – latches, mutexes, threading etc.


Database Performance

9

Disk

Server

CPU

RAM RAM

APP


Database Performance – Scale UP

10

Disk

Server

CPU

RAM RAM

APP

CPU

RAM RAM


Database – Scale OUT

11

Disk

Server

CPU

RAM RAM

APP

Server

CPU

RAM RAM

Server

CPU

RAM RAM


Scale UP – challenges 2011

12

• Scale UP

• Limited “max size”

• Appears unified, but is not, NUMA

• Slower CPUs

• 12.8-GB/sec per QPI

• Up to 88 Gb/sec of network (NEM)

• 8 PCIe slots

Sources: http://www.oracle.com/technetwork/articles/systems-hardware-architecture/sun-server-x2-8-architecture-1689083.pdf

http://www.oracle.com/technetwork/articles/systems-hardware-architecture/sun-server-x2-8-architecture-1689083.pdf


CPU Comparison 2011

13

X3-2• 8 cores per chip

• 2 chips max

• Server with 16 cores:

702 (SPECint®_rate2006)

• 35% capacity of x3-8

• 43.8 per core

• 1.8 x faster


• 8 chips max

• server with 80 cores:


• 2.8 X Capacity

• 24.8 per core

• 56% speed of x3-2

Sources:

http://www.spec.org/cpu2006/results/res2012q4/cpu2006-20121104-24909.html





CPU Comparison 2015

14


• 2 chips max



• 30% Capacity of X4-8

• 38.8 per core

• 45 MB CPU cache

• 1U


• 8 chips max



• 3.2 X capacity of x5-2

• 38.4 per core

• 40 MB CPU cache

• 4USources:

http://spec.org/cpu2006/results/res2014q4/cpu2006-20141007-32157.html




Scale OUT

16

• Requires Synchronisation Software – Oracle RAC

• Requires “interconnect” network

• Low latency critical

• Requires additional CPU resources for cross-node synchronisation

• Some operations (ex. locking) significantly increased latency (network vs. memory)


Adding CPU hardware

17

• Both Scale UP and Scale OUT increase total CPU capacity

• and IO network bandwidth

• Does not improve IO latency

• Does not improve IO requests capacity

• Does not improve application response time

• Does not make your batch job run faster


Adding CPU hardware

18

• Adds more workers, but you must keep them busy – at the same time

• Execute more tasks in parallel

• Parallelise existing tasks


Adding CPU hardware

19

• Unless more RAM is added

• Indirectly affects IO requests capacity

• Indirectly affects IO latency

20

Oracle RDBMS

Scale OUT with RAC


Why Scale OUT Oracle

21

• Natural High Availability

• Infinite scalability – realistically

• Smaller “block” of failure


Challenges with Oracle RAC

22

• Concurrent modifications latency increased drastically

• Requires block transfer (CPU + network)

• Requires partial disk flush (CPU + latency)

• Concurrent reads do not share RAM cache

• Each node has local copy

• Oracle RAC Cache Fusion Reads slow

• timings similar to SSD read speeds – 10,000x slower than memory reads

23

Practical Issues

Concurrent inserts


Data Block

24

• Oracle stores data in blocks, in rows

• Each segment (table) has independent blocks*

• Lowest granularity in RAC data exchange is block*

• Modifying any data requires the block to be ‘acquired’ in local SGA (cache)

Oracle Table Block

8 KiB (8192)

1, Christo Kutrovsky, Canada, 99$

2, Alex Gorbachev, Canada, 98$

3, Marc Fielding, Canada, 101$


Inserting Data

25

• Inserting a row is modifying the block

Oracle Table Block

8 KiB (8192)




4, Vanessa Simons, Canada, 199$


Block needs to move

26

• Block will bounce from node to node depending on where the insert runs

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

CPU

RAM RAM

APP

4, Vanessa Simons, Canada, 199$5, John Smith, Canada, 777$ 6, Scott Tiger, Canada, 333$


Inserting Data

27

• Well that’s not efficient, can’t each node insert into it’s own block?


Inserting Data

28

• Yes we can! But…

• ASSM

• Instances can “own” L1 space metadata, which maps to N blocks

• Larger segments

• To have multiple L1 space metadata, larger segments are required


ASSM – bitmapped space mgmt

29

ASSM

L3

L2

L2

L1

1

L1

2

L1

3

L1

4

L1

5

L1

6

L1

7

L1

8

BLOCKS

1 2 3 4

5 6 7 8

BLOCKS9 10 11 12

13 14 15 16


ASSM

30

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

CPU

RAM RAM

APP

4, Vanessa Simons, Canada, 199$ 5, John Smith, Canada, 777$ 6, Scott Tiger, Canada, 333$

L1

1

BLOCKS

L1

2

BLOCKS

L1

3

BLOCKS


Primary Key

31

• Primary Key – an index

• Each row has a very specific place

• Must insert into last block, always

• Sequentially generated

• … or is it?

Oracle Table Block

8 KiB (8192)





5, John Smith, Canada, 777$

6, Scott Tiger, Canada, 333$

7, ……………………..


Sequences

32

• Sequences can be

• Cached or NOcached

• Ordered or NOordered

• Cached NOordered with large cache are your best friend

• A sequence’s cache “sticks” with a RAC node

• cache 10,000 – 10,000 sequential values assigned to node 1


NO ORDER

33

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

CPU

RAM RAM

APP

?, Vanessa Simons, Canada, 199$ ?, John Smith, Canada, 777$ ?, Scott Tiger, Canada, 333$

10,001

10,002

20,001

20,002

30,001

30,002


Sequences

34

• Cached noordered 10,000 cache – almost perfect

• Oracle numbers “compress” zeroes

• 10,000 and 20,000 are next to each other

• Sequence dictionary updates moves blocks

• Order not maintained

• But what if it’s required?

• Is it required?

• When do you consider the row “inserted”?

• What if delayed due to lock/concurrency?


Index on date inserted

35

• Concurrently inserted records have same date or very close

• Specific ordering in blocks

• No flexibility on values

• Same for index on Country or any other “same value” column

Oracle Table Block

8 KiB (8192)





5, John Smith, Canada, 777$

6, Scott Tiger, Canada, 333$

7, ……………………..


Dates

36

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

CPU

RAM RAM

APP

?, Vanessa Simons, Canada, 199$ ?, John Smith, Canada, 777$ ?, Scott Tiger, Canada, 333$

11:09:01 11:09:01 11:09:01


Same Value Index

37

• Requires Partitioning

• Changes everything

• Partition key selection is non-trivial

• Local Indexes

• Lots of combinations

• Primary key local or global

• Depends on partition key and type

• Lots of considerations

• Data locality


Partition 4

BLOCKS

1 2 3 4

5 6 7 8

IDX

Date

IDX

PK

Partitioning

38

Partition 1

BLOCKS

1 2 3 4

5 6 7 8

IDX

Date

IDX

PK

Partition 2

BLOCKS

1 2 3 4

5 6 7 8

IDX

Date

IDX

PK

Partition 3

BLOCKS

1 2 3 4

5 6 7 8

IDX

Date

IDX

PK


Same Value Index

39

• Multiple partitions

• Create multiple blocks that have the “current” date

• “current” block will still move around the cluster, but decreases chances of “collisions”

• Still far from ideal


Same Value Index - Other

40

• Alternative solutions

• Prefix same value index with customer ID

• Prefix same value index with INSTANCE_ID

• “fake” local index

• Relies on skip scan

41

Application Partitioning


APPAPPApps

42

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

CPU

RAM RAM

APP

11:09:01 11:09:01 11:09:01


Apps

43

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

APP

Server

APP

Server

APP


Application partitioning

44

What if I told you it doesn’t have to be like that?


Application partitioning

45

• By Functionality

• Not all functions of the application require concurrent modifications

• Isolating by functionality is the easiest and fastest way to ‘fix’ RAC unfriendly activities

• Such functionality has inherently bad scalability properties, as number of cores increase, this will become bigger problem

• Cache fusion will mirror concurrently read data

• Multiple nodes do not ‘add’ to caching benefits


Apps

46

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

APP

Server

APP

Server

APP

Module(s)

With

Concurrent

Modifications


Application Partitioning

47

What if a single module exceeds the performance capacity of a single node?


More Performance

48

• Tune it

• What are you doing?

• Can you not do it?

• Can you do it more efficiently?

• Are you using the latest hardware?

• Should probably start with this

• Still need more?

© 2015 Pythian - Presentation for Oracle OpenWorld49

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

APP

Server

APP

Server

APP

Server

APP

Server

APP


Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

APP

Server

APP

Server

APP

Server

APP

Server

APP

Server

APP


Data Partitioning

51

• By Data

• Each application server processes a subset of the data

• By Customer ID

• By Region – data balancing challenges


Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

CPU

RAM RAM

Server

APP

Server

APP

Server

APP

Server

APP

Server

APP

Server

APP

Load Balancer


Data Partitioning

53

• Partition the Database to match

• Consider growth – have significantly more buckets than servers

• Local indexes everywhere

• Partition key and Primary Key generation can be challenging

• If partitioned by primary key, key generation from one node, must not send data to multiple partitions

• Secondary unique indexes still challenging


Data Partitioning – Primary Keys

56

• Unique Keys

• Primary Keys usually part of partition key

• Two level partitioning (range for historical, hash for concurrency) – global partitioned indexes


Sharded Data – why use RAC?

57

• Most apps will “just work”

• Few components require extra attention

• Not all data is sharded

• Reference / Lookup Data is shared

• Low concurrency data does not need to be sharded

• Redundancy – can ‘fail back’ to any node

• Caching (performance) implications


Sharded Data – why use RAC?

58

• Some data cannot be sharded one way

• Many to Many

• Buyer/Seller with portals for both customers and stores

• Manual sharding mistakes – loose data

• Non shardable queries? Code manually?

• Sharded queries and sorting

• Schema Changes/Indexing


Why use RAC?

59

• Run reports across all data


Reports Across all Data

60

• No replication

• Cross shard consistency – even after data restore

• No data movement

• Reuse cache from each node

• Parallel queries can leverage all nodes

• MPP on OLTP data

62

Final Thoughts


Final Thoughts

63

• RAC Works

• With a little help

• RAC Scales

• Your CPU

• Sometimes your RAM

• Not your disk

• RAC can be

• share everything

• share nothing

© 2009/2010 Pythian - Presentation for ABC Company64

The End

Thank You

Questions?

I blog at

http://www.pythian.com/news/author/kutrovsky/

Christo kutrovsky oracle rac solving common scalability problems

Software

Transcript of Christo kutrovsky oracle rac solving common scalability problems