Jonah H. HarrismyYearbook.com

Session #399

The Life of a Query(Oracle Edition)

Speaker Qualifications

• Currently the Sr. DBA at myYearbook.com

• Oracle DBA/developer starting with Oracle V7

• Database Internals Software Architect for 3 years

• Hobby is Researching Oracle Internals

• Speaker for IOUG, VOUG, NYOUG, SEOUC

• Technical Reviewer for IOUG SELECT & several O'Reilly books on SQL

• Blog about Oracle technology– http://www.oracle-internals.com/

• Developed patches for open-source databases: PostgreSQL, InnoDB, SAP DB, Firebird

Disclaimer

• This is my hobby• I’ve never been an Oracle insider• The material in this presentation has been

based on years of researching Oracle internals as well as analyzing network traffic and trace files.

• Do your own research!• Use at your own risk!

What’s in this for me?

• Get a better idea of what's happening behind-the-scenes.

• Use this information to help troubleshoot issues

• Use this information to help optimize queries or the system itself.

Oracle Architecture

Oracle Architecture

• It has layers!

• Each layer depends on the layer beneath it

• Developed primarily in the C programming language with platform-specific optimizations in assembly language

• Controls all facets of query execution, utility processes, etc.

Oracle Kernel Layers

Connection

• Client requests a connection to a serverSQL*Plus (OCI/UPI->NL [Network Layer])

• Network Naming (NN layer) resolves the network address/port to connect to (TNSNAMES, etc.) and determines which Network Transport (NT layer) adapter to use (TCP/IP, SDB/IP, ...)

• Oracle Client software configures itself to perform communication using the Transparent Network Substrate (NS layer) protocol on the selected transport.– The transport itself uses operating system dependent (OSD

layer) code to handle portability differences between OSes like Windows, Linux, and various UNIX flavors.

• Client connects to the listener

Listener Communication

• The Listener accepts the connection request and validates that the service requested by the client is available and that the client has access to it (via host-based access control)

• The Listener tells the client to reconnect to a different port or resend the request.

• The Listener creates another Oracle process/thread and passes the connection information to it.

Handshaking

• The new process/thread accepts the client request for service and begins to perform handshaking– Client and Server negotiate Additional Network Options

(ANO) such as authentication, encryption, data integrity, etc.

– Client and server negotiate a common version of the protocol to use; if none are compatible, you'll generally see things like ORA-03134: Connections to this server version are no longer supported

Authentication

• Client sends User Name, Terminal Name, Machine Name, Program Name, ...

• Server responds with challenge/response– Server sends client an encrypted key via

AUTH_SESSKEY– Client decrypts AUTH_SESSKEY using the user's

password, and then uses the secret to encrypt and hash the user's password and sends the result back to the server as AUTH_PASSWORD

– Server confirms valid/invalid password and proceeds accordingly.

Session Creation

• After successful authentication, finish session creation and perform relevant actions– Add session to global internal structures– Execute AFTER LOGON triggers

• Once session creation is complete, the server process waits for input from the client.

Client Sends a Query

• User issues a queryUPDATE mytable SET foo = 'baz' WHERE foo = 'bar';

• Oracle Client packages up the query text into a specific Two-Task Interface (TTI) subpacket (OALL*) with relevant flags set

• Oracle Client encapsulates the subpacket into a TNS data packet

• Client sends packet to the server.

Server Receives the Query

• Server receives TNS data packet

• Server checks to see if data contains a subpacket

• Server parses the subpacket

• Server performs the Oracle Programmatic Interface (OPI) function actions based on the subpacket– Parse

– Execute

– Fetch

– (about 170+ others)

Parse the Query

• Oracle performs a hash based on the query text and performs a lookup in the Library Cache

• If Oracle has already parsed the query and has a plan for it, re-use the plan (generally)

• If no plan is found, perform a "hard" parse– Tokenize the query– Parse the query using a recursive descent parser

• Started by Bob Miner and still contains a little of his code

– Build a parse tree representation of the statement• Includes objects referenced in the query

– Perform some type checking and data type coercion

• Does the query make sense syntactically?• Does the query make sense semantically?

Optimize the Query

• Goal is to determine how to best execute the query

• Query optimization is mathematical– Discrete Math (Set Theory, Logic, Graph Theory, ...)

– Application of Relational Algebra

– Plan costs are based on weighted graph calculations

– Best plan is defined as having the lowest cost calculation

• Optimization Process is– Query Rewrite

– Cost-based Query Optimization

– Execution Plan Generation

Rewrite the Query

• Transform to canonical form• Perform some basic optimizations

– Constant folding– Transitive Closure

• Determine whether the query (in whole or in part) can be optimized to use Materialized Views

Determine Access Paths

• Are there any indexes for columns specified in the predicate?

• If there are multiple indexes, build a plan for each

• If we are using an index, can we use that to optimize our join algorithm (hash, nested loop, external, ...)

Perform Cost-Based Optimization

• The goal is to determine– Which access methods to use (Index, FTS, etc.)– Optimal Join Order

• Uses a branch-and-bound algorithm to build plans by:– Performing join permutation (Join Ordering)– Determining which joins are not needed (Join Elimination)– Applying partitioning rules (Partition Elimination/Pruning)– Attempting to push down selection in joins (to reduce the

number of rows at each step)– Determining which indexes are available for use in the plan

and whether that changes the join algorithm (hash, nested loop, ...)

– Calculate the I/O and CPU costs for each step of the plan• Choose plan with the best overall graph weight.

Join Permutation

• Build graphs for joins in different orders– Join R S = (R, S) = (S, R)– Join R S T = ((R, S), T) = (R, (S, T))

• Join is a binary Relational Algebra operation• Follows mathematical rules for associativity

and commutativity

Join Elimination

• Generally follows the process– If a table is joined but none of its attributes (fields) are

projected (in the select-list), consider it for join removal

– If the table being joined is redundant due to PK/FK integrity constraints, is can be considered for removal.

– If the table being joined is redundant due to unique constraints, it can be considered for removal.

• If all conditions are met, it can be removed from the query plan

Partition Elimination

• Based on the concept of constraint exclusion• Drops (prunes) all partitions which could not

possibly be used in our plan– If you have range-based partitions for Q1, Q2, Q3,

and Q4 and query data BETWEEN '01-JAN-09' AND '15-JAN-09', the optimizer knows that the data could not exist in partitions Q2, Q3, and Q4 by excluding them based on their constraint (Q2>='2009-03-31' and Q2 < '2009-07-01', ...)

Selection Pushdown

• To try reduce the cost of a join, see if the number of rows in R or S can be reduced prior to the join by pushing down parts of the predicate to those individual tables prior to the join.– One area where this is beneficial is in nested loop

joins, whose basic cost is determined by cardinality(R) * cardinality(S)

– Another area where this is beneficial is in hash joins, where the server can reduce the cardinality of one of the tables significantly. This results in less CPU and memory required to build the hash table.

Build an Execution Plan

• Based on the best query plan, transform the plan into an execution plan for executing the query.

Execute the Plan

• For each node in the execution plan, perform the respective operation.– Table Scan

– Index Scan

– Join

– Qualification

– ...

Execute the Plan (Index Node)

• Find all rows in the index on mytable.foo where foo = 'bar'.

• Open the index

• Perform a search on the B*Tree

• Once the index entry is found, locate the data in the heap (table data) using the ROWID.

Perform B*Tree Search

• FUNCTION btree_search (x, k) i := 1 WHILE i <= n[x] AND k > key

i[x]

DO i := i + 1

IF i <= n[x] AND k = keyi[x] THEN

RETURN (x, i); IF leaf[x] THEN RETURN NIL ELSE kcbget(c

i[x]);

RETURN btree_search(ci[x], k);

END IF;

Retrieving a Block (kcbg*())

• Check the Buffer Cache• If the block isn't in the buffer cache, read it

from disk (sfrfb()-System File Read File Block)

• If the block is in the buffer cache, and no one has altered it without committing, use it.

• If the block is in the buffer cache, and someone has altered it without committing, build a before image of the block from UNDO and use it (kcbgtcr()-Kernel Cache Buffer GeT for Consistent Read).

Updating the Data

• Once the data has been found, update it

• Acquire a row-level lock by placing an entry in the interested transaction list (ITL). If the row is already locked, wait (or don't wait depending on what the user requested)

• The server generates an UNDO/REDO record containing the change vector for the record (ktugur()-Kernel Transaction Undo Generate Undo and Redo)– UNDO contains the column foo with value bar

– REDO contains the column foo with value baz

• The Oracle server returns a packet to the client regarding success/failure of the statement.

Committing the Data

• The client sends a commit message to the server, which the server processes as before. Because it's a command, it does not have to go through query planning.

• Flush the REDO/UNDO data to disk up to the point of the commit

• Increment the SCN• Fast Commit and Delayed Block Cleanout

– In Fast Commit mode, Oracle does not clean the ITL it used on the last transaction as a part of commit.

• The next request to read the block will check to see whether the transactions in the ITL are still in progress, if not, there's no reason to get a consistent read version of the block.

• If the next request is DML, it will itself perform ITL cleanup for the old transaction.

Fetching the Data

• The client sends the server a fetch request for N number of rows from a the cursor.

• The server marshalls the data to be sent over the network

• The server sends as many packets as are necessary to contain the data

• The client reads the data, unmarshalls it, performs any necessary encoding changes, and returns it to the application.

Questions?

Thank You

• Fill out evaluation– The Life of a Query– Session #399

• Further Information– jonah.harris@gmail.com– http://www.oracle-internals.com/