Steven Feuerstein - Programming With Collections

Copyright 2000-2009 Steven Feuerstein -

Collect Yourself:

Optimize PL/SQL Code with

Collections

Steven FeuersteinPL/SQL Evangelist, Quest Software

[email protected]

www.ToadWorld.com/SF

PL/SQL Webinars for Oracle Education


How to benefit most from this session

Watch, listen, focus on concepts and principles.

Download and use any of my the training materials:

You have my permission to use all these materials to do internal trainings and build your own applications.– But remember: they are not production ready.

– Modify them to fit your needs and then test them!

filename_from_demo_zip.sql

Download and use any of my scripts (examples, performance scripts, reusable code) from the same location: the demo.zip file.

http://www.ToadWorld.com/SFPL/SQL Obsession


What we will cover on collections

Review of "foundation" features

Indexing collections by strings

Working with collections of collections

MULTISET operators for nested tables

Then we will apply collections (overview):

– Data caching

– Bulk processing with FORALL and BULK COLLECT

– Table functions and pipelined functions


What is a collection?

A collection is an "ordered group of elements, all of

the same type." (PL/SQL User Guide and Reference)

– That's a very general definition; from collections, you can

build queues, stacks, lists, sets, arrays.

– Collections are single-dimensional and homogeneous, but

you can emulate multi-dimensional structures.

Collections are a critical feature in many of the newest

and most important features of PL/SQL.

– Yet they are greatly underutilized by PL/SQL developers.

abc def sf q rrr swq...1 2 3 4 22 23


Why use collections?

Generally, to manipulate in-program-memory lists of

information.

– Much faster than working through SQL.

Serve up complex datasets of information to non-

PL/SQL host environments using table functions.

Dramatically improve multi-row querying, inserting,

updating and deleting the contents of tables. Combined with BULK COLLECT and FORALL....

Emulate bi-directional, random access cursors.

Avoid mutating table trigger errors.


Three Types of Collections

Associative arrays (aka index-by tables)

– Can be used only in PL/SQL blocks.

– Similar to hash tables in other languages, allows you to

access elements via arbitrary subscript values.

Nested tables and Varrays

– Can be used in PL/SQL blocks, but also can be the

datatype of a column in a relational table.

– Part of the object model in PL/SQL.

– Required for some features, such as table functions

– With Varrays, you specify a maximum number of elements

in the collection, at time of definition.


About Associative Arrays

Unbounded, practically speaking.

– Valid row numbers range from -2,147,483,647 to

2,147,483,647.

– This range allows you to employ the row number as an

intelligent key, such as the primary key or unique index

value, because AAs also are:

Sparse

– Data does not have to be stored in consecutive rows, as is

required in traditional 3GL arrays and VARRAYs.

Index values can be integers or strings (Oracle9i R2

and above).assoc_array_example.sql

collection_of_records.sql


About Nested Tables

No pre-defined limit on a nested table.

– Valid row numbers range from 1 to

2,147,483,647.

Part of object model, requiring initialization.

Is always dense initially, but can become

sparse after deletes.

Can be defined as a schema level type and

used as a relational table column type.

nested_table_example.sql


About Varrays

Has a maximum size, associated with its type.

– Can adjust the size at runtime in Oracle10g R2.

Part of object model, requiring initialization.

Is always dense; you can only remove

elements from the end of a varray.

Can be defined as a schema level type and

used as a relational table column type.

varray_example.sql


How to choose your collection type

Use associative arrays when you need to...– Work within PL/SQL code only

– Sparsely fill and manipulate the collection

– Take advantage of negative index values and string indexing

Use nested tables when you need to...– Access the collection inside SQL (table functions, columns in

tables)

– Want or need to perform high level set operations

Use varrays when you need to...– If you need to specify a maximum size to your collection

– Access the collection inside SQL (table functions, columns in tables).


Handy Collection Methods

Obtain information about the collection– COUNT returns number of rows currently defined in

collection.

– EXISTS returns TRUE if the specified row is defined.

– FIRST/LAST return lowest/highest numbers of defined rows.

– NEXT/PRIOR return the closest defined row after/before the specified row.

– LIMIT tells you the max. number of elements allowed in a VARRAY.

Modify the contents of the collection– DELETE deletes one or more rows from collection.

– EXTEND adds rows to a nested table or VARRAY.

– TRIM removes rows from a VARRAY.


Useful reminders for PL/SQL collections

Memory for collections comes out of the PGA (Process Global Area) or UGA (User Global Area)– One per session, so a program using collections can

consume a large amount of memory.

Use the NOCOPY hint to reduce overhead of passing collections in and out of program units.

Encapsulate or hide details of collection management.

Don't always fill collections sequentially. Think about how you need to manipulate the contents.

Try to read a row that doesn't exist, and Oracle raises NO_DATA_FOUND.

nocopy*.*


PL/SQL in Shared Memory

Session 1 memory

(PGA/UGA)

emp_rec emp%rowtype;

tot_tab tottabtype;

Session 2 memory

(PGA/UGA)

emp_rec emp%rowtype;

tot_tab tottabtype;

System Global Area (SGA) of RDBMS Instance

Shared Pool

Large Pool

Reserved Pool

show_empscalc_totals upd_salaries

Select *

from emp

Shared SQL

Pre-parsedUpdate emp

Set sal=...

Library cache

Session 1Session 2

plsql_memory*.*


How PL/SQL uses the SGA, PGA and UGA

The SGA contains information that can be shared across schemas connected to the instance.

– From the PL/SQL perspective, this is limited to package static constants.

The User Global Area contains session-specific data that persists across server call boundaries

– Package-level data

The Process Global Area contains session-specific data that is released when the current server call terminates.

– Local dataplsql_memory.pkg

plsql_memory_demo.sql

PACKAGE Pkg isNonstatic_Constant CONSTANT PLS_INTEGER := My_Sequence.Nextval;Static_Constant CONSTANT PLS_INTEGER := 42;

END Pkg;


Expanded indexing capabilities for associative arrays

Prior to Oracle9iR2, you could only index by BINARY_INTEGER.

You can now define the index on your associative array to be:– Any sub-type derived from BINARY_INTEGER

– VARCHAR2(n), where n is between 1 and 32767

– %TYPE against a database column that is consistent with the above rules

– A SUBTYPE against any of the above.

This means that you can now index on string values! (and concatenated indexes and...)


Examples of New TYPE Variants

All of the following are now valid TYPE declarations in Oracle9i Release 2– You cannot use %TYPE against an INTEGER column,

because INTEGER is not a subtype of BINARY_INTEGER.

DECLARE TYPE array_t1 IS TABLE OF NUMBER INDEX BY BINARY_INTEGER; TYPE array_t2 IS TABLE OF NUMBER INDEX BY PLS_INTEGER;TYPE array_t3 IS TABLE OF NUMBER INDEX BY POSITIVE;TYPE array_t4 IS TABLE OF NUMBER INDEX BY NATURAL;TYPE array_t5 IS TABLE OF NUMBER INDEX BY VARCHAR2(64);TYPE array_t6 IS TABLE OF NUMBER INDEX BY VARCHAR2(32767);TYPE array_t7 IS TABLE OF NUMBER INDEX BY

employee.last_name%TYPE;TYPE array_t8 IS TABLE OF NUMBER INDEX BY

types_pkg.subtype_t;


Working with string-indexed collections

The syntax is exactly the same, but you should keep this in mind:

– The datatype returned by FIRST, LAST, NEXT and PRIOR methods is VARCHAR2.

– The longer the string values, the more time it takes Oracle to "hash" that string to the integer that is actually used as the index value.

If you are indexing by integer and find that your values are getting close to the limits (2**31 - 1 or -2**31 + 1), convert to a string index. assoc_array*.sql

assoc_array_perf.tst

int_to_string_indexing.sql

genaa.sql


Practical application for string indexing

I need to keep track of names used in my program.

– Specifically, in Quest Code Tester, we generate test code

and declare variables. So I need to make sure that I do not

declare the same variable more than once.

There are lots of ways to do this, but string-indexed

collections make it really easy!

FOR indx IN 1 .. l_variables.COUNTLOOP

If varname_already_usedTHEN

-- DO NOTHINGELSE

add_variable_declaration;mark_varname_as_used;

END IF;END LOOP; Without string indexing:

string_tracker0.*


The String Tracker package (V1)

First iteration: I only need to maintain one list of

names.

CREATE OR REPLACE PACKAGE BODY string_trackerIS

TYPE used_aat IS TABLE OF BOOLEAN INDEX BY maxvarchar2_t;g_names_used used_aat;

FUNCTION string_in_use ( value_in IN maxvarchar2_t ) RETURN BOOLEANIS BEGIN

RETURN g_names_used.EXISTS ( value_in );END string_in_use;

PROCEDURE mark_as_used (value_in IN maxvarchar2_t) ISBEGIN

g_names_used ( value_in ) := TRUE;END mark_as_used;

END string_tracker;

string_tracker1.*


Multi-level Collections

Prior to Oracle9i, you could have collections of records or objects, but only if all fields were scalars.

– A collection containing another collection was not allowed.

Now you can create collections that contain other collections and complex types.

– Applies to all three types of collections.

The syntax is non-intuitive and resulting code can be quite complex.

Oracle9i


String Tracker Version 2

The problem with String Tracker V1 is that it

only supports a single list of strings.

– What if I need to track multiple lists

simultaneously or nested?

Let's extend the first version to support

multiple lists by using a string-indexed, multi-

level collection.

– A list of lists....


The String Tracker package (V2)

CREATE OR REPLACE PACKAGE BODY string_trackerIS

TYPE used_aat IS TABLE OF BOOLEAN INDEX BY maxvarchar2_t;TYPE list_of_lists_aat IS TABLE OF used_aat INDEX BY maxvarchar2_t;g_list_of_lists list_of_lists_aat;

PROCEDURE mark_as_used (list_in IN maxvarchar2_t

, value_in IN maxvarchar2_t, case_sensitive_in IN BOOLEAN DEFAULT FALSE) IS

l_name maxvarchar2_t := CASE case_sensitive_in WHEN TRUE THEN value_in

ELSE UPPER ( value_in ) END;BEGIN

g_list_of_lists ( list_in ) ( l_name) := TRUE;END mark_as_used;

END string_tracker;

string_tracker3.*


Other multi-level collection examples

Multi-level collections with intermediate records and objects.

Emulation of multi-dimensional arrays

– No native support, but can creates nested collections to get much the same effect.

– Use the UTL_NLA package (10gR2) for complex matrix manipulation.

Four-level nested collection used to track arguments for a program unit.

– Automatically analyze ambiguous overloading.

multdim*.*

ambig_overloading.sql

OTN: OverloadCheck

multilevel_collections.sql


Encapsulate these complex structures!

When working with multi-level collections, you can easily and rapidly arrive at completely unreadable and un-maintainable code.

What' s a developer to do?

– Hide complexity -- and all data structures -- behind small modules.

– Work with and through functions to retrieve contents and procedures to set contents.

cc_smartargs.pkb:

cc_smartargs.next_overloading

cc_smartargs.add_new_parameter


Nested Tables unveil their MULTISET-edness

Oracle10g introduces high-level set operations on nested tables (only).

– Nested tables are “multisets,” meaning that there is no inherent order to their elements and duplicates are significant.

You can now…

– Check for equality and inequality

– Perform UNION, INTERSECT and MINUS operations

– Check for and remove duplicates

Works with nested tables of scalars, records, objects.

Oracle10g


Check for equality and inequality

Just use the basic operators….and NULLs have the usual disruptive impact.

Oracle10g

DECLARETYPE clientele IS TABLE OF VARCHAR2 (64);group1 clientele := clientele ('Customer 1', 'Customer 2');group2 clientele := clientele ('Customer 1', 'Customer 3');group3 clientele := clientele ('Customer 3', 'Customer 1');

BEGINIF group1 = group2 THEN

DBMS_OUTPUT.put_line ('Group 1 = Group 2');ELSE

DBMS_OUTPUT.put_line ('Group 1 != Group 2');END IF;

IF group2 != group3 THENDBMS_OUTPUT.put_line ('Group 2 != Group 3');

ELSEDBMS_OUTPUT.put_line ('Group 2 = Group 3');

END IF;END;

10g_compare.sql

10g_compare2.sql

10g_compare_old.sql


UNION, INTERSECT, MINUS

Straightforward, with the MULTISET keyword.

Oracle10g

BEGINour_favorites :=

my_favorites MULTISET UNION dad_favorites;

our_favorites := my_favorites MULTISET UNION DISTINCT dad_favorites;

our_favorites := my_favorites MULTISET INTERSECT dad_favorites;

our_favorites := dad_favorites MULTISET EXCEPT my_favorites;

END;

10g_setops.sql

10g_string_nt.sql

10g_favorites.sql

10g*union*.sql

SQL: UNION

SQL: UNION ALL

SQL: INTERSECT

SQL: MINUS


Distinct sets of values

Use the SET operator to work with distinct values, and determine if you have a set of distinct values.

Oracle10g

DECLAREkeep_it_simple strings_nt := strings_nt ();

BEGINkeep_it_simple := SET (favorites_pkg.my_favorites);

favorites_pkg.show_favorites ('FULL SET', favorites_pkg.my_favorites);

p.l (favorites_pkg.my_favorites IS A SET, 'My favorites distinct?');p.l (favorites_pkg.my_favorites IS NOT A SET, 'My favorites NOT distinct?');

favorites_pkg.show_favorites ('DISTINCT SET', keep_it_simple);

p.l (keep_it_simple IS A SET, 'Keep_it_simple distinct?');p.l (keep_it_simple IS NOT A SET, 'Keep_it_simple NOT distinct?');

END;

10g_set.sql

10g_favorites.pkg


Collections vs. Global Temporary Tables

Global temporary tables cut down on the

overhead of working with persistent tables.

– And you can use the full power of SQL, which is

their main advantage over collections.

GTTs still require interaction with the SGA.

So collections will still be faster, but they will

use more memory.

– GTTs consume SGA memory.

global_temp_tab_vs_coll.sql

Applying Collections

Data caching using packaged data

Turbo-charged SQL with BULK COLLECT

and FORALL

Table functions

I offer light coverage of these topics, simply

to ensure that you know what is possible.


Data Caching Options

Why cache data?

– Because it is static and therefore you want to avoid the performance overhead of retrieving that data over and over again.

Options for caching data:

– The SGA: Oracle does lots of caching for us, but it is not always the most efficient means.

– Package data structures: PGA memory has less access overhead than SGA.

– Oracle11g Function Result Cache

– Deterministic functions

Packaged collection caching

Prior to Oracle 11g, the best caching option for

PL/SQL programs involves declaring a package-level

data structure.

– It persists for the entire session.

– Usually a collection, to store multiple rows of data.

Why query information from the database (SGA) if

that data does not change during your session?

– Trivial example: the USER function

– More interesting: static tables

Instead, load it up in a package variable!Very simple example:

thisuser.*


FunctionPGA

Data Caching with PL/SQL Tables

First access

Subsequent accesses

PGAFunction

Database

Not in cache;

Request data

from database

Pass Data

to Cache

Application

Application

Requests Data

Data retrieved

from cache Data returned

to application

Application

Application

Requests Data

Data returned

to application

Data retrieved

from cache

Database

Data found in

cache. Database

is not needed.

emplu.pkg

emplu.tst

11g_emplu*.*


Turbo-charge SQL with bulk processing statements

Improve the performance of multi-row SQL operations by an order of magnitude or more with bulk/array processing in PL/SQL!

CREATE OR REPLACE PROCEDURE upd_for_dept (dept_in IN employee.department_id%TYPE

,newsal_in IN employee.salary%TYPE)IS

CURSOR emp_cur ISSELECT employee_id,salary,hire_dateFROM employee WHERE department_id = dept_in;

BEGINFOR rec IN emp_cur LOOP

adjust_compensation (rec, newsal_in);

UPDATE employee SET salary = rec.salaryWHERE employee_id = rec.employee_id;

END LOOP;END upd_for_dept;

Row by row processing:

elegant but inefficient


Use the FORALL Bulk Bind Statement

Instead of executing repetitive, individual DML statements, you can write your code like this:

Things to be aware of with FORALL:– You MUST know how to use collections to use this feature!

– Only a single DML statement is allowed per FORALL.

– New cursor attributes: SQL%BULK_ROWCOUNT returns number of rows affected by each row in array. SQL%BULK_EXCEPTIONS...

– Prior to Oracle10g, the binding array must be sequentially filled.

– Use SAVE EXCEPTIONS to continue past errors.

PROCEDURE upd_for_dept (...) ISBEGIN

FORALL indx IN list_of_emps.FIRST .. list_of_emps.LASTUPDATE employee

SET salary = newsal_inWHERE employee_id = list_of_emps (indx);

END;

bulktiming.sql

bulk_rowcount.sql

bulkexc.sql


Use BULK COLLECT INTO for Queries

DECLARETYPE employees_aat IS TABLE OF employees%ROWTYPE

INDEX BY BINARY_INTEGER;

l_employees employees_aat;BEGIN

SELECT *BULK COLLECT INTO l_employeesFROM employees;

FOR indx IN 1 .. l_employees.COUNT LOOP

process_employee (l_employees(indx));END LOOP;

END;

bulkcoll.sql

Declare a

collection of

records to hold

the queried data.

Fetch all rows into

collection

sequentially, starting

with 1.

Iterate through the

collection

contents with a

loop. WARNING! BULK COLLECT will not raise

NO_DATA_FOUND if no rows are found.

Always check contents of collection to confirm that

something was retrieved.


Limit the number of rows returned by BULK COLLECT

CREATE OR REPLACE PROCEDURE bulk_with_limit (deptno_in IN dept.deptno%TYPE)

ISCURSOR emps_in_dept_cur IS

SELECT *FROM empWHERE deptno = deptno_in;

TYPE emp_tt IS TABLE OF emps_in_dept_cur%ROWTYPE;emps emp_tt;

BEGIN OPEN emps_in_dept_cur;LOOP

FETCH emps_in_dept_cur BULK COLLECT INTO emps LIMIT 100;

EXIT WHEN emps.COUNT = 0;

process_emps (emps);END LOOP;

END bulk_with_limit;

Use the LIMIT clause with the

INTO to manage the amount

of memory used with the

BULK COLLECT operation.

WARNING!

BULK COLLECT will not raise

NO_DATA_FOUND if no rows

are found.

Best to check contents of

collection to confirm that

something was retrieved.bulklimit.sql


The Wonder Of Table Functions

A table function is a function that you can call in the FROM clause of a query, and have it be treated as if it were a relational table.

Table functions allow you to perform arbitrarily complex transformations of data and then make that data available through a query.– Not everything can be done in SQL.

Combined with REF CURSORs, you can now more easily transfer data from within PL/SQL to host environments.– Java, for example, works very smoothly with cursor

variables


Simple table function example

Return a list of names as a nested table, and

then call that function in the FROM clause.

CREATE OR REPLACE FUNCTION lotsa_names (base_name_in IN VARCHAR2, count_in IN INTEGER

)RETURN names_nt

ISretval names_nt := names_nt ();

BEGINretval.EXTEND (count_in);

FOR indx IN 1 .. count_inLOOP

retval (indx) := base_name_in || ' ' || indx;

END LOOP;

RETURN retval;END lotsa_names;

tabfunc_scalar.sql

SELECT column_valueFROM TABLE (

lotsa_names ('Steven', 100)) names;

COLUMN_VALUE ------------Steven 1 ... Steven 100


Streaming data with table functions

You can use table functions to "stream" data

through several stages within a single SQL

statement.

– Example: transform one row in the stocktable to two

rows in the tickertable. CREATE TABLE stocktable (ticker VARCHAR2(20),trade_date DATE,open_price NUMBER,close_price NUMBER

)/CREATE TABLE tickertable (

ticker VARCHAR2(20),pricedate DATE,pricetype VARCHAR2(1),price NUMBER)

/

tabfunc_streaming.sql


Streaming data with table functions - 2

In this example, transform each row of the

stocktable into two rows in the tickertable.

CREATE OR REPLACE PACKAGE refcur_pkgIS

TYPE refcur_t IS REF CURSORRETURN stocktable%ROWTYPE;

END refcur_pkg;/

CREATE OR REPLACE FUNCTION stockpivot (dataset refcur_pkg.refcur_t)RETURN tickertypeset ...

BEGININSERT INTO tickertable

SELECT *FROM TABLE (stockpivot (CURSOR (SELECT *

FROM stocktable)));END;/

tabfunc_streaming.sql


Use pipelined functions to enhance performance.

Pipelined functions allow you to return data

iteratively, asynchronous to termination of the

function.

– As data is produced within the function, it is

passed back to the calling process/query.

Pipelined functions can be defined to support

parallel execution.

– Iterative data processing allows multiple

processes to work on that data simultaneously.

CREATE FUNCTION StockPivot (p refcur_pkg.refcur_t) RETURN TickerTypeSet PIPELINED


Applications for pipelined functions

Execution functions in parallel.

– In Oracle9i Database Release 2 and above, use the PARALLEL_ENABLE clause to allow your pipelined function to participate fully in a parallelized query.

– Critical in data warehouse applications.

Improve speed of delivery of data to web pages.

– Use a pipelined function to "serve up" data to the webpage and allow users to being viewing and browsing, even before the function has finished retrieving all of the data.


Piping rows out from a pipelined function

CREATE FUNCTION stockpivot (p refcur_pkg.refcur_t)RETURN tickertypesetPIPELINED

ISout_rec tickertype :=

tickertype (NULL, NULL, NULL);in_rec p%ROWTYPE;

BEGINLOOP

FETCH p INTO in_rec;EXIT WHEN p%NOTFOUND;out_rec.ticker := in_rec.ticker;out_rec.pricetype := 'O';out_rec.price := in_rec.openprice;

PIPE ROW (out_rec);END LOOP;CLOSE p;

RETURN;END;

tabfunc_setup.sql

tabfunc_pipelined.sql

Add PIPELINED

keyword to header

Pipe a row of data

back to calling block

or query

RETURN...nothing at

all!


Table functions – Summary

Table functions offer significant new flexibility

for PL/SQL developers.

Consider using them when you...

– Need to pass back complex result sets of data

through the SQL layer (a query);

– Want to call a user defined function inside a

query and execute it as part of a parallel query.


Collections – don't start coding without them.

It is impossible to write efficient, high quality

PL/SQL code, taking full advantage of new

features, unless you use collections.

– From array processing to table functions, collections are

required.

Today I offer this challenge: learn collections

thoroughly and apply them throughout your

backend code.

– Your code will get faster and in many cases much

simpler than it might have been (though not always!).

Some Useful URLs...

E-Attendance link http://education.oracle.com/eattendance.html

(Class id : 2347771)

Course Evaluation link https://eval.oracle.com

Instructor Feedback link :

https://ougbsapex.us.oracle.com/pls/ougbsapex/f?p=120:17


http://education.oracle.com/eattendance.html

https://eval.oracle.com/

https://ougbsapex.us.oracle.com/pls/ougbsapex/f?p=120:17

Steven Feuerstein - Programming With Collections

Documents

Transcript of Steven Feuerstein - Programming With Collections