KC Server Development Nieuwe SQL en PL/SQL features in Oracle 10gR2 en R1 incl. reprise van ODTUG...

1KC Server Development – 15 november 2005 Oracle 10g’s Finest

KC Server Development

Nieuwe SQL en PL/SQL features in Oracle 10gR2 en R1

incl. reprise van ODTUG presentatie

15 november 2005


Agenda

10gR2 – Een paar hoogtepuntjes uit release 2 van Oracle 10g (juli 2005)

10gR1 – Een bonte verzameling features en functies uit 10gR2 (juni 2004)

De ODTUG 2005 presentatie: Oracle 10g’s Finest—The Top 3 SQL and PL/SQL

Features New in 10g Diner Workshop

AMIS LAB database (10gR2)


10gR2 Aandachttrekkers

Grenzen verkennen met DBMS_OUTPUT DML met Error Log – Constraint Violations

allowed SQL*Plus Autotrace op basis van DBMS_XPLAN Conditional PL/SQL Compilation 10x snellere transacties - Asynchronous Commit Small fry


Beyond DBMS_OUTPUTWat is het resultaat van dit code fragment?

A – Compilation Error B – Runtime Error C – 10 regels output D – 4 regels output en dan ORA-20000: ORU-10028 E – 10 regels output en dan ORA-20000: ORU-10028


Beyond DBMS_OUTPUTWat is het resultaat van dit code fragment?

Pre 10gR2: 10gR2:


Beyond DBMS_OUTPUT10gR2 Improvement for DBMS_OUTPUT

Output limit from 255 to 32767 SET SERVEROUTPUT ON SIZE UNLIMITED


Will the update statement always succeed?

create table emp ( empno number, ename varchar2(10), constraint emp_pk primary key (empno)); insert into emp values(1,'sam'); insert into emp values(2,'joe');

update emp set empno=empno+1;


Statement level Constraint CheckingWill this statement succeed?

add constraint sal_check check (sal < 6000)/

alter table empadd constraint sal_check check (sal < 6000)/

update empset sal = sal + 2000/


Oracle 10gR2 – DML with Error Log

begin dbms_errlog.create_error_log ('EMP' ,'ERROR_LOG_EMP‘ ) ;end;/

update empset sal = sal + 2000LOG ERRORS INTO ERROR_LOG_EMP ('salary raise')

REJECT LIMIT 1/


DML with Error Log - Errors logged for each failed record in the statement

Rowid of the recordthat failed the Updateor Delete Insert?

Columns mirroring thetable columns Hold the values that

would have been ifthe DML operation hadnot failed

Oracle Error numer andMessage

User defined tag


SQL*Plus Autotrace with DBMS_XPLAN


Conditional PL/SQL Compilation

create or replace procedure pasbegin $IF $$debug_code $THEN dbms_output.put_line( 'Our debug code' ); dbms_output.put_line( 'Would go here' ); $END dbms_output.put_line( 'And our real code here' );end;

alter procedure P compile plsql_ccflags = 'debug_code:true' reuse settings;



alter procedure P compile plsql_ccflags = 'debug_code:true' reuse settings;


Conditional PL/SQL Compilation - Use $error directive to ‘throw’ compilation errors

...$if $$plsql_optimize_level !=1 $then $error 'This program must be compiled at optimization_level 1' $end$end...



You can leave your - really slow - debug code in your application now—and turn it on and off at will.

You can program assertions as you might in C or Java. Each subprogram can test the values of its inputs, for example, and verify that

they meet some criteria. These tests can be active during the whole development cycle and inactive for production.

However, they remain as formal documentation of the conditions upon which the unit depends, and can be simply reactivated for debugging a production-time bug.

You can write database version-independent code Program one package with code sections for version X and version Y; during

compilation, depending on the ‘condition’, one of the two is compiled Check out the new DBMS_DB_VERSION supplied package.

The same applies to multi-locale, multi-customer, multi-platform One set of packages, conditional code sections per locale/customer/…

You can support best practices during unit testing. For production, helper subprograms will be declared in a package body. For calls

from a unit-testing framework, they are conditionally declared in the specification.



Use Compile Time variables to insert static strings in the PL/SQL code $$PLSQL_UNIT $$PLSQL_LINE

Use DBMS_DB_VERSION package to retrieve information about the compile time environment RELEASE , VER_LE_9_2 , VER_LE_10_1 , VER_LE_10_2

Instead of Session Level plsql_ccflags settings, you can also refer to Package variables (constants!)

Use package DBMS_PREPROCESSOR to retrieve the source as it is active after compilation

View USER_PLSQL_OBJECT_SETTINGS to retrieve the values of the flags at compilation time


What takes longer?

A – 300 small Commits

B – One Large Commit


What happens during Commit?

Log writer process flushes the contents of the REDO LOG buffers to the disk To each of the specified Redo Log files

Not until the O/S signals the successful completion of this write process (for each of the Redo Log files) is the commit considered complete Only then will no data have been lost upon immediate

database crash The Client has to wait for this write process to

complete!


Asynchronous Commit

With the statement COMMIT WRITE BATCH NOWAIT ;

You can instruct Oracle to NOT wait for the Redo Log writing before returning control to the Client

Consequences: Transaction ‘completes’ much faster You stand the risk of losing transactions

• Eventually, the database will crash and some committed transactions will have been lost!

Typical Usage Frequent, small, non-crucial transactions such as real-

time measurements, stocks tickers, visit-statistics etc.


Asynchronous CommitExample


10gR2 Small Fry

Transparent Data Encryption The data is stored encrypted on-disk, so that even if someone

steals your database, the information is protected .Net support

Stored Procedures in .Net languages C# and VB.NET XML DB: XSLT 2.0, XQuery, … Data Pump direct compress CLOB and BLOB over database links PL/SQL Data Mining Restore Points

easier Flashback to ‘logically labeled SCN’ Unicode 4.0 – 1226 new characters UTL_NA for Matrix calculations Rules Manager


10g Features and Functions

Data Pump “Insertable” External Table Merge improvements Partition Outer Join Regular Expressions (regexp) Flashback in overdrive

Database Recycle Bin


New Integrated Import/Export architecture New utilities expdp, impdp Uses External Table Data Pump Unload

High speed direct-path load/unload Serial speedup up to 10x for import and 2x for export

Complete server-site utility infrastructure Automatically scales using parallel execution Restart capabilities after stop or failure Job runtime and size estimations Progress monitoring Public documented APIs

Data Pump


External Table unload

Unload data with a simple CTAS Using pretty much any query

Combine multiple files from many source systems into one big “Create” to facilitate merging

Subsetting made easy Supports parallel unload and reload Does not support

BFILE (easy workaround using dbms_lob.filegetname in a query) LONG / LONG RAW (easy workaround using TO_LOB) ‘final object types’ – workable solutions


MERGE Optimizations

MERGE can be used as INSERT-only respective UPDATE-only operation

More efficient plans No outer join necessary (UPDATE-only) Can use ANTI-JOIN (INSERT-only)

Conditional INSERT and UPDATE branches Full flexibility for individual data filtering

Optional DELETE clause for UPDATE branch Implicit Data Maintenance


Example of Merge in 10g

Conditional Update, Delete and Insert Note: either update or insert can be omitted

MERGE INTO Products P – Destination tableUSING Product_Changes S – Source table ON (P.PROD_ID = S.PROD_ID) -- Search/Join condition WHEN MATCHED THEN UPDATE -- update if join SET P.PROD_LIST_PRICE = S.PROD_NEW_PRICE WHERE P.PROD_STATUS <> "OBSOLETE" -- Conditional UPDATEDELETE WHERE (P.PROD_STATUS = "OBSOLETE") WHEN NOT MATCHED THEN INSERT -- insert if not join (prod_id, prod_status,prod_new_price) values (...) WHERE S.PROD_STATUS <> "OBSOLETE" -- Conditional INSERT


Join Options

Cross Join (cartesian result set) Every record from A combined with every record in B

(Inner) Join Only records from A that match on join key with

records from B and vice versa Right Outer Join

All records from B with matching records from A where available and “empty” A records otherwise

Partition By Right Outer Join All records from B with – per partition in A - matching

records where available and “empty” A records otherwise


Example Joining – Voornamen en Achternamen

CROSS JOIN

INNER JOIN

RIGHT OUTER JOIN

PARTITION BY (geslacht)RIGHT OUTER JOIN

24 records

4 records

5 records

6 records


Partitioned Outer Join

New outer join syntax enabling easy specification and high performance for joins that "densify" sparse data.

To specify comparison calculations and to format reports reliably, best to return a consistent set of dimension members in query results

Yet data normally stored in "sparse" form: why waste space storing non-occurrence?

Ugly and slow SQL needed to add back rows for nonexistent cases into query output.

Most frequently used to replace missing values along time dimension.

Proposed to ANSI for SQL standard.


For each department, get the number of employees hired per year

select d.dname department, year, count(empno) "number of hires"from emp e right outer join ( select distinct extract(year from hiredate) year from emp ) years on extract(year from e.hiredate) = year right outer join dept d on (e.deptno = d.deptno)groupby year, d.dname


Partition Outer Join – For each department, get the number of employees hired per year

select d.dname department, year, count(empno) "number of hires"from emp e

partition by (deptno) right outer join ( select distinct extract(year from hiredate) year from emp ) years on extract(year from e.hiredate) = year right outer join dept d on (e.deptno = d.deptno)groupby year, d.dname


Partition Outer Join – For each department, get the number of employees hired per year

select d.dname department, year, count(empno) "number of hires"from emp e partition by (deptno) right outer join ( select distinct extract(year from hiredate) year from emp ) years on extract(year from e.hiredate) = year right outer join dept d on (e.deptno = d.deptno)groupby year, d.dname


POSIX compliant regular expression functionality with additional multilingual support similar to like, instr, substr, replace

Support for VARCHARs and CLOBs Four functions – SQL and PL/SQL:

regexp_like, regexp_instr, regexp_substr and regexp_replace

Regular Expressions

SELECT first_name, last_nameFROM employeesWHERE REGEXP_LIKE (first_name, ’^Ste(v|ph)en$’);

FIRST_NAME LAST_NAME-------------------- -------------------------Steven KingSteven MarkleStephen Stiles


Flashback

Flashback Query 9iR1 – primitive Had to open flashback cursors before doing any DML It worked, but was not “easy”


Flashback

Flashback Query 9iR2 – sophisticated No need to open cursors before doing modifications Can flashback in SQL, no packages needed Can flashback in modifications It worked and was much easier

• Could join the table with itself as of N minutes ago• Put update a set of rows, put them back as they were N

minutes ago• In SQL


Flashback

In 10g Flashback Query Flashback Table Flashback Row History Flashback Drop Flashback Database


Flashback – Flashback Row History

Instead of “show me the data as of”, you can say “show me all versions of the data between”

Select ename, sal from emp versions between timestamp a and b where ename = ‘SCOTT’

ENAME SAL---------- ----------SCOTT 3000SCOTT 3300SCOTT 3630…


Flashback – Flashback Row History

See related information about each row SCN range the row was “valid for” Time range (approx) the row was valid for The transaction id (XID) that modified the row The operation (I/U/D) that was performed on the row

select ename, sal, versions_operation, versions_starttime, versions_endtime, versions_startscn, versions_endscn, versions_xid from emp versions between timestamp &A and &B where empno = 7788 order by versions_startscn nulls first


New Connect By Features

CONNECT_BY_ROOT returns for the indicated column the value of the parent (prior) record

Function CONNECT_BY_ISLEAF returns 0 for non-leaf nodes and 1 for leaf-nodes

Functions CONNECT_BY_ISCYCLE and NO_CYCLE help prevent ‘end-less loops’

SELECT ename "Employee", CONNECT_BY_ROOT ename "Manager"FROM empCONNECT BY PRIOR empno = mgrstart with mgr is null

Employee Manager---------- ----------JONES KINGSCOTT KING


Other 10g Stuff

Alternative Quoting Mechanism Automatic execution of Bulk Collect when a For

Loop has been programmed Get the full Error Call-stack for the origin of an

exception: dbms_utility.format_error_backtrace Column Level VPD

Only enforce policy when user attempts to read specific columns

Fine Grained Audit on DML events


Oracle 10g’s Finest—The Top 3 SQL and PL/SQL Features New in 10g

Lucas JellemaAMIS, The Netherlands


Agenda

Oracle 10g Release 1 for SQL and PL/SQL What I will not discuss

SQL Model Clause The spreadsheet in your query

Expression Filters The query in your tables

DBMS_Frequent_Itemset Data Mining through PL/SQL


10g Features I will not discuss(but could also have picked)

Partition Outer Join Hierarchical Improvements (hear Tom Kyte, yesterday) Case Insensitive Query Regular Expressions (see Alan Wintersteen, session 10) Bulk DML Improvements – (FORALL) DBMS_SCHEDULER Alternative Quoting Returning Aggregate UTL_MAIL Data Pump Flashback Functionality (see David Anderson, right now) Merge Enhancements PL/SQL Compiler enhancements


Apology

For making some slight slide changes. No slight was intended, just better slides.


Oracle 10g SQL Model Clause

The SQL Model clause allows users to embed spreadsheet-like models in a SELECT statement in a way that was previously the domain of dedicated

multidimensional OLAP servers such as Oracle Express and Oracle9i OLAP or Spreadsheets such as Excel

Often, these models involve a series of macros that aggregate data over a number of business dimensions, over varying time periods, and following a set of complex business rules

The aim of the SQL Model clause is to give normal SQL statements the ability to create a multidimensional array from the results of a normal SELECT statement carry out any number of interdependent inter-row and

inter-array calculations on this array


The MODEL Clause – like processing the result set in a spreadsheet

select ename, deptno, sal, model_measure dept_totalfrom empwhere sal > 1000model ...order by deptno, ename

select <expression>from <tables><views><in line views>joinwhere

Model processing:•Update ‘cells’

• Through calculations across cells• Get value from preceding or following row• Multi-row aggregation (count, max, min, sum, avg)

• Create new cells (columns or measures)• Create new records

order by

(10g) Model

Result Set to Client

Outer Queryin case of

In-Line View

{


What does the MODEL clause do?

MODEL allows Any Column Value in the result set to be updated Columns to be added

• The MODEL clause may return more columns than the underlying tables contain

Rows to be added Choice between return all rows or only the new and

updated rows Values of cells to be derived using expressions that

• Reference existing cell-values• May contain functions and aggregations such as SUM, MAX,

MIN, AVG• May use queries to retrieve values from outside the result set


Model vs. Analytical Functions

MODEL 10g Calculate Columns using

inter-row calculations (direct cell-reference, max, min, sum, count, etc.)

Update cells Insert rows Suppress untouched rows Use reference models

Analytical Functions 8.1.6 EE, 9iR2 SE Calculate Columns using

inter-row calculations (lag, lead, first, last, rank, sum, max, min, count, avg,…)

No update of values No creation of rows No suppression of rows - More efficient

(performance) Easier syntax


Example Model ClauseAdd an additional derived column total_sal

select ename, deptno, sal, comm, total_salfrom empmodel main salary_modeldimension by (ename)measures (sal, comm, 0 as total_sal, deptno ) ignore navrules upsert ( total_sal[ANY]= sal[CV(ename)] + comm[CV(ename)])

dimensions

measures

Cells correspond with measuresCells referenced through dimensionsOnly measures can be inserted or updated


Example Model ClauseResult of adding column TOTAL_SAL

( total_sal[ANY]= sal[CV(ename)] + comm[CV(ename)])


Example Model ClauseCalculate (and exclusively return) Grand Total

select ename, total_salfrom empmodel return updated rows main salary_model dimension by ( ename ) measures (sal, comm, 0 as total_sal ) ignore nav rules upsert ( total_sal['GRAND']= sum(sal)[ANY] + sum(comm)[ANY] )


What happened with this last model?

total_sal['GRAND'] = sum(sal)[ANY] + sum(comm)[ANY] select ename

, total_salfrom empmodel return updated rows


Example Model ClauseIntroducing Partitioning

select deptno, ename, total_salfrom empmodel return updated rows main salary_model partition by (deptno) dimension by ( ename ) measures (sal, nvl(comm,0) comm , 0 as total_sal ) rules upsert ( total_sal['GRAND']= sum(sal)[ANY] + sum(comm)[ANY] )


Example Model Clause

SELECT SUBSTR(country,1,20) country, SUBSTR(prod,1,15) prod, year, salesFROM sales_viewWHERE country IN ('Italy','Japan')MODEL RETURN UPDATED ROWSPARTITION BY (country) –- for every country DIMENSION BY (prod, year) –- cells are referenced with theseMEASURES (sale sales) -- the value to be derivedRULES (sales['Bounce', 2002] = sales['Bounce', 2001] +sales['Bounce', 2000],sales['Y Box', 2002] = sales['Y Box', 2001],sales['2_Products', 2002] = sales['Bounce', 2002] + sales['Y Box', 2002])ORDER BY country, prod, year;


Example Model Clause

COUNTRY PROD YEAR SALES-------------------- --------------- ---------- ----------Italy 2_Products 2002 90387.54Italy Bounce 2002 9179.99Italy Y Box 2002 81207.55Japan 2_Products 2002 101071.96Japan Bounce 2002 11437.13Japan Y Box 2002 89634.83


Visualization of MODEL clause


MODEL Clause

Inside the MODEL clause, you can use nifty features like CV(dimension) to refer to the current value of a certain

dimension FOR loops over DIMENSION values

• e.g. sales[ ‘prodx’, for year in (2000,2001,2002)] = sales[CV(product), 1999 – (2002 - CV(year))]

• or even more interesting: sales[‘prodx’, for year in (select year from important_years)] = ….

Use symbolic references like ANY and ‘year > 2000’ Use ITERATE to evaluate rules a fixed number of

times Use Reference Models to retrieve supporting values

from


Example Model ClauseCreating new rows

SELECT cell numFROM DUALMODELDIMENSION BY (0 attr)MEASURES (0 cell)RULES ITERATE (10) ( cell[iteration_number] = iteration_number)


Pivoting the result set using the Model Clause

select deptno, ename, row_number() OVER ( PARTITION BY deptno ORDER BY sal desc NULLS LAST) rnkfrom emporderby deptno, rnk/

DEPTNO NR1 NR2 NR3---------- ---------- ---------- ---------- 10 KING CLARK MILLER 30 BLAKE ALLEN TURNER 20 SCOTT FORD JONES

This my goal

This is Step One


Pivoting the result set using the Model Clause

select deptno, nr1 , nr2, nr3from empmodel return updated rows partition by (deptno) –- do this thing for each Department dimension by (row_number() OVER (PARTITION BY deptno ORDER BY sal desc NULLS LAST) rnk ) -– each position gets its own result column measures (ename, lpad(' ',10) nr1 , lpad(' ',10) nr2 , lpad(' ',10) nr3) rules upsert ( nr1 [0] = ename [1] , nr2 [0] = ename [2] , nr3 [0] = ename [3] )


10g - Expression Filters

Oracle 10g Expression Filter Store filter expressions Select records that comply with the stored filters

• Or the filters that matchspecific records!

Filter expressions express• Customer interest• Regulation or business

rule• Match applicant with

jobs and vice versa Jobs have associated

profile== filter Application has profile

== expression filter


Expression Filters (10g)

“Given data, find consumers for it” Expression Filters like stored Where-clauses plus

framework for efficient dynamic evaluation What is an expression filter ?

Feature to manage and evaluate conditional expressions that describe users interests in data

Allows you to store conditional expressions in a column, which you can then use in the WHERE clause of a database query

Expression Filter includes: Expression datatype, SQL EVALUATE operator, Expression indexing (Enterprise Edition only) Administrative utilities


Overview of Expression Filters

Table Consumers

Consumer Interestspecified through “Filter” Clause, such as: 'Model=''Taurus'' and Price < 15000 and Mileage < 25000'

Object TypeCar4SaleModelYearPriceMileage

InterestCid Zip Phone Attr

ibut

e S

etC

ar4S

ale

Attribute Set Assignment -Linking column with filter clauseTo AttributeSet and Object Type

select *from consumerswhere evaluate ( interest , Car4Sale('Mustang', 2000, 18000 , 22000).getVarchar() ) = 1



Basic steps to create and use an expression column Define an attribute set Create user table with expression column (or add new varchar2

column to existing table) Assign attribute set to the expression column in the table Insert expressions in the table Apply the SQL EVALUATE operator

Applying the SQL evaluate Operator The EVALUATE operator returns 1 for an expression that

evaluates to true for the data item and 0 otherwise. The EVALUATE operator is used in the WHERE clause of a

standard SQL query or DML (UPDATE or DELETE) statement.


Overview of Expression FiltersUsing Expression Filter in Table Join

Table Consumers

Consumer Interestspecified through “Filter” Clause, such as: 'Model=''Taurus'' and Price < 15000 and Mileage < 25000'

Object TypeCar4SaleModelYearPriceMileage

InterestCid Zip Phone Attr

ibut

e S

etC

ar4S

ale

select consumers.*, inventory.*from consumers, inventorywhere evaluate ( interest , Car4Sale(model, year, price, mileage).getVarchar() ) = 1

Table InventoryMileageModel Year Price


Steps to set up Expression Filters

Create an object type as source for the Expression Attribute Set

Defining an attribute set based on the type

BEGIN DBMS_EXPFIL.CREATE_ATTRIBUTE_SET ( attr_set => 'Car4Sale‘ , from_type => 'YES‘ );END;

CREATE OR REPLACE TYPE Car4Sale AS OBJECT( Model VARCHAR2(20), Year NUMBER, Price NUMBER, Mileage NUMBER);



The Expression Attribute Set Defined as special Oracle Object Type that carries all valid

variables for the set Implicitly includes all the built-in functions Possible to add user-defined functions May contain XPath predicates defined on XMLType attributes Expressions must adhere to SQL Where Clause format

• Subqueries are not allowed! Not required to use all the defined elementary attributes (null is

acceptable) Two ways to create an attribute set

• use dbms_expfil.create_attribute_set in combination with dbms_expfil.add_elementary_attribute (automatically creates object)

• use an existing object type to create an attribute set dbms_expfil.create_attribute_set(attr_set => 'car4sale',from_type => 'yes'))


Typical Usage of Expression Filters

To screen incoming data Find matches with expressed interests or conditions

• We have found an item that may be exactly what you’re looking for

• A suspect has just entered the country Find non-matches

• This new piece of data does not meet one of (y)our standards• This record does not adhere to this business rule

To screen existing data for new interests, conditions, standards or rules Because of this new EU regulation, we have to

redesign these products…


Business Rule example

Business Ruleslabel table msg expression

Empempno ename job sal

Deptnodeptno dname loc

EMPDEPTEMPDEPTBONUS

EMP1DPT1EMP2DPT2BOS1

001011002012021

CREATE OR REPLACE TYPE BusinessRuleConditions AS OBJECT( Condition1 VARCHAR2(20), Condition2 VARCHAR2(20), Condition3 VARCHAR2(20), Condition4 VARCHAR2(20), Condition5 VARCHAR2(20));

BEGIN DBMS_EXPFIL.CREATE_ATTRIBUTE_SET ( attr_set => 'BusinessRuleConditions' , from_type => 'YES' );END;

BEGIN DBMS_EXPFIL.ASSIGN_ATTRIBUTE_SET ( attr_set => 'BusinessRuleConditions' , expr_tab => 'Business_Rules' , expr_col => 'br_expression' );END;


Inserting the business rules as Expressions

Business Ruleslabel table msg expression

Empempno ename job sal

Deptnodeptno dname loc

EMPDEPTEMPDEPTBONUS

EMP1DPT1EMP2DPT2BOS1

001011002012021

INSERT INTO Business_rules VALUES ('EMP1', '001','EMP', 'Condition1 =''SALESMAN'' or Condition2 is null');INSERT INTO Business_rules VALUES ('EMP2', '002','EMP', '(sysdate - to_date(condition3)) <8500');INSERT INTO Business_rules VALUES ('DPT1', '001','DEPT', 'Condition2 <>''SALES'' or Condition3 <> ''PARIS'' ');

Condition1 =''SALESMAN'' or Condition2 is nullCondition2 <>''SALES'' or Condition3 <> ''PARIS''(sysdate - to_date(condition3)) <8500012021


Data Mining from PL/SQL


PL/SQL API for Data Mining:DBMS_FREQUENT_ITEMSET

Oracle 9i and 10g EE contain Data Mining What used to be the Oracle Darwin product

Data Mining is statistically analyzing data, trying to discover patterns to be used for predictions

The Data Mining functionality is largely accessed through a Java API

Some Data Mining can directly be done from PL/SQL: dbms_frequent_itemset To find out from a large collection of items or facts,

which items have an increased chance of occurring together


Examples of questions suited for Data Mining with DBMS_FREQUENT_ITEMSET

Supermarket-basket analysis: if you buy pizza and cola, what other products are likely to be in your trolley

the occurrence of a certain disease in combination with behavioral or diet-related characteristics

the preference for a certain type of car in combination with properties such as income, profession, preferred make of father/neighbour/friend, age

the preference for a political candidate or party; you can think of some of the associated properties…(religion, income, skincolor, corporate associations etc.)

the occurrence of a software bug given some of the characteristics of the application, the organization, the technology and the programmer…


Business Case: Patterns in Movie Preferences

Three tables with movie data:


Business Case: Patterns in Movie Preferences Questions to investigate

Is there such a Frequent Itemset pattern that we can predict for someone of certain gender and age what movies he/she is likely to appreciate? If we know that someone enjoys Finding Nemo and Shrek, is there a distinct possibility that he will enjoy Sharktale? Who should we target in our marketing for a new movie, if that movie is akin to X? When my 5-year old nephew comes to visit, how on earth can I entertain him?


Performing Analysis

We need to call dbms_frequent_itemset Instructing how to find data – using a Cursor

Expression The cursor must return (id,value) where id is a ‘basket

identifier’ All values must be of the same data type

DBMS_FREQUENT_ITEMSET.FI_TRANSACTIONAL( cursor ( select id, value from …), 0.05 – threshold (minimal % of supporting records), 4 –- itemset min length, 6 –- itemset max length, null -- INCLUDING_ITEMS REF CURSOR, null -- EXCLUDING_ITEMS REF CURSOR ) RETURNS ANYDATASET


Simple transaction basket analysis

SELECT CAST (itemset as FI_number_NT) itemset, support, length, total_tranxFROM table( DBMS_FREQUENT_ITEMSET.FI_TRANSACTIONAL ( cursor ( SELECT id, mve_id_1 FROM movie_preferences mpe union SELECT id, mve_id_2 FROM movie_preferences mpe union SELECT id, mve_id_3 FROM movie_preferences mpe union SELECT id, 1000* age_category FROM movie_preferences mpe union SELECT id, case mpe.gender when 'M' then -1 else -2 end FROM movie_preferences mpe ) , 0.05 -- threshold , 4 –- itemset min length , 6 –- itemset max length , NULL , NULL ) ) orderby support desc, length


Interpretation of simple analysis

First record: 3 occurrences -2 : Female 5000: 21+ years 3: Shark Tale 5: Love Actually

Second record: 3 occurrences Female 12-16 years Shark Tale and Finding Nemo

…


Simple transaction basket analysis:better presentation of the findings

SELECT present_items( CAST (itemset as FI_number_NT)) itemset, support, length, total_tranxFROM table( DBMS_FREQUENT_ITEMSET.FI_TRANSACTIONAL ( cursor ( SELECT id, mve_id_1 FROM movie_preferences mpe union SELECT id, mve_id_2 FROM movie_preferences mpe union SELECT id, mve_id_3 FROM movie_preferences mpe union SELECT id, 1000* age_category FROM movie_preferences mpe union SELECT id, case mpe.gender when 'M' then -1 else -2 end FROM movie_preferences mpe ) , 0.05 , 4 , 6 , NULL , NULL ) )orderby support desc, length desc order


Simple transaction basket analysis: better presentation of the findings

SELECT present_items( CAST (itemset as FI_number_NT)) itemset, support, length, total_tranxFROM table( DBMS_FREQUENT_ITEMSET.FI_TRANSACTIONAL ( cursor ( SELECT id, mve_id_1 FROM movie_preferences mpe union SELECT id, mve_id_2 FROM movie_preferences mpe union SELECT id, mve_id_3 FROM movie_preferences mpe union SELECT id, 1000* age_category FROM movie_preferences mpe union SELECT id, case mpe.gender when 'M' then -1 else -2 end FROM movie_preferences mpe ) , 0.05 , 4 , 6 , NULL , NULL ) )orderby support desc, length desc order


Business Case: Our Nephew

Our nephew is visiting; he is 5 years old he enjoyed Finding Nemo he is male

What movie should we rent for him?


Simple transaction basket analysis:better presentation of the findings

SELECT select items.column_value "Recommended Movie"from <inline view> it, table(it.itemset) items, movies mvewhere support > 1 and set_contains_item( itemset, '2-6 years' ) = 1and set_contains_item( itemset, 'M' ) = 1and mve.title = items.column_valueand mve.title != 'Finding Nemo'


In-Line View for selecting Movie-Basket Combinations that include Finding Nemo

( SELECT CAST(itemset as FI_varchar_NT) itemset , support,length,total_tranx FROM table ( DBMS_FREQUENT_ITEMSET.FI_TRANSACTIONAL ( cursor ( SELECT mpe.id, title FROM movie_preferences mpe , movies mve where mve.id = mpe.mve_id_1 union SELECT mpe.id, title FROM movie_preferences mpe , movies mve where mve.id = mpe.mve_id_2 union SELECT mpe.id, title FROM movie_preferences mpe , movies mve where mve.id = mpe.mve_id_3 union SELECT mpe.id, acy.description FROM movie_preferences mpe, age_categories acy where acy.id = mpe.age_category union SELECT id, gender FROM movie_preferences ) , 0.05 , 4 , 6 , CURSOR( SELECT * FROM table(FI_VARCHAR_NT('Finding Nemo'))) , NULL ) ) ) it


Conclusions

10gR1 has some neat new SQL & PL/SQL features Enrichments of existing ones Some brand new ones

• Especially in the area of On-Line, Large Data Set interpretation Model Clause – spreadsheet in SQL

Inter-row, direct cell references, adding records Expression Filters – stored Filter Conditions

“Have (new) data find searchers” Matching, Alerting, Rule Enforcement

DBMS_FREQUENT_ITEMSET – data mining Exploit association patterns (“baskets”) buried in your

data for “fortune telling”


Onderwerpen voor ODTUG 2006 – 17-21 Juni Washington DC

Oracle 9i en 10g SQL en PL/SQL Oracle Text Oracle Spatial XML DB HTML DB Lite Express Edition

Oracle 10g WebForms JDAPI WebUtil PJC Java/J2EE integratie JavaScript

Oracle Warehouse Builder Oracle Discoverer …

Oracle JDeveloper 10.1.3 ADF ADF Faces JHeadstart

Oracle BPEL Oracle InterConnect XML Publisher Oracle en Open Source

ADF en Spring Hibernate en ADF MySQL vs. Express Edition

Business Rules Software Engineering

Oracle Development Tools & Jira, CVS, JUnit, Ant/Maven etc.

…

KC Server Development Nieuwe SQL en PL/SQL features in Oracle 10gR2 en R1 incl. reprise van ODTUG...

Documents

Transcript of KC Server Development Nieuwe SQL en PL/SQL features in Oracle 10gR2 en R1 incl. reprise van ODTUG...