Karen Cannell [email protected]

TH Technology

Karen [email protected]

http://www.thtechnology.com

How Do I Load Data …Let Me Count The WaysAPEX Data Loading Options

mailto:[email protected]

http://www.thtechnology.com/

http://www.thtechnology.com/

How Do I Load Data …

TH Technology

How Do I Load Data? : Agenda

Oracle Data Loading OptionsDBAs - Developers - End Users

APEX Data Loading OptionsFile UploadData Load Wizard2

XLS Upload – APEX ListenerWebSheets – Copy/Paste, Add Row

TH Technology


About Me … Karen Cannell ~ Consultant, TH Technology

SW Engineer 25+ years, Oracle since 1994. Building APEX apps for government, medical, engineering industries.Leveraging the Oracle 10g,11g (now 12c) suite of tools

Beginning Application Express, APress, 2011Agile Application Express, APress, 2011

Editor, ODTUG Technical JournalVolunteer to author ODTUG Journal Articles!

Using APEX since the HTMLDB beginning


TH Technology

About You … (Audience Background)

New to APEX?APEX Experience?Previous Tools?

APEX Training?Version 2.0? 3.1 ? 3.2 ? 4.1?


TH Technology

How DO I Load Data into Oracle?

Load Data Into Oracle7,340,000 hits

SQL LoaderOracle Database UtilitiesLoad Excel Data into Oracle


TH Technology


Load Data Into APEX317,000 hits

Import Excel Data …APEX Data LoaderLoad Text File into APEX


TH Technology


Load XLS Data Into APEX1,830,000 hits

Import Excel File …APEX Data LoaderLoad Excel Data …

Yikes!

TH Technology


Load Data into Oracle - Options

What Are the Options?Commercial Solutions - Excel-DB, QuickloadWarehouse Builder, ETL ToolsOracle Data Integratorhttp://www.oracle.com/technetwork/middleware/data-integrator/overview/index.html

Oracle Database UtilitiesOracle Data PumpSQL LoaderExternal Tables

http://www.oracle.com/technetwork/middleware/data-integrator/overview/index.html


TH Technology


Load Data into Oracle, cont’d

ODBC ConnectionsIDEs w/built-In Data LoadersCustom Code:

JavaPL/SQL Using UTL_FILEScripting from XLS ColumnsJava and PL/SQLPerlOther

TH Technology


Let’s Focus …

TH Technology


Load Data into APEX

Import Excel Data into Oracle …

APEX Data Loader

“Load Excel Data”

95% Say XLS but Really are CSV

The Apex Data Loader is a java based application that can bulk process the insert, update and delete on all object data into and

build queries to extract data out of salesforce.com using the Apex Web Services

(SOAP) API.

http://salesforce.com/

TH Technology


Our Problem

“Load Data into Oracle”40% Fixed-Format Text Files 60% Excel Spreadsheets Several XLS Formats200+ Files MonthlyLocations All Over the WorldSome Networked, Some NotVarying Volume of Data

TH Technology


Which to Choose?

It Depends ApplicationSourceTarget ( Table?)Amount of DataHow Often ? 1 or ManyAudience End UsersResources $$

Time $$

TH Technology


Which Make Sense for APEX ?

Oracle Database UtilitiesPL/SQL IDENo Additional Licenses PL/SQL Code No Complex SetupDeveloper-FriendlyEnd User Friendly

TH Technology


Oracle Database Utilities

Oracle Data PumpOracle DB to Oracle DBFaster than EXP and IMP

EXPDP 2x faster than EXPIMPDP 15-45x faster than IMP

Jobs can be RestartedSupports Network Import and Export

Load one instance form anotherRemote export

TH Technology


Oracle Database Utilities, cont’d

SQL LoaderExternal files into TablesVariety of formatsFilteringMultiple Table Load in One Load SessionConventionalDirect PathExternal Table Load

TH Technology


Oracle Database Utilities, cont’d

External TablesExternal files into TablesVariety of FormatsPreprocessing Capability

External Tables to PullRPT FilesCSV Files - That We Can Get

TH Technology


Oracle Database – Database Link

Don’t Forget the Database Link!SELECT …

FROM table@db_link

Simple, DirectRequires CommunicationAsk!

TH Technology


Use your IDE

SQL DeveloperPL/SQL DeveloperToadOthers …

Best for One-Time Load

TH Technology


SQL Developer – Data Load Option

Import Data … Wizard

TH Technology


SQL Developer – Import Data …

Import File Formats.XLS / XLSX.CSV.TSV (tab).DSV ( SQL

Server)

TH Technology



Import Wiz Detects• File type• Delimiter• Skip Row

TH Technology



INSERT or

GenerateINSERTScript)

TH Technology



SelectColumns toImport

TH Technology



Map Source to TableColumns

TH Technology



Verify

TH Technology



Finished!

XLS and XLSX* File Load in Very Few Clicks!*XLSX in SQL Developer 3.1

TH Technology



What About XLSX?• SQL Dev 3.0 – No• SQL Developer 3.1 – Yes

XLS/XLSX File Load in Very Few Clicks!

TH Technology



ProsOne Shot Data LoadDelimiter (, TAB, others)Flexible ColumnsFlexible MappingXLS FileCan be Faster than APEX Data Load

Wizard

ConsTable Must Be DefinedCannot Integrate into AppTedious to RepeatIDE SpecificOne Shot Data LoadNot for End Users

TH Technology


APEX Data Load/Unload Utility

APEX From the BeginningEasy Load of Spreadsheet or XML DataNew or Existing TableUpload File Or Copy/PasteDelimiter OptionColumn by Column Selection

TH Technology



TH Technology



1st Row Column Heading

TH Technology



TH Technology



Specify PK and Trigger

Finish

TH Technology


APEX Data Load Utility

ProsSimple: Point-Click-DoneFlexible FormatSpreadsheet ConvenienceCreates Table Copy/Paste Option

ConsMany StepsSingle TableNo XLS Upload (Must Copy/Paste or Save as CSV)One-TimeNot for End Users

TH Technology


What About the End User?

APEX Standard File Browse + Custom CodeData Load Wizard Wizard

APEX Manual Data Load Wizard End User-izedAPEX Web Sheet Data Grid

Copy/Paste, Data EntryAPEX Listener XLS Upload

File Browse + XLS2COLLECTION

TH Technology


APEX File Browse + Custom Code

Most Common (Still)File Browse ItemProcess (to Staging)Process To Destination Table(s)

MANY MANY OPTIONS …

All Require Custom Code

TH Technology



File Browse Item

WWV_FLOW_FILES – Original WayBLOB column – New Option w APEX 4+

TH Technology



WWV_FLOW_FILES – Traditional WayFile uploads to APEX Table WWV_FLOW_FILESMove Records to a Custom Table (i.e. Staging Table)Clean up!

Files accumulate in APEX_FILES Tablespace

TH Technology



BLOB Column – New w/ APEX 4File Stored in BLOB Column of Table Specified in Automatic Row Processing (DML) ProcessColumn Specified in Item SourceTable Must Have

BLOBFilenameMIME TypeCHARSET

Saves an Archiving Step

TH Technology



BLOB Column – New w/ APEX 4

TH Technology



WWV_FLOW_FILES

TH Technology



LOCAL_UPLOAD_FILES

TH Technology



Parse / Load ProcessRead Blob into local BLOB variableRead Records from BlobLoad Records into Destination Objects

TH Technology



Reusable Parse/ Load ProcessFile Browse Item

Load File to WWV_FLOW_FILEGeneric Parse File Package

Generic File Upload UtilityGeneric Store to APEX Collection

Process Code per Use/Table/Format

Custom Code, But Less of It

TH Technology


Generic Parse File Utility Package

Parse File to CollectionLoad Collection to Specified Table

Columns Must Match, or Adjust CodeReusable Code, Pages for Any Upload File/TableSaves One Part of Custom CodeCustom Processing from Stage Table Required

Custom Code, But Less of It

TH Technology



Parse File ProcessRead Blob into local BLOB variable

BEGIN SELECT blob_content INTO l_blob FROM wwv_flow_files WHERE name=p_file_name;EXCEPTION WHEN NO_DATA_FOUND THEN raise_application_error(-20000, 'File not found, id='||p_file_name);END;

Or BLOB column

from your BLOB table

TH Technology



Parse File ProcessRead Records from Blob Variable

TYPE varchar2_t IS TABLE OF VARCHAR2(32767) INDEX BY binary_integer; l_records VARCHAR2_T; … get_records(l_blob,l_records); … -- Initialize the APEX collection apex_collection.create_or_truncate_collection( p_collection_name);

TH Technology


PROCEDURE get_records(p_blob IN blob,p_records OUT varchar2_t) IS l_sep VARCHAR2(2) := chr(13)||chr(10); l_last INTEGER; l_current INTEGER;BEGIN IF (NVL(DBMS_LOB.INSTR(p_blob, utl_raw.cast_to_raw(l_sep),1,1),0)=0) THEN l_sep := chr(10); END IF; l_last := 1; LOOP l_current := dbms_lob.INSTR( p_blob, utl_raw.cast_to_raw(l_record_separator), l_last, 1 ); EXIT WHEN (nvl(l_current,0) = 0); p_records(p_records.count+1) := utl_raw.cast_to_varchar2( DBMS_LOB.SUBSTR(p_blob,l_current-l_last,l_last)); l_last := l_current+length(l_record_separator); END LOOP;END get_records;

TH Technology


LOOP l_current := dbms_lob.INSTR( p_blob, utl_raw.cast_to_raw(l_record_separator), l_last, 1 ); EXIT WHEN (nvl(l_current,0) = 0); p_records(p_records.count+1) := utl_raw.cast_to_varchar2( DBMS_LOB.SUBSTR(p_blob,l_current-l_last,l_last)); l_last := l_current+length(l_record_separator); END LOOP;

Loop to read records from the BLOB, line by line(separator to separator)

TH Technology



Parse File Process: Read from Blob,Load Collection

FOR i IN 2..l_records.count LOOP csv_to_array(l_records(i),l_record); l_seq_id := apex_collection.add_member(p_collection_name,'dummy'); FOR i IN 1..l_record.count LOOP apex_collection.update_member_attribute( p_collection_name=> p_collection_name, p_seq => l_seq_id, p_attr_number => i, p_attr_value => l_record(i)); END LOOP; END LOOP;

TH Technology



Parse File Process: Read Blob, Load Collection

Insert from Collection to Table … l_ddl := ‘INSERT INTO '||p_table_name||' '|| ‘SELECT '||v(p_columns_item)||' '|| ‘FROM apex_collections '|| ‘WHERE seq_id >= 1 and collection_name='''|| p_collection_name||'''';

EXECUTE IMMEDIATE l_ddl; …

TH Technology



Parse File Process: Read Blob, Load Collection, Insert,

Remove File -- Cleanup file! DELETE FROM wwv_flow_files WHERE name=p_file_name;

TH Technology



Process to Destination Table(s)

INSERT /* +append */ … SELECT … FROM staging

MERGE statement

Whatever is Required for Your Business

Custom Code

TH Technology


File Browse & Custom Code

ProsCustom CodeCustom Solution

ConsCustom CodeResourcesMaintenanceOne-of Solution

TH Technology


Data Load Wizard Wizard

The SQL Workshop Data Load Utility End-User-ized

Page Type

Creates a Shared Component

TH Technology



Wizard to Create A Data Load WizardCreate Page Data Loading

TH Technology



TH Technology



Define Table Lookups

TH Technology



Add TransformsTo Upper/Lower, R/L/Trim, Single Whitespace

TH Technology



Page/Region/Breadcrumb Names

TH Technology



Define Tabs (if any)

TH Technology



Define Branches

TH Technology



Confirmation

TH Technology


Data Load Wizard - Result

Working Upload for our USERS table

TH Technology


Data Load Wizard

End User Sees:Data Load ProgressionChance to Select Row to Include/IgnoreFailed Rows at the End

Chance to Back Out Load?Chance to Reload Clean?

TH Technology


Data Load Wizard

Upload OR Copy/PasteData Limit ==Specify DelimiterSpecify Enclosed BySpecify Currency, Group Separator, Decimal Characters

TH Technology


Data Load Wizard - Options

TH Technology


Data Load Wizard – Pro’s/Con’s

ProsFlexible FormDelimiter Option1st Row Heading OptCurrency – Group Separator – DecimalMapping Built InRepeatable for End User

Reusable for Developer

ConsToo Much for End User??No XLSCopy/Paste Size LimitUpload Size Limit ??Single Transform per Column

Home

http://localhost:8080/apex/f?p=4500:1000:6561482008021805

TH Technology



Is It Foolproof?No:

Ex: RPT file …. What does this means?

Too Many Columns?Too Few?

TH Technology



Can I trick it into taking an XLS?No:

TH Technology


Data Load Wizard

Can I Make it “Better”?

Try:Default/Eliminate All OptionsEliminate Load/Unload Column MappingCatch/Warn of File Type, # Columns Errors

Load the Collection Behind the Schemes

TH Technology


Data Load Wizard – Behind the Scenes

Usual Expected ItemsFILE_BROWSE

TH Technology


Data Load Wizard – P 2

Parse Uploaded DataSingle Process of Type Parse Uploaded DataList of ParametersFires upon Next

TH Technology


TH Technology



Process Uploaded DataSingle Process of Type Process Uploaded DataList of ParametersFires upon Next

TH Technology



Session InfoCollectionsCharacter ColumnsNumeric ColumnsDate Columns

TH Technology


TH Technology


Data Load Wizard WizardPros

In-AppStraightforwardReusableNo Custom Code for Data Load into DB

ConsToo Many End User Steps Still Need Custom Processing Code for Staging to Tables

IF get P1 Finish by Loading Collection,

have a Two-Step File Upload for End

Users!

TH Technology


APEX Listener XLS Upload

File Browse ItemLoad File to WWV_FLOW_FILE

Parsing FileCustom Code per Use/Table/FormatGeneric File Upload Utility

Custom Process to Destination Tables

TH Technology


Parse File to CollectionLoads Collection to Specified Table

Table Exist or NotOptions

Reusable for Any Upload File/TableSaves One Part of Custom CodeCustom Processing from Stage Table Required


TH Technology



Demo

TH Technology


XLS Upload via APEX Listener

Must Deploy via APEX ListenerCurrent Release 1.1.3…

Allows Upload of XLS File into a Collection

Best Reference: (from RMOUG 2010)http://krisrice.blogspot.com/2010/02/another-apex-listener-ea-more-knobs-to.html

http://krisrice.blogspot.com/2010/02/another-apex-listener-ea-more-knobs-to.html


TH Technology



Install APEX ListenerDeployEdit apex_config.xmlAdd Configuration Choices:

<entry key="apex.excel2collection">true</entry><entry key="apex.excel2collection.onecollection">true</entry><entry key="apex.excel2collection.name">EXCEL_COLLECTION</entry><entry key="apex.excel2collection.useSheetName">true</entry>

TH Technology



Configuration Options:

Turn Feature On: <entry key="apex.excel2collection">true</entry>

One Collection per Worksheet:

<entry key= "apex.excel2collection.onecollection">true|false</entry>

TH Technology



Configuration Options:

Specify Name for All Collections: <entry key="apex.useSheetName">true|false</entry>

false == sequential sheet nametrue == uses sheet name

Specify Name of Collection for All Collections:

<entry key="apex.excel2collection.collection.name">MY_COLLECTION</entry>

TH Technology



File Browse ItemWWV_FLOW_FILES option

Page Item Button Name or REQUEST = XLS2COLLECTION

Report Region:SELECT * FROM apex_collectionsWHERE collection_name = ‘MY_COLLECTION’

TH Technology



TH Technology



“Files ending with .xls or .xlsx will be processed.”

BUT.xls processMacro-Enabled .xls process.XLSX do NOT Process as of APEX Listener 1.1.3.243.11.40.XLSX will Process as of v2

Coming Soon!

TH Technology


XLSX Upload via APEX Listener

TH Technology


XLSX Upload via APEX Listener

SEVERE: service exception:org.apache.poi.poifs.filesystem.OfficeXmlFileException: The supplied data appears to be in the Office 2007+ XML. You are calling the part of POI that deals with OLE2 Office Documents. You need to call a different part of POI to process this data (eg XSSF instead of HSSF)

TH Technology



Formulas Translate Literally

Ex: IF(E2="","",VLOOKUP(E2,$AE$4:$AF$218,2,FALSE))

TH Technology



STILL Need Custom Code to Process Data from Collection to Database Table(s)

TH Technology



ProsXLS File – No Need to Save As CSVStraightforwardReusableLess Custom Code

ConsAPEX Listener DeploymentEmbedded FormulasNo XLSX - May Need a Save As Step

Will Have XLSX in V2

TH Technology


APEX Websheets

Data Grid Allows Copy/PasteSpreadsheet-Like EditsData Grid w/in Page

Custom Code Required to Transfer Data to Main Schema Tables

TH Technology


Editable Data in Websheet

TH Technology


APEX Websheets

Demo

TH Technology


Pull Data From Websheet into App

Look at APEX$_WS Tables:APEX$_WS_ROW – the dataAPEX$_WS_HISTORY – iff you care about who changed what when

Need to process from SQL Dev table to “real” tables

TH Technology


APEX WebSheets

ProsCopy/PasteSpreadsheet-Like Entry, EditsEnd User Can Build

ConsCustom Code to Create Websheet, Data Grid?Custom Code Required to Pull Data into Schema ObjectsCustom Code to Maintain Worksheet / Data GridMore Code Than Other Solutions ?

TH Technology


Our Solution

ProblemRPT FilesXLS filesDifferent Formats200+, MonthlyUsers Cannot “Save as CSV”Must Support EditsVarying VolumeLimited Network

SolutionTabular Forms - 4Macro to Save as RPTExternal Tables

Generated RPTLIS Output RPT Files

Upload File Process (to be retired)

Custom ProcessingRPT – LI Systems RPT – Spreadsheet Templates

TH Technology


APEX Data Load Options

TH Technology


Nice to Have … Future?

1-Step Data Load WizardCopy/Paste ORUpload XLSX

Easiest for Non-Networked, Low(er) Volume End Users

TH Technology


Some Last Thoughts …

Forms Create/Edit One Row

Tabular FormsEnter Many Rows

Java Solution.JSP to load XLS directlyPresume this means XLSX too!

TH Technology


How Do I Load Data ?

APEX-based SolutionWeigh Requirements

Source DataEnd User CapabilitiesResourcesTime

No One-Size-Fits-All

Apply Common Sense

TH Technology


TH Technology


Common Sense Test …

Two men play 5 games of checkers. Each wins the same # of games. There are no ties.

How can this be?

They are not playing each other.


TH Technology

References

Upload File into APEXoraexplorer.com – Ittichaihttp://oraexplorer.com/2007/11/apex-to-upload-text-file-and-write-into/More at avdeo.com In Oracle Milieuhttp://avdeo.com/2008/05/21/uploading-excel-sheet-using-oracle-application-express-apex/

http://oraexplorer.com/2007/11/apex-to-upload-text-file-and-write-into/

http://oraexplorer.com/2007/11/apex-to-upload-text-file-and-write-into/

http://avdeo.com/2008/05/21/uploading-excel-sheet-using-oracle-application-express-apex/




TH Technology

References

Oracle Data Integratorhttp://www.oracle.com/technetwork/middleware/data-integrator/overview/index.html

Oracle Database UtilitiesSQL Loader – External Tableshttp://www.oracle.com/technetwork/database/enterprise-edition/index-093639.html

SQL Developerhttp://www.oracle.com/technetwork/developer-tools/sql-developer/overview/index.html



http://www.oracle.com/technetwork/database/enterprise-edition/index-093639.html




TH Technology

References

APEX Listener XLS Upload – Kris Rice Bloghttp://krisrice.blogspot.com/2010/02/another-apex-listener-ea-more-knobs-to.html

A Java-and-PL/SQL based XLS to CSV Optionhttp://www.dynamicpsp.com/!go?ln=nfaq&faq=FAQ9



http://www.dynamicpsp.com/!go?ln=nfaq&faq=FAQ9

http://www.dynamicpsp.com/!go?ln=nfaq&faq=FAQ9

TH [email protected]

How Do I Load Data?

Let Me Count The Ways

Questions ?

Please fill out the

evaluations!

TH [email protected]

Thank You

Please fill out the

evaluations!

How Do I Load Data?

Let Me Count The Ways

Karen Cannell [email protected]

Documents

Transcript of Karen Cannell [email protected]