Karen Cannell kcannell@thtechnology.com

TH Technology

Karen Cannellkcannell@thtechnology.com

http://www.thtechnology.com

How Do I Load Data …Let Me Count The WaysAPEX Data Loading Options

How Do I Load Data …

TH Technology

How Do I Load Data? : Agenda

Oracle Data Loading OptionsDBAs - Developers - End Users

APEX Data Loading OptionsFile UploadData Load Wizard2

XLS Upload – APEX ListenerWebSheets – Copy/Paste, Add Row

TH Technology

About Me … Karen Cannell ~ Consultant, TH Technology

SW Engineer 25+ years, Oracle since 1994. Building APEX apps for government, medical, engineering industries.Leveraging the Oracle 10g,11g (now 12c) suite of tools

Beginning Application Express, APress, 2011Agile Application Express, APress, 2011

Editor, ODTUG Technical JournalVolunteer to author ODTUG Journal Articles!

Using APEX since the HTMLDB beginning

TH Technology

About You … (Audience Background)

New to APEX?APEX Experience?Previous Tools?

APEX Training?Version 2.0? 3.1 ? 3.2 ? 4.1?

TH Technology

How DO I Load Data into Oracle?

Load Data Into Oracle7,340,000 hits

SQL LoaderOracle Database UtilitiesLoad Excel Data into Oracle

TH Technology

Load Data Into APEX317,000 hits

Import Excel Data …APEX Data LoaderLoad Text File into APEX

TH Technology

Load XLS Data Into APEX1,830,000 hits

Import Excel File …APEX Data LoaderLoad Excel Data …

Yikes!

TH Technology

Load Data into Oracle - Options

What Are the Options?Commercial Solutions - Excel-DB, QuickloadWarehouse Builder, ETL ToolsOracle Data Integratorhttp://www.oracle.com/technetwork/middleware/data-integrator/overview/index.html

Oracle Database UtilitiesOracle Data PumpSQL LoaderExternal Tables

TH Technology

Load Data into Oracle, cont’d

ODBC ConnectionsIDEs w/built-In Data LoadersCustom Code:

JavaPL/SQL Using UTL_FILEScripting from XLS ColumnsJava and PL/SQLPerlOther

TH Technology

Let’s Focus …

TH Technology

Load Data into APEX

Import Excel Data into Oracle …

APEX Data Loader

“Load Excel Data”

95% Say XLS but Really are CSV

The Apex Data Loader is a java based application that can bulk process the insert, update and delete on all object data into and

build queries to extract data out of salesforce.com using the Apex Web Services

(SOAP) API.

TH Technology

Our Problem

“Load Data into Oracle”40% Fixed-Format Text Files 60% Excel Spreadsheets Several XLS Formats200+ Files MonthlyLocations All Over the WorldSome Networked, Some NotVarying Volume of Data

TH Technology

Which to Choose?

It Depends ApplicationSourceTarget ( Table?)Amount of DataHow Often ? 1 or ManyAudience End UsersResources $$

Time $$

TH Technology

Which Make Sense for APEX ?

Oracle Database UtilitiesPL/SQL IDENo Additional Licenses PL/SQL Code No Complex SetupDeveloper-FriendlyEnd User Friendly

TH Technology

Oracle Database Utilities

Oracle Data PumpOracle DB to Oracle DBFaster than EXP and IMP

EXPDP 2x faster than EXPIMPDP 15-45x faster than IMP

Jobs can be RestartedSupports Network Import and Export

Load one instance form anotherRemote export

TH Technology

Oracle Database Utilities, cont’d

SQL LoaderExternal files into TablesVariety of formatsFilteringMultiple Table Load in One Load SessionConventionalDirect PathExternal Table Load

TH Technology

Oracle Database Utilities, cont’d

External TablesExternal files into TablesVariety of FormatsPreprocessing Capability

External Tables to PullRPT FilesCSV Files - That We Can Get

TH Technology

Oracle Database – Database Link

Don’t Forget the Database Link!SELECT …

FROM table@db_link

Simple, DirectRequires CommunicationAsk!

TH Technology

Use your IDE

SQL DeveloperPL/SQL DeveloperToadOthers …

Best for One-Time Load

TH Technology

SQL Developer – Data Load Option

Import Data … Wizard

TH Technology

SQL Developer – Import Data …

Import File Formats.XLS / XLSX.CSV.TSV (tab).DSV ( SQL

Server)

TH Technology

Import Wiz Detects• File type• Delimiter• Skip Row

TH Technology

INSERT or

GenerateINSERTScript)

TH Technology

SelectColumns toImport

TH Technology

Map Source to TableColumns

TH Technology

Verify

TH Technology

Finished!

XLS and XLSX* File Load in Very Few Clicks!*XLSX in SQL Developer 3.1

TH Technology

What About XLSX?• SQL Dev 3.0 – No• SQL Developer 3.1 – Yes

XLS/XLSX File Load in Very Few Clicks!

TH Technology

ProsOne Shot Data LoadDelimiter (, TAB, others)Flexible ColumnsFlexible MappingXLS FileCan be Faster than APEX Data Load

Wizard

ConsTable Must Be DefinedCannot Integrate into AppTedious to RepeatIDE SpecificOne Shot Data LoadNot for End Users

TH Technology

APEX Data Load/Unload Utility

APEX From the BeginningEasy Load of Spreadsheet or XML DataNew or Existing TableUpload File Or Copy/PasteDelimiter OptionColumn by Column Selection

TH Technology

1st Row Column Heading

TH Technology

Specify PK and Trigger

Finish

TH Technology

APEX Data Load Utility

ProsSimple: Point-Click-DoneFlexible FormatSpreadsheet ConvenienceCreates Table Copy/Paste Option

ConsMany StepsSingle TableNo XLS Upload (Must Copy/Paste or Save as CSV)One-TimeNot for End Users

TH Technology

What About the End User?

APEX Standard File Browse + Custom CodeData Load Wizard Wizard

APEX Manual Data Load Wizard End User-izedAPEX Web Sheet Data Grid

Copy/Paste, Data EntryAPEX Listener XLS Upload

File Browse + XLS2COLLECTION

TH Technology

APEX File Browse + Custom Code

Most Common (Still)File Browse ItemProcess (to Staging)Process To Destination Table(s)

MANY MANY OPTIONS …

All Require Custom Code

TH Technology

File Browse Item

WWV_FLOW_FILES – Original WayBLOB column – New Option w APEX 4+

TH Technology

WWV_FLOW_FILES – Traditional WayFile uploads to APEX Table WWV_FLOW_FILESMove Records to a Custom Table (i.e. Staging Table)Clean up!

Files accumulate in APEX_FILES Tablespace

TH Technology

BLOB Column – New w/ APEX 4File Stored in BLOB Column of Table Specified in Automatic Row Processing (DML) ProcessColumn Specified in Item SourceTable Must Have

BLOBFilenameMIME TypeCHARSET

Saves an Archiving Step

TH Technology

BLOB Column – New w/ APEX 4

TH Technology

WWV_FLOW_FILES

TH Technology

LOCAL_UPLOAD_FILES

TH Technology

Parse / Load ProcessRead Blob into local BLOB variableRead Records from BlobLoad Records into Destination Objects

TH Technology

Reusable Parse/ Load ProcessFile Browse Item

Load File to WWV_FLOW_FILEGeneric Parse File Package

Generic File Upload UtilityGeneric Store to APEX Collection

Process Code per Use/Table/Format

Custom Code, But Less of It

TH Technology

Generic Parse File Utility Package

Parse File to CollectionLoad Collection to Specified Table

Columns Must Match, or Adjust CodeReusable Code, Pages for Any Upload File/TableSaves One Part of Custom CodeCustom Processing from Stage Table Required

Custom Code, But Less of It

TH Technology

Parse File ProcessRead Blob into local BLOB variable

BEGIN SELECT blob_content INTO l_blob FROM wwv_flow_files WHERE name=p_file_name;EXCEPTION WHEN NO_DATA_FOUND THEN raise_application_error(-20000, 'File not found, id='||p_file_name);END;

Or BLOB column

from your BLOB table

TH Technology

Parse File ProcessRead Records from Blob Variable

TYPE varchar2_t IS TABLE OF VARCHAR2(32767) INDEX BY binary_integer; l_records VARCHAR2_T; … get_records(l_blob,l_records); … -- Initialize the APEX collection apex_collection.create_or_truncate_collection( p_collection_name);

TH Technology

PROCEDURE get_records(p_blob IN blob,p_records OUT varchar2_t) IS l_sep VARCHAR2(2) := chr(13)||chr(10); l_last INTEGER; l_current INTEGER;BEGIN IF (NVL(DBMS_LOB.INSTR(p_blob, utl_raw.cast_to_raw(l_sep),1,1),0)=0) THEN l_sep := chr(10); END IF; l_last := 1; LOOP l_current := dbms_lob.INSTR( p_blob, utl_raw.cast_to_raw(l_record_separator), l_last, 1 ); EXIT WHEN (nvl(l_current,0) = 0); p_records(p_records.count+1) := utl_raw.cast_to_varchar2( DBMS_LOB.SUBSTR(p_blob,l_current-l_last,l_last)); l_last := l_current+length(l_record_separator); END LOOP;END get_records;

TH Technology

LOOP l_current := dbms_lob.INSTR( p_blob, utl_raw.cast_to_raw(l_record_separator), l_last, 1 ); EXIT WHEN (nvl(l_current,0) = 0); p_records(p_records.count+1) := utl_raw.cast_to_varchar2( DBMS_LOB.SUBSTR(p_blob,l_current-l_last,l_last)); l_last := l_current+length(l_record_separator); END LOOP;

Loop to read records from the BLOB, line by line(separator to separator)

TH Technology

Parse File Process: Read from Blob,Load Collection

FOR i IN 2..l_records.count LOOP csv_to_array(l_records(i),l_record); l_seq_id := apex_collection.add_member(p_collection_name,'dummy'); FOR i IN 1..l_record.count LOOP apex_collection.update_member_attribute( p_collection_name=> p_collection_name, p_seq => l_seq_id, p_attr_number => i, p_attr_value => l_record(i)); END LOOP; END LOOP;

TH Technology

Parse File Process: Read Blob, Load Collection

Insert from Collection to Table … l_ddl := ‘INSERT INTO '||p_table_name||' '|| ‘SELECT '||v(p_columns_item)||' '|| ‘FROM apex_collections '|| ‘WHERE seq_id >= 1 and collection_name='''|| p_collection_name||'''';

EXECUTE IMMEDIATE l_ddl; …

TH Technology

Parse File Process: Read Blob, Load Collection, Insert,

Remove File -- Cleanup file! DELETE FROM wwv_flow_files WHERE name=p_file_name;

TH Technology

Process to Destination Table(s)

INSERT /* +append */ … SELECT … FROM staging

MERGE statement

Whatever is Required for Your Business

Custom Code

TH Technology

File Browse & Custom Code

ProsCustom CodeCustom Solution

ConsCustom CodeResourcesMaintenanceOne-of Solution

TH Technology

Data Load Wizard Wizard

The SQL Workshop Data Load Utility End-User-ized

Page Type

Creates a Shared Component

TH Technology

Wizard to Create A Data Load WizardCreate Page Data Loading

TH Technology

Define Table Lookups

TH Technology

Add TransformsTo Upper/Lower, R/L/Trim, Single Whitespace

TH Technology

Page/Region/Breadcrumb Names

TH Technology

Define Tabs (if any)

TH Technology

Define Branches

TH Technology

Confirmation

TH Technology

Data Load Wizard - Result

Working Upload for our USERS table

TH Technology

Data Load Wizard

End User Sees:Data Load ProgressionChance to Select Row to Include/IgnoreFailed Rows at the End

Chance to Back Out Load?Chance to Reload Clean?

TH Technology

Data Load Wizard

Upload OR Copy/PasteData Limit ==Specify DelimiterSpecify Enclosed BySpecify Currency, Group Separator, Decimal Characters

TH Technology

Data Load Wizard - Options

TH Technology

Data Load Wizard – Pro’s/Con’s

ProsFlexible FormDelimiter Option1st Row Heading OptCurrency – Group Separator – DecimalMapping Built InRepeatable for End User

Reusable for Developer

ConsToo Much for End User??No XLSCopy/Paste Size LimitUpload Size Limit ??Single Transform per Column

TH Technology

Is It Foolproof?No:

Ex: RPT file …. What does this means?

Too Many Columns?Too Few?

TH Technology

Can I trick it into taking an XLS?No:

TH Technology

Data Load Wizard

Can I Make it “Better”?

Try:Default/Eliminate All OptionsEliminate Load/Unload Column MappingCatch/Warn of File Type, # Columns Errors

Load the Collection Behind the Schemes

TH Technology

Data Load Wizard – Behind the Scenes

Usual Expected ItemsFILE_BROWSE

TH Technology

Data Load Wizard – P 2

Parse Uploaded DataSingle Process of Type Parse Uploaded DataList of ParametersFires upon Next

TH Technology

Process Uploaded DataSingle Process of Type Process Uploaded DataList of ParametersFires upon Next

TH Technology

Session InfoCollectionsCharacter ColumnsNumeric ColumnsDate Columns

TH Technology

Data Load Wizard WizardPros

In-AppStraightforwardReusableNo Custom Code for Data Load into DB

ConsToo Many End User Steps Still Need Custom Processing Code for Staging to Tables

IF get P1 Finish by Loading Collection,

have a Two-Step File Upload for End

Users!

TH Technology

APEX Listener XLS Upload

File Browse ItemLoad File to WWV_FLOW_FILE

Parsing FileCustom Code per Use/Table/FormatGeneric File Upload Utility

Custom Process to Destination Tables

TH Technology

Parse File to CollectionLoads Collection to Specified Table

Table Exist or NotOptions

Reusable for Any Upload File/TableSaves One Part of Custom CodeCustom Processing from Stage Table Required

TH Technology

XLS Upload via APEX Listener

Must Deploy via APEX ListenerCurrent Release 1.1.3…

Allows Upload of XLS File into a Collection

Best Reference: (from RMOUG 2010)http://krisrice.blogspot.com/2010/02/another-apex-listener-ea-more-knobs-to.html

TH Technology

Install APEX ListenerDeployEdit apex_config.xmlAdd Configuration Choices:

<entry key="apex.excel2collection">true</entry><entry key="apex.excel2collection.onecollection">true</entry><entry key="apex.excel2collection.name">EXCEL_COLLECTION</entry><entry key="apex.excel2collection.useSheetName">true</entry>

TH Technology

Configuration Options:

Turn Feature On: <entry key="apex.excel2collection">true</entry>

One Collection per Worksheet:

<entry key= "apex.excel2collection.onecollection">true|false</entry>

TH Technology

Configuration Options:

Specify Name for All Collections: <entry key="apex.useSheetName">true|false</entry>

false == sequential sheet nametrue == uses sheet name

Specify Name of Collection for All Collections:

<entry key="apex.excel2collection.collection.name">MY_COLLECTION</entry>

TH Technology

File Browse ItemWWV_FLOW_FILES option

Page Item Button Name or REQUEST = XLS2COLLECTION

Report Region:SELECT * FROM apex_collectionsWHERE collection_name = ‘MY_COLLECTION’

TH Technology

“Files ending with .xls or .xlsx will be processed.”

BUT.xls processMacro-Enabled .xls process.XLSX do NOT Process as of APEX Listener 1.1.3.243.11.40.XLSX will Process as of v2

Coming Soon!

TH Technology

XLSX Upload via APEX Listener

TH Technology

XLSX Upload via APEX Listener

SEVERE: service exception:org.apache.poi.poifs.filesystem.OfficeXmlFileException: The supplied data appears to be in the Office 2007+ XML. You are calling the part of POI that deals with OLE2 Office Documents. You need to call a different part of POI to process this data (eg XSSF instead of HSSF)

TH Technology

Formulas Translate Literally

Ex: IF(E2="","",VLOOKUP(E2,$AE$4:$AF$218,2,FALSE))

TH Technology

STILL Need Custom Code to Process Data from Collection to Database Table(s)

TH Technology

ProsXLS File – No Need to Save As CSVStraightforwardReusableLess Custom Code

ConsAPEX Listener DeploymentEmbedded FormulasNo XLSX - May Need a Save As Step

Will Have XLSX in V2

TH Technology

APEX Websheets

Data Grid Allows Copy/PasteSpreadsheet-Like EditsData Grid w/in Page

Custom Code Required to Transfer Data to Main Schema Tables

TH Technology

Editable Data in Websheet

TH Technology

APEX Websheets

TH Technology

Pull Data From Websheet into App

Look at APEX$_WS Tables:APEX$_WS_ROW – the dataAPEX$_WS_HISTORY – iff you care about who changed what when

Need to process from SQL Dev table to “real” tables

TH Technology

APEX WebSheets

ProsCopy/PasteSpreadsheet-Like Entry, EditsEnd User Can Build

ConsCustom Code to Create Websheet, Data Grid?Custom Code Required to Pull Data into Schema ObjectsCustom Code to Maintain Worksheet / Data GridMore Code Than Other Solutions ?

TH Technology

Our Solution

ProblemRPT FilesXLS filesDifferent Formats200+, MonthlyUsers Cannot “Save as CSV”Must Support EditsVarying VolumeLimited Network

SolutionTabular Forms - 4Macro to Save as RPTExternal Tables

Generated RPTLIS Output RPT Files

Upload File Process (to be retired)

Custom ProcessingRPT – LI Systems RPT – Spreadsheet Templates

TH Technology

APEX Data Load Options

TH Technology

Nice to Have … Future?

1-Step Data Load WizardCopy/Paste ORUpload XLSX

Easiest for Non-Networked, Low(er) Volume End Users

TH Technology

Some Last Thoughts …

Forms Create/Edit One Row

Tabular FormsEnter Many Rows

Java Solution.JSP to load XLS directlyPresume this means XLSX too!

TH Technology

How Do I Load Data ?

APEX-based SolutionWeigh Requirements

Source DataEnd User CapabilitiesResourcesTime

No One-Size-Fits-All

Apply Common Sense

TH Technology

Common Sense Test …

Two men play 5 games of checkers. Each wins the same # of games. There are no ties.

How can this be?

They are not playing each other.

TH Technology

References

Upload File into APEXoraexplorer.com – Ittichaihttp://oraexplorer.com/2007/11/apex-to-upload-text-file-and-write-into/More at avdeo.com In Oracle Milieuhttp://avdeo.com/2008/05/21/uploading-excel-sheet-using-oracle-application-express-apex/

TH Technology

References

Oracle Data Integratorhttp://www.oracle.com/technetwork/middleware/data-integrator/overview/index.html

Oracle Database UtilitiesSQL Loader – External Tableshttp://www.oracle.com/technetwork/database/enterprise-edition/index-093639.html

SQL Developerhttp://www.oracle.com/technetwork/developer-tools/sql-developer/overview/index.html

TH Technology

References

APEX Listener XLS Upload – Kris Rice Bloghttp://krisrice.blogspot.com/2010/02/another-apex-listener-ea-more-knobs-to.html

A Java-and-PL/SQL based XLS to CSV Optionhttp://www.dynamicpsp.com/!go?ln=nfaq&faq=FAQ9

TH Technologykcannell@thtechnology.com

How Do I Load Data?

Let Me Count The Ways

Questions ?

Please fill out the

evaluations!

TH Technologykcannell@thtechnology.com

Thank You

Please fill out the

evaluations!

How Do I Load Data?

Let Me Count The Ways

Karen Cannell kcannell@thtechnology.com

Documents

Transcript of Karen Cannell kcannell@thtechnology.com

A process-based, terrestrial biosphere model of ecosystem dynamics (Hybrid v. 3.0) A. D. Friend, A.K. Stevens, R.G. Knox, M.G.R. Cannell. Ecological Modelling.

Congregational Development Workshop B.H. Carroll Institutecf2015.bhcarroll.edu/files/session-1-congregational... · 2015. 11. 4. · B.H. Carroll Institute . Linda Cannell, November

Cannell Capital Envivio Investor presentation July 20 2015

TeleCommunication Systems, Inc. & Cannell Capital LLC ...cannellcapital.com/wp-content/uploads/TSYS.pdfPatent Monetization Program Colossal Failure “Our collaboration with Ocean

WA#SEABIRDRESCUE:APICTORIALPENGUIN ADVENTURE# · Karlene’Bain’! ! Department)of)Parks)andWildlife! Walpole’Nornalup’National’Parks’Association ! Dr#Belinda#Cannell#! Murdoch(University,(Perth,(WA

Gene Therapy Maren Cannell Daniel Garzon Mariah Payne.

[XLS] · Web viewann pane candeland Candlish cani 37 croxteth rd cannell 13 bolton st george alfred Cannell william robert cannon 165 up parliament st thomas joseph cannor 151 up

Assessment of Ministry Preparation-Harriscf2015.bhcarroll.edu/files/session-2-toward-a... · Baylor’s Truett Divinity School; Linda Cannell, professor of Educational Ministries

Vitamin D and Autism A look at the evidence that links the two John J Cannell, MD /// Vitamin D Council.

Perkins 202 Dr. Michelle Crary – Staff Development and Accountability Coordinator Nori Cannell – Director – Guidance & Career and Technical Education.

Going to the Grid - TH TECHNOLOGY - Oracle … to the Grid: What the APEX 5.1 Interactive Grid Means or You and Your Users Karen Cannell kcannell@thtechnology.com TH Technology TH

Sheila Cannell Project Director

Indicators of Climate Change in the UK Melvin G R Cannell.

Worker Co-operatives and Trade Unions Bob Cannell bob@suma.coop.

APEX 5 IR Guts & Performance - TH TECHNOLOGYAPEX 5 IR Guts & Performance Karen Cannell TH Technology kcannell@thtechnology.com . TH Technology ... APEX 5 Interactive Reports: Performance

Grid Essentials - Amazon S3 · Grid Essentials: APEX 5.1 Interactive Grid Karen Cannell ... • Reengineered –CSS and JS, Widgets ... Layout. Interactive Grid: Essentials to Build

Vitamin D in Orthopedics John J Cannell, MD Executive Director, Vitamin D Council.

Hunt, D. F., Cannell, G., Davenhill, N. A., Horsford, S. A ...

Staff: Enhancing Teaching Rowena Pelik, Edinburgh Napier University Together with: Dr Pete Cannell, OU Scotland and topic lead.

Microbial Detection in Surface Waters Jarod Gregory ACCEND: Chemical Engineering B.S. & Environmental Engineering M.S. Jon Cannell Chemical Engineering.