INT-2: XQuery Levels the Data Integration Playing Field Carlo (Minollo) Innocenti DataDirect XML...

INT-2: XQuery Levels the Data Integration Playing Field

Carlo (Minollo) InnocentiDataDirect XML Technologies,

Program Manager

© 2007 Progress Software Corporation2 INT-2: XQuery Levels the Data Integration Playing Field

Agenda

The Problem Why XQuery XQuery for Java API (XQJ) XQuery for Data Integration XQuery Demos, Code Walk-throughs Summary


The user submits a request for a report

about their stock holdings

The web server needs to fetch user’s personal data, stock holdings and live stock data to compile a report to send

back to the userA public service offers

live (delayed) stock prices

A Typical Data Integration Problem

DB1 DB2 DB3

Different repositories are used for different parts of the information necessary to create a stock holdings report


TomCat server

Stock price web service

dBASE IV OpenEdge®

DatabaseSQL Server

Some implementation constraints…

HTML SOAPthrough AXIS

dBASE IVAPIs

JDBC

Java™/JSP codeaccessing the various

Java APIs and generating the HTML report

Java OpenClient or

JDBC


Data Source

Data Source

Data Source

Data Source

Data Source

Data Consumer

Data Consumer

Data Consumer

Data Consumer

Data Consumer

Data Source

Data Source

Data Source

Data Source

Data SourceEDI Message Web Service

RDBMSOpenEdgeDatabase

XML Document

<XML>

Data Access Layer

AJAXClient

DynamicHTMLClient

WebServiceClient

PublishingApps

RESTClient

A dangerous approach


Data Source

Data Source

Data Source

Data Source

Data Source

Data Consumer

Data Consumer

Data Consumer

Data Consumer

Data Consumer

Data Source

Data Source

Data Source

Data Source



XML Document

<XML>

AJAXClient

DynamicHTMLClient

WebServiceClient

PublishingApps

RESTClient

XML

XQuery

The XQuery Vision


Agenda



What is XQuery?

W3C Query Language for XML

• Native XML Programming Language• “The SQL for XML”• Designed to query, process, and create XML

High level functionality

• Find anything in an XML structure • Querying and combining data • Creating XML structures• Functions• User-defined function libraries


XQuery Basics

Path Expressions: Finding Data

doc("holdings.xml")/holdings/entry

FLWOR Expressions: Querying and Combining Data

for $h in doc("holdings.xml")/holdings/holding, $c in doc("companies.xml")/companies/companywhere $h/userid = "Minollo" and $c/ticker = $h/stocktickerreturn $c/name


XQuery Basics

Path Expressions, FLWOR Expressions, and XML Constructors

for $h in doc("holdings.xml")/holdings/entry, $c in doc("companies.xml")/companies/companywhere $h/userid = "Minollo" and $c/ticker = $h/stocktickerreturn <company ticker="{ $c/ticker }"> { $c/companyname } { $c/annualrevenues } </company>


Functions and Modules

A Function in a Library Module

module namespace stock="http://tagsalad.com/stocks";

declare function stock:companies($user as xs:string) { for $h in doc("holdings")/holdings/entry, $c in doc("companies")/companies/company where $h/userid = $user and $c/ticker = $h/stockticker return <company ticker="{ $c/ticker }">

… </company>};


Functions and Modules (2)

Importing and Using a Library Module

import module namespace stock="http://tagsalad.com/stocks";

stock:companies("Minollo")


Why XQuery?

Native Support for XML• Conventional programming and query languages are

not designed for XML• No more parse, navigate, cast, repeat – XML is the

native datatype

Designed for Data Integration• Native XML and non-XML data can be used the same

way• Vastly simplifies development when input includes

XML, relational, EDI…– Requires support from implementation for the data

sources you need


Why XQuery?

XML Output is Directly Useful• XML is becoming the industry standard for data

exchange• Dynamic Web Sites• Publishing Applications• Web Messages• We normally don’t exchange SQL tables or present

them to users!

Programmer Productivity• Readable, declarative code – transparent, easier to

maintain• 7 to 20 times less code than Java + SQL + JDBC +

XML APIs


Why XQuery?

Performance• Declarative code – can be optimized by the

XQuery Engine

• Relational database vendors and experts very involved in the design

• Actually performance depends on the implementation …


Benefits of XQuery

Data Integration is harder without XQuery!• Every data source is different

• Many applications use several languages and APIs to address data sources (e.g. Java+JDBC+DOM, SQL, Perl, XSLT…)

• Mediating among data sources accounts for a lot of code

• XQuery treats all data sources as XML


Benefits of XQuery

Processing XML is harder without XQuery!• Most programming languages don’t know XML

structures• Parse, navigate, cast, repeat• XML is the native data structure for XQuery

XML Reporting is harder without XQuery!• XML input and output may have very complex structure• Many different desired XML outputs• Data Integration, Native XML Processing are needed• XQuery gives full query processing for any XML input

and output


Agenda



What is XQJ?

XQuery API for Java (XQJ) – JSR 225 “The JDBC for XQuery”


Benefits of XQJ

Industry Standard, similar to JDBC• No need to learn a new proprietary API for each

product and each version

• Can build on existing JDBC knowledge

Lets XQuery fit into any Java architecture Queries can be created or parameterized at run-time

• Example: A portfolio for a given user at a given date

Interfaces are designed for use in J2EE applications• Example: Results can be retrieved as DOM, SAX,

StAX, or text


Agenda



An XQuery architecture


DataDirect XQuery

**

High performance Scalable Embeddable Plugs into any Java

architecture Accesses almost any data

source No dependency on servers Standards-based


HOLDINGS

USERID TICKER SHARES

Jonathan PRGS 23

Minollo PRGS 4000000

Jonathan AMZN 3000

Minollo AMZN 3000

<portfolio > <company ticker="AMZN"> <companyname>Amazon.com, Inc.</companyname> <annualrevenues>7780</annualrevenues> </company> <company ticker="EBAY"> <companyname>eBay Inc.</companyname> <annualrevenues>22600</annualrevenues> </company> <company ticker="IBM"> <companyname>Int'l Business Machines C</companyname> <annualrevenues>128200</annualrevenues> </company> <company ticker="PRGS"> <companyname>Progress Software</companyname> <annualrevenues>493.4</annualrevenues> </company></portfolio>

COMPANIES

TICKER NAME REVENUES

AMZN Amazon.com, Inc. 7780

EBAY eBay Inc. 22600

PRGSProgress Software 493.4

YHOO Yahoo! Inc. 10700

Highly optimized for relational sources Minimizes retrieval of data

• No more rows than needed• No more columns than needed

Uses database functionality • Joins• Sorting• Etc..

Optimizes for each SQL dialect Efficient JDBC retrieval

• Embeds DataDirect JDBC technology• Optimizations added to support XQuery

Supports incremental retrieval Optimizes for XML hierarchies

• Sort-merge algorithm• Minimal cost of XML construction

Leverages SQL library Supports hints

XQuery can be fast for relational data!


XQuery can be fast for XML files!

<?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <soap:Body> <GetQuotesResponse xmlns="http://swanandmokashi.com"> <GetQuotesResult> <Quote> <CompanyName>APPLE COMPUTER</CompanyName> <StockTicker>AAPL</StockTicker> <StockQuote>74.17</StockQuote> <LastUpdated>9/14/2006 4:01pm</LastUpdated> <Change>1.17</Change> <PercentChange>1.82%</PercentChange> <OpenPrice>N/A</OpenPrice> <DayHighPrice>N/A</DayHighPrice> <DayLowPrice>N/A</DayLowPrice> <Volume>0</Volume> <MarketCap>63.266B</MarketCap> <YearRange>47.87 - 86.40</YearRange> <ExDividendDate>21-Nov-95</ExDividendDate> <DividendYield>N/A</DividendYield> <DividendPerShare>0.00</DividendPerShare> </Quote> </GetQuotesResult> </GetQuotesResponse> </soap:Body></soap:Envelope>

General XQuery rewrites• Constant-folding, elimination of common

sub-expressions, loop rewrites, ordering rewrites, etc…

Document projection• XML construction accounts for much of

the cost• Don’t build parts of the document that

the query doesn’t need! Document streaming

• Discard parts of the document when no longer needed

• Makes memory usage near constant with size of file

Multiple Gigabytes can be queried

XQuery can use XML Converters

• EDI File

ISA+00+DATADIRECT+00+STYLUS2006+01+DATA DIRECT +01+STYLUS STUDIO +060504+1212+~+00503+200654321+0+I+:'GS+BF+DATADIRECT+STYLUS2006+20060504+121212+256+X+005030'ST+105+3389'BGN+28+1024+20060504+121212+GM'NM1+2L+4+Progress Software Corporation'N3+14 Oak Park Drive'N4+Bedford+MA+01730+US+AA'REF+1Z+PRGS'NM1+2L+4+Apple Computer, Inc.'N3+1 Infinite Loop'N4+Cupertino+CA+95014+US+AA'REF+1Z+AAPL'SE+11+3389'GE+1+256'IEA+1+200654321'

doc("adapter://EDI?ticker-request.edi")

<X12> <ISA> <ISA01>00</ISA01> <ISA02>DATADIRECT</ISA02> <ISA03>00</ISA03> <ISA04>STYLUS2006</ISA04> <ISA05>01</ISA05> <ISA06>DATA DIRECT </ISA06> <ISA07>01</ISA07> <ISA08>STYLUS STUDIO </ISA08> <ISA09>060504</ISA09> <ISA10>1212</ISA10> <ISA11>~</ISA11> <ISA12>00503</ISA12> <ISA13>200654321</ISA13> <ISA14>0</ISA14> <ISA15>I</ISA15> <ISA16>:</ISA16> </ISA> <GS> <GS01>BF</GS01> <GS02>DATADIRECT</GS02> <GS03>STYLUS2006</GS03> <GS04>20060504</GS04> <GS05>121212</GS05> <GS06>256</GS06> <GS07>X</GS07> <GS08>005030</GS08> </GS>

Convert non-XML format to XML on-the-fly!

• EDI message types• Comma-delimited or tab-delimited files• dBase• RTF• mbox• Batch conversions are supported• Custom conversions


declare function local:amazon-listing($isbn){ <tns:Request> <tns:Condition>All</tns:Condition> <tns:DeliveryMethod>Ship</tns:DeliveryMethod> <tns:FutureLaunchDate/> <tns:IdType>ASIN</tns:IdType> <tns:ItemId>{ $isbn }</tns:ItemId> <tns:ResponseGroup>Medium</tns:ResponseGroup> </tns:Request>};

let $loc := <location address="http://soap.amazon.com/onca/soap?Service=AWSECommerceService" soapaction="http://soap.amazon.com" />let $payload := local:amazon-listing("0395518482")return ws:call($loc, $payload)

Leverage existing SOA architecture in queries!

• Integrate queries with web services• Easily generate complex web service requests• Vastly increases the reach of your queries

XQuery can access Web Services


Questions

so far?



Agenda


TomCat server


dBASE IV OpenEdgeDatabase

SQL Server

A Data Integration problem

HTML SOAPthrough AXIS

dBASE IVAPIs

JDBC

Java/JSP codeaccessing the various

Java APIs and generating the HTML report

Java OpenClient or

JDBC


Data Source

Data Source

Data Source

Data Source

Data Source

Data Consumer

Data Consumer

Data Consumer

Data Consumer

Data Consumer

Data Source

Data Source

Data Source

Data Source



XML Document

<XML>

Data Access Layer

AJAXClient

DynamicHTMLClient

WebServiceClient

PublishingApps

RESTClient

A dangerous approach


Data Source

Data Source

Data Source

Data Source

Data Source

Data Consumer

Data Consumer

Data Consumer

Data Consumer

Data Consumer

Data Source

Data Source

Data Source

Data Source



XML Document

<XML>

AJAXClient

DynamicHTMLClient

WebServiceClient

PublishingApps

RESTClient

XML

XQuery

The XQuery Vision


The DataDirect XQuery Solution

TomCat server


dBASE IV ProgressDatabase

SQL Server

HTML


Step by step

XQuery to aggregate data from the multiple data sources

XQuery to publish an HTML or XSL-FO (PDF) report directly

Pipelining multiple XQueries with validation steps

Exposing an XQuery Web Service and consuming it from OpenEdge®


Step by step




Exposing an XQuery Web Service and consuming it from OpenEdge


Step by step






Agenda



Benefits of XQuery

Data Integration is harder without XQuery!• Every data source is different

• Many applications use several languages and APIs to address data sources (e.g. Java+JDBC+DOM, SQL, Perl, XSLT…)

• Mediating among data sources accounts for a lot of code

• XQuery treats all data sources as XML


Benefits of XQuery

Processing XML is harder without XQuery!• Most programming languages don’t know XML

structures• Parse, navigate, cast, repeat• XML is the native data structure for XQuery

XML Reporting is harder without XQuery!• XML input and output may have very complex structure• Many different desired XML outputs• Data Integration, Native XML Processing are needed• XQuery gives full query processing for any XML input

and output


DataDirect XQuery


Getting Started …

Examples & Tutorialshttp://www.xquery.com• XQuery Tutorial

• XQJ Tutorial

• DataDirect XQuery Tutorial


Questions?


Thank you foryour time

INT-2: XQuery Levels the Data Integration Playing Field Carlo (Minollo) Innocenti DataDirect XML...

Documents

Transcript of INT-2: XQuery Levels the Data Integration Playing Field Carlo (Minollo) Innocenti DataDirect XML...