Mazda Use of Third Generation Xml Tools

Third Generation XML Tools

Michael Leventhal

[email protected]

mailto:[email protected]

mailto:[email protected]

Generations

Ubiquity InfrastructureProgramming Standard

CorrectnessPerformance

Simplification

Commodity

Cost

3rd

2nd

1st

Reliability

Delivery Application Repository

Standardization Data DescriptionToolkits Transformation

Interfaces

Standards Presentation

Tiers

Tools Parsing Browsing Editing

Agenda Characterizing XML Tool Evolution – Semantic

Space of Change Key Ideas Underlying the Next Generation

XML Programming Models – Universal XML Development

High Performance XML, e.g., Streaming XML as Infrastructure – Reliability at the core of the

computing platform, merging of hardware and software, integration into the application framework

Whither else? Your feedback.

Schema to Java Generation

•Lessens development time and resource needs through removal of required of XML experience needed for generic XML APIs such as DOM and SAX.

•Superior type checking; most constraint and structure checking is done at compilation time as the Java classes have been generated from the schema.

•Performance worse than DOM with Validation

Generation of Java APIs from XML SchemasDevelopers program against these Schema-specific APIs

Schema to Java - Tools

•JAXB Implementation http://java.sun.com/xml/

•Castor http://www.castor.org/

•XML Binder for Java http://www.breezefactor.com

•Zeus http://zeus.enhydra.org/

•Schema2Java http://www.creativescience.com

•DocSOAP XDK XGen http://www.commerceone.com/developer

http://www.castor.org/

http://www.breezefactor.com/

http://zeus.enhydra.org/

http://www.creativescience.com/

Schema to Java Generation

XMLSchema Java

Classes

Generates

XMLInstance Java

Objects

Converts

XGen DTSchema

Compiler

XMLSchema

Java Classes

XGen RTRuntime

XMLInstance Java

Objects

Schema to Java Example<xsd:schema

targetNamespace="rrn:com:commerceone:schemas/xdk/samples/Car.xsd"xmlns="rrn:com:commerceone:schemas/xdk/samples/Car.xsd"xmlns:xsd="http://www.w3.org/2001/XMLSchema">

<xsd:element name="Car" type="CarType"/>

<xsd:complexType name="CarType"><xsd:sequence>

<xsd:element name="Manufacturer" type="xsd:string"/><xsd:element name="Model" type="xsd:string"/><xsd:element name="Year" type="xsd:int"/><xsd:element name="Color" type="xsd:string"/>

</xsd:sequence><xsd:attribute name="ForSale" type="xsd:boolean"/>

</xsd:complexType></xsd:schema>

<?xml version="1.0"?><prefix_0:Car xmlns:prefix_0="rrn:com.commerceone:schemas/xdk/samples/Car.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ForSale="true"><Manufacturer>Mazda</Manufacturer><Model>Miata MX-5</Model><Year>91</Year><Color>Blue</Color></prefix_0:Car>

<xsd:element name="Car" type="CarType"/>

<xsd:complexType name="CarType"> <xsd:sequence> <xsd:element name="Manufacturer" type="xsd:string"/> <xsd:element name="Model" type="xsd:string"/> <xsd:element name="Year" type="xsd:int"/> <xsd:element name="Color" type="xsd:string"/> </xsd:sequence> <xsd:attribute name="ForSale" type="xsd:boolean"/></xsd:complexType></xsd:schema>

<?xml version="1.0"?><prefix_0:Car xmlns:prefix_0="rrn:…/Car.xsd“ ForSale="true"><Manufacturer>Mazda</Manufacturer><Model>Miata MX-5</Model><Year>91</Year><Color>Blue</Color></prefix_0:Car>

Schema to Java Examplepackage com.commerceone.samples.xdk; import com.commerceone.schemas.xdk.samples.Car_xsd.Car;import com.commerceone.schemas.xdk.samples.Car_xsd.types.CarType; import com.commerceone.xdk.castor.types.*;import com.commerceone.xdk.castor.excp.ConstraintValidationException; import java.io.OutputStreamWriter;import java.io.Writer; public class CarCreator{ public static void main(String[] args) { try { CarCreator cc = new CarCreator(); Car car = cc.makeCar(); car.marshal(new OutputStreamWriter(System.out)); } catch(Exception e) { e.printStackTrace(); } }

public Car makeCar() throws ConstraintValidationException { Car car = new Car(); CarType ct = car.getCarTypeComplexType(); ct.setColor(new XString("Blue")); ct.setManufacturer(new XString("Mazda")); ct.setModel(new XString("Miata MX-5")); ct.setYear(new XInt(91)); ct.setForSale_Attribute(new XBoolean(true)); return car; }}

public class CarCreator{ public static void main(String[] args) { try { CarCreator cc = new CarCreator(); Car car = cc.makeCar(); car.marshal(new OutputStreamWriter(System.out)); } catch(Exception e) { e.printStackTrace(); } }

public Car makeCar()throws ConstraintValidationException { Car car = new Car(); CarType ct = car.getCarTypeComplexType(); ct.setColor(new XString("Blue")); ct.setManufacturer(new XString("Mazda")); ct.setModel(new XString("Miata MX-5")); ct.setYear(new XInt(91)); ct.setForSale_Attribute(new XBoolean(true)); return car; } }

Schema to Java in Web Services

SOAP-RPC<->Java DocSOAP<->Java

Schema to Java – Schema Feature Coverage

Overall Results

94%

64% 61% 54%

0%

10%

20%30%

40%

50%

60%

70%80%

90%

100%

XGen Breeze JAXB Castor

Schema to Java – Schema Feature Coverage

0%

20%

40%

60%

80%

100%

120%

Attribu

tes

Global

Elemen

t

Comple

xTyp

e

Simple

Type

Conte

nt M

odel

Group

Names

pace

Other

Breeze

Castor

JAXB

XGen

Schema to Java - Performance

Size Marshalling Unmarshalling Round Trip

First run Average after first run



1 Mb 1.8 1.3 19.9 10.4 21.7 11.7

3 Mb 4.9 3.7 41.0 29.9 46.0 33.6

5 Mb 8.9 7.0 60.1 50.5 69.0 57.5

Size Marshalling Unmarshalling Round Trip




1 Mb 1.0 0.4 5.8 3.5 6.8 3.9

3 Mb 3.0 1.3 13.2 10.6 16.2 11.9

5 Mb 4.2 3.2 20.8 17.5 25.1 20.7

DOM with Validation

Illustrative Schema to Java Tool

CPU

Secs

Schema to Java - Performance

Init Unmarshalling Marshalling

Peak Done Peak Done Peak Done

xCBL 1 meg 1730 1673 9845 9553 15455 9574

xCBL 3 meg 1734 1673 20764 17657 27765 17677

xCBL 5 meg 1769 1674 31556 25746 39183 25766

xCBL3.0 1 meg 1802 1674 23344 16835 24751 16875

xCBL3.0 3 meg 1790 1673 33142 25830 39134 25870

xCBL3.0 5 meg 1803 1673 45113 34700 52206 34741

MemoryKbs

Illustrative Schema to Java Tool

Rule of thumb for DOM: Memory use ~4X size of instance

Programming Models: XOBE

XMLSchema

XMLObjects

Instances of

Generated byXML constructors

http://www.research.avayalabs.com/user/wadler/planx/planx-eproceed/papers/E00-699879232.pdf

See Kempa, Martin and Linnemann, Volker, “On XML Objects”

Car makeCar() {Car myCar;myCar = <Car> <Manufacturer>Mazda</Manufacturer> <Model>Miata MX-5</Model> <Year>91</Year> <Color>Blue</Color> </Car>;Return myCar;}

•Uses XML Schema directly as definition of classes for XML objects; construct XML objects with parameterized XML expressions

•Eliminates distinction between string and object representation of XML documents

•Compiler statically guarantees validity; no need for special runtime checking or checking at marshalling

•Similar: ECMAScript (BEA XML Scripting), Water (http://www.waterlang.org)

Car makeCar() {Car myCar;myCar = <Car> <Manufacturer>Mazda</Manufacturer> <Model>Miata MX-5</Model> <Year>91</Year> <Color>Blue</Color> </Car>;Return myCar;

Stream Processing - ToolsBuilds complex XML processing on SAX events to achieve high-performance and ultra-low memory usage compared to tree-construction approaches•XSLT-style Transformation

•STX http://stx.sourceforge.net/ Stream Index (SIX)

•XST Smart Streaming

•XPath

•XMLTK http://sourceforge.net/projects/xmltk

•χαος http://www.research.ibm.com/xaos/applications.html

•SOAP Processing

•Axis http://ws.apache.org/axis/

•DocSOAP XDK http://www.commerceone.com/developer

Stream Processing-Algorithms

•Stream processing is not inherently faster than DOM tree-based approaches; improperly applied it can be slower than DOM. It should always be more memory-efficient than DOM. Some approaches consist mainly of limiting the application domain to problems that can usually be efficiently solved by a simple streaming algorithm. Example: XPP, simplified XML parser.

•Stream processing is not necessarily just using SAX events. It is a different approach to solving computational problems which uses a variety of sophisticated algorithms.

•SIX (XMLTK) uses event marker indices.

•Smart Streaming (XST) combines DOM and SAX and analyzing when to stream

•X-dag (χαος) converts backward constraints into forward constraints

Stream Processing – Simplifying the problem

Source: IBM Developer Works, XML in Java: Document models, Part 1: Performance,Dennis M. Sosnoski, September 2001

XPP: XML Pull Parser•Streaming•XML Subset•No Validation•No entities, PIs, Comments•Lazy, on-demand tree•Supports namespaces

http://www.extreme.indiana.edu/xgws/xsoap/xpp/

Document Modify

Stream Processing - XMLTK

StreamFunction

Description

xsort Sorts an XML Stream

xagg Computes an aggregate

xnest Groups elements based on key equality or number

xflatten Flattens collections

xdelete Removes elements or attributes

xpair Replicates an element multiple times,Pairing it with each element in a collection

xhead Retains only a prefix of a collection

xtail Retains only a suffix of a collection

file2xml Generates an XML stream for the dir filedirectory hierarchy

XMLTK Stream Processing Tools

Stream Processing - XMLTK

Data Size (KB)

Xalan (sec) Xsort (sec)

0.41 0.08 0.00

4.91 0.10 0.00

76.22 0.29 0.03

991.79 2.78 0.35

9671.42 29.42 3.54

100964.43 - 35.52

1009653.71 - 358.47

XMLTK XSort Performance vs Xalan Sort

•Sophisticated streaming algorithm can produce both low memory and high performance

Stream IndeX Algorithm

•Additional binary stream consisting of beginOffset, endOffset pairs for each XML element

•Allows characters to be skipped

•Small offsets can be skipped to reduce SIX stream and number of jumps

Smart Streaming

<ContractReferences> <ListOfContract> <Contract> <ContractID> <Identifier> <Agency> …… </Agency> <Ident>0088-858398</Ident> </Identifier> </ListOfContract>

</ContractReferences>

<Identifier> <Agency> …… <Ident>0088-858396</Ident></Identifier>

<Agency> ……</Agency><Ident>0088-858398</Ident>

Source XML Document<ContractReferences> <ListOfContract> <Contract> <ContractID> <Identifier> <Agency> …… </Agency> <Ident>0088-858398</Ident> </Identifier> </ListOfContract></ContractReferences>

<Identifier> <Agency> …… </Agency> <Ident>0088-858396</Ident></Identifier>

<Agency> ……</Agency>

<Ident>0088858398</Ident>

Target XML Document

Smart Streaming

<ContractReferences> <ListOfContract> <Contract> <ContractID> <Identifier> <Agency> …… </Agency> <Ident>0088-858398</Ident> </Identifier> </ListOfContract></ContractReferences>

<Identifier> <Agency> …… <Ident>0088-858396</Ident></Identifier>

<Agency> ……</Agency><Ident>0088-858398</Ident>

Streaming

Streaming

Buffering

Streaming

Smart Streaming - XST

<xst:templates version="1.0" …… <xst:template name="Agency" deferred="true"> <Ident><xst:val-of select="$Ident“></Ident> <xst:var name="AgencyDescription"/> <xst:if test="$AgencyDescription"> <AgencyDescription> <xst:val-of select="$AgencyDescription"/> </AgencyDescription> </xst:if> </xst:template> …… </xst:templates>

<xst:templates version="1.0" …… <xst:template name="Ident"> <xst:var name="Ident"/> </xst:template> ……</xst:templates>

<?xml version="1.0"?><xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"><xsl:template match="/"> <xsl:apply-templates select = "*"/></xsl:template><xsl:template match= "*" ><xsl:copy><xsl:copy-of select="@*"/> <xsl:apply-templates/> </xsl:copy></xsl:template>

<xsl:template match="Agency"> <Agency> <Ident><xsl:value-of select="//Ident"/></Ident> <xsl:variable name="AgencyDescription" select="//AgencyDescription"/> <xsl:if test="$AgencyDescription"> <AgencyDescription><xsl:value-of select="$AgencyDescription"/></AgencyDescription> </xsl:if> </Agency></xsl:template>

<xsl:template match="Ident"/></xsl:stylesheet>

XST XSLT

Streaming – Transformation Automation

Transformation Rules

SchemaAnalysis

Template Generator

StreamingTemplates

Template GenerationTemplate generation process can automate creation of streaming transformations.

•Transformation rules can be created in a visual tool like Contivo.

•Schema is analyzed to augment mapping rules.

•May be unnecessary to introduce YATL (Yet Another Transformation Language).

Streaming – Memory/CPU Performance

CPU Usage RatioXST XSLT XSLT/XST

1 3.065 5.608 1.8296900492 5.378 10.275 1.9105615473 8.122 15.362 1.8914060584 11.106 21.521 1.9377813795 14.04 30.945 2.2040598296 16.784 43.994 2.6211868457 20.41 320.461 15.701175898 23.624 out of memory9 26.458

10 29.87320 66.70630 106.98440 144.01750 186.67960 236.7970 out of memory

CPU Usage (s)No. Of ThreadsFile Size (K)

2000

CPU Usage Ratio XSLT/XST vs. No. of Threads (File size: 2000K)

0

0.5

1

1.5

2

2.5

3

1 2 3 4 5 6

CPU Usage RatioXSLT/XST

CPU Usage Ratio XSLT/XST vs. No. of Threads

(File size: 10K)

0

0.5

1

1.5

1

#REF!


(File size: 1000K)

0

2

1

#REF!


(File size: 200K)

0

0.5

1

1.5

1

#REF!


(File size: 20K)

0

2

1

#REF!

Streaming – Memory/CPU Performance

CPU Usage RatioXST XSLT XSLT/XST

1 0.521 0.832 1.5969289832 0.541 0.841 1.5545286513 0.601 0.941 1.5657237944 0.59 0.971 1.6457627125 0.651 1.042 1.6006144396 0.701 1.191 1.6990014277 0.741 1.191 1.6072874498 0.741 1.262 1.7031039149 0.761 1.252 1.645203679

10 0.812 1.382 1.70197044320 1.171 2.383 2.0350128130 1.653 2.704 1.63581367240 2.223 3.285 1.47773279450 2.563 3.775 1.4728833460 2.856 4.486 1.57072829170 3.265 5.017 1.53660030680 3.965 6.529 1.6466582690 3.986 6.769 1.698193678

100 4.397 6.94 1.578348874200 8.472 11.596 1.368744098210 8.943 17.054 1.906966342220 10.715 24.041 2.243677088230 12.628 34.35 2.720145708240 out of memory out of memory

10

File Size (K) No. Of ThreadsCPU Usage (s) CPU Us age Ratio XSLT/XST v s . No. of Threads

(Fi le s iz e: 2000K)

0

1

2

1

#REF!


(File size: 10K)

0

0.5

1

1.5

2

2.5

3

1 6 20 70 210

CPU Usage RatioXSLT/XST

CPU Us age Ratio XSLT/XST v s . No. of Threads


0

2

1

#REF!



0

1

2

1

#REF!


(Fi le s iz e: 20K)

0

2

1

#REF!

Web Services Infrastructure

By one estimate, less than 2% of all enterprise networks traffic in 2002 was XML. That will increase to about 25% of all LAN traffic by 2006.

Source: ZapThink, “Solving The Enterprise XML Processing Problem”

Much of that traffic will be due to Web Services. XML-based SOAP should replace most other message-oriented protocols.


Implications for 3rd Generation Tools

•XML has to become protocol-level safe and truly interoperable. There is a long way to go but W3C, WS-I and other consortiums are focusing intently on this.

•High performance and scalability will be at a premium.

•Market will bear higher prices for high-end tools.

•Interoperability and performance-related simplifications of XML and related standards will receive a second look.


Major Interoperability Issues Faced Today - Many

•Support for XML Schema

•Support for Namespaces

•Canonicalization of XML Documents

•XML Signing

•Links and References between Message Parts

•Handling of Document Subsets

•WSDL Binding to SOAP


Case Study: DocSOAP XDK Embedded in the Commerce One Conductor Web Services Platform

•First implementation of an XML document-centric Web Services toolkit

•High performance handling of large XML document message parts

•Highest level of support of XML Schema, namespaces, document fragments, inter-message references

•Major design emphasis on ensuring integrity of messages

Web Service Infrastructure

transportTransport headers

SOAP Header

SOAP Payload

Header BlockDOCSOAP

DOCUMENT

DOCUMENT

FRAMEWORK

FRAMEWORK

XGen RT

UniParser

APPLICATION

GeneratedSchema

API

DOMInterface

Schema

DOM

BeanInstance

Schema

unmarshall marshallvalidate

XGen DT

Commerce OneConductor


Images provided by and used with the permission of DataPower Technology

Case Study: DataPower Network Devices (XS40 XML Security Gateway and XA3 XML Accelerator)

• High-speed XML Encryption and Signing and XSLT transformation implemented in hardware and firmware

•Integrated with Altova and Excelon IDEs – applications leverage a common runtime in the network, software and hardware converge


Common Layer of XML Application Services, Development Tools and Runtimes in Software Frameworks such as .NET and J2EE

Sun JavaX – the JAXes

•JAXM - Messaging

•JAXP - Parsing

•JAX-RPC – Distributed Processing

•JAXB – Schema to Java Generation

•JAXR – XML Registries

Not Hot? (Debatable)

•Native XML Databases/XML-DBMS Convergence

•Publishing/XSL-FO

•Client apps/ECMAScript-JavaScript/ embedded Web Services

•Browsable XML

•Semantic Web/RDF/Topic Maps/Ontologies

Presentation

An HTML version of this presentation is available at:http://www.textscience.com/papers/ThirdGenerationXMLTools.htmlPowerpoint at:http://www.textscience.com/papers/ThirdGenerationXMLTools.ppt

The presenter, Michael Leventhal, can be reached at [email protected].

Mazda Use of Third Generation Xml Tools

Automotive

Transcript of Mazda Use of Third Generation Xml Tools