Download File

INT420 - Federated Database Solutions with Sybase Middleware

Lynn Weatherwax

Director of Engineering – Data IntegrationInfrastructure Platform Group, [email protected]

2

Abstract

Many companies today seek optimized ways to access structured and unstructured data in their enterprises, without extensive programming or costly database consolidation.

Virtual databases with links to distributed heterogeneous sources enable applications to consistently access and combine data in new ways.

Sybase solves this problem while supporting the optimized query capabilities and scalability of an enterprise class relational database, via ASE, Enterprise Connect Data Access, Enhanced Full Text Search, and ASE Web Services.

Leverage SQL and XML standards to make your data work for you in your enterprise.

3

Agenda

Enterprise Information Integration (EII)

Strategies for Integrating Federated Data

Federated Database Access

Document Access - Full Text Search

XML

Web Services

Summary and Resources

4


“Enterprise information integration (EII) is seeing a new surge of interest in 2003, largely because its virtual and/or federated approach to data integration has potential benefits.

Compared to other data integration technologies, solutions based on a vendor’s EII tool tend to be quick to deploy, inexpensive in terms of licensing and very flexible since they’re based on a metadata-based view of enterprise data.

In addition, EII relies on distributed queries to effect data integration in near real-time, thus delivering fresher data than traditional batch methods.

Many companies are evaluating vendors’ EII products in 2003, so 2004 should see an upsurge in EII implementations.”

Philip Russom, IdeaByte

5

Factors Driving EII Today

Changes in company structure Increased competition, requiring better intelligence Government regulations

Patriot Act Sarbanes-Oxley Act HIPPA

6

Every enterprise has federated data, due to

• Mergers and acquisitions

• Legacy mainframe applications

• Applications required to work with specific databases

EII Challenge

XML

DB2 UDB

SybaseMicrosoft

InformixVSAM

Oracle

Adabas

Documents

7

Option: Migrate All Data/Apps to One Database…

Sometimes this is feasible, but often there are time and $$ restrictions…

And what about the next acquisition, or the next undiscovered information island?

XML

DB2 UDB

My CentralizedDatabase

Microsoft

Informix

VSAM

Oracle

Adabas

Documents

8

Enterprise Information Integration Requirements

Leverage existing hardware and software investment Ensure secure access Integrate legacy data and applications while they

remain in place – short-term or long-term Metadata transparently aggregates data, no

programming required Components that work together, forming a complete,

end-to-end solution Supported in standard client/server environments, yet

fully e-business ready (supports XML, Java, etc.)

9

Agenda





XML

Web Services


10


Data Access Document Access - Full Text Search Messaging/Data Replication XML Web Services

11

Enterprise Connect Data AccessEnterprise Connect Data Access

Make all your enterprise data look like one database

Any Any mainframe mainframe datadata

OracleOracleMS SQL ServerMS SQL ServerSybaseSybaseDB2 UDB for OS/390, NT/UNIXDB2 UDB for OS/390, NT/UNIXAS/400AS/400InformixInformixODBC-enabled databasesODBC-enabled databasesDocuments/FilesDocuments/Files

CustomCustomSpecialty Data StoresSpecialty Data Stores

ASE/CISMainframe Connect

Distributed

Proxy Tables

Specialty Data Store

Toolkit

Any ClientOr BrowserAny Client

Or Browser

12

Mainframe Connect - Extending the Mainframe

EnterpriseConnect VSAM

IMSDatacom IDMSAdabas

Mainframe

SybaseASE

ReplicationServer

EnterprisePortals,

App Servers

Direct Connect

for OS/390

DB2

Mainframe Applications and Transactions

Any Client

Any Client

Mainframe as a Client(Client Options)

Access to any MFData/Applications (Server Options and XJS/390)

Dynamic Accessto DB2(DB2 UDB for OS/390 Option)

“Gatewayless”configuration

13

XJS/390 Enterprise Integrator

ASE/CIS

VSAM IMSDatacom IDMS Sequential Adabas

MainframeSoap, XML, HTML, and Scripts

ASE

ReplicationServer

EnterprisePortals, App

Servers

Direct Connect

forOS/390

DB2

Mainframe Applications, and Transactions


Or Browser

MQSeries

MQSeriesAccess to MF Data and Applications (Server Options and XJS/390)

CICS Programs, Data HTTP, MQ, Web Services

IMS Programs, Data, HTTP, MQ

Navigate CICS Transactions

Integrate with MainframeCONNECT

No data is too complex to web- or message- enableNo data is too complex to web- or message- enable

“Gatewayless”configuration

14

Scenario – Customer Service

Problem: Phone Service Two-Step

• Taking 20+ minutes/call to help customer buy new options

• Customer data from IMS mainframe “green screens”

• Regional service data from distributed Informix databases

Goal: Faster Service

• Present all data at once in easy-to-use GUI application so customer service operators can provide faster service

• Must leverage existing application logic and data

InformixRegional service data

Mainframe IMS customer data

15

Solution – Customer Service

• OS/390 IMS customer data

• Informix local telephone service data

Enterprise ConnectMainframe Connect

GUI Customer Service application.

Solution:

<2 minutes/call with GUI customer service application transparently accessing mainframe and distributed data

Proxy Tables

16

Data Access – How It Works

Set up connectivity to remote servers Define remote servers and security in ASE-CIS Automatically set up proxy tables mapped to remote

heterogeneous tables (federated data) Set up security as desired Metadata retained within ASE-CIS Now queries work with the proxy tables as if they were local Create stored procedures, views for proxy tables as needed Federate data is accessible transparently via SQL

Enterprise ConnectMainframe Connect

17

Agenda


Technologies for Integrating Federated Data



XML

Web Services

Resources

18

Document Access – Full Text Search

Store and index documents for quick search PDF, Microsoft Word, many other formats supported Ability to join database data with results of full text search Automated indexing, backup, restore

19

ASE – Enhanced Full Text Search Option (EFTS)ASE – Enhanced Full Text Search Option (EFTS)

Full Text Search of Wide Variety of DocumentsCombined Search Results with Data from Other Tables

ASE/CIS Full Text Search Engine

• Documents stored as binary in user-created source tables with identity column for unique id

• CIS index tables (proxy tables) mapped to text search engine collections


Or Browser

Collections (Directories with text indexes and unique identifiers)

Supports over 200 formats including Microsoft Word, XML, SGML, PDF

20

Scenario – Resume Search

Problem: Find Resumes with Certain Skills, Match with other Data on Potential Employees

> Scan individual resume documents in various formats for candidate with appropriate skills

> Once candidate is located, query ASE database containing other information on the candidate

> Assemble information by hand in report for interviewers

DB2 database with other candidate info

Resume Documents

21

Solution – Resume Search

ASE-CIS

Resume Search Application

Solution: • Single point of access

for both documents and other data

• One query locates matching documents based on relevance, joins this data w/ related federated data on candidates, presents complete info to end user

EFTS Option for ASE

ASE database with documents and other data on candidates

Text indexes

Collection

22

Enhanced File Text Search – How It Works

Setup Store documents in ASE as binary data Set up EFTS remote server defined as server class “sds” Run stored procedures that extract and index the text of each

document into “collections” Proxy tables w/in ASE-CIS mapped to indexes called

“collections” Each document has unique id in database As documents are changed/updated, triggers track changes

so indexes can be kept in sync

23

Enhanced File Text Search – How It Works

Run-time Query for particular text string is sent through proxy

mapping to EFTS for query against collection Document ids returned, based on relevance (degree of match

with query) If desired, join with other data Return data to client application Client application may then display lists of docs or actual doc

content as desired

24

EFTS – Advanced Capabilities

Topics Groupings of information that represents a concept or subject

area. Precompiled query made available for run-time searches by

any user Clustering

Group similar documents Query by Example

Search for documents that have similar phrases or are similar to docs already indexes

Custom Thesaurus Build a custom thesaurus with synonyms you define

25

Enhanced Full Text Search – More Features

Sample Text Load program Loads data into text or image columns into a source table Delivered as source code

Stored procedures to help automate EFTS configuration Alteration of cluster capabilities Backup and recover text indexes Display information about a text index Optimize indexes

26

ASE 12.5 Content Management Option(External File System Support – EFS)

File System Directory Access SQL access to file system directories and their underlying files is

enabled via proxy tables Each directory entry is visible as a row in a proxy table Regular file content is available as an image column in the proxy

table

Combine with Full Text Search to index and search on documents external to ASE via ASE queries

27

File System Directory Access – How it Works

Create a proxy table for the directory The directory pathname must reference directory visible to/searchable by ASE Proxy table is created which maps column names to file attributes and content Option to include subdirectories

Attributes of directory contents converted to SQL data types Content of regular files returned via content column Always treated as an image datatype

SQL Commands supported Insert causes new file to be created in directory Update allows filename change Delete causes file to be deleted Select returns columns as required Readtext will retrieve contents of the file Writetext will modify contents of the file

File content may be queried with the select statement

28

File Access

SQL access to individual files within a file system enabled via proxy tables

Contents of file must be readable characters, 255 char max records delimited with newline

Useful for exporting ASE data to other applications that cannot otherwise be accessed via CIS

SQL alternative to bcp out or bcp in Create proxy_table command creates a proxy table with one

column called “record”, 255 varchar To specify column names and datatypes use the create

[existing] table command

29

File Access – How It Works

Example, to read individual records within the ASE error log file:

create proxy_table errorlog external file at "/usr/sybase/ase12_5/install/errorlog"

select record from errorlog where record like "%server%"

This query will return all rows from the file that match the like pattern. Rows longer than 255 bytes will be truncated. To specify longer rows:

create existing table errorlog(

record varchar(512) null )external file at "/usr/sybase/ase12_5/install/errorlog"

In this case records up to 512 bytes in length will be returned

30

Agenda





XML

Web Services

Resources

31

Support for Storing, Searching and Querying XML documents

Key functionalities:

XQL to query XML documents Store parsed XML documents in ASE Query parsed XML documents Enhanced Full-Text Search is XML-aware

Benefits Simple: Java classes to extract contents General: extraction can be generalized Optimized: store once, query many times Tight integration with SQL language

ASE 12.5 as an XML Server

32

New XML Features with ASE 12.5.1

Native XML Engine Lets you more efficiently store, query, manage and exchange

XML data in its native format Based on W3C, XML and other open Internet standards Integrates SQL and XML Data Management functionalities

natively in the same server reduces cost of ownership reduces number of moving parts in a complex IT infrastructure provides a solid data management platform to exploit the

power of XML while leveraging existing IT investments.

33

ASE 12.5.1 Native XML Engine

Native XML Processor Running in ASE Implements XPath – a significant subset of XQuery language

standard for querying XML data Upwardly compatible w/ XQL provided w/ ASE 12.5

Supports for xml extension of select statements Map SQL statements to standard SQLX-XML Java-based forsqlj functions allow mapping of SQLX-XML back to

SQL script that recreates the data

34

More on 12.5.1 XML Features

Native XML Support Store and Retrieve well formed XML data Schema independent

SQL and XML Duality Complete interoperability between SQL and XML at language and storage

level

XML Indexing Renders high query performance Self defined indexes – No user input necessary

XPATH & XQUERY Support Granular results – Returns documents, document fragments, elements Wild card support for complex queries

35

How Does this Relate to Federated Data?

Combine ASE and XML Management option with ECDA to Query XML documents located in heterogeneous databases Use SQL to extract and join data from heterogeneous databases, and convert

it automatically to XML format

ASE/CIS + ECDA Distributed Heterogeneous

Data

Proxy Tables


Or BrowserSQL

XML

XML

SQL

36

Agenda





XML

Web Services


37

What are Web Services?

Self-contained, modular applications that can be described, published, located, and invoked over a network, generally, the Web.

Generally associated with SOAP, WSDL and UDDI protocol specifications.

Enable loose coupling of independent applications

38

What can web services do?

Allow applications to share data. Discrete units of code; each handles a limited set of tasks. Based on XML, the universal language of Internet data exchange

Can be called across platforms and operating systems, regardless of programming language.

39

What About Data Available from Web Services?

ASE Web Services Option for ASE 12.5.1 Expose ASE functionality (Stored procedures within

ASE) as Web Services – ASE as Web Services Producer

Extend ASE to access external Web Services and use the results in the SQL constructs seamlessly – ASE as a Web Services Consumer

Application data (e.g. Peoplesoft) exposed as Web Services can participate in the Federated Data world

40

ASE Web Services Producer

ASE

ASE Web Services

Producer

Any SOAP Client

Input: Soap

Output: Soap

Input: JDBC

Output: JDBC

41

ASE Web Services Consumer

ASE ASE Web ServicesConsumer

SOAP Server

2. RPCEvent

3. SOAPrequest

4. SOAPresponse5. Results

1. select

6. Results

42

Agenda





XML

Web Services


43

Sybase EII MiddlewareComplete Access to Federated Data

Sybase EII Middleware Enterprise Connect Data Access and Mainframe Connect ASE Options for EFTS, Content Management, XML

Management, Web Services

Benefits Leverage existing hardware/software investment Leverage existing IT skills Leverage Sybase support for and experience with

heterogeneous environments

44

The Big Picture

How Today’s Sybase Solutions Enable Information Liquidity

45

For More Information

Product Documentation - see Enterprise Connect Data Access, Mainframe Connect, ASE Component Integration Services, XML Management Option, Web Connect

http://www.sybase.com/support/manuals/

Product Information and White Papers

http://www.sybase.com/products/middleware

Newsgroups

sybase.public.gateways

sybase.public.omni

46

Sybase Developer Network (SDN)

Developer software, services, up-to-date technical information:

White papers and documentation Collaboration with other developers and Sybase engineers Code samples and beta programs Technical recordings Free software Join today: www.sybase.com/developer or visit SDN at

TechWave’s Technology Boardwalk

47

SDN Presents CodeXchange

Forum for exchanging samples, tools, scripts, etc. New features enable community collaboration

Download samples created by Sybase or external users Leverage contributions of others to extend Sybase tools Contribute code or start your own collaborative / open source project with input

from other product experts

Any SDN member can participate Log in using your MySybase account via SDN

Join the collaboration already underway http://www.codexchange.sybase.com or via SDN at

www.sybase.com/developer

A New Benefit for Sybase Developers

48

Check Out These Related TechWave Sessions!

ASE111 - Integration, XML, and Web Services--Why are They Important to a DBA?

ASE114 - Querying and Transforming XML with XQuery, XPath, and SQLX

INT404 - Integrating Adaptive Server Enterprise with Web Services

Boardwalk demos

INT420Federated Database Solutions with Sybase Middleware

Lynn Weatherwax

Director of Engineering – Data IntegrationInfrastructure Platform Group, [email protected]

Q & A

Download File

Documents

Transcript of Download File