Post on 15-Dec-2014
description
INT420 - Federated Database Solutions with Sybase Middleware
Lynn Weatherwax
Director of Engineering – Data IntegrationInfrastructure Platform Group, SybaseLynn.Weatherwax@sybase.com
2
Abstract
Many companies today seek optimized ways to access structured and unstructured data in their enterprises, without extensive programming or costly database consolidation.
Virtual databases with links to distributed heterogeneous sources enable applications to consistently access and combine data in new ways.
Sybase solves this problem while supporting the optimized query capabilities and scalability of an enterprise class relational database, via ASE, Enterprise Connect Data Access, Enhanced Full Text Search, and ASE Web Services.
Leverage SQL and XML standards to make your data work for you in your enterprise.
3
Agenda
Enterprise Information Integration (EII)
Strategies for Integrating Federated Data
Federated Database Access
Document Access - Full Text Search
XML
Web Services
Summary and Resources
4
Enterprise Information Integration (EII)
“Enterprise information integration (EII) is seeing a new surge of interest in 2003, largely because its virtual and/or federated approach to data integration has potential benefits.
Compared to other data integration technologies, solutions based on a vendor’s EII tool tend to be quick to deploy, inexpensive in terms of licensing and very flexible since they’re based on a metadata-based view of enterprise data.
In addition, EII relies on distributed queries to effect data integration in near real-time, thus delivering fresher data than traditional batch methods.
Many companies are evaluating vendors’ EII products in 2003, so 2004 should see an upsurge in EII implementations.”
Philip Russom, IdeaByte
5
Factors Driving EII Today
Changes in company structure Increased competition, requiring better intelligence Government regulations
Patriot Act Sarbanes-Oxley Act HIPPA
6
Every enterprise has federated data, due to
• Mergers and acquisitions
• Legacy mainframe applications
• Applications required to work with specific databases
EII Challenge
XML
DB2 UDB
SybaseMicrosoft
InformixVSAM
Oracle
Adabas
Documents
7
Option: Migrate All Data/Apps to One Database…
Sometimes this is feasible, but often there are time and $$ restrictions…
And what about the next acquisition, or the next undiscovered information island?
XML
DB2 UDB
My CentralizedDatabase
Microsoft
Informix
VSAM
Oracle
Adabas
Documents
8
Enterprise Information Integration Requirements
Leverage existing hardware and software investment Ensure secure access Integrate legacy data and applications while they
remain in place – short-term or long-term Metadata transparently aggregates data, no
programming required Components that work together, forming a complete,
end-to-end solution Supported in standard client/server environments, yet
fully e-business ready (supports XML, Java, etc.)
9
Agenda
Enterprise Information Integration (EII)
Strategies for Integrating Federated Data
Federated Database Access
Document Access - Full Text Search
XML
Web Services
Summary and Resources
10
Strategies for Integrating Federated Data
Data Access Document Access - Full Text Search Messaging/Data Replication XML Web Services
11
Enterprise Connect Data AccessEnterprise Connect Data Access
Make all your enterprise data look like one database
Any Any mainframe mainframe datadata
OracleOracleMS SQL ServerMS SQL ServerSybaseSybaseDB2 UDB for OS/390, NT/UNIXDB2 UDB for OS/390, NT/UNIXAS/400AS/400InformixInformixODBC-enabled databasesODBC-enabled databasesDocuments/FilesDocuments/Files
CustomCustomSpecialty Data StoresSpecialty Data Stores
ASE/CISMainframe Connect
Distributed
Proxy Tables
Specialty Data Store
Toolkit
Any ClientOr BrowserAny Client
Or Browser
12
Mainframe Connect - Extending the Mainframe
EnterpriseConnect VSAM
IMSDatacom IDMSAdabas
Mainframe
SybaseASE
ReplicationServer
EnterprisePortals,
App Servers
Direct Connect
for OS/390
DB2
Mainframe Applications and Transactions
Any Client
Any Client
Mainframe as a Client(Client Options)
Access to any MFData/Applications (Server Options and XJS/390)
Dynamic Accessto DB2(DB2 UDB for OS/390 Option)
“Gatewayless”configuration
13
XJS/390 Enterprise Integrator
ASE/CIS
VSAM IMSDatacom IDMS Sequential Adabas
MainframeSoap, XML, HTML, and Scripts
ASE
ReplicationServer
EnterprisePortals, App
Servers
Direct Connect
forOS/390
DB2
Mainframe Applications, and Transactions
Any ClientOr BrowserAny Client
Or Browser
MQSeries
MQSeriesAccess to MF Data and Applications (Server Options and XJS/390)
CICS Programs, Data HTTP, MQ, Web Services
IMS Programs, Data, HTTP, MQ
Navigate CICS Transactions
Integrate with MainframeCONNECT
No data is too complex to web- or message- enableNo data is too complex to web- or message- enable
“Gatewayless”configuration
14
Scenario – Customer Service
Problem: Phone Service Two-Step
• Taking 20+ minutes/call to help customer buy new options
• Customer data from IMS mainframe “green screens”
• Regional service data from distributed Informix databases
Goal: Faster Service
• Present all data at once in easy-to-use GUI application so customer service operators can provide faster service
• Must leverage existing application logic and data
InformixRegional service data
Mainframe IMS customer data
15
Solution – Customer Service
• OS/390 IMS customer data
• Informix local telephone service data
Enterprise ConnectMainframe Connect
GUI Customer Service application.
Solution:
<2 minutes/call with GUI customer service application transparently accessing mainframe and distributed data
Proxy Tables
16
Data Access – How It Works
Set up connectivity to remote servers Define remote servers and security in ASE-CIS Automatically set up proxy tables mapped to remote
heterogeneous tables (federated data) Set up security as desired Metadata retained within ASE-CIS Now queries work with the proxy tables as if they were local Create stored procedures, views for proxy tables as needed Federate data is accessible transparently via SQL
Enterprise ConnectMainframe Connect
17
Agenda
Enterprise Information Integration (EII)
Technologies for Integrating Federated Data
Federated Database Access
Document Access - Full Text Search
XML
Web Services
Resources
18
Document Access – Full Text Search
Store and index documents for quick search PDF, Microsoft Word, many other formats supported Ability to join database data with results of full text search Automated indexing, backup, restore
19
ASE – Enhanced Full Text Search Option (EFTS)ASE – Enhanced Full Text Search Option (EFTS)
Full Text Search of Wide Variety of DocumentsCombined Search Results with Data from Other Tables
ASE/CIS Full Text Search Engine
• Documents stored as binary in user-created source tables with identity column for unique id
• CIS index tables (proxy tables) mapped to text search engine collections
Any ClientOr BrowserAny Client
Or Browser
Collections (Directories with text indexes and unique identifiers)
Supports over 200 formats including Microsoft Word, XML, SGML, PDF
20
Scenario – Resume Search
Problem: Find Resumes with Certain Skills, Match with other Data on Potential Employees
> Scan individual resume documents in various formats for candidate with appropriate skills
> Once candidate is located, query ASE database containing other information on the candidate
> Assemble information by hand in report for interviewers
DB2 database with other candidate info
Resume Documents
21
Solution – Resume Search
ASE-CIS
Resume Search Application
Solution: • Single point of access
for both documents and other data
• One query locates matching documents based on relevance, joins this data w/ related federated data on candidates, presents complete info to end user
EFTS Option for ASE
ASE database with documents and other data on candidates
Text indexes
Collection
22
Enhanced File Text Search – How It Works
Setup Store documents in ASE as binary data Set up EFTS remote server defined as server class “sds” Run stored procedures that extract and index the text of each
document into “collections” Proxy tables w/in ASE-CIS mapped to indexes called
“collections” Each document has unique id in database As documents are changed/updated, triggers track changes
so indexes can be kept in sync
23
Enhanced File Text Search – How It Works
Run-time Query for particular text string is sent through proxy
mapping to EFTS for query against collection Document ids returned, based on relevance (degree of match
with query) If desired, join with other data Return data to client application Client application may then display lists of docs or actual doc
content as desired
24
EFTS – Advanced Capabilities
Topics Groupings of information that represents a concept or subject
area. Precompiled query made available for run-time searches by
any user Clustering
Group similar documents Query by Example
Search for documents that have similar phrases or are similar to docs already indexes
Custom Thesaurus Build a custom thesaurus with synonyms you define
25
Enhanced Full Text Search – More Features
Sample Text Load program Loads data into text or image columns into a source table Delivered as source code
Stored procedures to help automate EFTS configuration Alteration of cluster capabilities Backup and recover text indexes Display information about a text index Optimize indexes
26
ASE 12.5 Content Management Option(External File System Support – EFS)
File System Directory Access SQL access to file system directories and their underlying files is
enabled via proxy tables Each directory entry is visible as a row in a proxy table Regular file content is available as an image column in the proxy
table
Combine with Full Text Search to index and search on documents external to ASE via ASE queries
27
File System Directory Access – How it Works
Create a proxy table for the directory The directory pathname must reference directory visible to/searchable by ASE Proxy table is created which maps column names to file attributes and content Option to include subdirectories
Attributes of directory contents converted to SQL data types Content of regular files returned via content column Always treated as an image datatype
SQL Commands supported Insert causes new file to be created in directory Update allows filename change Delete causes file to be deleted Select returns columns as required Readtext will retrieve contents of the file Writetext will modify contents of the file
File content may be queried with the select statement
28
File Access
SQL access to individual files within a file system enabled via proxy tables
Contents of file must be readable characters, 255 char max records delimited with newline
Useful for exporting ASE data to other applications that cannot otherwise be accessed via CIS
SQL alternative to bcp out or bcp in Create proxy_table command creates a proxy table with one
column called “record”, 255 varchar To specify column names and datatypes use the create
[existing] table command
29
File Access – How It Works
Example, to read individual records within the ASE error log file:
create proxy_table errorlog external file at "/usr/sybase/ase12_5/install/errorlog"
select record from errorlog where record like "%server%"
This query will return all rows from the file that match the like pattern. Rows longer than 255 bytes will be truncated. To specify longer rows:
create existing table errorlog(
record varchar(512) null )external file at "/usr/sybase/ase12_5/install/errorlog"
In this case records up to 512 bytes in length will be returned
30
Agenda
Enterprise Information Integration (EII)
Technologies for Integrating Federated Data
Federated Database Access
Document Access - Full Text Search
XML
Web Services
Resources
31
Support for Storing, Searching and Querying XML documents
Key functionalities:
XQL to query XML documents Store parsed XML documents in ASE Query parsed XML documents Enhanced Full-Text Search is XML-aware
Benefits Simple: Java classes to extract contents General: extraction can be generalized Optimized: store once, query many times Tight integration with SQL language
ASE 12.5 as an XML Server
32
New XML Features with ASE 12.5.1
Native XML Engine Lets you more efficiently store, query, manage and exchange
XML data in its native format Based on W3C, XML and other open Internet standards Integrates SQL and XML Data Management functionalities
natively in the same server reduces cost of ownership reduces number of moving parts in a complex IT infrastructure provides a solid data management platform to exploit the
power of XML while leveraging existing IT investments.
33
ASE 12.5.1 Native XML Engine
Native XML Processor Running in ASE Implements XPath – a significant subset of XQuery language
standard for querying XML data Upwardly compatible w/ XQL provided w/ ASE 12.5
Supports for xml extension of select statements Map SQL statements to standard SQLX-XML Java-based forsqlj functions allow mapping of SQLX-XML back to
SQL script that recreates the data
34
More on 12.5.1 XML Features
Native XML Support Store and Retrieve well formed XML data Schema independent
SQL and XML Duality Complete interoperability between SQL and XML at language and storage
level
XML Indexing Renders high query performance Self defined indexes – No user input necessary
XPATH & XQUERY Support Granular results – Returns documents, document fragments, elements Wild card support for complex queries
35
How Does this Relate to Federated Data?
Combine ASE and XML Management option with ECDA to Query XML documents located in heterogeneous databases Use SQL to extract and join data from heterogeneous databases, and convert
it automatically to XML format
ASE/CIS + ECDA Distributed Heterogeneous
Data
Proxy Tables
Any ClientOr BrowserAny Client
Or BrowserSQL
XML
XML
SQL
36
Agenda
Enterprise Information Integration (EII)
Strategies for Integrating Federated Data
Federated Database Access
Document Access - Full Text Search
XML
Web Services
Summary and Resources
37
What are Web Services?
Self-contained, modular applications that can be described, published, located, and invoked over a network, generally, the Web.
Generally associated with SOAP, WSDL and UDDI protocol specifications.
Enable loose coupling of independent applications
38
What can web services do?
Allow applications to share data. Discrete units of code; each handles a limited set of tasks. Based on XML, the universal language of Internet data exchange
Can be called across platforms and operating systems, regardless of programming language.
39
What About Data Available from Web Services?
ASE Web Services Option for ASE 12.5.1 Expose ASE functionality (Stored procedures within
ASE) as Web Services – ASE as Web Services Producer
Extend ASE to access external Web Services and use the results in the SQL constructs seamlessly – ASE as a Web Services Consumer
Application data (e.g. Peoplesoft) exposed as Web Services can participate in the Federated Data world
40
ASE Web Services Producer
ASE
ASE Web Services
Producer
Any SOAP Client
Input: Soap
Output: Soap
Input: JDBC
Output: JDBC
41
ASE Web Services Consumer
ASE ASE Web ServicesConsumer
SOAP Server
2. RPCEvent
3. SOAPrequest
4. SOAPresponse5. Results
1. select
6. Results
42
Agenda
Enterprise Information Integration (EII)
Technologies for Integrating Federated Data
Federated Database Access
Document Access - Full Text Search
XML
Web Services
Summary and Resources
43
Sybase EII MiddlewareComplete Access to Federated Data
Sybase EII Middleware Enterprise Connect Data Access and Mainframe Connect ASE Options for EFTS, Content Management, XML
Management, Web Services
Benefits Leverage existing hardware/software investment Leverage existing IT skills Leverage Sybase support for and experience with
heterogeneous environments
44
The Big Picture
How Today’s Sybase Solutions Enable Information Liquidity
45
For More Information
Product Documentation - see Enterprise Connect Data Access, Mainframe Connect, ASE Component Integration Services, XML Management Option, Web Connect
http://www.sybase.com/support/manuals/
Product Information and White Papers
http://www.sybase.com/products/middleware
Newsgroups
sybase.public.gateways
sybase.public.omni
46
Sybase Developer Network (SDN)
Developer software, services, up-to-date technical information:
White papers and documentation Collaboration with other developers and Sybase engineers Code samples and beta programs Technical recordings Free software Join today: www.sybase.com/developer or visit SDN at
TechWave’s Technology Boardwalk
47
SDN Presents CodeXchange
Forum for exchanging samples, tools, scripts, etc. New features enable community collaboration
Download samples created by Sybase or external users Leverage contributions of others to extend Sybase tools Contribute code or start your own collaborative / open source project with input
from other product experts
Any SDN member can participate Log in using your MySybase account via SDN
Join the collaboration already underway http://www.codexchange.sybase.com or via SDN at
www.sybase.com/developer
A New Benefit for Sybase Developers
48
Check Out These Related TechWave Sessions!
ASE111 - Integration, XML, and Web Services--Why are They Important to a DBA?
ASE114 - Querying and Transforming XML with XQuery, XPath, and SQLX
INT404 - Integrating Adaptive Server Enterprise with Web Services
Boardwalk demos
INT420Federated Database Solutions with Sybase Middleware
Lynn Weatherwax
Director of Engineering – Data IntegrationInfrastructure Platform Group, SybaseLynn.Weatherwax@sybase.com
Q & A