Post on 15-Jan-2016
description
1
Grid DAIS:Database Access and Integration Services
Greg RiccardiFlorida State University riccardi@cs.fsu.edu
2
Overview of Presentation Goals of DAIS Conceptual model of Grid database
access Examples of client-service interactions
Discovery and creation of servicesAsynchronous query processing and datasetsUpdating from datasets
Representing Sky Query in DAIS Other topics/issues
3
Goals of DAIS The group seeks to promote standards for the
development of grid database services, focusing principally on providing consistent access to existing, autonomously managed databases. Provide service-based access to existing data management
systems. Accommodate several widely used data management paradigms
(e.g., relational, object, XML) within a consistent framework. Provide sufficient information about itself to allow the service to
be used given the specification of the service and the metadata provided by the service.
Peacefully coexist with other Web and Grid Service standards. Be orthogonal to Grid authentication and authorization
mechanisms. Support higher-level information-integration and federation
services.
4
Desirable Properties of DAIS Systems OGSI/A compliant
Letter and Spirit Plugability/Extensibility
Different kinds of data resources Many access mechanisms Evolvable
Easy to understand and apply Existing standards/designs Tooling
GridServices and WebServices applicable Supports current technology Access AND integration
Integration of different models at the data level Implementable Integrateable into customer scenarios Technology independent
5
The Model – External Artifacts
DBMS
DB
Resultset
External data resource manager
External data resource
External data set
External = external to the OGSI compliant grid
6
The Model – Logical Artifacts
DBMS
DB
Resultset
data resource manager
data resource
data activity session
data request
data set
7
Data Resource Manager External data resource manager (edrm)
A data management system such as a relational database management system or a file system
Data resource manager (drm) A grid service that represents the external data
resource manager Binds to an existing edrm Supports management operations such as start and
stop Mainly out of scope of DAIS. A place holder for
interaction with other working groups
8
Data Resources External Data Resource (edr)
A data construct managed by the external data resource manager, for example, a database or a directory structure.
An external data resource manager may manage many external data resources
Data Resource (dr) A grid service that represents an external data resource Represents the point of contact to the data structures managed
by the edrm. Exposes meta-data about the structure of the edr Defines the the target for queries across the edr Can act as a notification source for notifications associated with
the edr Is bound to existing or newly created edr Has similarities with a data set. More of which later.
9
Data Sets External Data Set (eds)
Data logically separated from an external data resource manager
Could be a snapshot (query) of a relational database or data generated by some process prior to being inserted into a database
Will be typed and identifiable Data Set (ds)
A service wrapper for the eds Exposes meta data about the type, description, format of the eds Immutable Exposes simple data access operations depending on the type
of data. getAllData, createIterator, getTuple, getFile, getByte, etc.
Can be moved while maintaining its handle and data identity Can be copied or replicated while maintaining its data identity Can be delivered to a data manager for persistence
Query and update could be supported
10
Putting It Together Logical Artifact = Service
ds
eds
dr
edredrm
drm das
External world
DAIS world
create
bind/create
create
bind
create
create
bind/create
requester
data request
locate access data
11
Exploiting The Logical Artifacts: Data Sets
dsdr
edr
das
ds ds dr
edr
das
queryinsert/update
create
move copy
referencelaunch launch
GSH GSH
move service
reference create
copy service
reference create
target details
target details
Analyst1
12
Client-Server Interaction Patterns
A G
Q
S + R
AG
Q + D
S
RC
A G
Q + U
S
A G
Q
S
UP
Retrieve Update/Insert
AG
Q
S
DCR
A G
Q + D
S
IP
U
1.
2.
3.
4.
6.
5.
Pipeline
A
G = PQ1 + D
S1 U/R
G = CS2
Q2
A
G = PQ1
S1 U/R
G = CS2
Q2 + DI
7.
8.
13
Examples of client-service interactions Discovery and creation of services
createcreate
A dr
das
Create
GSH
Query
Result
drmCreate
GSH
Database
ExternalDataResourceManager
11
2
23
14
Examples of client-service interactions Asynchronous query processing and
datasets
Database
crea
te
A das
dsC
Query
Id
Id
Get
Result
3
1
2
5
4
15
Examples of client-service interactions Updating from datasets
Database
ds
dasAUpdate(Id)
Status
Get Result2
1
3
16
Example of performance estimation
Database
preparedasA
Query
Status
Estimate
Statistics
17
SkyQuery Cross Match Query
18
Cross Match Estimation in DAIS
ClientQuery Manager
Query
StatisticsSpatialQuery
StatisticsPr
epar
e
das
Database
Prep
are
das
DatabasePr
epar
e
das
Database
SpatialQuery
SpatialQuery
Statistics
19
Cross Match in DAIS
ClientQuery Manager
Query
Id1Run
Query
Result
MatchId1
MatchId2 Id2Id3
getId3
Que
rycr
eate
das
ds3
Database
Que
rycr
eate
das
ds2
DatabaseQ
uery
crea
te
das
ds1
Database
20
Other topics and issues for DAIS Data provenance management Transaction management Fault tolerance Security, logging, auditing Supporting many concurrent users Establishing the identity and provenance
of datasets Creating pipelines and other workflows Querying streams of data