IVOA Interop, Beijing, China, May 2007 1

IVOA Data Access LayerWorking Group Sessions

Doug Tody (NRAO/NVO)Markus Dolensky (ESO/EuroVO)

Data Access Layer Working Group

INTERNATIONAL VIRTUAL OBSERVATORY ALLIANCE

IVOA Interop, Beijing, China, May 2007 2

Table Access ProtocolInterface Analysis/Options

(from the DAL perspective)

IVOA Interop, Beijing, China, May 2007 3

Protocol Consistency

• Motivation– TAP and SSAP, SLAP, SIA-V2, etc. (the second generation

protocols) should be consistent where possible– Promotes code sharing at all levels– Simplifies service frameworks as well as client apps– Sharing reduces overall system complexity– Future ADQL integration into other DAL services will be easier

• Guideline– Take basic DAL service profile and semantics as a starting

point, only deviate for TAP where there is a need to do so.

IVOA Interop, Beijing, China, May 2007 4

DAL Service Profile

• Standard Profile– Intended as starting point for any 2ndGen DAL service– Joint GWS/DAL/Registry standards effort

• Standard Operations– queryData find available datasets (may be virtual)– (getData) get a single dataset (synchronous)– stageData start async job to generate/stage datasets– getCapabilities get service metadata (capabilities)– getAvailability get service availability, status

IVOA Interop, Beijing, China, May 2007 5

Table Access Protocol

• Assumptions– Scope is a single service at a single site– Distributed queries are handled at a higher level

• Requirements– Simple usage for a simple query is a requirement– Large (asynchronous) queries must however be supported– A service may manage a "table set" - multiple tables– ADQL-based queries may query multiple tables in one operation

IVOA Interop, Beijing, China, May 2007 6

Basic Query

• Purpose– Execute a single query and return immediate results

• Goals– Basic usage very simple

• essentially a generalized cone search– Synchronous GET (POST also possible)– Returns a VOTable directly (or other table format)– Both ADQL and data model-based queries provide

IVOA Interop, Beijing, China, May 2007 7

Basic Query

• ADQL query– queryData(query="SELECT * FROM a WHERE snr>2.5", format="csv")

• Example GET translation– http://...REQUEST=queryData&QUERY=<url-encoded-ADQL>&FORMAT=csv– Details such as operation name may vary

• Equivalent DM-based query– queryData(table="a", snr="2.5/", format="csv")

• Classic Cone Search– queryData(table="a", POS="180.0,1.0", SIZE="0.1")

IVOA Interop, Beijing, China, May 2007 8

Async Query

• Purpose– Execute a single query asynchronously and stage the

results

• Approach– Variation on stageData as planned for SIAP etc.– Differs only in the content of the stageData request– VOSpace used to stage output (or input)

IVOA Interop, Beijing, China, May 2007 9

Async Query

• Canonical StageData– Normally (for SIA etc.) proceeded by a queryData

• queryData response defines virtual datasets (as at present)• these are computational tasks which can be performed

– Provides a declarative approach to job specification– QueryData response can include cost estimation

• stageData request– Is a POST; returns a UWS JobID– Lists one or more tasks to be performed– Includes data staging (e.g., VOSpace) information– UWS mechanisms (polling, messaging) used to monitor job progress

IVOA Interop, Beijing, China, May 2007 10

Async Query

• TAP StageData Variant– Skips (normally) the cost estimation step

• not needed except for very large queries• can be hard to estimate query costs with SQL

– Request is parameter-based rather than declarative

• stageData request– Request includes the ADQL query directly as the task specifier– Data staging functionality is common– Job control and monitoring facilities are common

IVOA Interop, Beijing, China, May 2007 11

Metadata Queries

• Purpose– Mechanism used by client to query “database” metadata– At issue here is "dataset content" metadata (tables, columns, etc.)

• Possible Approaches– Uniform interface to query both table data and metadata– Dedicated operation for each query

• Registry Integration– In either case table/column metadata can be generated to– provide for registry-based caching and discovery

IVOA Interop, Beijing, China, May 2007 12

Metadata Queries

• Uniform Query Interface– Represents database metadata as tables which can be queried– Same interface used to query both table data and metadata– This is same approached used for the SQL INFORMATION_SCHEMA

• Examples– queryData(table="SCHEMA.tables", format="xml")– queryData(table="SCHEMA.columns", tableName="a")

• Advantages– Re-use of existing table query mechanism

• all related software at all levels can be reused• features such as FORMAT options come for free

– Adding more metadata does not require an interface change• tables, columns, views, functions, indexes, etc.

IVOA Interop, Beijing, China, May 2007 13

Metadata Queries

• Dedicated Operations– Defines a custom operation to query each form of table metadata

• e.g., getTables, getColumns– Could potentially still re-use query response mechanisms

• e.g., FORMAT options and parameter

• Advantages– Slightly simpler query, e.g., to get list of tables

• Disadvantages– Standard data query interface cannot be used to access metadata– Less easily extended as need to extend service interface

IVOA Interop, Beijing, China, May 2007 14

Interface Summary

• Operations– simple synchronous query (queryData variant)– table metadata query (probably also queryData)– asynchronous query (stageData variant)– getCapabilities (standard)– getAvailability (standard)

• Protocol– HTTP details consistent with other services (multiple requests,– error handling, file formats, etc.)

IVOA Interop, Beijing, China, May 2007 15

Some Key TAP Issues• Parameterized queries (POS,SIZE etc.)

– Do we go only the ADQL route, or do we also provide a parameter-based query for the simplest use cases?

• Unified table data/metadata query– Do we provide a uniform interface for table data and table metadata

queries?

• Form of interface– Can we agree to standardize the form of an interface at the level of the

HTTP protocol, e.g., the details of how multiple operations, parameters, error responses, etc., are handled, and use this for all the IVOA data services?

• Grid capabilities– Do we wish to have a phased development for TAP which does not fully

specify the grid capabilities in the initial version?– Will this be sufficiently useful?