The prototype TNG Long-Term Archive ––

Interoperability issues and interactions with the Italian GRID

project

F.Pasian, R.Smareglia – INAF/O.A.Trieste

L.Benacchio – INAF/O.A.Padova

“TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

Garching, 10-14 June 2002

Towards an Italian Virtual Observatory?

Joining the International Virtual Observatory Alliance

Struggling to convince your funding agencies that it is worth to devote manpower and funds to coordinate efforts within an international VO activity, while they are still not necessarily

convinced it is useful to develop archives

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

2

“Telescopio Nazionale Galileo” (TNG)

• 3.5 m Nasmyth telescope - NTT-based - La Palma• First light in 1998 - AaT since the beginning• Long integration / testing / commissioning phase• First generation instruments completed in 2000:

– optical imager (OIG)– low-resolution imager/spectrograph (DOLORES)– high resolution spectrograph (SARG)– near-IR camera/spectrograph (NICS)

• ADOPT@TNG, DIMM, Meteo• Assisted observations

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

3

The TNG LTA pilot project

• A pilot project is currently being finalized, aiming at demonstrating the feasibility of technical solutions for the construction of the Long-Term Archive (LTA) for the TNG building a prototype LTA

• Interfaces to TNG “archiving at the telescope” (AaT)• End of pilot phase foreseen to be end of June 2002• Financed by CNAA, results to be submitted to INAF

– INAF board on computing, archives and networks to produce a report in early July

4“TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

Garching, 10-14 June 2002

Scope of the pilot project

• Provide suggestions / plans / prototypes to: – allow the long-term storage of scientific and technical data

from the TNG – guarantee accessibility by the TNG Observatory staff and

by the scientific community to original and derived data – test practically the feasibility of deriving value-added

information from the stored data– provide tools supporting life cycle of observing proposals – provide interoperability with a number of existing

international data centers and archives – guarantee harmonization with other Italian projects related

to archiving of data of astrophysical interest

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

5

n a m e re so lverb a s ic in te ro p era b ilityre com m en d a tio ns

In te rop e rab ility

D a ta In te rfa ceD o cum e nt

D a ta M o d e l

R D B M S

O O D B M S

p a ra lle l R D B M S

D a tab a se

o n -line

q u a s i-o n -line

o ff-line

D a ta s to re

A rch ive s tru c tu red e fin it ion

T e chn ica l d a ta

S c ien ce d a ta

In g es tion to o ls

U IF

R e q u es t se rv ic ing

R e trie va l to o ls N e tw o rk

te le m e trysc ie nce fra m es

v isu a liza tion

ca lib ra tion

co n s iste n cy ch e cks

q u a lity co n tro l

su p po rt to T A C

d a ta m in ing

in s tr. tre n d an a lys is

p u b lic ou trea ch

A d d it ion a l se rv ices

T N G L o T A rP ro to type

F in a lR e p o rt

N a tio na l a rch ivesD a ta m in ingp ro je c ts

C o ord ina tion

Product Tree

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

6

Interoperability

• Direct participation in “Interoperability WG” of OPTICON

• Prototype LTA uses standards (e.g. VOTable) defined within “Interoperability WG” aims at being compatible with AVO development

• Catalogue of observations available at CDS for basic-level interoperability with other archives

• More in future LTA

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

7

Lessons (being) learned

• Data quality is an issue:– inconsistencies in data (esp. scientific exposures)– some information missing (documentation, observation

schedule, etc.)– improved observing procedures are possible – AI analysis of technical / engineering data identifies some

possible problems in T&I systems data quality (see G.Longo’s talk on Friday)

• An archive must be developed in synch with the observatory:– late development of LTA (even if prototype) late discovery of

inconsistencies

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

8

Coordination with Italian research Grid

• The Italian astronomical community will not build its own AstroGrid

• It will rather coordinate with nation-wide initiatives to foster a Grid infrastructure:– eScience Grid– IG-BIGEST

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

9

The eScience Grid in Italy (1/2)

• Proposal submitted in October 2001 to Ministry of Education University & Research (MIUR)

• Bid for ~15 M€ of available ~30 M€ funds; answer soon

Title: “Enabling Platforms for High-Performance Computational Grids Oriented to Scalable Virtual Organizations”Strategic Programme on Enabling IC Technologies for Knowledge Society

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

10

The eScience Grid in Italy (2/2)

• Objectives:– R&D Grid technological development project (CNR&INFN)– Research on optical networks (CNIT)– Deployment of an Italian eScience Grid infrastructure

(INFN) for:• Astrophysics• Biology• Computational Chemistry• Geophysics• Earth Observation

– International integration by INFN (following requirements)

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

11

Astrophysics in the Italian eScience Grid

• Four demonstrators:– distributed computing– accessing and processing VST data – accessing archives and databases (inter alia the prototype

TNG LTA)– accessing observational facilities as “data nodes”

• About 1 M€ (1/15 of the total MIUR request)

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

12

The Italian Grid for Business, Industry, Government, EScience&Technology IG-BIGEST

• New initiative leveraging from MIUR proposal and aiming at the common development of the IG-BIGEST

• Italian Participation to the Expression of Interest for an Integrated Project "ENABLING GRIDS AND E-SCIENCE IN EUROPE" (EGEE) in the context of 6th EU Framework

• All major Italian Scientific National Research Institutions, Universities and Computing Centres reached general consensus:

– to take part to the large European Initiative EGEE having CERN as leading partner and INFN as the organization representing the Italian Grid eScience comunity

– to have only one large Integrated Project aiming at the development and deployment of the Grid Infrastructure in Europe as Geant does for the Networking

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

13

The participants and the Italian Grid eScience infrastructure

• High bandwidth network provided by Garr-G Geant• High performance computing infrastructures provided by

National Research Institutions, Universities and Super- Computing Centers. Participants :

– INFN, CNR, INAF, INFM, INGV, ENEA……– Computing Centers: CINECA, CILEA, CASPUR….– Universities

• Università di Lecce, Università di Napoli, Università di Pisa, Università di Roma Tor Vergata, Centro Fermi Roma1, Università di Cosenza, Università di Genova, Università di Padova, Università di Politecnico Torino, Università di Salerno, Università di Messina

• Formal participation has been finalized

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

14

Conclusions (and roadmap?)

• TNG LTA Pilot Project has coordination duties (of national archives and with international activities)

• TNG prototype LTA is VO-compliant; so will be future archives (TNG LTA, VST, LBC, etc.)

• INAF board on computing, archives and networks will provide recommendations

• The goal is to have the Italian community joining AVO Phase-B

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

15

DTWRITEDTWRITEWR

user applicationuser application

TAGTAG

TDBTDB

Message Exchange Network

Data Network

Staging Disk

External Disk

DTINDTIN

DTFLUSHDTFLUSH

DTOUTDTOUT

database Data store

AaT

UIF

AaT

UIF

WSS-std instr. non-std instr.

User Disk

VERIFYVERIFY

commands/messagestelemetryscientific datadata descriptors

on-line archive

External processesExternal processes

AaT processAaT process Storageon-linestorage

off-line storage

observer’s CDsoff-line archives CDs

AS

IWS

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

16

INSTRUME

EXP_ID

OBSERVER

DATE_OBS

EXPSTART

EXPTIME

RA

DEC

AIRMASS

CRDELT1

CRDELT2

DETOFF1

DETOFF2

EXP_ID

DET_ID

EXP_ID

INS_MODE

OBS_MODE

OBS_TYPE

EXP_TYPE

FLT_ID

…

…

DET_ID

BITPIX

PIXSIZE

NAXIS

DETSIZ1

DETSIIZ2

GRM_ID

LMP_ID

SLT_ID

…

…

…

…

…

EXP_ID

$_filename

DATE_CR

NAXIS1

NAXIS2

$_path_fts

CD_ID

wss_sess_id

DATE_CR

TIME_CR

DATE_MD

TIME_MD

$_path

CD_ID

…

SA_FILE MDBT

SA_DET_IDSA_INS SA_DET SA_EXP

FITS database shared with Technical database

CD_DATE

NCOPIES

CD_ID

SA_CD

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

17

CD_DATE

NCOPIES

CD_ID

SESS_ID

DATE_CR

TIME_CR

DATE_END

TIME_END

$path

CD_ID

CD_TAB

MDBT

shared with FITS database

Technical database

VALUE

TAG

TAG

ACRON

TAG

HASHTAB

AAA database

ACRON

TAG

HASHTAB

XYZ database

…

STR

ACRONTAG

HASHTAB

CUB database

…

TIME

VALUE

TAG

TAG

STR

TIME

VALUE

TAG

TAG

STR

TIME

Garching, 10-14 June 2002 “TOWARD AN INTERNATIONAL VIRTUAL OBSERVATORY”

18

Cyclic operations scheme

from: di Serego et al., 26 Jan 1995