ASTRO-WISE- federation OmegaCEN

Virtual Survey System sept 04

ASTRO-WISE- federation OmegaCEN

ASTRO-WISE- federation OmegaCEN

AstroWise a Virtual Survey System

OmegaCAM – Lofar – AstroGrid –((G)A) VO

AstroWise a Virtual Survey System

OmegaCAM – Lofar – AstroGrid –((G)A) VO

Edwin A. ValentijnEdwin A. Valentijn


VSTVST


Paranal Paranal

Mid 2005Mid 2005

20072007


Large Data VolumeLarge Data Volume

• Wide-field imaging = vast amounts of data– VST sees equivalent of Southern sky with 0.2” pixels in 3 years. 100 Tbyte of image data and Tbytes of source list data A project like KIDS (1000 Sq Deg) has >10^6 8Mpix raw data

images

• Wide-field imaging = vast amounts of data– VST sees equivalent of Southern sky with 0.2” pixels in 3 years. 100 Tbyte of image data and Tbytes of source list data A project like KIDS (1000 Sq Deg) has >10^6 8Mpix raw data

images

• Handling of the data is non-trivial– Pipeline data reduction– Calibration with very limited resources– Things change in time:

–Physical changes (atmosphere, various gains)–Code (new methods, bugs)–Human insight in changes

– Working with source lists

Science can only be archive based

• Handling of the data is non-trivial– Pipeline data reduction– Calibration with very limited resources– Things change in time:

–Physical changes (atmosphere, various gains)–Code (new methods, bugs)–Human insight in changes

– Working with source lists

Science can only be archive based


ASTRO-WISE Mission Virtual Survey SystemASTRO-WISE Mission Virtual Survey System

• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification image/calibration pipelines– Execution of source extraction algorithms- catalogues– Archiving or regenerate on the fly dynamically – Paradigm: no static data releases but dynamic on request data– federated to link different data centers

• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification image/calibration pipelines– Execution of source extraction algorithms- catalogues– Archiving or regenerate on the fly dynamically – Paradigm: no static data releases but dynamic on request data– federated to link different data centers

• Dynamical archive continuously grows, can be used for – small or large science projects– generating and checking calibration data– exchanging methods, scripts and configuration

• Dynamical archive continuously grows, can be used for – small or large science projects– generating and checking calibration data– exchanging methods, scripts and configuration

raw pixel data pipelines/cal files catalogues raw pixel data pipelines/cal files catalogues


Keys -SolutionKeys -Solution

• Procedurizing– Data taking at telescope for both science and

calibration data• Observing Modes: —Stare —Jitter —Dither —SSO

• Observing Strategies: —Stan —Deep —Freq —Mosaic

– Full integration with data reduction– Design- ADD – Data model (classes) defined for data reduction and

calibration– View pipeline as an administrative problem

• Procedurizing– Data taking at telescope for both science and

calibration data• Observing Modes: —Stare —Jitter —Dither —SSO

• Observing Strategies: —Stan —Deep —Freq —Mosaic

– Full integration with data reduction– Design- ADD – Data model (classes) defined for data reduction and

calibration– View pipeline as an administrative problem


Calibration proceduresCalibration procedures

Sanity checksSanity checks

Quality controlQuality controlCalibration proceduresCalibration procedures

Image pipelineImage pipeline

Source pipelineSource pipeline


AstroWise PipelinesAstroWise Pipelines

Photometric pipelinePhotometric pipeline

Bias pipeline

Flatfield pipeline

Image pipeline

Source pipeline


DB – engineDB – engine

Long search: Objectivity, CERN HEP July 2002 Oracle contract

– reference – licenses - consultancy

Oracle 9i -> 10g:- Full oo support + Python to db persistency- Terabyte scalability through “partitioned tables”- Administration tools- federation through “Advanced replication” to evolve into cross

site links (=references= pointers)– Interoperability: STREAMS connects to others: SyBase, MySQL– Python I/F SQL – OCI- Oracle db (Python binding)

Long search: Objectivity, CERN HEP July 2002 Oracle contract

– reference – licenses - consultancy

Oracle 9i -> 10g:- Full oo support + Python to db persistency- Terabyte scalability through “partitioned tables”- Administration tools- federation through “Advanced replication” to evolve into cross

site links (=references= pointers)– Interoperability: STREAMS connects to others: SyBase, MySQL– Python I/F SQL – OCI- Oracle db (Python binding)


Concepts of federationConcepts of federation

• Federation maintained by a single database- Oracle9i• Full history tracking by linking (joints, references)

– of all input that went into result – providing on-the fly reprocessing

• Dynamical archive - Context as object attributes– Project: Calibration, Science, Survey, Personal– Owner: Pipeline, Developer, User– Strategy: —Standard —Deep —Freq — Mosaïc– Mode: —Stare —Jitter — Dither —SSO

– Time: Time stamping VO I/F Publish• Software standards

– Classes/data model/procedures– 00 – inheritance/ persistency

– Python scripts/ c-libraries USER Python

• Federation maintained by a single database- Oracle9i• Full history tracking by linking (joints, references)

– of all input that went into result – providing on-the fly reprocessing

• Dynamical archive - Context as object attributes– Project: Calibration, Science, Survey, Personal– Owner: Pipeline, Developer, User– Strategy: —Standard —Deep —Freq — Mosaïc– Mode: —Stare —Jitter — Dither —SSO

– Time: Time stamping VO I/F Publish• Software standards

– Classes/data model/procedures– 00 – inheritance/ persistency

– Python scripts/ c-libraries USER Python


Contents of federationContents of federation

• Raw data– Observed images– Ancillary information

• Calibration results– Calibration files time stamped

• Reduced images– Single observation– Coadded images

• Software– Methods (pipelines) for processing calibration– Configuration files

• Source lists – catalogues– Extracted source information– Associated among different data objects

• Raw data– Observed images– Ancillary information

• Calibration results– Calibration files time stamped

• Reduced images– Single observation– Coadded images

• Software– Methods (pipelines) for processing calibration– Configuration files

• Source lists – catalogues– Extracted source information– Associated among different data objects

SourceList

RA0, Dec0

t0

V filterSource

ListV filter

t2

RA0?, Dec0?

SourceList

V filterRA0?, Dec0?

t1

SS

C

I

TES

A

A

O

B filterRA0, Dec0

SourceList

SourceList

V filterRA0, Dec0

SS

C

I

TES

A

A

OS

S

C

I

TES

A

A

O

SourceList

B filterRA0, Dec0

SourceList

B filterRA1, Dec1

SourceList

V filterRA0, Dec0

SourceList

B filterRA1, Dec1

SourceList

B filterRA0, Dec0

SourceListB - VRA0, Dec0

SourceList

V filterRA0, Dec0

t0

SourceList

V filterRA0?, Dec0?

t1

SourceList

V filter

t2

RA0?, Dec0?

SourceList

V filterRA0?, Dec0?

tn

Proper motion

Red quasars

Supernovae

Tbyte source lists brains make the associations

Tbyte source lists brains make the associations

Link -lists

as fast as possible


StatusStatus

• From abstract design to working prototype FIRST VIRTUAL LIGHT

17 April 2003

– [email protected]– INT

• Federated – USM– OAC

• Summer 2004

– First massive ingest

• From abstract design to working prototype FIRST VIRTUAL LIGHT

17 April 2003

– [email protected]– INT

• Federated – USM– OAC

• Summer 2004

– First massive ingest


VST - Virtual Survey TelescopeVST - Virtual Survey Telescope


New componentsNew components• Python binding to Oracle – 5LS ۷• Scripting paradigm-۷

– Wrappers– Make metaphor– Persistency, links as attributes

• Fileserver ۷• Parallelisation code ۷• Catalogue class/ associating HTM ۷• Federating with Oracle Streams• User interfaces

– Tell me everything tool<- >on the fly reprocessing

• Python binding to Oracle – 5LS ۷• Scripting paradigm-۷

– Wrappers– Make metaphor– Persistency, links as attributes

• Fileserver ۷• Parallelisation code ۷• Catalogue class/ associating HTM ۷• Federating with Oracle Streams• User interfaces

– Tell me everything tool<- >on the fly reprocessing


New technologies-paradigmNew technologies-paradigm

• raw data centralized – on the fly reprocessing = VSS

• sociology- project management• ++ • Python at top layer(s) = VSS • parallel – IP - seti@home• user tunable pipelines = VSS• ++ db• 5 LS – 5 lines script = VSS• Distribution – VO = VSS

• raw data centralized – on the fly reprocessing = VSS

• sociology- project management• ++ • Python at top layer(s) = VSS • parallel – IP - seti@home• user tunable pipelines = VSS• ++ db• 5 LS – 5 lines script = VSS• Distribution – VO = VSS


development issuesdevelopment issues

• Indexing Terabytes on the sky• Associating entries • World wide linking entries/distribution

– (referencing) – registries– centralized – replication – p2p– Other p2p’s?

• Visualisation - imaging • Visualisation -multidimensional data• VO world - Euro-VO

• Indexing Terabytes on the sky• Associating entries • World wide linking entries/distribution

– (referencing) – registries– centralized – replication – p2p– Other p2p’s?

• Visualisation - imaging • Visualisation -multidimensional data• VO world - Euro-VO

ASTRO-WISE- federation OmegaCEN

Documents

Transcript of ASTRO-WISE- federation OmegaCEN