Lofar Survey workshop 8/3/2007 Lofar -Surveys and AstroWise OmegaCEN NOVA – Kapteyn Institute -RuG...

Lofar Survey workshop 8/3/2007

Lofar -Surveys and AstroWise

Lofar -Surveys and AstroWise

OmegaCENNOVA – Kapteyn Institute -RuG

8 March 2007Lofar Survey Workshop

OmegaCENNOVA – Kapteyn Institute -RuG

8 March 2007Lofar Survey Workshop

Edwin A. ValentijnEdwin A. Valentijn


“Moore’s law” for N-body simulations

“Moore’s law” for N-body simulations

Courtesy Simon White


Instrument: e-scienceInstrument: e-science

= share resources beyond workstation• Compute power:

– AstroWise grid (RuG, Leiden, MPE, Bonn, INAF)– EGEE-3

• Storage:– AstroWise fileservers (Leiden, RuG, Nijmegen, MPE, Bonn,

INAF)– BiG GRID (RuG- NikHef- SARA)

• Database: all meta + calibration +source data– AstroWise - Oracle

• Application programs + doc’s• Web services• Interoperability: EGEE- EURO-VO

= share resources beyond workstation• Compute power:

– AstroWise grid (RuG, Leiden, MPE, Bonn, INAF)– EGEE-3

• Storage:– AstroWise fileservers (Leiden, RuG, Nijmegen, MPE, Bonn,

INAF)– BiG GRID (RuG- NikHef- SARA)

• Database: all meta + calibration +source data– AstroWise - Oracle

• Application programs + doc’s• Web services• Interoperability: EGEE- EURO-VO


SDSS

ROSAT

2MASS

FIRST GAIA

Scattered ArchivesScattered Archives


ROSATFIRST GAIA

SDSS2MASS

Euro-VO


ManipulationManipulation

X-matchX-match

VisualizationVisualization

Direct links to:Direct links to:

CDS Aladin

ALADINALADIN


TOPCAT VOTable viewTOPCAT VOTable view


Multi-tier models for LHC & LOFAR

Multi-tier models for LHC & LOFAR

Large Hadron Collider• Tier-0 (CERN):

– copy of all raw data• Tier-1 (~8-10 centers)

– archive 1/n fraction of raw data & reconstructed data

– regular re-processing of the raw data

– archiving data from Tier-2 centers

– provide central grid services: grid accessible computing and data resources

– support coordination• Tier-2 (~100 centers):

– data analyses– no data archiving

Large Hadron Collider• Tier-0 (CERN):

– copy of all raw data• Tier-1 (~8-10 centers)

– archive 1/n fraction of raw data & reconstructed data

– regular re-processing of the raw data

– archiving data from Tier-2 centers

– provide central grid services: grid accessible computing and data resources


– data analyses– no data archiving

LOFAR• Tier-0 (RUG): central processor

– lots of processing, limited archive• Tier-1 (~6-8 centers)

– specialized science analysis– advanced processing of (large)

data sets– dedicated archive, also for Tier-2

centers– provide central grid services:

grid accessible computing and data resources


– individual scientists– data analyses– no data archiving

LOFAR• Tier-0 (RUG): central processor

– lots of processing, limited archive• Tier-1 (~6-8 centers)

– specialized science analysis– advanced processing of (large)

data sets– dedicated archive, also for Tier-2

centers– provide central grid services:

grid accessible computing and data resources


– individual scientists– data analyses– no data archiving


AstroWise paradigmAstroWise paradigm

“Classical” paradigmForward chaining

Target processing - AweBackward chaining

waterfall model TIER architecture

User hunts upstream

driven by input raw data Driven by query of user

Process in pipelineworkflow

Process in bits and pieces on the flyBackward chaining

Operators push data User pulls data

Results in releases Provide information system

Static archives – publish Dynamic archives –publish Internet

Raw data - obsolete Raw data is sacred


Virtual Survey TelescopeVirtual Survey Telescope

NL:Leiden, Nijmegen, GroningenEU: Napoli, Munchen, Bonn, Heidelberg, Paris Santiago

NL:Leiden, Nijmegen, GroningenEU: Napoli, Munchen, Bonn, Heidelberg, Paris Santiago

raw pixel data pipelines/cal files catalogues

all integrated in one information system100% data lineage

1 dec 2006 delivered !• distributed services

• processing GRID Storage GRID • Methods/services GRID

raw pixel data pipelines/cal files catalogues

all integrated in one information system100% data lineage

1 dec 2006 delivered !• distributed services

• processing GRID Storage GRID • Methods/services GRID


Current Instruments/projectsCurrent Instruments/projects

• VST- OmegaCAM– GT, KIDS, VESUVIO, OmegaWhite, OmegaTrans

• VISTA– UltraVISTA, VIKING via Edinb.

• SuprimeCAM @ Subaru• [email protected], WFC@ING• ACS@HST Coma legacy survey• Panstarrs• Lofar

– Surveys– EoR

• VST- OmegaCAM– GT, KIDS, VESUVIO, OmegaWhite, OmegaTrans

• VISTA– UltraVISTA, VIKING via Edinb.

• SuprimeCAM @ Subaru• [email protected], WFC@ING• ACS@HST Coma legacy survey• Panstarrs• Lofar

– Surveys– EoR

mailto:[email protected]

mailto:[email protected]


Lofar StorageLofar Storage


Astro-Wise VO Properties Benefits integrated dynamic db

Astro-Wise VO Properties Benefits integrated dynamic db

• on-the fly re-processing• 5LS: 5 Lines Script• All bits are traced• Administration for parallel processing compute GRID SETI@home• Global solutions –astrometry/photometry• Build–in workflow• Fully user tunable – own provided script• Context: projects/surveys, instruments, mydb• Publish directly in EURO-VO

• on-the fly re-processing• 5LS: 5 Lines Script• All bits are traced• Administration for parallel processing compute GRID SETI@home• Global solutions –astrometry/photometry• Build–in workflow• Fully user tunable – own provided script• Context: projects/surveys, instruments, mydb• Publish directly in EURO-VO


TARGET diagramLofar Survey/ ST ACS

TARGET diagramLofar Survey/ ST ACS


www.astro-wise.orgwww.astro-wise.org

http://www.astro-wise.org/

http://www.astro-wise.org/


www.ASTRO-WISE/portalwww.ASTRO-WISE/portal


Web services- object viewerWeb services- object viewer


Web services- object makercompute GRID

Web services- object makercompute GRID


Planning 2007Planning 2007

• ADD with Astron - enable communities

• Publish WENSS in VO• connect to EGEE• AstroWise at Astron• Build e-SOC at RC RuG• Source extraction -> source

parameters services • Involve Lofar community

• ADD with Astron - enable communities

• Publish WENSS in VO• connect to EGEE• AstroWise at Astron• Build e-SOC at RC RuG• Source extraction -> source

parameters services • Involve Lofar community


e-science me-sciencee-science me-science

Lofar Survey workshop 8/3/2007 Lofar -Surveys and AstroWise OmegaCEN NOVA – Kapteyn Institute -RuG...

Documents

Transcript of Lofar Survey workshop 8/3/2007 Lofar -Surveys and AstroWise OmegaCEN NOVA – Kapteyn Institute -RuG...