1

The EIS Experience: Lessons LearnedThe EIS Experience: Lessons Learned

May 12, 2005

2

EIS 1996-2004

Original mission: – Limited-term experimental project– Preparation of data sets for VLT commissioning and early

operation– Coordination with ESO community (SWG, visitors) – Train & disseminate tools as possible

In 1999 mission expanded– Support long-term public surveys (1997-present; 8 years)– Develop infrastructure for public surveys – Develop image processing engine – Develop survey software system

– Ramp-up to VST and VISTA

BackgroundBackground

Mission: R&D & operation; science & software; but ...

3

Community ParticipationCommunity Participation

OPC (approval)

Survey Working Group (design, oversight)

– 29 participants (2 committees)

– 24 institutes

– 7 member states

Visitor Program (operation/development)

– 43 participants

– 15 institutes

– 7 member states

Full & broad community involvement in every aspect of the project

4

EIS Team responsibilities:EIS Team responsibilities:

Observations (preparation/observing) Software Development:

– Infrastructure (GUIs, database, architecture)– Image processing engine– Scientific algorithms– System integration & tests

Hardware procurement Preparation, verification & delivery of survey products

WEB development & maintenance

Publications & reports

Recruitment of visitors & administration of intense visitor program

Composition:

– 5-6 FTE/year (astronomers & software developers)

5

EIS SurveysEIS Surveys

6

Summary of Summary of surveyssurveys (07/97- )(07/97- )

9 surveys in < 6 years (FORS, FLAMES & VIMOS) 13 strategies 3 Telescopes (NTT, VLT, 2.2m) 5 imagers (EMMI, SUSI2, SOFI, ISAAC, WFI) 22 filters 240 nights in visitor mode alone done by the team 73,000 frames of raw data; 37,000 science exposures

– WFI 7,000 science; 23,000 total– SOFI 18,200 science; 31,000 total– ISAAC 11,800 science; 21,000 total

27 public releases (several data products, software, zeropoints) Observations still ongoing (SOFI 09.04; WFI: 02.05)

7

Phase-1 (Phase-1 (03.97-10.9903.97-10.99))

Best-effort

Simple image reduction pipeline using

– Available packages (IRAF, Eclipse)

– Tools (Drizzle, LDAC, SExtractor)

– Wrapper: shell scripts

Limited software development (adaptations, bug-fixing,

small-scale new developments)

Help from experts in-house and in the community

8

Limitations of best-effort approach:Limitations of best-effort approach:

Problems with the data:– Calibration (loss of flux in 1998 release associated to Jitter)– Serious problems with the astrometric calibration of WFI – Rapid increase ( > 6 x) in data volume in 1999 – Reductions mostly manual; no history

Over-reliance on a few people making operation vulnerable & fatigue Departure of key team members => 6 months interruption in reductions

& development Different environments for optical and infrared Unsuitable hardware & software

9

April ‘99 April ‘02 April ‘04

SOFTWARE

Phase-2Phase-1

EIS Data volume: time evolutionEIS Data volume: time evolution

Project split into 2 phases

10

Lessons learned Lessons learned

• Best-effort approach one-off (not 24 times in 6 years) unsustainable in the long-term error-prone, hard to recover disruptive, leaves no legacy impedes progress of development (developers become operators)

• Need of framework and stable core group to preserve know-how and inherit code

• Resource-limited operation requires large degree of automation

• Handling large amount of data/information big challenge in survey context (differs from data-in/data-out problem)

Requirement:1) develop new image processing code2) develop integrated reduction system

11

Procedures standardized and accessible via GUIs

Access to data/information transparent to user (SE/DAL)

Integrated environment (CVS, ARS, Database, Web)

Common optical/IR image processing engine (90,000 lines of C-code)

System Wrapper (Python > 400,000 lines of code)

XML technology (configuration; logs; Web; database contents)

Self-describing products with quality parameters

Uniqueness, versioning and history of products

EIS data reduction system EIS data reduction system (06.00-09.04)(06.00-09.04)

Medium-size project (by industrial standards) done by non-specialists

12

13

ReleasesReleases Maximum interval between end of survey and release < 3 years Products

– Night (XMM, DPS, PF)– Stacks– Mosaics– Source catalogs (SExtractor, DAOPHOT)– Catalogs of clusters of galaxies, quasars, low-mass stars– WFI zero-points for 150 nights over 5 years– WFI data covering 29 square degrees

Highlights 2004– Release of EIS/MVM code– 9 releases in 4 months; 11 releases in 2004 (1 release/month)– Data releases with product logs and READMES– 7000 ISAAC frames; 3000 WFI frames

14

Raw data

– Requests: 1367

– Products: 84589

– Volume: 6.1Tb

Survey products

– Requests: 672

– Products: 9292

– Volume: 0.93 Tb

Software: ~ 62 users

Total of last 46 months

– Requests: 2039 (44/mo)

– Products: 93851 (2040/mo)

– Volume: 7 Tb

Over project lifetime

– 27 releases

– 40 requests/mo

– Products: > 100,000

– 12,500 prod/year; 34 prod/day

Data Request StatisticsData Request Statistics

15

Project LegacyProject Legacy

High-performance, instrument-independent image processing pipeline

(tested for all ESO imagers; publicly available)

Integrated, end-to-end data reduction system to monitor & reduce multiple

surveys from a single desktop

Survey infra-structure (WG, database, Web interface, release)

Blueprint for a modern data reduction & analysis system

System easily adaptable for different applications

16

SummarySummary

Over 25 man-years of development

System supports

– optical/infrared, single/multi-chip instruments

– configurable workflows for un-supervised operations

System consists of several pipelines

– Image processing

– Photometric calibration

– Stack & Mosaics

– Catalog (DAOPHOT, SExtractor)

– Science applications (plug-ins)

Mature, extensively-tested system before UKIDSS, VST and VISTA commissioning

Work in progress

EIS Data Reduction System