Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #1PACS Overview and new developments in Herschel Data Processing Stephan Ott Herschel Science Data Processing Development Manager Herschel Science Data Processing Coordinator Herschel Science Centre on behalf of all contributors of the Herschel mission Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #2PACS Overview Overview of Herschel Data Processing Major improvements during the last year What can HSC/NHSC and HIPE do for your Herschel Data? Further plans and milestones What can you do for HIPE? Questions? NGC 1999 observed by Herschel ESA / PACS & SPIRE Consortium, Tom Megeath, NASA/JPL-Caltech/Univ. of Toledo Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #3PACS Overview of Herschel Data Processing Rosette cloud, credits ESA/PACS & SPIRE Consortium/HOBYS Key Programme Consortia Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #4PACS Overview of Herschel Data Processing The system combines for the first time data retrieval, pipeline execution and scientific analysis in one single environment All tools for data reduction and analysis, e.g. also the expert applications for e.g. instrument calibration are part of the Data Processing System. Therefore the community has access to the same system as the instrument experts The Herschel Data Processing software is coded in Java/Jython to be license free and portable for different operating systems Formal support is provided for Windows XP, Vista and Windows 7,Linux, Mac OS X 10.5 ("Leopard") and Mac OS X 10.6 ("Snow Leopard") Herschel Science Centre (ESA), the Instrument Control Centres (HIFI, PACS and SPIRE) and NHSC jointly manage and contribute to the Herschel Data Processing System Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #5PACS Overview of Herschel Data Processing Herschel Data Processing is a major project, with over 200 contributors and currently 60 full-time equivalents working on calibration, coding, documentation, quality control, testing and tutoring Very distributed development we work 24/5 (and often 24/7) on Herschel Data Processing System 300 data processing related software tickets are raised and resolved each month A major HIPE version is released around each three months. This is currently HIPE 5.0 Minor HIPE versions (currently 5.2) are released around each three weeks. These serve also as the operational version to generate the standard products All releases can be downloaded viaLatest HIPE developer releases are available via Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #6PACS Major Improvements during the last year Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #7PACS Improvements for user tools, HIPE and pipeline system User friendliness and performance Pre-installation steps for installer Notification of updated user releases HIPE perspectives have been simplified Autocompletion in the console Jobs can be shown and cancelled individually via status progress indicator Option to gather user feedback Restructuring of documentation Support of plug-ins Updates of documentation start-up page, links to data reduction guides from Help menu Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #8PACS Improvements for PACS Interactive pipeline scripts Restructuring of menus reflecting observation type and astronomer's science case (extended/point source, raster/pointed,...) Overhaul of interactive pipeline scripts, convenient end-end processing to publication-quality spectra and maps chopped spectroscopy (line scans, SEDs) photometer scan maps Two-stage with masking high signals before highpass filter Madmap fully available and working on scan/cross-scan pairs Significant improvements in the SSO astrometry chopped photometry Spectrum explorer / PACS products integration: Spectrum explorer and tools (fitting,...) can now be used for all PACS products Wavelength calibration based on in-orbit data / polynomial fits to grating position-wavelength relation Cross-scan pipeline processing combines two observations into a higher quality mosaic Reference documentation and data reduction guide Reduction of size of user build Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #9PACS Improvements for PACS products Range Spectroscopy, pointed Chop/Nod Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #10PACS Improvements for PACS products Line Spectroscopy, pointed Chop/Nod Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #11PACS Improvements for PACS products Photometry Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #12PACS What can HSC/NHSC and HIPE do for your Herschel Data? Rosette cloud, credits ESA/PACS & SPIRE Consortium/HOBYS Key Programme Consortia Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #13PACS Herschel Pipeline Processing and Data Quality Control Pipelines are executed on the ESAC Grid to produce Herschel Products to different reduction levels Level 0raw data Level 1instrumental and satellite effects removed Level 2scientific analysis can be performed The data are available in the Herschel Science Archive after the processing is finished (usually on the same day of reception of the data from the satellite) The fast data quality control cycle by the Technical Assistants and Instrument Calibration Scientists takes a few days data quality control is a combination of automatic screening and manual inspection quality control reports are electronically distributed to experts Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #14PACS Bulk and on-demand reprocessing Currently HSC performs a bulk reprocessing of all data taken up to now twice a year using an even HCSS version (HCSS 2, 4, 6, 8, ) Bulk reprocessing with HCSS 4 is nearly complete Automatic quality control is performed using the latest quality control flags Spot-checks are performed on a few observations Users can request reprocessing of observations with the latest operational release using on-demand processing Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #15PACS Herschel Helpdesk Contact the HSC helpdesk or the NHSC helpdesk for any questions or suggestions you have on Herschel Data Processing (and beyond) Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #16PACS Further Plans and Milestones Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #17PACS Next Steps and Schedule March 2011Release of HIPE 6.0 March 2011start of bulk reprocessing of all data taken up to now with HCSS 6 14 18 March 2011Data Processing ESAC Spring 2011Release of eHSA 4.0 ~ May 2011Release of HIPE 7.0 ~ August 2011Release of HIPE 8.0 ~ November 2011Release of HIPE 9.0 ~ March 2012Release of HIPE 10.0 Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #18PACS Foreseen improvements for user tools, HIPE and pipeline system Reduce HIPE freezes and hangs Improvement of calibration and data reduction pipelines Improvement of user friendliness Data manager for Herschel spectra to improve the robustness of spectrum explorer interface Taskvariable association and useful tooltips Simplified data input/output. Access to reprocessed data. Addition of useful example scripts to the build Harmonization of final spectra and VO. Workflow between Herschel products/HIPE and VO Combination of data from the different Herschel instruments Publication quality plots Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #19PACS Foreseen improvements for user tools, HIPE and pipeline system Upgrade to Jython 2.5 Improved automatic testing to increase confidence in developer builds and speed-up delivery cycle for releases to the community Improvement of code quality so the system will be maintainable with the reduced manpower we can expect during post-operations Improvements to quality control framework, especially to support the bulk reprocessing exercises Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #20PACS Foreseen improvements for PACS Calibration and mapping of extended emission Tool to handle unchopped spectroscopy, including spectroscopy of extended sources and models of PSFs Flatfielding for spectroscopy Improved removal of artefacts including striping from scan maps and elongated images for moving target observations Documentation how to perform end-to-end processing of calibrated full-range spectra with PACS and SPIRE Improvement of quality control criteria and rules Implementation of browse products and postcards for the Herschel Science Archive Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #21PACS What can you do for HIPE? Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #22PACS Provide your feedback on documentation on HIPE on HIPE crashes Participate in the Herschel Data Processing Users and Interest Groups. You can subscribe atParticipate in the HIPE user/developer workshops Reply to the DP Questionnaire Collaborate to HIPE development as contributor and tester Data Processing Workshop NHSC, Pasadena, CA 02 04 February 2011 Stephan Ott VG #24PACS Your questions to me?