The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex...
Transcript of The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex...
![Page 1: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/1.jpg)
EsoReflex , NEON School Garching, May 2016
The ESO Recipe Flexible Workbench EsoReflex
Sabine Moehler, Wolfram Freudling
http://www.eso.org/sci/software/pipelines/reflex_workflowshttp://www.eso.org/sci/software/pipelines/reflex_workflows
![Page 2: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/2.jpg)
EsoReflex , NEON School Garching, May 2016
EsoReflex Project
Recommended environment to run ESO VLT pipeline “recipes”
Gives users enough information how to run recipes
Reflex is based on the Kepler Workflow engine(https://kepler-project.org), which provides a graphical user interface (Java)
In case of problems please contact [email protected] (include information about workflow, data, operating system, see also last slide)
Make pipeline recipes accessible to general users
![Page 3: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/3.jpg)
EsoReflex , NEON School Garching, May 2016
Design Goals
Document dependencies
Organize data
Run pipeline with single click
Monitor progress
Do bookkeeping
Allow pre-defined interaction*
Allow insertion of user procedures in several command line languages
*currently interactive python windows (python needed for installation)
![Page 4: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/4.jpg)
EsoReflex , NEON School Garching, May 2016
Workflow driven Data ReductionA&A 559, A96 (2013)
Forchi, V., Reflex User Manual,
VLT-MAN-ESOs-19000-5037
Reflex tutorials
http://eso.org/reflex
Pipeline manuals
![Page 5: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/5.jpg)
EsoReflex , NEON School Garching, May 2016
Basic Reflex Workflow
![Page 6: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/6.jpg)
EsoReflex , NEON School Garching, May 2016
Reflex Data Organisation
![Page 7: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/7.jpg)
EsoReflex , NEON School Garching, May 2016
Reflex OCA Rules
Processing steps imply Data Organization
Data organization defined in “OCA rules”: text file
Three types of rules: Classification ( „This is a Raw Dark“) Organization („These Raw Darks are processed together”) Association (“select Biases based on properties of Raw Darks“)
classifying, organizing and associating astronomical databased on their meta-data (FITS keywords).
![Page 8: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/8.jpg)
EsoReflex , NEON School Garching, May 2016
untick to deselect file
![Page 9: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/9.jpg)
EsoReflex , NEON School Garching, May 2016
Reflex OCA Rules
Processing steps imply Data Organization
Data organisation defined in “OCA rules”: text file
Three types of rules: Classification ( „This is a Raw Dark“) Organization („These Raw Darks are processed together”) Association (“select Biases based on properties of Raw Darks“)
DO produces DataSets: set of science files to be processed together, plus all additional files needed for processing
Each file in DataSet has a category (e.g. “raw bias”) and a purpose action1/action2/... (e.g. „MasterBias/MasterDark“)
classifying, organizing and associating astronomical databased on their meta-data (FITS keywords).
![Page 10: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/10.jpg)
EsoReflex , NEON School Garching, May 2016
Reflex Data Organisation
category
purpose
![Page 11: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/11.jpg)
EsoReflex , NEON School Garching, May 2016
![Page 12: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/12.jpg)
EsoReflex , NEON School Garching, May 2016
Reflex Data Organisation
Status of Datasets
Inspect Datasets
Select which Datasets to process
Grey = Incomplete Dataset(calSelector recommended for download)
![Page 13: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/13.jpg)
EsoReflex , NEON School Garching, May 2016
Reflex FitsRouter
![Page 14: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/14.jpg)
EsoReflex , NEON School Garching, May 2016
FitsRoutersorting by category
routing by category is explicit: specify what recipe needs
each recipe needs well defined input (e.g. category raw biases) and creates well defined output (e.g. master bias)
relations determine work”flow”
![Page 15: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/15.jpg)
EsoReflex , NEON School Garching, May 2016
Pipeline Recipes
![Page 16: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/16.jpg)
EsoReflex , NEON School Garching, May 2016
SofSplitter:Sorting by Purpose
Sorting by Purpose is implicit Purpose defined in OCA rules Assumes that each recipe has at least one unique file category
![Page 17: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/17.jpg)
EsoReflex , NEON School Garching, May 2016
Basic Reflex Workflow
Information between actors* is passed by tokens
Reflex uses Set Of Files/Parameters (SOF/SOPs) as tokens
SOFs include files, categories (like “normal” SoFs) + purpose
Data Organizer organizes data in “DataSets”
DataSets are SOFs that include everything needed to process one set of science observations
*actor: A step in a workflow, i.e. a program that needs some external input to run.
SOP
DataSet
![Page 18: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/18.jpg)
EsoReflex , NEON School Garching, May 2016
Looping over Datasets,Output Organisation
![Page 19: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/19.jpg)
EsoReflex , NEON School Garching, May 2016
Looping over Datasets,Output Organisation
![Page 20: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/20.jpg)
EsoReflex , NEON School Garching, May 2016
• Double-click on entry to change
• Avoid trailing blanks
• DO scans all files in RAW_DATA_DIR and CALIB_DATA_DIR
• Book keeping allows to skip already executed processing steps (if all files and parameters are the same as before)
• “Lazy Mode”
Directories
![Page 21: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/21.jpg)
EsoReflex , NEON School Garching, May 2016
Tools
“Animate at Runtime” to follow progress
![Page 22: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/22.jpg)
EsoReflex , NEON School Garching, May 2016
Parameters
Double-click on entry to change
Important:GlobalPlotInteractivityRecipeFailureMode
![Page 23: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/23.jpg)
EsoReflex , NEON School Garching, May 2016
Saving a Workflow
XML can be exchanged with others (ASCII file, “Save as” should be used only locally)
Paths to python scripts and OCA rules are saved together with the workflow
To share: Make sure that the pipeline is installed in the target system Export as XML Edit the XML and change the python scripts/OCA file paths Open the workflow and change the data paths
![Page 24: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/24.jpg)
EsoReflex , NEON School Garching, May 2016
Complex Workflows
![Page 25: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/25.jpg)
EsoReflex , NEON School Garching, May 2016
Complex Workflows
![Page 26: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/26.jpg)
EsoReflex , NEON School Garching, May 2016
Existing Reflex Workflows
http://www.eso.org/sci/software/pipelines/
![Page 27: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/27.jpg)
EsoReflex , NEON School Garching, May 2016
Upcoming Reflex Workflows
Most new instruments pipelines will come with workflows
VIMOS (imaging)HAWK-IVIRCAMSPHEREAMBER
ESPRESSOFLAMES-GIRAFFE
GRAVITYMATISSE
![Page 28: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/28.jpg)
EsoReflex , NEON School Garching, May 2016
Advanced Topics
Recipe parameters
Re-executing a recipe
Command line interface
Sharing workflows
Debugging workflows
![Page 29: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/29.jpg)
EsoReflex , NEON School Garching, May 2016
Recipe parameters (I)
The interactive windows use some default values as starting points. Right click on actor, select “Open Actor”, and change the INIT_* variables to change them
![Page 30: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/30.jpg)
EsoReflex , NEON School Garching, May 2016
Recipe parameters (II) Right click on actor, select “Open Actor”
Right-click on RecipeExecuter, select “Configure Actor”
![Page 31: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/31.jpg)
EsoReflex , NEON School Garching, May 2016
Re-executing a recipe
Sometimes a recipe fails due a number of factors: bad parameters, wrong data, software bugs, etc...
The workflow reacts to the failure of the recipe depending on parameter “Recipe Failure Mode”
The Product Explorer allows to open the bookkeeping directory for a given product file with right-click on the file and open xterm
Then you can re-execute the recipe with the same parameters and output path
./cmdline.txt
![Page 32: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/32.jpg)
EsoReflex , NEON School Garching, May 2016
Re-executing a recipe
Re-execute the recipe with same parameters and output path:
./cmdline.txt To change parameters edit the file
<recipe>.rc
To use defaults just call esorex esorex <recipe> data.sof
![Page 33: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/33.jpg)
EsoReflex , NEON School Garching, May 2016
Command line interface examples
List all available workflows: esoreflex –l
Load kmos workflow: esoreflex kmos
Process all new data in my directory without interaction: esoreflex –n xshooter -RAW_DATA_DIR=/data/xshooter_data
Rerun all previously failed datasets with nonstandard workflow parameter:
esoreflex –n muse –RAW_DATA_DIR=/data/muse_data \
-SelectDatasetMode=failed –recomputeWCS=true
![Page 34: The ESO Recipe Flexible Workbench EsoReflexEsoReflex , NEON School Garching, May 2016 EsoReflex Project Recommended environment to run ESO VLT pipeline “recipes” Gives users enough](https://reader033.fdocuments.net/reader033/viewer/2022041601/5e30c70915a73436bd7fe5cf/html5/thumbnails/34.jpg)
EsoReflex , NEON School Garching, May 2016
Additional Information
Reflex page http://www.eso.org/sci/software/reflex/
Data Reduction FAQ
http://www.eso.org/sci/data-processing/faq.html
ESO pipeline page http://www.eso.org/sci/software/pipelines/
In case of problems please report to [email protected] and include: log (or description) of error/problem, data set id (or list of raw files), pipeline version, operating system