Dominik Stokłosa

Poznań Supercomputing and Networking Center, Supercomputing Department

INGRID 2008 Lacco Ameno, Island of Ischia , ITALY, April 9-11

Workflow management in Remote Instrumentation Infrastructure – based on e-VLBI experiences

Introduction to the e-VLBIIntroduction to the e-VLBI

PSNC in EXPReS - FABRIC PSNC in EXPReS - FABRIC

Grid – VLBI design & technologyGrid – VLBI design & technology

SummarySummary

Outline

Introduction to the e-VLBIIntroduction to the e-VLBI

VLBI is a technique, in which physically independent and widely

separated radio telescopes observe the same region of sky

simultaneously, in order to generate very high-resolution

continuum and spectral-line images of cosmic radio sources

Telescopes are usually separated by thousands of kilometres

Data from each telescope are digitally sampled and stored locally,

using high-capacity magnetic tape systems and magnetic disk-array

systems

Data are sent and correlated at the central point (JIVE)

The total flow of data into the central processor is approximately 10-

100 Terabytes per single observation, after processing this is reduced

to 10-100 Gbytes.

Introduction to EXPReSIntroduction to EXPReS

EXPReS – the objective is to create a production-level

“electronic” VLBI (e-VLBI) service, in which the radio telescopes

are reliably connected to the central data processor at JIVE via a

high-speed optical-fibre communication network.

Project Details

Three years, started March 2006

International collaboration

Funded at 3.9 million EUR

FP6, Contract #026642

PSNC in EXPReSPSNC in EXPReS

EXPReS – a Real-time e-VLBI Radio Telescope

- JRA1: Future Arrays of Broadband Radio-Telescopes

on Internet Computing (FABRIC)- Grid – VLBI collaboration- Grid Workflow management

- Grid Routing

Creating solution for incorporating Grid resources for

distributed correlation using existing infrastructure.

Once upon a time (1)Everything was slow Telescopes collected data on tapes Sent via postal mail

Hard drive arrays slightly improved the situation

The entire cycle could easily require 6 months or more

Once upon a time (2)

Hardware correlator; the EVN MkIV data correlator at JIVE dedicated, purpose designed/built hardware a super computer; ~100 T ops/sec

Today / In the near future

Data can be transferred over the network

Each stage of the process can be speeded up

GRID resources

Software correlator

e-VLBI - electronic VLBI

e-VLBI System - architecture

e-VLBI System – control flow

Workflow Manager module (WFM)

CCF – correlator configuration file

WFM – phase 1

Definition of radio telescopes – automatically based on theobservation schedule

WFM – phase 2

Definition of file servers (each file sever is responsible for capturing data from RT)

WFM – phase 3

Definition of correlation nodes and data flows between components

WFM – properties

Definition of resource properties

WFM (1)

The WFM application has several purposes:

•Workflow creation – a user is given a list of building blocks. Each block represents the available resource. It is the user’s responsibility to design and construct the experiment’s workflow from the available building blocks.

•Workflow submission – the scenario prepared in the previous step can be submitted to the system.

•Workflow management and monitoring – allows to manage the executed workflow.

WFM (2)

Obrazek z WFM -> przeplywy danych

Link definition

Type definition

Radio telescope

File server

Correlator

Properties definition

Dynamic forms

Parameters stored in Json

format

PSNC in EXPReSPSNC in EXPReS

EXPReS – a Real-time e-VLBI Radio Telescope

- JRA1: Future Arrays of Broadband Radio-Telescopes

on Internet Computing (FABRIC)- Grid – VLBI collaboration- Grid Workflow management

- Grid Routing

Creating solution for incorporating Grid resources for

distributed correlation using existing infrastructure.

Thank you for your attention

http://www.expres-eu.org/d.stoklosa@man.poznan.pl

EXPReS is made possible through the support of the EC, 6th FP, Contract #026642