Deliverable 7.3 Workflow Manager

Deliverable 7.3

Workflow Manager

Poznao Supercomputing and Networking Center

D7.3 Workflow Manager Page2

Table of contents 1 Introduction ............................................................................................................ 3 2 E-VLBI Correlation Platform ................................................................................. 3

2.1 Liferay introduction......................................................................................... 3

2.2 NEXPReS model ............................................................................................. 5

2.2.1 User Roles and Permission ...................................................................... 5

2.2.2 Experiments and Workflow portlets ........................................................ 5

2.2.3 Telescope, Correlator, external tools Portlets .......................................... 6

2.3 Visualization of the experiment ...................................................................... 6

3 Correlation testbed ................................................................................................. 8 4 How to conduct an experiment – short user manual .............................................. 9

4.1 System log-in................................................................................................... 9

4.2 Experiments page (and portal navigating) .................................................... 10

4.3 Experiment creation ...................................................................................... 11

4.4 Experiment’s workflow ................................................................................. 12

4.5 Experiment submission ................................................................................. 12

5 Software ............................................................................................................... 13

Table of figures

Figure 1. Fabric Workflow Manager Application ......................................................... 3

Figure 2. NEXPReS module architecture ...................................................................... 5 Figure 3. Example of correlation's graph ....................................................................... 7 Figure 4. WP7 testbed based on partner's resources ...................................................... 8 Figure 5. Sign in portlet ................................................................................................. 9

Figure 6. Welcome screen.............................................................................................. 9 Figure 7. Experiments page ......................................................................................... 10 Figure 8. Creating new experiment .............................................................................. 11 Figure 9. Workflows portlet ......................................................................................... 12


1 Introduction This document describes e-VLBI Platform – software developed within WP7 –

Correlation in a shared infrastructure. It contains a short overview of previous

approach prepared in EXPReS project, presents a new web based perspective for data

correlation, provides description of the current testbed and finally describes the

content of the software components.

2 E-VLBI Correlation Platform

The Workflow Manager Application (WFM) has been constructed in EXPReS project

within Fabric activity. The idea was to create a prototype system for software

correlation. The WFM was a main interface between users and the system and has

been constructed as a standalone Java window-based application.

Figure 1. Fabric Workflow Manager Application

User can create so called observation workflows based on an observation schedule.

The application allows also defining correlation parameters which controls the

behaviour of the software correlator. The completed observation workflow is then

submitted for further processing.

The role of WFM application within NEXPReS project has been significantly

extended. The WFM has been transformed from a single user application to a

multiuser and multipurpose application for all VLBI actors. The application became a

central place not only to manage and control a distributed correlation platform, but

also to monitor the entire experiment process.

PSNC has created an e-VLBI Correlation Platform which will be used within VLBI

community to manage and monitor experiments. The WFM application has been

rewritten using web based technologies and nested in the Correlation Platform as a

portlet.

2.1 Liferay introduction

Experiences from the EXPReS Fabric project lead to a new approach for handling e-

VLBI experiments. Instead of a single point of entry for Principal Investigator,

allowing only submission of the correlation using specified resources, focus was put

to facilitate whole process of setting up the observation, execution, monitoring and

validating correlation process. A need for a multi-user program arose. It should


provide various, pluggable tools and allow easy development, integration and

deployment of new applications. A research was done about available solutions (WP7

State of the Art on Web Portals and Web Technologies:

http://www.jive.nl/nexpres/lib/exe/fetch.php?media=nexpres:wp7_state-of-the-

art_on_web_portals_and_web_technologies-final.pdf) and Liferay Portal

(http://www.liferay.com) was chosen as a platform for NEXPReS project.

Liferay Portal is an enterprise web platform for building business solutions that

delivers immediate results and long-term value (from Liferay website). It contains

out-of-the-box set of portlets providing basic needs for a web platform – user

management, login functionality, alerts and announcements, collaboration tools such

as chats, documents and so on. It is also a typical portlet container, which allows

writing own applications and deploying them inside the portal. Deployed applications

can be added to portal pages as portlets – modular, pluggable user interface

components. Administrators can prepare and set up which sets of portlets will be

visible on each page of the portal.

A great deal of influence on deciding on portal solution was collaboration and multi-

user approach. Using provided user management tools we can model each type of

actor as a specific user role, create communities, organizations and user groups. For

each such unit a specific page or view can be created, containing only needed

applications. This allows to create a dedicated page layouts for different user types.

Cooperation between scientists during observation is fulfilled by adaptation of the

Liferay portlets, which provide a chat functionality, mail (also connected with

external mail services) and message boards. System generated information is also

displayed to interested parties using alerts and announcements portlet.

Crucial requirement of the Workflow Manager in Liferay portal was intuitive and

responsive user interface. From technologies supported by the portal, JSF (Java

Server Faces - http://javaserverfaces.java.net/) and ICEFaces

(http://www.icefaces.org/) were chosen. JSF is a Web application framework and

simplifies building user interfaces. It supports Model-View-Controller architecture

pattern and is written in Java. ICEFaces on the other hand extends JSF framework

with Ajax support, which allows development of rich internet applications.

Asynchronous updates from server to client is provided by Automatic Ajax and Ajax

Push functionalities. This allows to update data in the user’s browser without

reloading whole page, depending on the state of the model on the server. Objects

during e-VLBI observation will be constantly changing, updating the statuses of the

components, executing steps of the correlation process. This changes will be

immediately displayed to all interested users in their Liferay portal instance.

http://www.liferay.com/

http://javaserverfaces.java.net/

http://www.icefaces.org/


2.2 NEXPReS model

Figure 2. NEXPReS module architecture

2.2.1 User Roles and Permission

During design of the e-VLBI Platform we had to take into account the structure and

functionalities of the Liferay Portal. One of such feature is user support. After

evaluating NEXPReS use cases we have decided to create 5 NEXPReS roles:

PI – Principal Investigator

SS – Support Scientist

TO – Telescope Operator

CO – Correlator Operator

Admin – e-VLBI Platform Administrator

Each of the four user types (beside Admin) has an unique perspective in the portal –

providing default view, customized for this particular role, containing all portlets

required by the user. User roles also allow to enforce permissions on whole e-VLBI

Platform –e.g. general information about NEXPReS displayed for not logged or

unregistered visitors, displaying monitoring information of the correlation process to

logged, non-correlator operators and finally providing interactive controls for

correlator operators.

2.2.2 Experiments and Workflow portlets

In e-VLBI Platform information about experiments (VLBI observations) will be

displayed using two portlets: Experiments and Workflow. Experiments portlet displays

all existing observations in the system. It also allows to create new experiment, by

uploading VEX file describing e-VLBI observation and specifying additional


information. General information of the experiment (Status, time to start) is being

displayed to the user.

By selecting the experiment more information about the experiment is available. Most

important is graphical representation of the experiment in Workflow portlet (see

Figure 9. Workflows portlet). Interaction with the experiment is possible using this

view: before experiment starts it allows modifications of the whole process and during

the observation it provides monitoring information about experiment.

2.2.3 Telescope, Correlator, external tools Portlets

Pluggable architecture of e-VLBI Platform allows adding new modules, providing

new functionalities to whole application. Some of the distinguished portlets, which

will be needed during VLBI observation are:

Telescope Portlet - allowing Telescope Operators to operate and communicate

with their telescope module, others can monitor progress and status of

telescope operations during observation.

Correlator Portlet - allowing Correlator Operators set up of the correlation

process, specifying parameters of the correlation. It also allows monitoring of

the correlation process to all interested parties.

Fringe Plots – graphical module used to display fringe plots required to update

and adjust telescopes and correlation parameters

2.3 Visualization of the experiment

Initial experiment upload to Workflow Manager contains only a VEX file and

author’s information. Basing on this file the correlation representation is created by

Workflow Manager – both on server side and in user tools, to visualise the flow of

data and resources taking part in e-VLBI experiment.

On client side the convenient Workflows portlet was developed. It comprises

both Java portlet and Adobe Flash technologies, thereby addressing standards support

(Java portlet can be deployed in any portlet’s container) as well as platform-

interoperation (embedding binary Flash content).

Portlet embedded in Liferay portal uses JSF (Java Server Pages) GUI libraries

– “Java-based Web application framework intended to simplify development

integration of web-based user interfaces” (source: Wikipedia). JSF facilitates mapping

from/to web application to/from Java model classes on server and extended with

AJAX libraries (e.g. IceFaces) results with responsive and intuitive user interfaces.

On user selection in web browser, the VLBI experiment’s representation is

rendered inside Workflows portlet. This portlet is intended as a tool for graphical

design of e-VLBI workflow – a graph representing data flow within telescopes,

translation nodes and correlators during correlation process. e-VLBI workflow

visualises resources taking part in experiment (see Figure 3. Example of correlation's

graph) and - by connecting them with each other – user defines destination/sources of

data for services carrying out the correlation. Changes made in correlation’s workflow

are reflected in model classes and saved on server side. Designed graph can be edited

and parameters of experiment can be changed. When correlation time is met –


workflow’s modifications are no longer possible, and the experiment is started on

infrastructure.

Figure 3. Example of correlation's graph

Representation of experiment’s workflow is rendered in Workflows portlet

using Flex/ActionScript technologies, a web-enabled toolkit supporting visualisation

of graph-like data. To enable data synchronisation between server-side model and

Flash client application an Adobe’s BalzeDS messaging technology was used.

BlazeDS allows for RPC-based communication from ActionScript clients to server as

well as for PUSH-like transfer of data from server to clients. The data sent is in AMF

format – a binary protocol enhancing light and fast communication.

On the server side experiment is passed to Experiment Manager Unit (EMU) which

ensures coherence between correlation definition and infrastructure resources. EMU

schedules experiments to start automatically – the proper model objects of workflow

are created, passed to workflow management module. The correlation is started by

invoking web services of Fabric system.


3 Correlation testbed Resources assigned by the project partners to the NEXPReS project have been set up

to form a testbed for e-VLBI Platform developed by PSNC within work package. The

testbed consists of (see also Figure 4. WP7 testbed based on partner's resourcesFigure

4):

computational resources which will be used during correlation process,

translation. These have been provided by PSNC, Jive and Ventspils.

Translation services which serve the telescope data

Correlation services which handle correlation process

VLBI Broker deployed at PSNC to handle experiments

E-VLBI Platform – replacement of Fabric Workflow Manager

Figure 4. WP7 testbed based on partner's resources

A current version of e-VLBI Platform has been deployed on local installation at

PSNC. This local installation is available to all project partners. The idea is to keep

this software operable till the end of the project duration. This site will be updated

whenever a new version of the e-VLBI Platform is available. This will allow us to

conduct experiments at any time and also use this installation for demonstration

purposes.


4 How to conduct an experiment – short user manual This section describes actions needed to be taken by user in order to create

experiment and submit it for correlation. The process is taken from user point of view

and therefore presented in form of short manual illustrated with live e-VLBI Platform

system screenshots.

The general scenario when running a correlation with e-VLBI Platform is to log

into portal, navigate to page containing experiment related-portlets (management and

workflows) where new experiment can be created and its workflow designed. The

required user interaction ends at this point, as Experiment Manager Unit (EMU)

running on server side takes responsibility for running and monitor correlation

process. Steps needed to conduct experiment are described in more detail below.

4.1 System log-in

In web browser (preferably Mozilla Firefox 9.0+ or Google Chrome 14.0+)

navigate to e-VLBI Platform main page

Click “Log in” link in upper right corner

Figure 5. Sign in portlet

Provide your user credentials in the form that will be displayed (to obtain user

and password contact your e-VLBI Platform administrator)

On successful log in you will be redirected to portal’s welcome site –

displaying general information.

Figure 6. Welcome screen


4.2 Experiments page (and portal navigating)

Figure 7. Experiments page

After successful log in you are located on e-VLBI Portal’s main page. To get

back there (in case of any navigation problems) – click NEXPReS logo

located in upper left page corner.

Page on which experiment-related portlets are placed is called “Experiments”.

It is listed among other sites on top horizontal bar. To navigate to this page

just click on the page name. Other sites can be accessed the same way.

The Experiments site contains 2 portlets:

o Experiments – main entry point to manage all your experiments –

allows for creation and selection or experiment to be edited in other

portlets. For user’s convenience view is divided into 2 tabs, grouping

experiments by their status:

Active – the experiments that are to be started in the future

(status SCHEDULED) and currently running (status

RUNNING). Some of them may still require users’ interaction

(e.g. approving telescopes configuration or correlation

workflow customization)

Archive – on this tab user can access all experiment that are no

longer running

o Workflows – this portal is intended to present correlation in graphical,

user-friendly way in form of workflow’s graph. The important thing to

note is that workflows portal is not independent – it is connected with

Experiments portal. Content rendered in Workflows comes from

experiment selected in Experiments window.


4.3 Experiment creation

Figure 8. Creating new experiment

On Experiments site, in Experiments portlet press “Create new” button

An experiment wizard form is displayed containing:

o Name – provide name under which experiment will be accessible in e-

VLBI Platform,

o Description – provide a short description of the experiment,

o VEX file – upload of file containing observation definition,

o Start date – specify the time at which correlation process should be

started.

When all information in wizard are provided – press “OK”.

On successful experiment creation it will be displayed among other

experiments in “Active” tab of Experiments portlet, with status set to

SCHEDULED.


4.4 Experiment’s workflow

Figure 9. Workflows portlet

Select one of the experiments from list in Experiments portlet

In Workflows portlet a visualization of correlation graph will be presented

Detailed information on resources taking part in correlation process

(telescopes, translation nodes and correlation nodes) can be obtained by

selecting elements of correlation graph. Information will be presented in right

panel on portlet, listing resource’s name, abbreviation and description.

4.5 Experiment submission

When creating an experiment (see 4.3) a start date of correlation has to be

specified.

Experiments due to start in future are listed on “Active” tab of Experiments

portlet (for more details see 4.2), with status set to SCHEDULED.

No user interaction is needed to start correlation process at time specified –

Experiment Manager Unit (EMU) monitors start times of experiments and

takes responsibility for running it at given time.

When experiment has been started by EMU, its status changes from SCHEDULED to

RUNNING.


5 Software The table below presents the content of the attached CD-ROM. This includes all client

and server components of new e-VLBI Platform .

LP COMPONENT / MODULE /OTHER SHORT DESCRIPTION

Actionscript

1.

nexpresWorkflowFlex

ActionScript/Flex project that builds

graphical (Flash technology) part of

Workflows management system in e-VLBI

Platform. Displays infrastructure of the

experiment (radio telescopes, translation

nodes, correlation nodes) and properties of

each node that takes a part in work.

Common

1. kiwiCommon

KIWI common library – set of general

purpose utility classes

2. misc

Main project containing all related libraries

and Ant build script for e-VLBI Platform

3.

nexpresModel

Model for e-VLBI Platform i.e. to describe

VLBI experiment and to serve internal

persistence for infrastructure resources

description

Fabric

1.

Correlation Node

Manages all computations on the underlying

high performance computers. It provides an

abstraction layer for correlation

computations, hence the WfM will be able

to work with both different hardware

infrastructures and different software

correlator.

2. expresRemote

Remote client stubs for communication

purpose between modules

3. misc

Project containing all related libraries and

Ant build script for Fabric components

4. vlabCommon

VLab common library – set of general

purpose utility classes

5.

VLBI Broker

Handles correlation process on the server

side: “talks” to Translation Nodes and

Correlation Nodes decides when to submit

new portion of data for the correlation.

Portlets

1.

nexpres-portlet

Contains GUI portal parts – all portlets

composing NEXPReS e-VLBI Platform –

JSF resources and web toolkit libraries. This

module covers also Flex resources and

configuration files.


Workflows

1.

experimentManager (EMU)

Module responsible for management of

experiments. All newly created experiment

should be registered with EMU. It monitors

for experiment-related events and notifies all

interested parts. EMU is responsible for

starting experiment at time given.

2.

Workflow Manager (WfM)

WfM handles all actions related to

correlation workflow – translates data from

experiment description into workflow

format and – after experiment start –

monitors workflow execution.

The current versions of the software components is always available at workpackage

SVN repository hosted by PSNC. The repository location is as follows:

kiwi.man.poznan.pl/svn/nexpres.

Deliverable 7.3 Workflow Manager

Documents

Transcript of Deliverable 7.3 Workflow Manager