SDMCenter

A Quick Update on the TSI and PIW workflows

SDM All Hands March 2-3, 2005

https://www-casc.llnl.gov/sdm/

Terence Critchlow,

Xiaowen Xin, Bertram Ludaescher, Ilkay Altintas

Mladen Vouk, Zengang Cheng, Lots of Others….

UCRL-PRES-209990

SDMCenterWeb page is up and running

Example workflows available for download

Documentation exists and is improving

List of (somewhat dated) publications

https://www-casc.llnl.gov/sdm/

SDMCenterTSI-1 Workflow

Web interface for Simulation configuration submission

Error notification

Identify config files

Are files consistent Move files to correct

locationsSubmit batch

request at NERSC

Check on job status Delay Generate

directory listingAre there

new files? Identify

complete filesJob still

running?

Count files

Empty list

Transfer files to HPSS Transfer completed

correctly

Transfer files to SBTransfer completed

correctly

Delete file

Extract

Get Variables

Remap coordinates

Create Chem vars Create neutrino vars

Derive other vars Write diagnostic file

Generate plots

Tool-1

Tool-2

Tool-3

Tool-4

Generate thumbnails

Generate movie

Update notebook

Update web page

Update notebook

Update web page

Delay

Update notebook

Update web page

Update notebook

Update web page

SDMCenter

Delay

TSI-1 Workflow

Web interface for Simulation configuration submission

Error notification

Identify config files

Are files consistent Move files to correct

locationsSubmit batch

request at NERSC

Check on job status Delay Generate

directory listingAre there

new files? Identify

complete filesJob still

running?

Count files

Empty list

Transfer files to HPSS Transfer completed

correctly

Transfer files to SBTransfer completed

correctly

Delete file

Extract

Get Variables

Remap coordinates

Create Chem vars Create neutrino vars

Derive other vars Write diagnostic file

Generate plots

Tool-1

Tool-2

Tool-3

Tool-4

Generate thumbnails

Generate movie

Update notebook

Update web page

Update notebook

Update web page Update

notebook

Update web page

Update notebook

Update web page

Remote job submission - done

SDMCenterTSI-1 Workflow

Web interface for Simulation configuration submission

Error notification

Identify config files

Are files consistent Move files to correct

locationsSubmit batch

request at NERSC

Check on job status Delay Generate

directory listingAre there

new files? Identify

complete filesJob still

running?

Count files

Empty list

Transfer files to HPSS Transfer completed

correctly

Transfer files to SBTransfer completed

correctly

Delete file

Extract

Get Variables

Remap coordinates

Create Chem vars Create neutrino vars

Derive other vars Write diagnostic file

Generate plots

Tool-1

Tool-2

Tool-3

Tool-4

Generate thumbnails

Generate movie

Update notebook

Update web page

Update notebook

Update web page

Delay

Update notebook

Update web page

Update notebook

Update web page

Real-time file transfer - done

SDMCenterTSI-1 Workflow

Web interface for Simulation configuration submission

Error notification

Identify config files

Are files consistent Move files to correct

locationsSubmit batch

request at NERSC

Check on job status Delay Generate

directory listingAre there

new files? Identify

complete filesJob still

running?

Count files

Empty list

Transfer files to HPSS Transfer completed

correctly

Transfer files to SBTransfer completed

correctly

Delete file

Extract

Get Variables

Remap coordinates

Create Chem vars Create neutrino vars

Derive other vars Write diagnostic file

Generate plots

Tool-1

Tool-2

Tool-3

Tool-4

Generate thumbnails

Generate movie

Update notebook

Update web page

Update notebook

Update web page

Delay

Update notebook

Update web page

Update notebook

Update web page

Data analysis – not yet started

SDMCenterTSI-1 Workflow

Developed modular, configurable, actors that can be used elsewhere File filtering

• Use regular expressions to filter file list

File transfer • User specified command transfers files between two remote

locations

SSH • Executes remote command, catches errors

Retry • Repeat an action until successful or until user defined limit

is reached

SDMCenter

Top level view of TSI workflow

SDMCenterSubmit actor

Submit job using llsubmit

Return the job number

Debug actor lets you test without actually submitting job

SDMCenterPIW Workflow

Added logging Modularized

workflow a bit more

Added debugging and testing code

New visualization actor

SDMCenterNext Steps

Working through deadlock / race condition problems in Ptolemy PN director Discovered because of test suite / debug actor

Need to be able to “merge” multiple streams

Data analysis section of TSI-1 workflow

More documentation

SDMCenterNext talks

Steve Parker SCIRun and SPA integration status

Mladen Vouk On Large Data Flow Scientific Workflows: An

Astropysics Case Study

Bertram Ludaescher Kepler / SPA Extensions for Scientific Workflows –

Now and Upcoming