Arsenic Groundwater Contamination-Bangladesh Encheng Zhou Xiaowen Kan.
SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence...
-
Upload
kristin-hardy -
Category
Documents
-
view
220 -
download
0
Transcript of SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence...
![Page 1: SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence Critchlow, Xiaowen Xin, Bertram.](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649e245503460f94b126bf/html5/thumbnails/1.jpg)
SDMCenter
A Quick Update on the TSI and PIW workflows
SDM All Hands March 2-3, 2005
https://www-casc.llnl.gov/sdm/
Terence Critchlow,
Xiaowen Xin, Bertram Ludaescher, Ilkay Altintas
Mladen Vouk, Zengang Cheng, Lots of Others….
UCRL-PRES-209990
![Page 2: SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence Critchlow, Xiaowen Xin, Bertram.](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649e245503460f94b126bf/html5/thumbnails/2.jpg)
SDMCenterWeb page is up and running
Example workflows available for download
Documentation exists and is improving
List of (somewhat dated) publications
https://www-casc.llnl.gov/sdm/
![Page 3: SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence Critchlow, Xiaowen Xin, Bertram.](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649e245503460f94b126bf/html5/thumbnails/3.jpg)
SDMCenterTSI-1 Workflow
Web interface for Simulation configuration submission
Error notification
Identify config files
Are files consistent Move files to correct
locationsSubmit batch
request at NERSC
Check on job status Delay Generate
directory listingAre there
new files? Identify
complete filesJob still
running?
Count files
Empty list
Transfer files to HPSS Transfer completed
correctly
Transfer files to SBTransfer completed
correctly
Delete file
Extract
Get Variables
Remap coordinates
Create Chem vars Create neutrino vars
Derive other vars Write diagnostic file
Generate plots
Tool-1
Tool-2
Tool-3
Tool-4
Generate thumbnails
Generate movie
Update notebook
Update web page
Update notebook
Update web page
Delay
Update notebook
Update web page
Update notebook
Update web page
![Page 4: SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence Critchlow, Xiaowen Xin, Bertram.](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649e245503460f94b126bf/html5/thumbnails/4.jpg)
SDMCenter
Delay
TSI-1 Workflow
Web interface for Simulation configuration submission
Error notification
Identify config files
Are files consistent Move files to correct
locationsSubmit batch
request at NERSC
Check on job status Delay Generate
directory listingAre there
new files? Identify
complete filesJob still
running?
Count files
Empty list
Transfer files to HPSS Transfer completed
correctly
Transfer files to SBTransfer completed
correctly
Delete file
Extract
Get Variables
Remap coordinates
Create Chem vars Create neutrino vars
Derive other vars Write diagnostic file
Generate plots
Tool-1
Tool-2
Tool-3
Tool-4
Generate thumbnails
Generate movie
Update notebook
Update web page
Update notebook
Update web page Update
notebook
Update web page
Update notebook
Update web page
Remote job submission - done
![Page 5: SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence Critchlow, Xiaowen Xin, Bertram.](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649e245503460f94b126bf/html5/thumbnails/5.jpg)
SDMCenterTSI-1 Workflow
Web interface for Simulation configuration submission
Error notification
Identify config files
Are files consistent Move files to correct
locationsSubmit batch
request at NERSC
Check on job status Delay Generate
directory listingAre there
new files? Identify
complete filesJob still
running?
Count files
Empty list
Transfer files to HPSS Transfer completed
correctly
Transfer files to SBTransfer completed
correctly
Delete file
Extract
Get Variables
Remap coordinates
Create Chem vars Create neutrino vars
Derive other vars Write diagnostic file
Generate plots
Tool-1
Tool-2
Tool-3
Tool-4
Generate thumbnails
Generate movie
Update notebook
Update web page
Update notebook
Update web page
Delay
Update notebook
Update web page
Update notebook
Update web page
Real-time file transfer - done
![Page 6: SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence Critchlow, Xiaowen Xin, Bertram.](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649e245503460f94b126bf/html5/thumbnails/6.jpg)
SDMCenterTSI-1 Workflow
Web interface for Simulation configuration submission
Error notification
Identify config files
Are files consistent Move files to correct
locationsSubmit batch
request at NERSC
Check on job status Delay Generate
directory listingAre there
new files? Identify
complete filesJob still
running?
Count files
Empty list
Transfer files to HPSS Transfer completed
correctly
Transfer files to SBTransfer completed
correctly
Delete file
Extract
Get Variables
Remap coordinates
Create Chem vars Create neutrino vars
Derive other vars Write diagnostic file
Generate plots
Tool-1
Tool-2
Tool-3
Tool-4
Generate thumbnails
Generate movie
Update notebook
Update web page
Update notebook
Update web page
Delay
Update notebook
Update web page
Update notebook
Update web page
Data analysis – not yet started
![Page 7: SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence Critchlow, Xiaowen Xin, Bertram.](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649e245503460f94b126bf/html5/thumbnails/7.jpg)
SDMCenterTSI-1 Workflow
Developed modular, configurable, actors that can be used elsewhere File filtering
• Use regular expressions to filter file list
File transfer • User specified command transfers files between two remote
locations
SSH • Executes remote command, catches errors
Retry • Repeat an action until successful or until user defined limit
is reached
![Page 8: SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence Critchlow, Xiaowen Xin, Bertram.](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649e245503460f94b126bf/html5/thumbnails/8.jpg)
SDMCenter
Top level view of TSI workflow
![Page 9: SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence Critchlow, Xiaowen Xin, Bertram.](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649e245503460f94b126bf/html5/thumbnails/9.jpg)
SDMCenterSubmit actor
Submit job using llsubmit
Return the job number
Debug actor lets you test without actually submitting job
![Page 10: SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence Critchlow, Xiaowen Xin, Bertram.](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649e245503460f94b126bf/html5/thumbnails/10.jpg)
SDMCenterPIW Workflow
Added logging Modularized
workflow a bit more
Added debugging and testing code
New visualization actor
![Page 11: SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence Critchlow, Xiaowen Xin, Bertram.](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649e245503460f94b126bf/html5/thumbnails/11.jpg)
SDMCenterNext Steps
Working through deadlock / race condition problems in Ptolemy PN director Discovered because of test suite / debug actor
Need to be able to “merge” multiple streams
Data analysis section of TSI-1 workflow
More documentation
![Page 12: SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 Terence Critchlow, Xiaowen Xin, Bertram.](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649e245503460f94b126bf/html5/thumbnails/12.jpg)
SDMCenterNext talks
Steve Parker SCIRun and SPA integration status
Mladen Vouk On Large Data Flow Scientific Workflows: An
Astropysics Case Study
Bertram Ludaescher Kepler / SPA Extensions for Scientific Workflows –
Now and Upcoming