Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas...
-
Upload
dwayne-leonard -
Category
Documents
-
view
218 -
download
0
Transcript of Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas...
![Page 1: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/1.jpg)
Applications for the Grid
Here at GGF1:
Gabrielle Allen, Thomas, Dramlitsch,
Gerd Lanfermann, Thomas Radke,
Ed Seidel
Max Planck Institute for Gravitational Physics,
(Albert Einstein Institute)
![Page 2: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/2.jpg)
THE GRID: Dependable, consistent, pervasive access
to high-end resources
CACTUS is a freely available, modular,
portable and manageable environment
for collaboratively developing parallel, high-
performance multi-dimensional simulations
![Page 3: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/3.jpg)
Working Cactus Grid Tools Remote monitoring and steering
of simulations (thorn HTTPD). Just adding point-and-click
visualization to Web Server, and embedded visualization.
Remote visualization ... data streamed to local site for analysis with local clients "Window into simulation"
Notification by Email at start of simulation, including details of where to connect (URLs) for other tools.
Checkpointing and restart between machines.
![Page 4: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/4.jpg)
Grand PictureRemote steering and monitoring
from airport
Origin: NCSA
Remote Viz in St Louis
T3E: Garching
Simulations launched from Cactus PortalGrid enabled
Cactus runs on distributed machines
Remote Viz and steering from Berlin
Viz of data from previous simulations in
SF caf₫
DataGrid/DPSSDownsampling
Globus
http
HDF5
IsoSurfaces
![Page 5: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/5.jpg)
Developing Cactus Grid Tools Cactus Simulation Portal
(Astrophysics Simulation Collaboratory, ASC) - Version 2 next month. Lots of ideas, users are designing it.
Testbeds/VMRs - CVMR (big production machines), Egrid (test and development), EU Astrophysics.
Dynamic scenarios ... Production Level Cactus Worm (Migration Tool) and more.
Non-demo distributed simulations ... cheaper, faster, bigger
Notification, Info Services, Data management ... requested by users
![Page 6: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/6.jpg)
Grid Applications
How can applications really exploit Grids? The Grid is much more than Portals, distributed computing,
and remote visualization ... we should be thinking of new application scenarios
Cactus Worm (Simulation Migrator) was a prototype of such classes of applications, this is a thorn (module) which can be added to any application for automatic migration between machines (using checkpoint files) Dynamically locate new resources Move appropriate data files between old and new machine Get hold of appropriate executable for central repository Restart application on new machine
![Page 7: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/7.jpg)
Grid Applications Execution Staging
Use most appropriate (virtual) machine (fastest, cheapest, biggest) Need it by Wednesday, only use 200 SUs, need 1GB.
Set parameters on the basis of available machines ... eg anywhere between 70<nx<150 but I want to run it right now
Simulation Redistribution Put new grids on different machines (AMR, Multigrid) Adapt automatically to varying processor speeds and application
loads Slow Startup ... start application now on a slow resource, and move
to faster resources as they become available
![Page 8: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/8.jpg)
Grid Applications Simulation Migration
Move to more appropriate machines (faster, cheaper) Move to new resource at end of queue time Convergence testing
Send coarser grid to different resources, either at start of simulation or dynamically at user request, or following some event
Look-Ahead Spawn a downsampled resolution to a different resource to predict the
future Cloning/Multiple Universe
Dynamically initiate a cloned version with changed parameters
![Page 9: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/9.jpg)
Grid Applications Spawn Independent Tasks
Analysis tasks Don't need to be performed in line with simulation, send to different
available (free) machines Vector
Send to a vector pool of resources ... e.g. Calculate FFT of each grid variable, send each one to different machine
![Page 10: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/10.jpg)
Grid Applications Spawn Independent Tasks
Pipeline Series of tasks performed on same data
![Page 11: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/11.jpg)
Grid Applications Task Farming
Automated parameter studies Genetic parmater studies
Use heuristic algorithms to cover large parameter spaces Manage embarrassing parallel applications
Make use of Condor, Entropia Grid scripting languages ... everything available from the command
line. Many other possibilities ...
![Page 12: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/12.jpg)
Dynamic Grid Computing
NCSA
Go!
Clone job with steered parameter
Queue time over, find new machine
Add more resources
Found a horizon,try out excision
Look forhorizon
Calculate/OutputGrav. Waves
Calculate/OutputInvariants
Find bestresources
Free CPUs!!
SDSC RZG
SDSC
LRZ Archive data
![Page 13: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/13.jpg)
Users View Has To Be Easy!
![Page 14: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/14.jpg)
Cactus Grid Application Toolkit (GAT)
Cactus Application
[Black hole collisions, chemical engineering, cosmology, hydrodynamics]
Grid Tools
[Worm, Spawner,
Taskfarmer]
Cactus Programming APIs
[Grid variables, parameters, scheduling, IO, interpolation,
reduction, driver]
Grid Application Toolkit
[Information services, resource broker, data
management, monitoring]
YOUR GRID
Distributed resources, different
OS, software
GridService
GridService
GridService
![Page 15: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/15.jpg)
Grid Application Toolkit (GAT)
Application developer should be able to build simulations with tools that easily enable dynamic grid capabilities
Want to build programming API to: Query/Publish to Information Services
Application, Network, Machine and Queue status Resource Brokers
Where to go, how to decide?, how to get there? Move and manage data, compose executables
Gsiscp, gsiftp, streamed HDF5, scp, GASS, ... Higher level grid tools:
Migrate, Spawn, Taskfarm, Vector, Pipeline, Clone, ... Notification
Send me an Email, SMS, fax, page, when something happens.
Much more.
![Page 16: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/16.jpg)
Cactus GAT: Example
![Page 17: Applications for the Grid Here at GGF1: Gabrielle Allen, Thomas, Dramlitsch, Gerd Lanfermann, Thomas Radke, Ed Seidel Max Planck Institute for Gravitational.](https://reader030.fdocuments.net/reader030/viewer/2022032612/56649ebe5503460f94bc893f/html5/thumbnails/17.jpg)
Cactus GAT: Example