The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van...
-
date post
19-Dec-2015 -
Category
Documents
-
view
219 -
download
0
Transcript of The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van...
![Page 1: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/1.jpg)
The Ibis Project:Simplifying Grid Programming & Deployment
Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost,
Ceriel Jacobs,Frank Seinstra, Roelof Kemp, Kees
Verstoep
Vrije Universiteit Amsterdam
![Page 2: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/2.jpg)
The ‘Promise of the Grid’
Efficient and transparent (i.e. easy-to-use) wall-socket computing over a distributed set of resources [Sunderam ICCS’2004, based on Foster/Kesselman]
![Page 3: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/3.jpg)
● Performance & scalability
● Heterogeneous● Low-level & changing
programming interfaces
● writing & deploying grid applications is hard
Reality: ‘Problems of the Grid’
● Connectivity issues
● Fault tolerance
● Malleability
Wide-Area Grid Systems
UseUserr
!
![Page 4: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/4.jpg)
The Ibis Project
● Goal:● drastically simplify grid
programming/deployment
● ``Grids as promised’’
![Page 5: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/5.jpg)
Approach
● Minimal assumptions about execution environment● Virtual Machines (Java) deal with heterogeneity
● Use middleware-independent APIs
● Different programming abstractions
● Designed for dynamic/hostile grid environments
● Modular and flexible: can replace Ibis components by external ones
![Page 6: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/6.jpg)
Global picture
![Page 7: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/7.jpg)
Grid programming
● Programming models:● Message passing (RMI, MPJ)
● Divide-and-conquer (Satin)
● IPL (Ibis Portability Layer)● Java-centric “run-anywhere” library, sent with
application● Can handle fault-tolerance, malleability
● SmartSockets● Solve connectivity problems automatically
(firewalls, NAT, addressing problems)
![Page 8: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/8.jpg)
Grid deployment
● JavaGAT: Java Grid Application Toolkit● Make applications independent of underlying grid
● Used for file copying, resource discovery, job submission & monitoring, user authentication
● API is currently standardized (SAGA)
● Zorilla P2P system● Job management, gossiping,
clustering, flood scheduling
![Page 9: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/9.jpg)
Ibis applications● e-Science (VL-e)
● Brain MEG-imaging● Mass spectroscopy
● Multimedia content analysis● Video processing● Many HPC applications
● SAT-solver, N-body, grammar learning, ….
![Page 10: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/10.jpg)
European users
● D-Grid: Workflow engine for astronomy● U. Erlangen: grid file system● INRIA: ProActive on Ibis RMI● U. Patras: Jylab scientific computing● UPC Barcelona: Grid Superscalar● HITACHI: Peta-scale data management
Grid’5000
![Page 11: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/11.jpg)
DAS-3272 nodes
(AMD Opterons)
792 cores1TB memoryLAN: Myrinet 10G Gigabit
EthernetWAN
(StarPlane): 20-40 Gb/s
OPN
See CCGrid’08 session 11 (Verstoep)
![Page 12: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/12.jpg)
Gene sequence comparison in Satin (on
DAS-3)
Speedup on 1 cluster Run times on 5 clusters
• 78% efficiency on 5 clusters (with 1462 WAN-msgs/sec)
![Page 13: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/13.jpg)
Multimedia content analysis
● Analyzes video streams to recognize objects
● Extract feature vectors from images● Describe properties (color, shape)● Data-parallel task
● Compute on consecutive images● Task-parallelism on a grid
See CCGrid’08 SCALE challenge & sessions 12+16
![Page 14: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/14.jpg)
High Resolution Video Processing
● Realtime processing of CineGrid movie data● 3840x2160 (4xHD) @ 30 fps = 1424 MB/sec
● Multi-cluster processing pipeline ● Using DAS-3, StarPlane and Ibis
![Page 15: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/15.jpg)
Summary● Goal: Simplify grid
programming/deployment
● Key ideas in Ibis● Virtual machines (JVM) deal with
heterogeneity● High-level programming abstractions
(Satin)● Handle fault-tolerance, malleability,
connectivity problems automatically (Satin, SmartSockets)
● Middleware-independent APIs (JavaGAT)● Modular
![Page 16: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/16.jpg)
Future work?
● Clouds as promised (CCGrid’09 Shanghai)
● Exaflop as promised (CCGrid’10 Melbourne)
● Cold fusion as promised● Stock market prediction as promised● Indiana Jones 5 as promised (2032?)
![Page 17: The Ibis Project: Simplifying Grid Programming & Deployment Henri Bal, Jason Maassen, Rob van Nieuwpoort, Thilo Kielmann, Niels Drost, Ceriel Jacobs, Frank.](https://reader036.fdocuments.net/reader036/viewer/2022062516/56649d3f5503460f94a18e79/html5/thumbnails/17.jpg)
More information● Ibis can be downloaded from
● http://www.cs.vu.nl/ibis
● Papers:● Satin [PPoPP’07], SmartSockets [HPDC’07],
Gossiping [HPDC’07], JavaGAT [SC’07],MMCA [IEEE Multimedia’07, CCGrid’08],DAS-3/StarPlane [CCGrid’08]
• Ibis tutorials• CCGrid’08