1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science...
-
Upload
kory-french -
Category
Documents
-
view
219 -
download
0
Transcript of 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science...
![Page 1: 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk.](https://reader036.fdocuments.net/reader036/viewer/2022062410/56649f285503460f94c40478/html5/thumbnails/1.jpg)
1
Condor BirdBathSOAP Interface to Condor
Charaka GoonatilakeDepartment of Computer Science
University College [email protected]
![Page 2: 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk.](https://reader036.fdocuments.net/reader036/viewer/2022062410/56649f285503460f94c40478/html5/thumbnails/2.jpg)
2
Background
![Page 3: 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk.](https://reader036.fdocuments.net/reader036/viewer/2022062410/56649f285503460f94c40478/html5/thumbnails/3.jpg)
3
Motivations Adding Web Service interfaces to Condor daemons
Facilitate the development of third-party applications capable of interacting with Condor (remotely).– E.g. build higher-level application specific scheduler that submits jobs to
multiple Condor pools based on application semantics
– These can be built using a wide range of languages/SOAP packages
– BirdBath has been tested on:• Java (Apache Axis, XSUL)• Python (ZSI)• C# (.Net)• C/C++ (gSOAP)
Condor accessible from platforms where its command-line tools are not supported/installed
![Page 4: 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk.](https://reader036.fdocuments.net/reader036/viewer/2022062410/56649f285503460f94c40478/html5/thumbnails/4.jpg)
4
The (Hypothetical) Big Picture
Schedd Scheddflocking
Schedd
SOAP
Globus/gridSAMCondor-G
Client
Site A Site B Site C
Site D
SOAP
![Page 5: 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk.](https://reader036.fdocuments.net/reader036/viewer/2022062410/56649f285503460f94c40478/html5/thumbnails/5.jpg)
5
Condor SOAP Interfaces
Central manager
Execution machine
Submission machine(s)
CollectorSchedd
Execution machine
Execution machine
Execution machine
Begin/commit/abort transaction Submit Job Obtain Job ClassAds Get/Send file Removed/hold/release job
Obtain resource ClassAds Obtain all daemon ClassAds
Shadow
Negotiator
Now available in Condor 6.7.5
![Page 6: 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk.](https://reader036.fdocuments.net/reader036/viewer/2022062410/56649f285503460f94c40478/html5/thumbnails/6.jpg)
6
Clients for BirdBath
WSDLs available that describe each daemon interface
SOAP libraries (stub code) can be generated automatically from WSDLs
Custom Client
holdJob()
SOAP library
Submission machine(s)
Schedd
![Page 7: 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk.](https://reader036.fdocuments.net/reader036/viewer/2022062410/56649f285503460f94c40478/html5/thumbnails/7.jpg)
7
Example: Query Collector
Obtain information (ClassAds) about all resources that have over 512MB RAM
collectorLocator = new CondorCollectorLocator();
collector = locator.getcondorCollector(new URL(“http://kotturoti.cs.ucl.ac.uk:9618”);
classAds = collector.queryStartdAds(“Memory>512”);
![Page 8: 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk.](https://reader036.fdocuments.net/reader036/viewer/2022062410/56649f285503460f94c40478/html5/thumbnails/8.jpg)
8
Example: Job Submission
1. Create New Transaction2. Create New JobId3. Send Files ({input files, binaries})4. Create Job Description5. Submit Job6. Commit Transaction
Transaction based process
Uses two phase commit to promote reliability and robustness
Files transferred in chunks – Base64 encoded SOAP attachments
DAGMan jobs supported
![Page 9: 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk.](https://reader036.fdocuments.net/reader036/viewer/2022062410/56649f285503460f94c40478/html5/thumbnails/9.jpg)
9
OMII Collaboration: gridSAM plugin Job Submission & Monitoring Web Service that submits to range of
resource managers (e.g. Condor, SGE, Globus) Uses JSDL (Job Submission Description Language) for
standardised job description
GridSAMGridSAM
CondorCondor SGESGE ForkFork ……
ClientClient
SOAP
JSDLJSDL
<?xml version="1.0" encoding="UTF-8"?> <JobDefinition xmlns="http://…"> <JobDescription> <Application> <Executable>/bin/echo</Executable> <Argument>hello world</Argument> </Application> </JobDescription></JobDefinition>
GridSAMGridSAM
CondorCondor SGESGE ForkFork ……
ClientClient
SOAP
JSDLJSDL
![Page 10: 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk.](https://reader036.fdocuments.net/reader036/viewer/2022062410/56649f285503460f94c40478/html5/thumbnails/10.jpg)
10
gridSAM plugin Existing Condor plugin
uses shell-based submission via command-line tools and monitoring of log files
GridSAMGridSAM
CondorCondor SGESGE ForkFork ……
ClientClient
Condor-WSCondor-WS Our plugin accesses Condor via the SOAP interfaces
Submissionmachine(s)
Schedd
Middle-tier
JSDLJSDLSOAP
SOAPClassAdClassAd
Building JSDL to ClassAd conversion libraries
![Page 11: 1 Condor BirdBath SOAP Interface to Condor Charaka Goonatilake Department of Computer Science University College London c.goonatilake@cs.ucl.ac.uk.](https://reader036.fdocuments.net/reader036/viewer/2022062410/56649f285503460f94c40478/html5/thumbnails/11.jpg)
11
Future Work Bundle plug-in into gridSAM
– Also provide separate JSDL to ClassAd conversion library
Extensions to other daemons (what functionality to expose?)– For example, allow checkpoint data to be retrieved through the shadow
in order to migrate jobs across pools
Roll out daemons on e-minerals mini-grid