Post on 16-Dec-2014
description
Apache Airavata
GSOC 2013
Knowledge and Expertise
Computational Resources
Scientific Instruments
Algorithms and Models
Archived Data and Metadata
Advanced Science Tools
Target Community: Science Gateways Enabling & Democratizing Scientific Research
What does Apache Airavata do?• Compose, manage, execute,
and monitor distributed, computational workflows.
• Wrap legacy command line scientific applications with Web services.
• Run jobs on computational resources ranging from local resources to computational grids and clouds.
• Manage provenance data.
Workflow Interpreter
Application Factory
Message Box
Registry
Apache Airavata
API
Lorem ipsum
insolens
p1m5
duo x
End
Use
rsG
atew
ay
Dev
elop
er
Scientific Applicati
on
Core Developer
Computational Resources
Apache Airavata
Apache Airavata ComponentsComponent Description
XBaya Workflow graphical composition tool.
Registry Service Insert and access application, host machine, workflow, and provenance data.
Workflow Interpreter Service
Execute the workflow on one or more resources.
Application Factory Service (GFAC)
Manages the execution and management of an application in a workflow
Messaging System WS-Notification and WS-Eventing compliant publish/subscribe messaging system for workflow events
Airavata API Single wrapping client to provide higher level programming interfaces.
Hi, I’m Nolram. I’m a computational
physicist. I run computational
experiments everyday
This is how typically I run my experiments
Scientific Application
Another ScientificApplication
First I collect my observed data
And then pass data to my applications & get
the result
This is starting to become a very tiring
task
How can I make this much simpler…?
Logically, this is how my life would be
made easier…
Is it possible to automate this flow
sequence without my guidance?
Scientists from many different fields face this
problem everyday.
The solution is to use a workflow-powered science gateway to
manage the experiment online.
What is a workflow you ask? Well, you just saw one in
our previous animation…
We introduce Apache Airavata, a system capable of composing, managing, executing, and monitoring
small to large scale applications and workflows
Want to see how it works?
A Typical Workflow
Apache Airavata
I will handover my data & my experiment details (the workflow)
to the Airavata server
The Gateway
Airavata will complete the experiment & return me the results
Results
Progress of the experiment
… and while I wait for results, Airavata will notify me with
progress updates of my experiment
Let’s look closely how Airavata manages workflows.
The GatewayResults
Experiment progress
Apache Airavata
Let’s look closely how Airavata manages workflows.
The GatewayResults
Experiment progress
Airavata main has 4 components…
The Gateway
1. Workflow InterpreterSteer the workflow execution
2. The GFacSteer science app executions & data
transfers
Workflow Interpreter
GFac
Message Box
Registry
3. The RegistryDefines the available applications &
records all results of experiments
4. The Message BoxRecords the progress of the workflow
execution
A Stable API for Airavata
Apache Airavata
Lorem ipsum
insolens
p1
m5
duo x
End
Use
rsG
atew
ay D
evel
oper
Scientific Application
Computational Resources
Application Registration
UIApplication Developer
A1
Airavata Service Interface
(wraps client API)
Service Map XML
Airavata Server
Service Map to AWSDL
Web Based workflow composerWorkflow
Developer
Get AWSDL
Put XWF
A2
A3
W1W2
W3
Web Based Experiment
BuilderExperimentBuilder
Get WI’s
E1
E2
E3
W4
Shred Workflow
Inputs
Launch Workflow
Web Based Workflow MonitorWatch Progress
M1M2
Get Workflow Graph
M3 Monitor Workflow
Goal of the project
• Design Web-Based interfaces for Airavata:– Application Registration– Workflow Construction– Workflow Execution– Workflow Monitoring
• Provide an opportunity for GSoC to understand Distributed System in action
• Scope for Research and Software Engineering papers
Data Model
• Application Description – User describes inputs and outputs of the
application. – Currently this information is captured in Service
Map Schema. – This schema is stored in Airavata Registry as XML.
Also the schema utility generates a application service WSDL from this schema using the Airavata WSDL Generator.
Application Registration
UI
Application Developer
A1
Service Map XML
Service Map to AWSDL
Web Based workflow composer
WorkflowDeveloper
Get AWSDL
A2
W1
W2
XML
Aira
vata
Ser
ver A
PI
Launch & Manage Jobs
Notify progress of job or workflow execution
Real-Time Monitoring
Messaging Subsystem
Registry
Execute & Manage Computations
Workflow Interpreter
Application Factory (Gfac)
Application Desc
Application Desc
Workflow
A peek at one of the cluster
InterconnectNodes
Scheduling ‘qsub’ batch jobs on the cluster worker node
worker node
worker node
worker node
worker node
SGE MASTER node
Queue-A Queue-B Queue-C
A S
lot 1
A S
lot 2
B S
lot 1
C S
lot 1
C S
lot 2
C S
lot 3
B S
lot 1
B S
lot 2
B S
lot 3
B S
lot 1
C S
lot 1
C S
lot 2
A S
lot 1
B S
lot 1
C S
lot 1
Queues
Policies
Priorities
Share/Tickets
Resources
Users/Projects
JOB Y JOB ZJOB X
JOB U
JOB OJOB N
System characteristics
System statusResources
Job policiesResources
Resource Matching
JOB
User
User policiesGroupsRolesDepartmentsProjects
Selection Scheduling
Simplified Gateway Architecture
One time Gateway Community Setup
Community Account Grid Certificate username, password
Gateway InterfaceGateway Server
Compute Servers
Gateway Authentication
Fetch
Community C
reden
tialGrid
Prox
y
Job Submit or File Transfer request
Output
Proxy, Job Request
Job Status, Output
Step 0
Step 1
Step 2,3,,
XSEDE User Portal
ParamChem
BioVLab
GridChem
DES
VLAB
UltraScan
Apache Airavata 1.0
NSG
ParamChem
BioVLab
CIPRES
POPLAR
GridChem
DES UltraScan
VLAB
…
…
…
Apache Airavata 2.0
Apache Airavata 1.0
Apache Airavata 1.0
Apache Airavata 1.0
Apache Airavata 1.0
Apache Airavata 1.0