AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas...
-
Upload
clifford-booth -
Category
Documents
-
view
217 -
download
1
Transcript of AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas...
![Page 1: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/1.jpg)
agINFRA science gateway for workflows and integrated services
07/02/2012
Robert Lovas
MTA SZTAKI
![Page 2: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/2.jpg)
Why workflows?
• ‘Orchestration of Tasks’ (not only in sequence; organized in Direct Acyclic Graphs)
• Data-driven• To exploit large computational resources / process large data sets• To make the complex applications run faster
– By applying paralellization techniques on them
• Paralellization Techniques:– Indepent tasks can be executed concurrently– Execute tasks against LARGE datasets parameter study, domain
decomposition, etc.
![Page 3: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/3.jpg)
Parameter study / domain decomposition
GEN
SEQ
COLL
SEQSEQSEQ
Generates input
parameter space
Evaluates the results of the
simulation
Parameter sweep jobs
3
![Page 4: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/4.jpg)
Liferay-based WS-GRADE/gUSE portal for agINFRA:– http://aginfra-portal.lpds.sztaki.hu/liferay-portal-6.0.5/– Open Registration for project participants– X509 Certificate authentication required to be able to submit jobs NEW: robot certs– agINFRA VO is accessible (ca. 5000 CPU cores, >50 TB storage)
![Page 5: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/5.jpg)
Access modes
5
WS-PGRADEWF
DeveloperUI
gUSE DCI Bridge
DCI 1
DCI 2
DCI n
ApplicationSpecific
User Interface
ExistingApplicationSpecific UI
WS-PGRADEEnd-User
UI
Remote API
BES interface
ASM API
A
B
C
D
E
BES interface
ASM API
WS-PGRADE UI
Customized UI
Other, existing UI
gUSE Workflow engine
agINFRA VOagINFRA VO
Volunteers’ computersVolunteers’ computers
![Page 6: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/6.jpg)
Workflow building blocks (glossary)• “Jobs” operating on data• Jobs can be:
– Grid-enabled applications (e.g. AgrovocTagging)
– Web-services
– NEW: REST services
– Another workflows (embedded)
• “Ports” representing inputs and outputs for the jobs– Available port types:
• Value• Local file e.g. from the scientists laptop• Remote File (gsiftp, lfc) in the Grid• Database Queries (SQL)• ….
– Extensions or improvement might be required (e.g. Drupal / Dublin Core / SPARQL / CIARD RING support?) 6
![Page 7: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/7.jpg)
agINFRA overview
![Page 8: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/8.jpg)
Services to be integrated
Harvesting, validation, transformation
![Page 9: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/9.jpg)
The Organic.Edunet Ingest Workflow
![Page 10: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/10.jpg)
Schematic Representation of the AGRIS workflow
![Page 11: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/11.jpg)
Cross-community workflows identified at the Athens
![Page 12: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/12.jpg)
DEMONSTRATION I.
![Page 13: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/13.jpg)
AgroTagger
![Page 14: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/14.jpg)
First demo application
![Page 15: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/15.jpg)
First demo application - details
![Page 16: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/16.jpg)
Job details
![Page 17: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/17.jpg)
Inputs and outputs
![Page 18: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/18.jpg)
Monitoring of execution
18
![Page 19: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/19.jpg)
Successful execution of AgrovocTagging application
![Page 20: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/20.jpg)
DEMONSTRATION II.
Harvesting workflow
![Page 21: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/21.jpg)
ARIADNE aggregation panel
- Select schedulingLink to the workflow interface
-Invoke the workflow-Check the status of the workflow
-Stop the workflow-Add metadata for the
aggregation
ARIADNE aggregation panel
- Select schedulingLink to the workflow interface
-Invoke the workflow-Check the status of the workflow
-Stop the workflow-Add metadata for the
aggregation
gUSE WS-PGRADEgUSE WS-PGRADE
Harvesting- Add parameters of
agDataHarvesters web service
Harvesting- Add parameters of
agDataHarvesters web service
Metadata Validation vs target
schema - Add parameters of
agMetadataValidation web service
Metadata Validation vs target
schema - Add parameters of
agMetadataValidation web service
Target Validation- Add parameters of
agTargetValidation web service
Target Validation- Add parameters of
agTargetValidation web service
Target schema?
Target schema?
No
• Stop the process for the specific target• send message or store logs and send them
through the gUSE API
Yes
Transformation- Get parameters for the
transformation and invoke agMetadataTransformation
web service
Transformation- Get parameters for the
transformation and invoke agMetadataTransformation
web serviceNo
Store metadata on the GRID (agINFRA VO)
Store metadata on the GRID (agINFRA VO)
YesValid?Valid?
starting a pre-defined procedure as an agINFRA workflow
![Page 22: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/22.jpg)
Integration of multiple components/services
![Page 23: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/23.jpg)
Details in demo…
![Page 24: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/24.jpg)
Plan: Continue the development of aggregation workflow
![Page 25: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/25.jpg)
Plan: Further integration
+ volunteers…
![Page 26: AgINFRA science gateway for workflows and integrated services 07/02/2012 Robert Lovas robert.lovas@sztaki.mta.hu MTA SZTAKI.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f255503460f94c3bb31/html5/thumbnails/26.jpg)
07/02/2012
Questions?
26