Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de...
Transcript of Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de...
![Page 1: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/1.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
Tracking provenance information with the OPUS job controller
Mathieu Servillat
Observatoire de Paris - LUTHParis Astronomical Data Centre
1
![Page 2: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/2.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
OPUS: Observatoire de Paris UWS Server
• Main features• IVOA standards
• Universal Worker System (UWS)• Provenance data model
• REST architecture • Python micro-framework: bottle.py
• Collaborative development • Git server at PADC (gitolite)• GitHub: https://github.com/mservillat/OPUS
• Tests and quality• Unit tests with unittest and webtest• Activity history with logging
• Prototype available• https://voparis-uws-test.obspm.fr
2
![Page 3: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/3.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
Job management at PADC
Available structure• Work cluster (Tycho)• Job scheduler (SLURM)
PADC projects• Web based clients
• Data access• Online data processing• Wrap simulation codes
Need a simple interface to computational resources
3
![Page 4: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/4.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
Inner workings
• Separate job controller from work cluster• Wait for work cluster signals• Avoid (too many) status queries to work cluster
4
![Page 5: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/5.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
Server main classes
5
SQLAlchemySQLitePostgreSQL…
SLURMLocal
ActivityDescriptionWADLJSON
![Page 6: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/6.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
IVOA Provenance Data Model diagram
Description side
Configuration
Groups ofactivities
Groups ofentities
(same in W3C)
6
![Page 7: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/7.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
Serializations - ActivityDescription
VOTable DataLink Service DescriptorUWS Job Description LanguageProvenance ActivityDescription
7
![Page 8: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/8.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
Client main features
Javascript based + Python Flask• UwsLib.js: sends AJAX requests to the server• uws_manager.js: handles and displays responses
• Integration with Bootstrap3• HTML pages with specified <div> elements
(id=joblist, parameters, results…)
Job definition editor• Interface to create the ActivityDescription file
• parameters, used and generated entities• bash execution script
Job manager• list jobs, create jobs, control jobs, view results
8
![Page 9: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/9.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
Definition of the gammapy_maps job
OPUSObservatoire de Paris UWS Server
9
![Page 10: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/10.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
Submission of a gammapy_maps job
10
• OPUS reads the ActivityDescription file to generate a form
• This form also carries the Datasets Obscore description
![Page 11: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/11.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
Tracking of Provenance informations
Results and Provenance
11
![Page 12: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/12.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
UWSdatabase
ActivityDescription
(VOTable)
ProvenanceFiles
(VOTable, XML,
JSON, SVG)
+ =Provides Provenance files
12
![Page 13: Paris Astronomical Data Centre Observatoire de Paris ... · Mathieu Servillat (Observatoire de Paris) 29 Aug 201829 Aug 2018 OPUS: Observatoire de Paris UWS Server • Main features](https://reader033.fdocuments.net/reader033/viewer/2022061001/60b07f69c86c616f755c73de/html5/thumbnails/13.jpg)
Mathieu Servillat (Observatoire de Paris) 29 Aug 2018Mathieu Servillat (Observatoire de Paris) 29 Aug 2018
Serializations - W3C PROV formats
13