Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query...
Transcript of Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query...
![Page 1: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/1.jpg)
Anastasia GalkinJochen KlarGal Matijevič Kristin RiebeHarry Enke
ADASS 2019
Daiquiri - Python based framework for the publication of scientific databases
![Page 3: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/3.jpg)
Scientific Databases hosted @AIP
3
• CosmoSim https://www.cosmosim.org
• RAVE https://www.rave-survey.org
![Page 4: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/4.jpg)
Scientific Databases hosted @AIP
4
• CosmoSim https://www.cosmosim.org
• RAVE https://www.rave-survey.org
• Gaia@AIP https://gaia.aip.de
![Page 5: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/5.jpg)
Scientific Databases hosted @AIP
5
• CosmoSim https://www.cosmosim.org
• RAVE https://www.rave-survey.org
• Gaia@AIP https://gaia.aip.de
• MuseWIDE https://musewide.aip.de
![Page 6: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/6.jpg)
Scientific Databases hosted @AIP
6
• CosmoSim https://www.cosmosim.org
• RAVE https://www.rave-survey.org
• Gaia@AIP https://gaia.aip.de
• MuseWIDE https://musewide.aip.de
• APPLAUSE https://www.plate -archive.org
![Page 7: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/7.jpg)
7
![Page 8: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/8.jpg)
8
![Page 9: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/9.jpg)
9
![Page 10: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/10.jpg)
10
![Page 11: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/11.jpg)
11
![Page 12: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/12.jpg)
12
![Page 13: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/13.jpg)
13
![Page 14: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/14.jpg)
Daiquiri – from a user perpectiveA framework for the publication of scientific databases
14
• Allows for highly customizable web applications
• Based on a common easily maintainable code base
• Separated into an app and the daiquiri library
• Features:– SQL web interface to relational databases
– User management and user registration work flow
– Metadata management and access control
– First impression in-browser plotting
– Table download
– Archive file service
– VOTable upload
• Employs VO protocols and standards
![Page 15: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/15.jpg)
Daiquiri features for the provider
15
• SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL – pgSphere integration, ADQL support
• Customizable data query forms• Contact messages to the support staff• DOI integration and landing pages for databases and tables • OAI-PMH interface for harvesting services• File service and zip-archive creation• Registration and log-in using Oauth2 (twitter, GitHub, Google, ORCID)• WordPress integration for project presentation and documentation• VO protocols: Table Access Protocol (TAP), Cone search, UWS • File access, filtering and download for observatory archives (MUSE WIDE)• Cut-out service for images and datacubes
open source and available on GitHub and PyPy
![Page 16: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/16.jpg)
16
![Page 17: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/17.jpg)
Language and framework
17
Python and Django• Python: todays preferred scripting language, widely used in astronomy• Django: full MVC framework with everything included, huge community• Django REST framework: de-facto standard for REST interfaces in Django• django-allauth: local and social authentication, registration work flows• astropy: community python library for Astronomy• Queryparser and ANTLR for translation between ADQL and PostgresQL• ...
Front end: AngularJS and Bootstrap• AngularJS 1: awesome since Daiquiri v1• Bootstrap 3: responsive layout, mobile friendly
Back end: • PostgresQL, MariaDB• Asynchronos tasks with Celery, Redis, RabbitMQ• Systemd, linux pipes
![Page 18: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/18.jpg)
Asynchronous jobs
18
Celery and RabbitMQ and old-school unix pipes
Downloading tables
Celery and RabbitMQ• Celery asynchronous task queue queue in Python, widely adopte• RabbitMQ: message broker in Erlang• redis: in-memory data structure store for task results, can also be used for caching• systemd: new init system for Linux, make it easy to deploy daemons
![Page 19: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/19.jpg)
Try django-daiquiri!
19
as a userProduction version of the Gaia@AIP Services: gaia.aip.de or plate-archive.org
as a providerhttps://github.com/django-daiquiri/daiquiri
as dockerhttps://github.com/django-daiquiri/daiquiri-docker-compose
![Page 20: Daiquiri - Python based framework for the …...Daiquiri features for the provider 15 • SQL query interface (with examples, job list, plotting) - full SQL syntax of PostgresQL –](https://reader034.fdocuments.net/reader034/viewer/2022042621/5f6e12c482f45c7bb463cdb2/html5/thumbnails/20.jpg)
Questions?
20
Anastasia Galkin
github.com/django-daiquiri
escience.aip.de