Circumnavigating the Antarctic with Python and Django during ACE 2016
88
Python and Django during the Antarctic Circumnavigation Expedition (ACE) PyCon UK, 2017. Carles Pina i Estany ([email protected])
-
Upload
carles-pina-estany -
Category
Software
-
view
663 -
download
1
Transcript of Circumnavigating the Antarctic with Python and Django during ACE 2016
Python and Django during the AntarcticCircumnavigation Expedition (ACE)
PyCon UK, 2017. Carles Pina i Estany ([email protected])
What do we do for work?
CarlesC++/Qt software engineer at MendeleySome Python (scripts at work, evenings and weekends, Python code Dojos, etc.)Using GNU/Linux for a long timeLong time ago: system administrator / network administrator
2 / 45
What do we do for work?
CarlesC++/Qt software engineer at MendeleySome Python (scripts at work, evenings and weekends, Python code Dojos, etc.)Using GNU/Linux for a long timeLong time ago: system administrator / network administrator
JenWorked for 3.5 years at the British Antarctic Survey (BAS) as a data managerNow works for the Swiss Polar Institute, following up on the data management of ACEMore into wildlife and science (especially birds) than programming
2 / 45
What were our roles on the ship?As a team we:
managed the data of all 22 science projects
set up the data management recording system (Django);set up the data storage and back-ups;liaised with scientists to back up data, record metadata in a database and createmetadata records;
telecommunications setup:
onboard network;e-mail system;file uploader / Dropbox equivalent;
connecting science equipment.
3 / 45
What is ACE?
Antarctic Circumnavigation Expedition
4 / 45
Copyright: Swiss Polar Institute 5 / 45
R/V Akademik Tryoshnikov
6 / 45
7 / 45
8 / 45
Why is ACE different?Usually science expeditions focus on a small area of the Antarctic
9 / 45
Why is ACE different?Usually science expeditions focus on a small area of the AntarcticOnly other similar expedition was Discovery II in the 1950's
9 / 45
Why is ACE different?Usually science expeditions focus on a small area of the AntarcticOnly other similar expedition was Discovery II in the 1950'sSampled and studied marine and terrestrial habitats
9 / 45
Why is ACE different?Usually science expeditions focus on a small area of the AntarcticOnly other similar expedition was Discovery II in the 1950'sSampled and studied marine and terrestrial habitatsFirst expedition of the Swiss Polar Institute
9 / 45
Why is ACE different?Usually science expeditions focus on a small area of the AntarcticOnly other similar expedition was Discovery II in the 1950'sSampled and studied marine and terrestrial habitatsFirst expedition of the Swiss Polar InstituteIt was organised very quickly
9 / 45
Science
10 / 45
Copyright Swiss Polar Institute 11 / 45
12 / 45
13 / 45
14 / 45
Data management
15 / 45
What is it?Backup all the collected dataKeep a log of what's happening:
ultimate goal - to be able to describe where, when, by whom and for what reason thedata were collected
Keep a database of all samples collectedCreate metadata records of all data sets so they can be discovered
16 / 45
What is it?Backup all the collected dataKeep a log of what's happening:
ultimate goal - to be able to describe where, when, by whom and for what reason thedata were collected
Keep a database of all samples collectedCreate metadata records of all data sets so they can be discovered
Why?Backup to avoid losing data and to archive it properly for the futureThe data can be made publicly available after the 2 year embargoMake data easier to discover and usable by others in the future
16 / 45
We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)
17 / 45
We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computers
17 / 45
We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboard
17 / 45
We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboardDesign and implement an e-mail system that works with an unstable 128 kbit connectionfor 80 people
17 / 45
We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboardDesign and implement an e-mail system that works with an unstable 128 kbit connectionfor 80 peopleCreate a way to upload videos for the news (we had journalists on board)
17 / 45
We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboardDesign and implement an e-mail system that works with an unstable 128 kbit connectionfor 80 peopleCreate a way to upload videos for the news (we had journalists on board)Download data (ice images, weather models) for the scientists
17 / 45
We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboardDesign and implement an e-mail system that works with an unstable 128 kbit connectionfor 80 peopleCreate a way to upload videos for the news (we had journalists on board)Download data (ice images, weather models) for the scientistsHelp scientists to connect different equipment (like snow flake counter)
17 / 45
We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboardDesign and implement an e-mail system that works with an unstable 128 kbit connectionfor 80 peopleCreate a way to upload videos for the news (we had journalists on board)Download data (ice images, weather models) for the scientistsHelp scientists to connect different equipment (like snow flake counter)Fix a winch (software part)
17 / 45
We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboardDesign and implement an e-mail system that works with an unstable 128 kbit connectionfor 80 peopleCreate a way to upload videos for the news (we had journalists on board)Download data (ice images, weather models) for the scientistsHelp scientists to connect different equipment (like snow flake counter)Fix a winch (software part)...
17 / 45
science-data-managementJen said: Can you help me build a database?
18 / 45
science-data-managementJen said: Can you help me build a database?
I answered: yes! (and thought of MySQL and create table...)
18 / 45
science-data-managementJen said: Can you help me build a database?
I answered: yes! (and thought of MySQL and create table...)
Jen also meant a web system to enter data in the database!
18 / 45
19 / 45
Django
20 / 45
I knew:
Python
21 / 45
I knew:
PythonFlask
21 / 45
I knew:
PythonFlaskMySQL
21 / 45
I knew:
PythonFlaskMySQLSQLAlchemy
21 / 45
I knew:
PythonFlaskMySQLSQLAlchemyMustache
21 / 45
I knew:
PythonFlaskMySQLSQLAlchemyMustache
But... How could I create the forms to enter data into the database?
21 / 45
I knew:
PythonFlaskMySQLSQLAlchemyMustache
But... How could I create the forms to enter data into the database?
I asked Fran... and he said...
21 / 45
I knew:
PythonFlaskMySQLSQLAlchemyMustache
But... How could I create the forms to enter data into the database?
I asked Fran... and he said...
Django!
21 / 45
I knew:
PythonFlaskMySQLSQLAlchemyMustache
But... How could I create the forms to enter data into the database?
I asked Fran... and he said...
Django!(I thought... nooooo...)
21 / 45
I knew:
PythonFlaskMySQLSQLAlchemyMustache
But... How could I create the forms to enter data into the database?
I asked Fran... and he said...
Django!(I thought... nooooo...)
Fran created an example... and I loved it!
21 / 45
One week of DjangoI went back home and really enjoyed learning Django:
Writing Django models is easier than MySQL create table (with foreign keys, etc.)Easier to change the models and migrateWe needed authentication of usersPermissions for tables were usefulTemplates are very well integrated with modelsAll the admin part that we used has a good default UIEasy to extend
22 / 45
One week of DjangoI went back home and really enjoyed learning Django:
Writing Django models is easier than MySQL create table (with foreign keys, etc.)Easier to change the models and migrateWe needed authentication of usersPermissions for tables were usefulTemplates are very well integrated with modelsAll the admin part that we used has a good default UIEasy to extend
And it has very good off-line documentation!
22 / 45
Django modelsclass StorageCrate(models.Model): name = models.CharField(max_length=255, unique=True) location = models.CharField(max_length=255) description = models.CharField(max_length=255, null=True, blank=True) comment = models.TextField(null=True, blank=True)
23 / 45
Django modelsclass StorageCrate(models.Model): name = models.CharField(max_length=255, unique=True) location = models.CharField(max_length=255) description = models.CharField(max_length=255, null=True, blank=True) comment = models.TextField(null=True, blank=True)
It creates the forms automaticallyIf the model didn't exist: creates the tables in the databaseIf the model already existed: creates the migration codeDeals with foreign keys, relations, etc.It creates relational database models like we would have done
23 / 45
science-cruise-data-managementhttp://github.com/cpina/science-cruise-data-management
24 / 45
25 / 45
26 / 45
27 / 45
28 / 45
29 / 45
In retrospectDjango was a very good tool to do what we had to do:
Without Django we would have taken much longer!Django didn't get in the way as much as I had thoughtDjango commands were very easy to implement and useful for accessing the models fromcommand line parameters, widely used for many of our scriptsDjango developers are very nice and helpful! Thanks for the concise error messages:
main.Project.title: (fields.E120) CharFields must define a 'max_length' attribute.
30 / 45
In retrospectDjango was a very good tool to do what we had to do:
Without Django we would have taken much longer!Django didn't get in the way as much as I had thoughtDjango commands were very easy to implement and useful for accessing the models fromcommand line parameters, widely used for many of our scriptsDjango developers are very nice and helpful! Thanks for the concise error messages:
main.Project.title: (fields.E120) CharFields must define a 'max_length' attribute.
Django documentation is very well written!Admin forms were easy to adapt and change
30 / 45
31 / 45
What is Django?From the Django website:
Django makes it easier to buildbetter Web apps more quickly andwith less code.
32 / 45
What is Django?From the Django website:
Django makes it easier to buildbetter Web apps more quickly andwith less code.
Data manager thinks:
We need to build a database
32 / 45
What is Django?From the Django website:
Django makes it easier to buildbetter Web apps more quickly andwith less code.
Data manager thinks:
We need to build a database
Django for data managers?32 / 45
Very useful Django features for usAdmin forms were easy to adapt and change (we could work around something if itwasn't possible)commands are fantastic!
33 / 45
Very useful Django features for usAdmin forms were easy to adapt and change (we could work around something if itwasn't possible)commands are fantastic!
Django things that were not so helpful...Some modules use resources (CSS, js, etc.) from the Internet (e.g. debug module). Thisdidn't work for usHard to change (only via changing the template?) the main admin page
33 / 45
pip and rsyncpip wasn't "bad connection friendly":
No progress barWe didn't see how to easily see the URIs being download (e.g. apt-get --show-uris todownload them overnight)When bad networking: error messages weren't clear
34 / 45
pip and rsyncpip wasn't "bad connection friendly":
No progress barWe didn't see how to easily see the URIs being download (e.g. apt-get --show-uris todownload them overnight)When bad networking: error messages weren't clear
man rsync, --progress:
This option tells rsync to print information showing theprogress of the transfer. This gives a bored user something towatch.
I'd say "it helps my impatience".
34 / 45
pip and rsyncpip wasn't "bad connection friendly":
No progress barWe didn't see how to easily see the URIs being download (e.g. apt-get --show-uris todownload them overnight)When bad networking: error messages weren't clear
man rsync, --progress:
This option tells rsync to print information showing theprogress of the transfer. This gives a bored user something towatch.
I'd say "it helps my impatience".
Python documentationMore comprehensive off-line documentation could have been better (we relied on examplesand source code)
34 / 45
Django closing the circleWe wanted to thank Django somehow for all the fantastic help.
I thought of fixing at least one bug that we experienced during the expedition
35 / 45
Django closing the circleWe wanted to thank Django somehow for all the fantastic help.
I thought of fixing at least one bug that we experienced during the expeditionBug #28120 reported and fixed! (thanks for the quick code review, merge...)
35 / 45
Python/Django ideas if you go onan expedition
Take lots of code with you (e.g. Calibre, other projects, etc.). This was more useful thanbooks and documentation
36 / 45
Python/Django ideas if you go onan expedition
Take lots of code with you (e.g. Calibre, other projects, etc.). This was more useful thanbooks and documentationUse an IDE like Pycharm that allows you to navigate code of your libraries easier
36 / 45
Python/Django ideas if you go onan expedition
Take lots of code with you (e.g. Calibre, other projects, etc.). This was more useful thanbooks and documentationUse an IDE like Pycharm that allows you to navigate code of your libraries easierDownload the offline Django documentation: you will feel almost online!
36 / 45
Python/Django ideas if you go onan expedition
Take lots of code with you (e.g. Calibre, other projects, etc.). This was more useful thanbooks and documentationUse an IDE like Pycharm that allows you to navigate code of your libraries easierDownload the offline Django documentation: you will feel almost online!
Actually also download Wikipedia!
36 / 45
Appendix: Internet on the ship
37 / 45
IntroductionWe had 2 Iridium systemsInitially one for calls, another one for InternetThis is, 128 kbits for (officialy) 80 peopleThe Iridium connections are very unstable
38 / 45
L. Wood, SaVi: satellite constellation visualization, First Annual CCSR Research Symposium(CRS 2011), Centre for Communication Systems Research, 30 June 2011.http://arxiv.org/abs/1204.3265
39 / 45
Iridium66 Iridium satellites
40 / 45
Iridium66 Iridium satellitesIridium name: originally supposed to be 77 satellites: atomic number of Iridium
40 / 45
Iridium66 Iridium satellitesIridium name: originally supposed to be 77 satellites: atomic number of IridiumThere are spare satellites for when there are problems
40 / 45
Iridium66 Iridium satellitesIridium name: originally supposed to be 77 satellites: atomic number of IridiumThere are spare satellites for when there are problemsSignal on the north and south pole! But VERY slow and unstable: it connects anddisconnects because satellites are not geostationary
40 / 45
Iridium66 Iridium satellitesIridium name: originally supposed to be 77 satellites: atomic number of IridiumThere are spare satellites for when there are problemsSignal on the north and south pole! But VERY slow and unstable: it connects anddisconnects because satellites are not geostationarySatellites are around 780 Km above the Earth (GPS and Inmarsat around 20,000 Km)
40 / 45
Iridium66 Iridium satellitesIridium name: originally supposed to be 77 satellites: atomic number of IridiumThere are spare satellites for when there are problemsSignal on the north and south pole! But VERY slow and unstable: it connects anddisconnects because satellites are not geostationarySatellites are around 780 Km above the Earth (GPS and Inmarsat around 20,000 Km)Sometimes the satellites are visible: Iridium flares!
40 / 45
Iridium66 Iridium satellitesIridium name: originally supposed to be 77 satellites: atomic number of IridiumThere are spare satellites for when there are problemsSignal on the north and south pole! But VERY slow and unstable: it connects anddisconnects because satellites are not geostationarySatellites are around 780 Km above the Earth (GPS and Inmarsat around 20,000 Km)Sometimes the satellites are visible: Iridium flares!They are launching new satellites
40 / 45
41 / 45
42 / 45
43 / 45
Thank you!Questions?
44 / 45
ContactCarles Pina i Estany ([email protected])Jen Thomas ([email protected])
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
SlidesSlides have been written using Markdown and rendered using remarkjs (https://remarkjs.com)
Slides: https://github.com/cpina/ace-it-presentation / https://cpina.github.io/ace-it-presentation/
ACE: http://spi-ace-expedition.ch/
45 / 45
