A Personal Cloud Storage App written in Python ... · Cloud Based File Sync Local File System...

James Abel – PyBay Aug 2017

Latus®A Personal Cloud Storage App written in Python

www.github.com/jamesabel/latus

James Abel

Aug 12, 2017

[email protected]

@jamesabel

www.abel.co

Latus is a registered trademark of James Abel

http://www.github.com/jamesabel/latus

mailto:[email protected]

http://www.abel.co/


Intro

• James Abel

• HW/SW consultant

• Former Intel Principal Engineer

• Some Python for several years (otherwise C/C++, ASM, etc.)

• Python ‘go to’ programming language for the last few years


Agenda• What is latus?• Why should I care?• Cloud based File Synchronization

• File system events• Event database• What’s in the cloud• Synchronization

• Encryption• Task Bar Application• Using latus• Next Steps• Summary

latus is on github at www.github.com/jamesabel/latus



What is latus?

• File storage and synchronization across your computers• AKA cloud storage• Automatically mirrors files across your computer and to the cloud• Make files accessible• Inherent backup

• App that runs in the background (‘client sync’)

• Open Source (GPLv3)

• “Zero Knowledge” Encryption

• Inherent versioning

• Python 3.6

• Utilizes lots of awesome packages!

Latus is a cloud file sync app written in Python


Why should I care?• You like cloud storage file sync, but you want ..

• Open Source• Python• Zero Knowledge Encryption to the cloud• Control over exactly where files are stored in the cloud (e.g. for regulatory, compliance

and/or business requirements)• Control over versioning history• Control over what takes up space in the cloud in the long term• Custom integrated applications• ‘Independent Web’

• Provide a customized branded (‘white label’) cloud storage/sync offering

• Or just example uses of several awesome packages and capabilities

Open Source File Sync written in Python!


What is file sync?

Computer A Computer B

The Interneta.txt

“latus” folder “latus” folder


What is file sync?

Computer A Computer B

The Interneta.txt a.txt

“latus” folder “latus” folder


Cloud Based File SyncFile System Events Database Cloud Synchronization

Local File System (PC/Mac)

WatchdogEvent Database(AWS DynamoDB + local SQLite cache)

Events

Sync Engine

File Cloud Storage (AWS S3)

Files

Computer A

Cloud

Computer B



Cloud Based File Sync



Events

Sync Engine


Files(encrypted)

Computer A

Cloud

Computer B


Step 1: File “a.txt” created





Events

Sync Engine


Computer A

Cloud

Computer B


Step 2.a.:“a.txt Created” Event

Step 2.b.:“a.txt” to S3

Files(encrypted)


Step 3:“a.txt” mirrored




Events

Sync Engine


Computer A

Cloud

Computer B


Files(encrypted)

Events

Files(decrypted)


mivui – Monotonically Increasing Value• Events are strictly sequenced

• Monotonically Increasing Value in Micro-Seconds (uS) since Epoch as an Integer

• Equivalent to: int(round(time.time()*1E6))

• Integer allows reliable comparison and DB indexing/sort key

• Server provides mivui• Optionally can be locally created (but is then not absolutely guaranteed to be monotonically

increasing)

• http://api.abel.co/miv

{

"mivui": 1502435085769804,

"toc": 0,

"mivf": 1502435085.769804,

"dur": 0.0018029212951660156,

"success": true,

"mivs": "1502435085.769804“

}

http://api.abel.co/miv


File System Event Database Items

• originator – node (computer) where this event was detected (normally a UUID)

• mivui – monotonically increasing integer in micro-seconds from epoch (from server)

• detection – how was event detected – initial scan, watchdog, periodic poll

• event_type – create, delete, move, modify

• file_hash – SHA512 of file

• file_path – file path in the latus folder

• mtime – mod time of the file

• size – file size (in bytes)

• src_path – source in the case of moves (otherwise unused)


Cloud and Local event databases

Cloud database is ‘the’ database – local is a cache

Cloud(AWS DynamoDB)

Local Computer Nodes(SQLite cache)a b

AWS

SQLite

AWS DynamoDB


Synchronization

• Sync is hard

• Compute (algorithm) on the client

• All nodes see the events on all other nodes and use the same algorithm

• ‘winner’ is based on ordered events• Default algorithm is that the most recent event (across all nodes) wins

• Other algorithms are possible

• Files are available in the cloud to provide them locally• Encrypted in the cloud


Sync example$vi a.txt

$vi b.txt

$mv a.txt c.txt

$rm b.txt

DynamoDB

S3

class DetectionSource(IntEnum):

unknown = 0

initial_scan = 1

watchdog = 2

periodic_poll = 3

class LatusFileSystemEvent(IntEnum):

created = 1

modified = 2

deleted = 3

moved = 4

Winner!


Encryption• Keep unencrypted files off the internet/cloud• Uses the most excellent Python cryptography package

• Key exists on client nodes (computers)• Local computer is assumed secure

• In the cloud file contents are stored as AWS S3 objects• S3 object names are SHA512 hashes of the files contents ‘salted’ with encryption key• Thwarts dictionary lookup attacks

• For security, the encryption key should not be transferred over the internet• Keep the key ‘out of band’• Once generated, a key can be exported to a file (simple JSON format)• USB stick works well (put it in a safe place!)

• On first run:• Makes a new encryption key if no one exists

or• Asks the user for an existing key to be imported


boto3• Amazon’s Python library to access Amazon Web Services (AWS)• AWS credentials stored on the local file system or passed in explicitly to

boto3 APIs• ~/.aws/credentials

[default]

aws_access_key_id = YOUR_ACCESS_KEY

aws_secret_access_key = YOUR_SECRET_KEY

region=us-west-1 # means Northern California

• Access AWS

import boto3

s3 = boto3.resource('s3’)

# Upload a new file

data = open('test.jpg', 'rb')

s3.Bucket('my-bucket').put_object(Key='test.jpg', Body=data)


AWS S3 (with encryption)

‘a’

This object’s contents contains the bytes corresponding to a file with contents ‘b’

latusstorage S3 bucket

James Abel – Pyninsula Mar 2017

latus is a Task Bar App …


What is a Task Bar Application?

• A taskbar is an element of a graphical user interface which has various purposes. It typically shows which programs or applications are running on the device, as well as provide links or shortcuts to other programs or places, such as a start menu, notification area, and clock.

https://en.wikipedia.org/wiki/Taskbar

• AKA System Tray

https://en.wikipedia.org/wiki/Graphical_user_interface_elements

https://en.wikipedia.org/wiki/Graphical_user_interface

https://en.wikipedia.org/wiki/Computer_program

https://en.wikipedia.org/wiki/Mobile_app

https://en.wikipedia.org/wiki/Hyperlink

https://en.wikipedia.org/wiki/Shortcut_(computers)

https://en.wikipedia.org/wiki/Start_menu

https://en.wikipedia.org/wiki/Notification_area

https://en.wikipedia.org/wiki/Clock

https://en.wikipedia.org/wiki/Taskbar


PyQt

• Qt (/kjuːt/ "cute"[7][8][9]) is a cross-platform application framework that is used for developing application software that can be run on various software and hardware platforms with little or no change in the underlying codebase, while still being a native application with native capabilities and speed.• https://en.wikipedia.org/wiki/Qt_(software)

• What is PyQt?• PyQt is a set of Python v2 and v3 bindings for The Qt Company's Qt application

framework and runs on all platforms supported by Qt including Windows, OS X, Linux, iOS and Android.

• https://riverbankcomputing.com/software/pyqt/intro

• Currently using PyQt5

https://en.wikipedia.org/wiki/Help:IPA_for_English

https://en.wikipedia.org/wiki/Qt_(software)#cite_note-7



https://en.wikipedia.org/wiki/Cross-platform

https://en.wikipedia.org/wiki/Application_framework

https://en.wikipedia.org/wiki/Application_software

https://riverbankcomputing.com/software/pyqt/intro

http://www.qt.io/

https://riverbankcomputing.com/software/pyqt/intro


Hello World Task Bar App

class HelloWorldSystemTray(QSystemTrayIcon):

def __init__(self):

import icons

icon = QIcon(QPixmap(':icon.png'))

super().__init__(icon)

menu = QMenu()

menu.addAction("About").triggered.connect(self.about)

menu.addAction("Exit").triggered.connect(self.exit)

self.setContextMenu(menu)

def about(self):

about_box = QDialog()

layout = QGridLayout(about_box)

layout.addWidget(QLabel('hello world'))

about_box.setLayout(layout)

about_box.show()

about_box.exec_()

def exit(self):

QApplication.exit()

app = QApplication(sys.argv)

app.setQuitOnLastWindowClosed(False) # so popup dialogs don't close the system tray icon

system_tray = HelloWorldSystemTray()

system_tray.show()

app.exec_()


• Latus Task Bar App

class LatusSystemTrayIcon(QSystemTrayIcon):

def __init__(self, app, latus_appdata_folder):

self.app = app

self.latus_appdata_folder = latus_appdata_folder

menu = QMenu(parent)

menu.addAction("Open Latus Folder").triggered.connect(self.open_latus_folder)

menu.addAction("Preferences").triggered.connect(self.preferences)

menu.addAction("Export Latus Key").triggered.connect(self.export_latus_key)

menu.addAction("About").triggered.connect(self.about)

menu.addAction("Nodes").triggered.connect(self.nodes)

menu.addAction("Exit").triggered.connect(self.exit)

self.setContextMenu(menu)

def start_latus(self):

self.sync = latus.aws.sync_aws.Sync(self.latus_appdata_folder)

def show(self):

QSystemTrayIcon.show(self)

def exit(self):

self.sync.request_exit()

QApplication.exit()

app = QApplication(sys.argv)

system_tray = LatusSystemTrayIcon(app, latus_appdata_folder)

system_tray.start_latus()

system_tray.show()

app.exec_()


Preferences

• Local preferences kept in preferences.db• SQLite DB

• Latus crypto key

• Latus folder path

• AWS configuration (credentials, region)

• Accessible via icon pulldown


Using latus• Can be cloned and executed from source

• www.github.com/jamesabel/latus

• latus as a native app• latus uses many packages – it can be a challenge to freeze/install• cryptography and PyQt can be problematic to freeze

• Embedded Python interpreter tends to work best

• Freezers/Installers• Windows: pynsist, osnap using the ‘embedded’ Python interpreter

• MacOS: osnap currently

latus can be run for source or as a native app for Windows and MacOS



Hacking latus

• Fork from• www.github.com/jamesabel/latus

• Run regression tests• Uses py.test, localstack

• Has create, delete, move, etc.

• But can use more tests. ☺

• Pull requests

• File issues



Help – get a cool T Shirt!

* While supplies last** If you help ☺


Some Next Steps …

•Add AWS pubsub to get file system events instantly (AWS SNS/SQS)

•Cloud storage efficiency enhancements• Select folders for ‘offline only’

•Node management• Filter out special files (e.g. “.*”)•Code review ☺

Try out latus! Its on github at www.github.com/jamesabel/latus



Summary and Thank You

• latus is an Open Source Cloud Storage app in Python 3.6

• Hack on latus at SF Python Project Night, chat at Pyninsula or Bay Piggies

• Thank you• Glyph• Thomas Kluyver

Try out latus! Its on github at www.github.com/jamesabel/latus


A Personal Cloud Storage App written in Python ... · Cloud Based File Sync Local File System...

Documents

Transcript of A Personal Cloud Storage App written in Python ... · Cloud Based File Sync Local File System...