PITHOS

Architecture and .NET Technologies

Panagiotis Kanavos

Object storage similar to Amazon S3/Azure Blob storage

A service of Synnefo Written in Python Clients for Web, Windows, iOS,

Android, Linux

Synnefo

Client API

REST API based on OpenStack Object Storage API

Accounts, Containers without Folders GET for data, object info PUT, POST for uploads and data

updates

StructureA

ccount Container

Folder1/Object1

Folder1/Object2

Container

Folder1/Object1

Folder2/Object2

API Extensions

Block Storage Partial Upload/Download Permissions, Versions Metadata Queries UUIDs for Object IDs Object updates (copy, move)

API Characteristics

No folders! Placeholder directory object hold

metadata Block updates ONLY Merkle hashing to detect modified

blocks Hash using SHA256

Merkle Hashing

Top Hash

Hash of #1-2

Hashes

Block #1 Hash

Block #2 Hash

Hash of #2-3

Hashes

Block #3 Hash

Block #4 Hash

Download Process

Get Hashmap from server

Calculate local hashmap

Find different blocks

Download blocks

Patch local file with blocks

Upload Process

Calculate local hashmap

PUT to server

Server responds with missing block hashes

PUT missing blocks at

container level

Repeat from #2

Server responds 201

Pithos Client

Multiple accounts per machine Synchronize local folder to Pithos

account Detect local changes and upload Detect server changes and download Calculate Merkle Hash for each file

Client Architecture

UI

WPF

MVVM

Caliburn Micro

Core

File Agent

Poll Agent

Network Agent

Status Agent

Networking

CloudFiles

HttpClient

Storage

SQLite

SQL Server

Compact

Technologies

.ΝΕΤ 4, due to Windows XP support req Visual Studio 2012 + Async Targeting Pack UI - Caliburn.Micro Concurrency - TPL, Parallel, Dataflow Network – HttpClient Hashing - OpenSSL – Faster than native

provider for hashing Storage - NHibernate, SQLite/SQL Server

Compact Logging - log4net

The challenges

Handle hundreds of file events Hashing of many large files Multiple slow connections to the

server Unstable network Yet it shouldn’t hang Minimal UI with enough info for the

user

Event Handling

• Listen• Wait for Idle

File Agent

Events Handling(2)

Use producer/consumer Store events in ConcurrentQueue Process ONLY after an idle timeout

Merkle Hashing

Why I hate Game of Thrones Asynchronous reading of blocks Block hashing in parallel Use OpenSSL to gain SSE2 etc Concurrency throttling Watch the memory consumption!

Memory Leaks in a Managed Environment!

4ΜΒ Blocks? Large Memory but … Quickly reading 2GB in 64ΚΒ blocks? Downloading 600ΜΒ in x KB blocks? Huge number of small objects awaiting

collection during CPU/IO intensive processing

Poor Garbage Collector can’t keep up!

Hashing 100% CPU?

Multicore is nice but … Blocks the system when processing large

files! Throttle parallel block hash ops Improvements:

Dynamically throttle «large» files «Throttling» of File Read Ops

Multiple slow network calls Every call a Task Concurrent REST calls per account

and shared folder Task.WhenAll to process results at

end of poll

Unstable network

Use System.Net.Http.HttpClient Store downloaded blocks

in .pithos.cache folder Check and reuse orphans Asynchronous Retry of calls

Resistance to crashes

Use Transactional NTFS where available

Thanks MS for killing it! Modify a copy File.Replace otherwise

Should not hang

Use independent agents Asynchronous operations wherever

possible Use async/await for more readable

code Must always .ConfigureAwait(false)! BE CAREFULL of async void

Minimal UI

Use WPF, MVVM Use Progress to update the UI

Part of .NET 4.5, backported to 4 The Icon is the Shell! Lack of good WPF Notification Icon Problematic Data Binding in menus

SQLite or Compact CE?

Initially SQLite -> Staleness problems (DUH !) Write Ahead logging, means you can see

stale data Switch to SQL Compact to allow

concurrent updates (duh ?) Really needed better caching? Akavache?

A Document DB is better suited

Next Steps

File Manager UI General Cleanup (DUH!) Bring back Unit Tests (Duh ?) Mock Server

WebAPI? scriptcs? Yumm! Create a separate Pithos library Windows RT, Windows Phone clients

AFTER the cleanup

Links for Pithos

Pithos trialhttp://pithos.okeanos.io

Synnefo Documentationhttp://www.synnefo.org/docs/synnefo/latest/index.html

Pithos API Documentationhttp://www.synnefo.org/docs/pithos/latest/index.html

Pithos Windows Clienthttps://code.grnet.gr/projects/pithos-ms-client

Useful Links

Parallel FX Team bloghttp://blogs.msdn.com/b/pfxteam

Caliburn.Microhttp://caliburnmicro.codeplex.com/

Ayende’s BufferPoolhttp://ayende.com/blog/4827/answer-stopping-the-leaks

Useful Books

C# 5 in a Nutshell, O’Riley Parallel Programming with .NET,

Microsoft

• Pro Parallel Programming with C#, Wiley

• Concurrent Programming on Windows, Pearson

• The Art of Concurrency, O’Reilly