Online storage for the masses and the case of pithos
Transcript of Online storage for the masses and the case of pithos
![Page 1: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/1.jpg)
1
Online storage services for the masses and the case of Pithos
Nectarios KozirisCSLAB-NTUA / Vice Chair GRNET
GridNets 2009, AIT-Athens, 9 /9/2009
www.pithos.gr
![Page 2: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/2.jpg)
2
Why Online Storage?
• Hard Disk costs are decreasing per TB (Sept 09: 80$/TB)
• Speed ….? 10s+MB/sec? 100+MB/sec with SSDs!
• MTTF: 1.000.000-1.500.000h –> Annual replacement ~1%
But.. true annual replacement up to 10%!(FAST07: ‘Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you?’)
• Many redundancy options
• Portability?– USB sticks 64GB common today (~120$)
![Page 3: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/3.jpg)
3
Pros
• Access anyplace, anytime
• Reliability
• Multiple workplaces, multiple PCs (sync between)
• Sharing/collaboration
• Security?
• Backup to the net/restore anywhere
• Access from online applications
• Access from different devices (mobiles…)
![Page 4: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/4.jpg)
4
Open issues• Ease of use• Security• Privacy• Provider reliability (xdrive..)
• Speed (uploading takes time….)
• Integration with local OS/applications/websites• Filesize limits• Multiplatform access (mobiles/netbooks etc)• No clear business model!
Still a long way to go….
![Page 5: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/5.jpg)
5
Online Storage for users
• ADrive: www.adrive.com– 50GB free/2GB per file
• Microsoft Skydrive: SkyDrive.live.com – 25GB free/50MB per file
• Mozy (Home Free 2GB) & 3.75€/month unlimited space• Box.net
– 1GB free
• DropBox (uses Amazon S3): www.dropbox.com – 2GB free, 50GB at 9,9$/month, 100GB at 19,9$/month
• Livedrive.com – 100GB /£39.95 per year
• Esnips 5GB free• Humyo, Infinit, YouSendIt, Wuala, etc..
Remember Xdrive.com?
– 2005: 5GB free, 50GB at $9.95/month
– AOL acquired it 2005 / 5M$ selling/ closed it on January 12, 2009.
![Page 6: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/6.jpg)
6
Online Storage as a Service: Amazon S3
• unlimited storage through a simple web services interface • scalability, high availability, and low latency at commodity
costs. • REST-style HTTP interface // SOAP interface • 5GB file size per bucket in S3• DropBox, SmugMug, Slideshare, Twitter, Woot.com and many
other sites use Amazon S3 to host images/videos/files.
AWS: +Elastic Block Store (EBS) for EC2 VMs
![Page 7: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/7.jpg)
7
SmugMug example
• 2006: SmugMug spent $6,000 per TB of space
• S3 since April 2006 - Savings of US$500,000 per year by using S3
• Despite outages/slowdowns of S3 in 2006 continued.
• saved almost $1 million in storage costs in 12 months…
![Page 8: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/8.jpg)
8
GRNET• GRNET is the greek academic, education and research
network (NREN)
• Offers services since 1995
• 1998 GRNET company formed (fully publicly owned)
• 2002 GRNET2 upgrade, 2004 GRNET2 B’ phase (λ based)
• 2008 GRNET3 upgrade (10G, dark fiber)
• 2009 GRNET4
Virtualization at all levels:Networking, Storage, Computing (VMs)
![Page 9: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/9.jpg)
9
GRNET3 DF Network
• Basic characteristics– 35 PoPs– Includes:
• 8410 km• MANs Attica &
Thessaloniki• DF local loops at 33
cities
![Page 10: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/10.jpg)
10
Pithos• From Wikipedia: Pithos (plural pithoi) is the ancient Greek word (πίθος, πίθοι)
for a large storage jar of a characteristic shape.
• Both Storage for users and Storage as a Service from GRNET
– At the current stage, 22 universities have Shibbolethised their AA process (total number of possible users > 300,000)
1. Personal Online Storage:– 50GB/user– Versioning (per file)– Backup– Groups– Sharing– Full text Search and tagging– No filesize limit/no file expiration– no traffic-bandwidth limit
www.pithos.gr and more…
![Page 11: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/11.jpg)
11
Pithos
(more:)– rich web GUI client (access via the REST API) based on GWT– desktop client (access via the REST API) – Firefox addon (via REST)
2. Storage as a Service:– REST API– WebDAV interface
A truly scalable online file storage infrastructure
![Page 12: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/12.jpg)
12
GSS software for Pithos
• Stands for GRNET Storage Service
• It’s the software behind PITHOS• Provides a virtual disk (storage) accessed over the network
– Users are able to upload, share, and index their files
– Inspired by Amazon S3, but going beyond…
• Targeting research & academic community– > 300,000 users
• Funded by the national project HellasGrid (www.hellasgrid.gr)
• Open Source Code http://code.google.com/p/gss/(designed and code developed by www.ebs.gr)
![Page 13: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/13.jpg)
13
User Functionality• offers users a file system abstraction, complete with file/
folder hierarchical structures
• Basic functions– Usual file system operations, e.g. copy, move, etc.– Delete plus “trash-bin” functions– 50GB free space per user, easily extendible
• Advanced functions– File sharing with selected users, user groups or public– Automatic file versioning, e.g. keep history of the file– Text search and tagging capabilities for file names or
folders– Shibbolized authentication, aka no need for extra
credentials– Access over IPv6
![Page 14: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/14.jpg)
14
GUISecure Web Access
File Sharing
User Credentials
Available Resources
Versioning
IPv6 Support
Public access (if enabled)
Search Capabilities
Properties
Flexible File Naming
![Page 15: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/15.jpg)
15
![Page 16: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/16.jpg)
16
Technical Details• All GSS functionality provided on the application
layer via application server farms– Implemented in Java, dual stack (IPv6)
• Multiple access interfaces– Direct API (REST), web-based client (AJAX), desktop client
(Tcl/Tk), WebDAV.• Use Shibboleth (shibboleth.internet2.edu) for AA– Use student credentials from university catalogues
• Infrastructure– 12 servers (IBM x3550, 2.0 GHz 16GB RAM), SAN (140TB, SATA
disks), 10Gbps connection (to GRNET backbone)
![Page 17: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/17.jpg)
17
webdav interface (XP/Vista/MacOS/Linux)
![Page 18: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/18.jpg)
18
FireGSS plugin for Firefox using REST
![Page 19: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/19.jpg)
19
Another Pithos Firefox plugin…
![Page 20: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/20.jpg)
20
Pithos Architecture
![Page 21: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/21.jpg)
21
Logical Topology - Components
• Cluster of GSS servers (handling main functionality and APIs)
• Solr server(s) ( for indexing and search functionality)
• DB server (for storing meta-data)
• GUI web clients
![Page 22: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/22.jpg)
22
![Page 23: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/23.jpg)
23
Shibboleth-based Access
![Page 24: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/24.jpg)
24
Current Status
• Beta testing Completed (over 1500 users)• Performance testing
– On going process
• Enrich functionality– Open source effort http://code.google.com/p/gss/– Take advantage of REST API for third-party
applications – Firefox Plugins
![Page 25: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/25.jpg)
25
Issues to discuss further
• Cost per TB? – Commodity Disks…
• Scalability– Scalable metadata structures
– Fast, symmetric access-NGNs (FTTH)
• Reliability– software based
• Storage for VMs (Block Store etc)
![Page 26: Online storage for the masses and the case of pithos](https://reader034.fdocuments.net/reader034/viewer/2022050614/55494c2db4c90566498b46b7/html5/thumbnails/26.jpg)
26
For more info:
http://pithos.grnet.gr
http://www.pithos.gr
http://code.google.com/p/gss/
contact: [email protected]/~nkoziris
www.pithos.gr