Fast, Inexpensive Content- Addressed Storage in Foundation Sean Rhea*Russ Cox, Alex Pesterev*...
-
Upload
ariel-kennedy -
Category
Documents
-
view
213 -
download
0
Transcript of Fast, Inexpensive Content- Addressed Storage in Foundation Sean Rhea*Russ Cox, Alex Pesterev*...
Fast, Inexpensive Content-Addressed Storage in Foundation
Sean Rhea* Russ Cox, Alex Pesterev*Meraki, Inc. MIT CSAIL
*Work done while at Intel Research, Berkeley.
“Digital Dark Ages?”
• Users increasingly store their most valuable data digitally– Wedding/baby photographs– Letters (now called email)– Diaries, scrapbooks, tax returns
• Yet digital information remains especially vulnerable
• Terry Kuny: “We are living in the midst of digital Dark Ages”– Hard drives crash– Removable media evolve (e.g., 5 ¼” floppies)– File formats become obsolete (e.g., WordStar, Lotus 1-2-3)– What will the world remember of the late 20th century?
As a community, we’re not bad at storing important data over the long term.
We’ve only just begun to think about how we’ll interpret that data 30 years from now.
For Example…
• Viewing an old PowerPoint presentation– Do we still have PowerPoint at all? And Windows?– Does the presentation use non-standard fonts/codecs?– Has some newer application overwritten a shared
library with an incompatible version (“DLL Hell”)?
• Not just a Microsoft problem: consider a web page– Even current IE/Safari/Firefox don’t agree on formatting– All kinds of plugins necessary: sound, video, Flash
The Foundation Idea
• Make daily backups of entire software stack– Archives users’ applications, OS, and configuration state
• Don’t worry about identifying dependencies– Just save it all: “Every byte, every night”
• To recover an obscure file, boot the relevant stack in an emulator– View file with the application that created it
Foundation FAQ
• Why preserve the entire disk?– Preserve software stack dependencies: preserve the data with the right
application, libraries, and operating system as a single unit– Works for all applications, not just ones designed for preservation
• Why daily images?– Want to preserve machine state as close as possible to last write of
user’s data (i.e., preserve image before something changes)– Also allows recovery from user errors
• Why emulate hardware?– Much better track record than emulating software– Software example: OpenOffice emulating Microsoft Word (yikes)– Hardware emulators available today for Amiga, PDP-11, Nintendo…
I would love to give a talk about why Foundation is a great solution to the
digital preservation problem.
Really, though, I think it’s just a pretty good start.
Instead, I’m going to talk about a fun problem we had to solve to make it work.
Every Byte, Every Night?Indefinitely? Really?
• Plan 9 did exactly that– Archive changed blocks every night to optical jukebox– Found that storage capacity grew faster than usage
• Later with Content-Addressable Storage (Venti)– Automatically coalesces duplicate data to save space– Required multiple, high-speed disks for performance
• Challenge for Foundation: provide similar storage efficiency on consumer hardware– “Time Machine model”: one external USB drive
Talk Outline
• Introduction– What is Foundation?– Review of Content-Addressed Storage (Venti)
• Contributions– Making Cheap Content-Addressed Storage Fast– Avoiding Concerns over Hash Collisions
• Related Work
• Conclusions
Venti Review
• Plan 9 file system was two-level– Spinning storage, mostly a normal file system– Archival storage, optical write-once jukebox
• Venti replaced optical jukebox– Still write-once– Chunks of data named by their SHA-1 hashes
“Content-Addressable Storage (CAS)”– Automatically coalesces duplicate writes
5:h( )16:7:8:9:
h( )2
reads 1st blockreads 2nd block
User’s Hard Drive External USB Drive
Hash Offset
Data Log
seen it before?
0:1:2:3:h( )04:
RAM
ArchivalProcess
Summary
h( )
appendto log
update index
appendhash to
summary,h( ) ,h( )
reads 4th block
no logwrite!
h( ),
Venti Review
Venti Review
User’s Hard Drive External USB Drive
Hash Offset
Data Log
0:h( )41:2:h( )33:h( )04:h( )7
5:h( )16:h( )67:h( )58:h( )29:
RAM
Summary
h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( )
RestoreProcess
lookup hashof 1st block
map hash to log offset
read blockfrom log
restore block
Crash!
Final step (not shown): archivesummary in data log as well
Notes on Venti• The Good News:
– CAS stores each block with particular contents only once– Changing any one block and re-archiving uses only one
more block in archive– Adding a duplicate file from a different source uses no
additional storage
• The Bad News:– Synchronous, random reads to on-disk index
reads 4th block
User’s Hard Drive External USB Drive
Hash Offset
Data Log
seen it before?
0:1:2:3:h( )04:
5:h( )16:7:8:9:
RAM
ArchivalProcess
Summary
h( ),h( ) ,h( )
h( )2
Venti Review
Have to seek to theright bucket
Venti Review
User’s Hard Drive External USB Drive
Hash Offset
Data Log
RAM
Summary
h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( )
RestoreProcess
lookup hashof 1st block
map hash to log offset
0:h( )41:2:h( )33:h( )04:h( )7
5:h( )16:h( )67:h( )58:h( )29:
Have to seek to theright bucket
Notes on Venti• The Good News:
– CAS stores each block with particular contents only once– Changing any one block and re-archiving uses only one
more block in archive– Adding a duplicate file from a different source uses no
additional storage
• The Bad News:– Synchronous, random reads to on-disk index– Best case, one-disk performance for 512-byte blocks:
one 5 ms seek per 512 bytes archived = 100 kB/s
– That’s 12 days to archive a 100 GB disk!– Larger blocks give better throughput, less sharing
Notes on Venti (con’t.)• Venti’s solution: use 8 high-speed disks for index
– Untennable in consumer space– Wears disks out pretty quickly, too
• The “compare-by-hash” controversy:– Fear of hash collisions: two different blocks with same
hash breaks Venti– May be very unlikely, but cost (data corruption) is huge
Does CAS really require a cryptographically strong hash?
Talk Outline
• Introduction– What is Foundation?– Review of Content-Addressed Storage (Venti)
• Contributions– Making Cheap Content-Addressed Storage Fast– Avoiding Concerns over Hash Collisions
• Related Work
• Conclusions
Making Inexpensive CAS Fast
• The problem: disk seeks– Secure hash randomizes an otherwise sequential disk-
to-disk transfer– To reduce seeks, must reduce hash table lookups
• When do hash table lookups occur?1. When writing data, to determine if we’ve seen it before
2. When writing data, to update the index
3. When reading data, to map hashes to disk locations
2. Updating the Index
• After appending a block to the data log, must update the index– Psuedorandom hash causes a seek
User’s Hard Drive External USB Drive
Hash Offset
Data Log
0:1:2:3:h( )04:
5:h( )16:7:8:9:
RAM
ArchivalProcess
Summary
h( )
appendto log
update indexUpdating the Index
Have to seek to theright bucket
reads 2nd block
2. Updating the Index
• After appending a block to the data log, must update the index– Psuedorandom hash causes a seek
• Easy to fix: use a write-back index cache– Store index writes in memory– Flush to disk sequentially in large batches– On crash, reconstruct index from the data log
3. Mapping Hashes to Disk Locations During Reads
• To restore disk– Start with the list of original blocks’ hashes– Lookup each block in index– Read block from data log and restore to disk
User’s Hard Drive External USB Drive
Hash Offset
Data Log
RAM
Summary
h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( )
RestoreProcess
lookup hashof 1st block
map hash to log offset
0:h( )41:2:h( )33:h( )04:h( )7
5:h( )16:h( )67:h( )58:h( )29:
Have to seek to theright bucket
3. Mapping Hashes to Disk Locations During Reads
• To restore disk– Start with the list of original blocks’ hashes– Lookup each block in index– Read block from data log and restore to disk
• Observation: data log is mostly ordered– Duplicate blocks often occur as part of duplicate files
Ordering in Data Log
User’s Hard Drive External USB Drive
Hash Offset
Data Log
RAM
Summary
h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( )
0:h( )41:2:h( )33:h( )04:h( )7
5:h( )16:h( )67:h( )58:h( )29:
3. Mapping Hashes to Disk Locations During Reads
• To restore disk– Start with the list of original blocks’ hashes– Lookup each block in index– Read block from data log and restore to disk
• Observation: data log is mostly ordered– Duplicate blocks often occur as part of duplicate files– Idea: add another index, ordered by log offset– Read-ahead in this index to eliminate future lookups
in original index
Offset Hash0:h( )1:h( )2:h( )3:h( )4:h( )5:h( )
6:h( )7:h( )8:9:
10:11:
read blockfrom log(seek!)
read blockfrom log
(no seek!)
Index by Offset
User’s Hard Drive External USB Drive
Hash Offset
Data Log
RAM
Summary
h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( ),h( ), h( ), h( )
RestoreProcess
lookup hashof 1st block
map hash to log offset (seek!)
Crash!
Hash Offset
prefetch hashes for next few offsets from
secondary index(seek!)
new index, sorted by offset
h( )0h( )1h( )2
h( )3h( )4
restore block 0:h( )41:2:h( )33:h( )04:h( )7
5:h( )16:h( )67:h( )58:h( )29:
lookup hashof 2nd block
find log offsetin secondary
index – no seek!
1. Is a Block New, or Duplicate?
• Optimization for reads also helps duplicate writes– Index misses on first duplicate block– Hits on subsequent blocks rewritten in same order
• Doesn’t help for new data– Every lookup in primary index fails– Still suffer a seek for every new block
1. Is a Block New, or Duplicate?
• Idea: use a Bloom filter to identify new blocks– Lossy representation of the primary index– Uses much less memory than index itself
• For any given block, Bloom filter tells us:– It’s definitely new append to log, update index– It might be duplicate lookup in index
• If it really is a duplicate, we get the prefetch benefit• Otherwise, called a “false positive”• Using enough memory keeps false positives at ~1%
Results
• Do these optimizations pay off?– Buffering index writes is an obvious win– Bloom filter is, too: removes 99% of seeks when
writing new data– Both trade RAM for seeks
• Benefit of secondary index less clear– If duplicate data comes in long sequences, it reduces
index seeks to two per sequence– If duplicate data comes in little fragments, it doubles
the number of index seeks– Need traces of real data to answer this question
Results (con’t.)
• Research group at MIT has been running Venti as its backup server for two years– We looked at 400 nightly snapshots– Simulated archiving and restoring these in both Venti
and Foundation
Venti Foundation
Average archival speed < 1 MB/s 20.1 MB/s
% time spent seeking 96% 10%
Average restore speed 1.2 MB/s 13.6 MB/s
% time spent seeking 95% 58%
Talk Outline
• Introduction– What is Foundation?– Review of Content-Addressed Storage (Venti)
• Contributions– Making Cheap Content-Addressed Storage Fast– Avoiding Concerns over Hash Collisions
• Related Work
• Conclusions
Eliminating “Compare by Hash”
• Some worried that same SHA-1 doesn’t imply same contents (i.e., hash collisions are possible)– Even if very rare, consequences (corruption) too great
• Stepping back a bit, CAS as a black box:– Give it a data block, get back an opaque ID– Give it an opaque ID, get back the data block
• Do we care that the ID is a SHA-1 hash?– What if the “opaque” ID was just the block’s location
in the data log?
Using Locations As IDs
• Pros+ Reads require no index lookups at all+ System can still find potential duplicates using
hashing (with a weaker, faster hash function)
• Cons– Need another mechanism to check integrity– Since hash untrusted, must compare suspected
duplicates byte-by-byte
• Others have claimed these byte-by-byte comparisons are a non-starter
2nd Disk Arm to the Rescue
• Once we eliminate most index reads (via our previous optimizations), the backup disk is otherwise idle while backing up duplicate data
• Can instead put it to work doing byte-by-byte comparisons of suspected duplicates
Foundation
Venti By Hash By Value
Archival < 1 MB/s 20.1 MB/s 15.4 MB/s
Restore 1.2 MB/s 13.6 MB/s 15.0 MB/s
Talk Outline
• Introduction– What is Foundation?– Review of Content-Addressed Storage (Venti)
• Contributions– Making Cheap Content-Addressed Storage Fast– Avoiding Concerns over Hash Collisions
• Related Work
• Conclusions
Related Work
• Apple Time Machine– Duplicates coalesced at file level via hard links
• Netapp WAFL, ZFS– Copy-on-write coalesces blocks at the FS level– Misses duplicates that come into system separately
• Data Domain Deduplication FS– Very similar to Foundation, in enterprise context– Depends on collision-freeness of hash function
• Lots of other Content-Addressed Storage work– LBFS, SUNDR, Peabody
Conclusions
• Consumer-grade CAS works now– A single, external USB drive is enough– Just have to be crafty about avoiding seeks
• Lots of uses other than preservation– E.g., inexpensive household backup server that
automatically coalesces duplicate media collections
• Doesn’t require a collision-free hash function