TiBS

Fermilab – HEPiX-HEPNTRay Pasetes

October 22, 2003

Problem

• AFS full backups > 40 hours

• UNIX full backups > 20 hours

• Operator assisted tape mounts

• Mandate for automated operations

Investigation

• March – July 2003

• Software evaluated included– Veritas Netbackup– SyncSort– IBM/Tivoli TSM– Amanda– Teradactyl/TiBS

Criteria

• Support for IBM AFS and OpenAFS

• Reduce backup windows < 24 hours

• Complete automation

• Scalable

• Reliable

• Affordable

EvaluationAFS/

OpenAFS

< 24Window

Automated Scalable Reliable Affordable

Veritas No Yes Yes Yes Yes No

SyncSort No Yes Yes Yes Yes No

IBM Yes Yes Yes Yes Yes No

Amanda Yes Yes Yes Yes Yes No*

TiBS Yes Yes Yes Yes Yes Yes

Notes: Veritas

• No support for OpenAFS

• No ongoing development for AFS

• No bug fixes for current AFS offering

• Pricing– Per seat, by OS, #CPUs, and CPU type– Number of tape drives

• Different pricing for SAN attached drives

– Lots of add-on feature charges

Notes: SyncSort

• No support for AFS

• Pricing– Per seat, UNIX or Windows– Number of tape drive slots in library

• Different pricing for SAN attached drives

– Pricing came in packs• May not fit current equipment.• I.E. 170 tape slot pack for 120 slot library• 25 pack UNIX licenses for 18 UNIX systems

Notes: IBM

• Actively supporting AFS and OpenAFS

• Can do single file restores for AFS

• Pricing– Vague pricing unless we were willing to

commit to significant purchase– Per seat, OS, CPU type

Notes: Amanda

• AFS support enhanced by internal work

• Pricing– The support effort to maintain the product was

not available

Notes: TiBS

• Support for AFS and OpenAFS

• Pricing– Client

• Per OS, irregardless of clients• Enterprise (all supported OSes)

– Server• Per process (number of parallel backups)

Decision

• July 2000, decision to deploy TiBS• Production -- August 2000

– 400GB of data, all AFS– Only part of CSI backups migrated

• Currently, ~1.6TB– 140+ total systems

• AFS, HFS+, NTFS, UNIX file systems.

– IRIX, Linux, OSX, Solaris, Tru-64, W2K, XP– Backups for CSI and SCS migrated

• Backup Window ~< 5 hours

TiBS: How does it work?

TiBS == True incremental Backup System• First backup is a full backup over the network• All subsequent backups are incrementals over

the network• Subsequent weekly and full tapes generated

from previous Fulls, Weeklies and incrementals at the server, without client participation

• All data staged to spooling disk first

More on TiBS

• Incremental data – Mirrored in disk cache– Merged with previous incrementals daily

• Does allow for direct-to-tape backup– Can not use merge process

• One-pass restore

Additional Benefits

• AFS– Reports on volumes that need salvaging– Consistency check between vldb and fileservers– Look up individual AFS files before restoration

• General– Hostaudit reports any new partitions/disks on a

backup client– Merge process auto-checks integrity of tapes– Lost tapes can be reconstructed from previous tapes– User initiated backups -- laptops– Support has been excellent

Cons

• May require large caching disk– Typically 10% of total data backed up– Cache needs to be at least as large as largest

filesystem

• Requires a larger library– Or more aggressive tape handling– Reliance on previous tapes to create data

Keep in Mind

• Non-traditional scalability factors– Size of individual filesystems vs. overall

storage– Disk/cache and tape speed vs. network speed– CMU-CS backs up 7TB using a single sparc,

512MB memory, 280GB cache, 4DLT8Ks.

• Higher reliance on previous backups

• Failed backups have higher significance

More on Teradactyl

• Roots are from CMU– Birthplace for AFS– CMU ECE and H&SS departments are beta

sites

• Very open to customizing solutions– K5/KCA authentication

Summary

• Originally purchased for CSI AFS backups and UNIX server backups (20 systems total)

• Proved to be very scalable on current hardware and has expanded to two groups, 140+ systems

• We are now backing up desktops, laptops, Windows and Mac OSX as well as AFS and traditional UNIX systems

• Does not require enterprise version of Linux OS

Additional Information

• FNAL– http://www-oss.fnal.gov/csi/afs/stats/tibs/html/bindex.html– Ray Pasetes <rayp@fnal.gov>– Kevin Hill <kevin@fnal.gov>

• Vendor– http://www.teradactyl.com