Flexibility, Manageability and Performance in a Grid Storage Appliance

John Bent, Venkateshwaran Venkataramani, Nick Leroy, Alain Roy, Joseph Stanley, Andrea Arpaci-Dusseau, Remzi

Arpaci-Dusseau, and Miron Livny

University of Wisconsin

Two trends

Data setsPerformance

• Storage appliances address both trends.

Storage Appliances: + and -

Storage Appliances: Great for basic file service Easy to manage: Plug in and it works Good performance: Specialized just for I/O Reliable and available too

Storage Appliances for the Grid: Mismatch? Inflexible: Few, specific protocols (e.g., NFS) Costly: 10x the cost of PC + a few disks Difficult to integrate: Just one piece of the puzzle

A Solution: NeST

NeST: A Storage Appliance for the Grid

Flexible: Multiple simultaneous protocols Virtual protocol layer

Low-cost: Use commodity machines Dynamic adaptation

Grid-aware: Integrate w/ higher-level systems Design specifically for the Grid

Outline

Introduction General architecture Design goals

Flexibility Low-cost Grid-aware features

NeST in the Grid example Conclusions

Physical storage layer

Chirp HTTP Grid FTP NFSCommon protocol layer

Dispatcher

Transfer Mgr Concurrency ModelsStorage Mgr

Control flowDatal flow

NeST: Protocol layer

Physical network layer Virtualizes different protocols

Mediates access to network

Physical storage layer

Chirp HTTP Grid FTP NFSCommon protocol layer

Dispatcher

Transfer Mgr Concurrency ModelsStorage Mgr

Control flowDatal flow

NeST: Dispatcher

Physical network layer Mediates interaction between other components

Gathers information, advertises

Physical storage layer

Chirp HTTP Grid FTP NFSCommon protocol layer

Dispatcher

Transfer Mgr Concurrency ModelsStorage Mgr

Control flowDatal flow

NeST: Storage manager

Physical network layer Space management

Access control

Virtualizes physical storage

Physical storage layer

Chirp HTTP Grid FTP NFSCommon protocol layer

Dispatcher

Transfer Mgr Concurrency ModelsStorage Mgr

Control flowDatal flow

NeST: Transfer manager

Physical network layer Implementsscheduling policies

Chooses concurrency model

Outline

Introduction General architecture Design goals

Flexibility Low-cost Grid-aware features

NeST in the Grid example Conclusions

Flexibility: Multiple protocols

Problem: How to support multiple protocols? One approach: Just a Bunch of Servers (JBOS)

Problems with JBOS Lack of control (scheduling) Painful administration No shared code Larger memory footprint nfsd

wu-ftpd

httpd

JBOS Server

NeST: Flexibility By Design

NeST: Integrate protocols and gain advantage Implementation like VFS

Integration introduces new challenges Different protocols allow different auth models More expensive to add a new protocol Less fault isolation

NeST vs JBOS

Chirp GridFTP HTTP NFS0

5

10

15

20

25

30

35

Ser

ver

band

wid

th (

MB

/s)

0

5

10

15

20

25

30

35

Total

• For each protocol, NeST is comparable to JBOS server.

wu-

ftpd

Apa

che

linu

x nf

sd

Linux cluster- Dual PIII- 1 GB Ram- linux 2.2.19Each protocol- 4 clients- 10 MB files

Exerting scheduling control

Different scheduling policies FCFS Cache-aware [USENIX ‘02] Proportional share

Proportional share scheduling Allows administrators to set protocol proportions

e.g. favor NFS Very difficult in JBOS

Proportional share

0

5

10

15

20

25

30

35

Ser

ver

band

wid

th (

MB

/s)

FCFS 1:1:1:1 1:2:1:1 1:1:1:4Scheduling configuration

• In most cases, achieves Jain’s metric of fairness > 0.98 (1 is “fair”).

Linux cluster- Dual PIII- 1 GB Ram- linux 2.2.19Each protocol- 4 clients- 10 MB files

Outline

Introduction General architecture Design goals

Flexibility Low-cost Grid-aware features

NeST in the Grid example Conclusions

Low-Cost: New challenges Desire: Run on arbitrary OS on arbitrary PC

Software-only, user-level storage appliance Currently on Linux (release 0.9) and Solaris (beta)

Problem: Portable performance Performance under load is platform / workload dependent

Threads or processes on some systems, events on others May also be workload dependent (e.g. whether in cache)

NeST approach: Dynamic adaptivity Simultaneously support multiple concurrency models Monitor performance using each model Bias towards better model over time

Adaptive Concurrency

0

1

2

3

Solaris: 1K files

Ave

tim

e pe

r re

ques

t (se

c)

Eve

nts

Thr

eads

Linux: 10 MB files

Eve

nts

Thr

eads

0

1

0.5

1.5

Ave

tim

e pe

r re

ques

t (m

s)

Ada

ptiv

e

Ada

ptiv

e

• Dynamic adaptation approaches “ideal” without static information.

Outline

Introduction General architecture Design goals

Flexibility Low-cost Grid-aware features

NeST in the Grid example Conclusions

Grid-Aware Mechanisms

Basic functionality Users and groups: Dynamic creation/deletion

does not need administrative intervention Access control: Generic AFS-style ACLs

Advanced functionality QoS: Preferential scheduling Advertises into global scheduling systems Flexible protocol and authentication mechanisms Self-cleaning storage guarantees: Lots

Storage guarantees: Lots

Characteristics of Lots: Capacity: Total amount of data lot can store Duration: Time for which data is guaranteed to exist Set of files: Multiple files may co-exist within lot

Self-cleaning Expired lots become “best-effort” lots

Lot management Either default set created by administrator, OR

use resource management protocol to create before usage Implementation: File system quotas

Advantage: Integrates cleanly with local access methods Disadvantage: Performance hit for large writes

Outline

Introduction General architecture Design goals

Flexibility Low-cost Grid-aware features

NeST in the Grid example Conclusions

NeST in the Grid

LinuxNeST

SolarisNeST

Home Remote

Global Execution Manager

computenode

computenode

computenode

computenode

Condor Tribune

Advertisement

NeST in the Grid

N N

Home Remote

Global Execution Mgr

GridFTP

3)

3) Global coordinates xfer1)

1) Home submits jobs

6)

6) Global terminates space5) 5) Global coordinates xfer

NFS4)

4) Global starts jobs2)

2) Global reserves space

Conclusions

NeST: A storage appliance for the Grid Gain manageability Without sacrificing performance

Design goals: Flexibility: Virtual protocol architecture Low-cost: Adaptation mechanisms Grid-aware: Space management

Current status: release 0.9 available Future work

Hot deployable NeSTs, lot management extensions

Questions?

http://www.cs.wisc.edu/condor/nest