Tailoring NAS Proxies for Virtual Machines

[email protected]

[email protected]

Sponsored by:

&

mailto:[email protected]

mailto:[email protected]

• Traditional tradeoff: Local vs. Central storage: – Central storage migrates very simply.

• Just complete last work from old location

–Then start at the new location.

• Extracts non-local latency tax on every access.

– Purely local storage makes migration a challenge.

• Must migrate VM image and local storage.

• And synchronize them.

2 Xen Summit Aug 2, 2011 –

Caitlin.Bestler/[email protected]

• SSD vs HDD: –SSDs provide most IOPS per $.

•SSDs need to be on host, or at least near the VM, or network latencies minimize SSD speed.

–HDDs provide most TBs per $.

•HDDs provide capacity, not processing speeds.

Xen Summit Aug 2, 2011 –


Host

Device Domain

DomU

VFS

File System

Block Driver

Network Driver

File System

Block Backend SSD Proxy

External NAS

Server(s)

App

NAS Client Block Driver

External SAN

Server(s)

Network Driver

Remote Block Access

Local Block Driver

Local Block Device

• A very common solution currently features a Block Backend that co-ordinates a local block device (typically SSD) with remote block device (still NAS or SAN).

• Problem:

– Any cache solution only stores a portion of a VM‟s storage locally.

– With NAS Proxy the “portion stored locally” can be understood as files and directories.

• A block-oriented solution never sees more than a collection of blocks.

• Many solutions are enhanced or made possible with File-orientation.

4



• DomU can support both NAS

access and Virtual Disk Access – Most VMs will prefer directly-attached boot

drive.

• Backend can map virtual drive to

NAS.

• Meanwhile NAS Client can access

NAS Proxy via local internal

network.

• NAS Proxy can utilize local SSD

drive to optimize either.

5

Host

Device Domain

DomU

VFS

NAS Client

Network Driver

Network Driver

NAS Proxy

Network Backend

External NAS

Server(s)

App

Block Driver

Local Block Device

File System

Block Driver

BlockBackend



• NAS Benefits

– Files enable understanding of context:

– Deduplication of files is far easier than deduplication of block.

– Hierarchical Storage Management works on files, not blocks.

• Traditional NAS Proxy Caveats

– Traditional NAS is unaware of VM-specific semantics.

– Excessive syncing – Historical assumptions are that files are

shared by default.

– Proxy overhead for non-cached files – Historically, the only easy

configuration was to proxy all NAS access.

• VM Storage Proxy provides full benefits of NAS without

limitations of traditional NAS Proxies.



• Distributed NAS protocols such as NFSv4 and

DFS/CIFS have features that make NAS proxies far

more efficient.

• They allow efficient caching via exclusive delegation

and oplocking (“opportunistic locking”).

– Proxy can temporarily claim a file.

• They have advanced session features to protect

against data loss from connection loss.

– File reverts to central NAS server if Proxy dies.

– System recovers from network partitions and temporary

outages of distributed servers and/or proxies as long as

they all have persistent redundant storage.



• Client can simply mount from NAS server.

• NAS server can refer select directories to a NAS “Server” that is actually the NAS Proxy.

– NAS Proxy/Server must act as a NAS Server for NAS replication procedures.

• Client does not need to identify itself.

– NAS server can infer identity from IP address.

• Advantage that Proxy/Server is enabled/disabled for each directory from the central NAS Server.

9

VM Client

NAS Server

NAS Proxy / Server

All Directories

Cached Files Directory access

Referral

Replication



• NAS Proxy can act as a NAS

client and claim an exclusive

delegation on a file.

• It can then treat its cached copy

as the copy.

• It must replicate it back to the

central NAS server before

ending the delegation.

• Keep-alive messages track the

NAS client with the delegation.

• Protocol rules already cover

loss of contact/failure

scenarios.

10

VM Client

NAS Server

NAS Proxy / Server

All Directories

Cached Files

Claim Exclusive Delegation

DelegationGranted

Local copy canbe treated as

THE copy.



• Generic NAS Proxy does not understand that it is the sole writer for the VM Private Directory.

• NAS Proxy management is not integrated with VM Management. – The NAS system must detect that the VM using its VM Private

Directory has migrated.

• Proxies only optimize storage for selective files – but all traffic goes through them. – If there are directories/files that are not worth caching the clients

would ideally just go directly to the real server.

– NAS Referrals can be applied selectively, but Management Plane controls are awkward for dynamically controlling referrals.

• Optimized cache sharing across multiple clients is challenging without understanding of VM life-cycle.



• Allow VM‟s files to be stored and archived as

files.

• Allow storage associated with commodity servers

to be deployed effectively for VMs in combination

with NAS storage.

• Don‟t replicate to local storage those files that

probably won‟t be accessed anyway.

• Allow very thin provisioning of local storage.

– SSDs are designed to provide IOPs cost effectively,

not storage.

– If IOPS are not needed, avoid cost of redundant

storage. 13



• The VM NAS Proxy is a specialized NAS Proxy that is optimized to support VMs and the Hypervisor Platform supporting them:

• A VM Private Directory is maintained for each VM:

– A per VM directory in the NAS global name space that holds files that are “local” for this VM.

– A “platform” sub-directory hold virtual disk images and other files used by the Hypervisor and/or Dom0.

– A “mount” sub-directory holds NAS mount points that are available only to this VM.

• These are implicitly named. The client does not know the path to its VM Private Directory because it does not know which VM it is.

• A “proxy” sub-directory holds files used by the VM NAS Proxy for this VM, such as the NAS Response Cache.



VM NAS Proxy as a VM. Virtualization Host

Virtualization Management

NAS VMS

Hypervisor Platform VM NAS ProxyVM

NAS

/vmexports/vm13478

/platform/mounts

/foo` /bar/vm13483

/platform/mounts

/foo/bar

VM13478

/foo/bar

VM NAS Proxy for a Rack

15

Rack (or other enclosure)

Virtualization Host

Virtualization Management

NAS VMS

Hypervisor Platform

VM NAS ProxyVM

NAS

/vmexports/vm13478

/platform/mounts

/foo` /bar/vm13483

/platform/mounts

/foo/bar

VM13478

/foo/bar



• Virtualization Management

– Unchanged, but new information is available to it.

• NAS VMS (Virtualization Management System)

– Extends Virtualization Management to assign VM NAS Proxies.

• Hypervisor Platform

– The Hypervisor and supporting domains.

• NAS system with NFSv4/v4.1 and/or DFS/CIFS.

– Unchanged, but optional enhancements are defined.

• VMs

– Unchanged, but optional enhancements for NAS Client are

defined.

• VM NAS Proxy

– A specialized NAS Proxy optimized for support of VMs



• A VM NAS Proxy can control what is stored locally through several techniques:

– Which sub-directories of the VM‟s private directory are referred to it.

– Whether it provides local storage for specific files within those directories.

– Which blocks of a file that is stored locally are actually stored locally, vs. being accessed over the network.

– pNFS-aware clients can be enabled to access network block servers directly.



• VM will be migrated from Current Host

to a new Host.

• VM has public IP address, which

Virtualization Management will migrate.

• VM has private IP Address which it will

use in new network without realizing that

it is on a new network.

• New VM NAS Proxy will have same

private IP address on the new network.

• New VM NAS Proxy does not respond

to the VM‟s private IP Address until it is

ready.

19

NAS

Current Home

Virtualization Host

10.1.1.1192.168.1.1

Hypervisor Platform10.1.3.115192.168.1.115

VM NAS Proxy

VM

10.1.1.2192.168.1.2

Planned Home

Virtualization Host

10.1.2.1192.168.1.1

Hypervisor Platform

10.1.3.115192.168.1.115

VM Storage Proxy

10.1.2.2192.168.1.2



• By assigning VM NAS Proxies to an Internal Network migration of a VM can be made transparent to the VM. – All Internal Networks can look alike.

– Just as almost all home networks are 192.168.1.*

• NAS client does not need to know about the migration. – NAS clients will see a dropped connection to a NAS server.

– NAS clients will assume the NAS server is simply rebooting.

– Recovery will be complete and transparent to client layers.

– No change to the IP address of the VM NAS Proxy the Client uses.

• As far as the client is concerned it just reconnected to the same server – It even has the same response cache.

– The response cache is stored within the VM Private Directory.



• VM will be migrated from Current Host to a new Host. – 10.1.3.115 is migrated.

• VM is still instructed to use the old VM NAS Proxy (10.1.1.2). – It will continue to use it until explicitly

redirected to the new VM NAS Proxy (10.1.2.2).

– This is very network inefficient.

– Current RTT measurements and congestion window will be wildly incorrect.

– VM NAS Proxy must accept clients from anywhere.

21

NAS

Current Home

Virtualization Host

10.1.1.1

Hypervisor Platform

10.1.3.115

VM NAS Proxy

VM

10.1.1.2

Planned Home

Virtualization Host

10.1.2.1

Hypervisor Platform10.1.3.115

VMNAS Proxy

10.1.2.2

Cont

inue

d ac

cess



• Distributed NAS features for Referral and Delegation have nothing to do with each other. – Referral/redirection is a Management Plane

feature.

• It is used to refer/redirect clients to other NAS servers.

– Delegation/oplocking is a Control Plane feature.

• It is used by NAS clients.

• A VM NAS Proxy integrates them in collaboration with NAS servers.



• VM NAS Proxies use Delegation/Oplocking to

exclusively claim specific sub-directories

within a VM Private Directory.

– This is an enhancement to a NAS server.

• Supporting NAS servers will then

automatically direct referrals for the claimed

sub-directories to that VM NAS Proxy for the

duration of the delegation/oplock.

– Reverting is automatic, preferable to having to

reverse a management plane edit.

24



• Only the correct clients are referred,

• Other clients are refused access.

• Allowed access types:

– Hypervisor Platform access a „platform‟ sub-

directory for virtual disk images and snapshots.

– VM itself accesses NAS mounts in a „mount‟ sub-

directory.

– VM NAS Proxy keeps its own data in a „proxy‟

sub-directory.

• This is where the NAS Response Cache is kept.

25



VM NAS Proxy

NASServer

NASServer

13

2

Locally Cached

Files

• 1 – VM NAS Proxy takes exclusive delegation on a directory.

• 2 – Client referencing delegated directory is referred to VM NAS Proxy

• 3 – Client accesses directory on VM NAS Proxy

26



• Progression of Solutions for Controlling File Locations:

– Without NFSv4 Referrals/DFS Redirections: • Load balancing requires editing each client‟s configuration.

• Or directing all traffic through a proxy.

– Using just standard Referrals/Redirections: • Central control of which directories are referred

• But only via Management Plane edits of exports.

– Integrating Delegation to allow Automatic Referrals: • Dynamic control over which directories are referred is

exercised by the VM NAS Proxy and/or VM NAS Management.

• Dynamic referral automatic ends when delegation ends.


Caitlin.Bestler/[email protected] 27

• One of the features of the VM NAS Proxy is that it can dynamically control which of a VM‟s private directories are referred to it. – But standard granularity for referral is an entire directory.

• Partial Referrals is a new technique to publish alternate referrals for subsets of a directory: – Each partial referral identified by a regular expression.

– The alternate server is then specified for that regular expression, which can be “revert to default” to ignore the main referral.

– This information is published as named attributes for the directory.

• If a NAS client does not understand them then it will simply go to the default server for the directory

• The default server then acts as a proxy for the actual server.



VM NAS Proxy

NASServer

NASServer

1

2A

2B

Locally Cached

Files

• 1 – VM NAS Proxy advertises portions of directory that Client should access directly on the home NAS server.

• 2a – Client continues to access non-excluded portions of directory on the VM NAS Proxy, which caches them locally.

• 2b – But now accesses files covered by partial referral on the home NAS Server

30



VM NAS Proxy

NASServer

NASServer

1

2 3

Locally Cached

Files

• 1 – VM NAS Proxy advertises portions of directory that Client should access directly on the home NAS server.

• 2 – Client does not understand or ignores the partial referral, so all access for the directory goes through the VM NAS Proxy

• 3 – When files are not stored locally access is proxied via the NAS Server.

31



• When the VM‟s NAS Client indicates that it understands pNFS, the VM NAS Proxy can supply pNFS layouts that mix references to its own storage and NAS storage.

• Referencing pNFS blocks directly is safe, because the VM NAS Proxy is the only client allowed to update the file while it is claimed.

• pNFS-aware clients can access unmodified and uncached blocks without overhead of going through the proxy.

Locally Cached Blocks

Logical Block #

Physical Block #

Logical Block #

Physical Block #

pNFS Layout

Logical Block #

Physical Block #

Logical Block #

Logical Block #

Volume ID

Physical Block #

Volume ID

Physical Block #

Volume ID

Physical Block #

Volume IDLogical Block #

Proxy pNFS Layout

Logical Block #

Physical Block #

Logical Block #

Logical Block #

Local Volume ID

Physical Block #

Volume ID

Physical Block #

Volume ID

Physical Block #

Local Volume ID

Logical Block #

Physical Block #


Physical Block #


Physical Block #



Caitlin.Bestler/[email protected] 33

• Named attributes can identify files that are

to be stored only in the local cache.

– The file is never updated on the NAS server.

• Regular expression can easily identify

common temporary files.

• These files would be stored only on local

SDD

– never consuming any network bandwidth.



• VM NAS Proxy is only

on the path for a VM

accessing storage

when it wants to be.

– Control is fully dynamic.

• Access to Non-

optimized

directories/files can

bypass the Proxy.

• VM NAS Proxy is only

identified as referral

target for the correct

clients. 36

Access to referred directories, files

Default Access, may be referred to VM Storage ProxyAccess to non-referred directoriesAccess to partially un-referred files

Direct access to un-cached blocks

VM

VM NAS PRoxy

pNFS Block Server

NASServer



Tailoring NAS Proxies for Virtual Machines

Technology

Transcript of Tailoring NAS Proxies for Virtual Machines