SAN101 Brocade

7/27/2019 SAN101 Brocade

1/46

SAN Workshop


2/46

SAN Design PrinciplesAugust 2007

Legal Disclaimer

All or some of the products detailed in this presentation may still be underdevelopment and certain specifications, including but not limited to, release dates,prices, and product features, may change. The products may not function asintended and a production version of the products may never be released. Even if aproduction version is released, it may be materially different from the pre-releaseversion discussed in this presentation.

NOTHING IN THIS PRESENTATION SHALL BE DEEMED TO CREATE A WARRANTY OFANY KIND, EITHER EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, INCLUDINGBUT NOT LIMITED TO, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESSFOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT OF THIRD PARTY RIGHTSWITH RESPECT TO ANY PRODUCTS AND SERVICES REFERENCED HEREIN.

Brocade, the Brocade B-weave logo, Brocade, Fabric OS, File Lifecycle Manager,MyView, Secure Fabric OS, SilkWorm, and StorageX are registered trademarks andthe Brocade B-wing symbol and Tapestry are trademarks of Brocade CommunicationsSystems, Inc. or its subsidiaries, in the United States and/or in other countries.FICON is a registered trademark of IBM Corporation in the U.S. and other countries.All other brands, products, or service names are or may be trademarks or servicemarks of, and are used to identify, products or services of their respective owners.


3/46


Agenda

SAN Design Principles Other SAN Design Considerations


4/46

SAN Design Principles


5/46


Terminology

SAN: The network for block-level storage connectivity

Fibre Channel SAN: SAN based on SCSI over Fibre Channel (FCP) foropen systems; includes switching elements (switches and directors),optics, cabling, and routers. FC represents more than 99% of the SANinstalled base.

Functional SAN: Grouping of Fibre Channel networks based on

function and/or physical location. Typical functional SANs:

Disk SAN. Connectivity between individual hosts and disk LUNs. In

most shops this is the biggest and most challenging SAN. Generally

one or more paired fabrics (A/B).

Disk Replication SAN. Distance connectivity between storage

subsystems for the purposes of data replication.

Tape SAN. Connectivity between tape users (hosts) and tape (real

and/or virtual).


6/46


Terminology(continued)

Physical Fabric: Network of interconnected Fibre Channelswitches/directors. Services are distributed throughout all switches.

Virtual Fabric: Logical grouping of ports in the physical fabric withdedicated subset of fabric services (naming, zoning, etc.)

Hardware-based (e.g. Brocade M-series LPARs)

Software-based (e.g. Brocade Virtual Fabrics, Cisco VSANs) Routed SAN: Two or more fabrics connected via Fibre Channel

inter-fabric routers. As in the IP/Ethernet world, it is not possible toscale a connectivity model indefinitely without moving to ahierarchical design. (The Internet is the most famous example.)

SAN routers are more analogous to firewalls than to simple IProuters.


7/46


Terminology(continued)

SAN Size

Number ofusableports for all host and storage connectivity in all

fabrics In the Disk SANthere are always at least two independent sides for

redundancy (side A and side B). There may be multiple fabrics ineach side.

In these discussions, when referring to the SAN size, we will generally

be referring to one side Ascalable SAN designis able to meet the changing connection and

bandwidth requirements of the business in a non-disruptive manner

Fabric Size:

Number ofusableports for all host and storage connectivity in onephysical fabric

In the Disk SANthere are always at least two independent fabrics forredundancy (fabric A and fabric B)

Ascalable fabric designis able to grow to the desired number ofconnections in a non-disruptive manner


8/46


Engineering Requirements

In a bridge design the stakeholders have different requirements

Architectwants it to look slick and people to say Wow, look at that bridge!

Engineer wants it to be safe, never break and last 200 years

Users want it to have enough lanes to prevent congestion and always beopen

Ownerwants it to fit within budget, be easy and cost-effective to maintainand not have to worry about replacement for a long time

The SAN design is an engineering project just like a bridge


9/46


What Does the Fibre Channel SAN do?

Its the REALLY IMPORTANT network that provides the paths for I/Otransactions (host-to-disk, host-to-tape, disk-to-disk, etc.)

No SAN no storage No storage no applications

No applications no business function

No business function no customers

No customers . no revenue

No revenue well thats bad for everybody, including Brocade!

The SAN should be invisible to the rest of the IT infrastructure

The best SAN is a silent SAN. Nobody should even know its there!

So, the most important thing about the SAN is that it has to work ALLTHE TIME

Its all about AVAILABILITY!


10/46


SAN Business Requirements

Availability/reliability - Always on 99.999% availability (5 minutes per year of down time)

It must be as reliable as the old direct-attached technologies!


11/46


Storage Transaction Reliability

What a pain when it happens during your web-based IP transaction Storage transactions cannot tolerate this Storage transactions need to be reliable with consistent response times


12/46


Key Business Requirement Cost!

1. Connectivity costs - Port and bandwidth. Hardware, software. One time

and recurring. Well understood. Mostly up front.2. Management costs - The people to run the SAN on a day-to-day basis.

These costs are influenced by:

Rate of change

Complexity/simplicity of architecture

Quality of equipment

Number of points of potential congestion to be managed

Use of and quality of software tools

Outages (think of all the people that get involved when there is an unplannedoutage!)

3. Business function impact costs

Outages Lost opportunities

In the end it has to be about cost-effectively meetingthe requirements of the business


13/46


14/46


SAN/Storage Management Costs Stuff

Not easy or practical to define industry standard costs although many seek these mythical numbers andsome experts claim to know them!

Its all about the specific stuff in your environment:

Quantity: How much stuff do you have? Quality: Do you have good stuff of bad stuff?

Diversity: Do you have one kind or many kinds of stuff?

Change: Do you have to make changes to your stuff all the time

or does it stay the same? Complexity: Is your stuff complex or simple?


15/46


Basic Design Inputs

1. SAN size - This is the biggie. How many points of connectivityto design for? How many years out? How quickly will it grow?

2. I/O workloads - The workloads drive the need for the data pathsin the first place (zero I/Os is easy to design for!):

Bandwidth Data transferred per unit time (MB/sec). Congestion inthe SAN data paths is caused by bandwidth demands. Mostapplications dont drive sustained high bandwidth (but backup does!)

Transaction rate I/Os per unit time (IOPs). Congestion at storageports may be caused by high IOPs. Applications like databases candrive high IOPs. IOPs themselves do not cause congestion in the SANdata paths.

Rarely are these workloads well known. At a minimum one needs toseparate the high workload (bandwidth and IOPs) users fromeverybody else.


16/46


Additional Design Inputs

3. Asset reuse

Will existing switches/directors be reused?

Driven by things like: depreciation schedule, maintenancecosts, applicability in new design, compatibility

4. Migration plan

What inventory (hosts and storage) will be migrated? What application outages are acceptable?

Are changes occurring with the storage subsystems also?

Are there power/cooling/space constraints?


17/46


Key SAN Design Decisions

1. Type of storage

a) Total amount of storageb) Number of target ports

c) Fan-in ratio

2. Fabric size and number of fabrics

3. Selection of switching elementsa) Number of ports

b) Bandwidth

c) RAS

4. Architectural fabric modela) Flat

b) Core-edge

c) Mesh

d) Network-centric


18/46


Type of Storage

The S in SAN stands for Storage!

Large Tier 1 storage subsystems that allow many front-end target

ports (32, 64, and larger) provide connectivity flexibility

Smaller Tier 2/3 storage subsystems have limited front-end ports (4

to 8), which constrain connectivity flexibility

Host-to-storage fan-in ratio determines the required number of

storage target ports: Should be driven by workloads (bandwidth and IOPs)

Typical industry rule of thumb historically 7:1 at 1Gbit

This increased to 12:1 at 2Gbit as storage devices became more

advanced In 4Gbit environments, 18:1 is not uncommon

However, real world production deployments range from 1:1 to 64:1

Consult storage array vendor for recommendations


19/46


Size/Number of Fabrics

Manageability

The size and number of fabrics that have to be managed will impact

management costs Size of an individual fabric and number of fabrics must be balanced.

One fabric is easier to manage than many but a single large andcomplex fabric (versus several smaller, simpler fabrics) increases risk ofhuman error.

Manageability applies to both physicaland virtualfabrics. Virtual fabricsmust be managed similarly to physical fabrics.

Today, the largest production fabrics have more than 4,000 connectionsbut the typical fabric size in a large SAN is 1,000 to 2,000 ports.

Large SANs can be built using a managed unitof SAN strategy

AManaged unit represents a well understood unit of SAN capacity

(ports and bandwidth) AND behavior. When we build this we knowhow it will behave!

Resource stranding is not an issue when the manage unit is BIG.These are SAN CONTINENTS, not SAN islands!


20/46


Size/Number of Fabrics (continued)

Risk

A balanced approach to fabric size and risk must be taken

If a large fabric fails the impact to the business is morewidespread

The overall storage provisioning and data path managementprocess may become more complex as the fabric size increases

Using well-understood managed units reduces overall risks. Logical partitioning of fabrics (zoning, virtual fabrics,

administrative domains, etc.) minimizes some but not all risks ina physical fabric

Fabric design must be resilient to withstand the failure of

individual components without compromising the overall fabricavailability


21/46


Size/Number of Fabrics (continued)

Physical isolation (separate physicalfabrics)

Separate physical locations (different data centers) Business reasons (some big project wants their own stuff)

Very different change control needs (business critical 7x24 systemsversus all the others)

Different RAS specifications (blade center embedded switches versus

directors) Incompatible I/O workloads (tape versus disk)

Virtual isolation (separate virtualfabrics)

Minimize RSCN broadcasts

Increase security (restrict management or connectivity into a certain

fabric) Provide granularity of administration

Appropriate for some functionality but not a substitute for separate

physical fabrics


22/46


Switch Selection

Number of ports

Larger port-count allows larger fabrics to be built Larger port-count makes host and storage locality (HASL) easier to

maintain in flat/ collapsed-coredesigns

Bandwidth

Individual ports have bandwidth limitations (2Gb, 4Gb, ) Port cards and individual port card processors have limitations that

may limit the aggregate port bandwidth. This is over-subscriptionat

the component(ASIC, port card, supervisor card, etc.) level

Entire switch has finite bandwidth that may also limit the aggregate

port bandwidth. This is over-subscriptionat the switching element

level


23/46


Switch Selection (continued)

RAS (Reliability, Availability, Serviceability)

Directors are 5-Nines, switches may be less

Choice is driven by tolerance to outages at the physical fabriclevel and individual port level

Storage 5-Nines

Some hosts may not need 5-Nines Not advisable mixing different RAS levels in a fabric since

fabric RAS is impacted when element failures occur

Fabric instability affects all connections

Isolation technologies (e.g. Brocade Access Gateway) mitigate

these risks


24/46


Architectural Model

When more than one switch is required the manner in

which they are inter-connected is defined by thearchitectural model

Inter-Switch Links (ISLs) are used to connect the

switches using in one of the following models:

Flat/collapsed core model

Meshmodel

Core-edgemodel

Network-centricmodel


25/46


Architectural Model (continued)

The architectural model deployed should support a storage-

centric approach: Any host should be able to access storage capacity in any storage

subsystem (any port-to-any port is NOT required)

Storage ports should be stable and not have to be moved

Data paths should be easily and consistently defined and easy to

troubleshoot. Connecting a host and provisioning storage should not bea science project!

A properly designed SAN should not require a significant amount of

inter-fabric routing within the data center

Routing should be used to address specific functions (e.g. DR, e-

vaulting, etc.) Layer 2 switching functionality should be the cornerstone of all I/O

transactions


26/46



FLAT/COLLAPSED core model

Easy to manage No ISLs (may use formanagement)

Single director may be able to meet all connectivityneeds (collapsed core)

Expansion is done by adding switches

Expansion may require storage port moves

Switches can be any size

About 4 is maximum reasonable size

1,000 ports when using a 256-port switch

All host-to-storage connections contained within theswitch (HASL)

High performance and easy to understand but

Least flexible and scalable

Best fit for static environments

Host and storage place may be optimized within theswitch (e.g. local switching)

A Side


27/46



MESH model

Usually becomes a MESS!

Not recommend except in very smallenvironments

Wouldnt mesh more than 3-4 switches

Hard to expand a fabric usually justadd another

Try to keep host and storage local(HASL) to a switch but difficult due tonumber and size of switches

Difficult to manage traffic resulting and

congestion ISLs consume lots of ports. ISL port

consumption grows by n*(n-1)

A Side


28/46



CORE-EDGE model Storage connections are stableand centralized at the core

Flexible host access to storage LUNs

A familiar tiered design Edge hosts (one hop to storage)

Core hosts (zero hops to storage)

Second core may be added forgrowth (best to do it in thebeginning!)

Can be built using large and/or small

switches Size of core dictates scalability of

fabric

Well-defined managed unit

ISLs

ISLs

ISLs

CORE

switch

EDGE

switches

EDGE

hosts

CORE

hosts

A Side

Disk Targets


29/46



Network Centric Model

One large physical fabric Physical fabric must be partitioned using

virtual fabrics for manageability, stability,performance and risk mitigation

Routing fabric only contains ISLs fromother edge fabrics

Edge fabrics can either be flat or core-edge All ports have access to every other port via

the core fabric (routing between virtualfabrics is required to get any-to-any)

Servers use zero to four hops to storage

Can be built using large and/or small

switches Storage connections are stable but

scattered over multiple fabrics

Very complex

Fabric 1

Fabric 2

Fabric 3

Fabric 4

Routing

Fabric

Storage TargetsStorage Targets

Storage Targets

Storage Targets


30/46


Design Principles

1. Minimize the number of fabrics that have to be

managed (virtual and physical) Fewer things are easier to manage

Sometimes you have to build more (physical constraints, specialsecurity, etc.)

Allows storage targets to be more easily shared

Doesnt introduce resource isolation

Make them big enough and inter-fabric routing is not needed

Large disk storage subsystems with many target portsenable alldisk LUNs to be accessible on all fabrics

Must balance; manageability and risk

KEEP IT SIMPLE to maximize AVAILABILITY

BIG Managed Units (SAN Continents)


31/46


Design Principles(continued)

2. Minimize the number of switches per fabric (8-

12) Fewer things are easier to manage

Easier to visualize on an computer screen or piece of paper

Less inter-switch coordination required during fabric events

Less fabric disruption Blade centers with embedded switches are an exception; use

isolation technologies (e.g. Access Gateway) to overcome

Use large switching elements for large SANs


32/46



3. Limit the fabric size to about 2,000 target and initiatorconnections

Minimizes per-fabric risk

These are the biggest fabrics running today in environments with themost demanding AVAILABILITY requirements

Approaches practical limits for zones, zone sets and port aliases

Approaching limits of management software tools

Actual limits are much higher: production fabrics exist with over4,000 usable ports. However, these are exceptional cases and notthe rule.

If more ports are needed, simply deploy additional big managedunits and using routing if needed. Localizing most flows within a

2,000 port group is generally easy to do.


33/46



4. Utilize switches with a high level of reliability,

availability and serviceability (RAS) Even with redundant data paths you dont want paths to be lost

its a big pain in the neck!

Shared storage target ports should always connect to switchingelements with high RAS

The CORE should be high RAS

If the fabric has demanding RAS requirements, all switchesshould have similar RAS specifications

FC switches with high RAS are a cornerstone of a highly

AVAILABLE storage environment Elements with high RAS characteristics are fundamental to HA

environments


34/46


Basic Design Principles(continued)

5. Ensure that over-subscription does not cause

congestion and degrade performance Where over-subscription is used, ensure that workloads(bandwidth) are well understood

Design conservatively

Classify hosts in at least two categories to isolate high

bandwidth users

Use trunking and DPS to maximize ISL utilization and minimize

management


35/46



6. Use a core-edge model for large environments

The biggest fabrics can be built using a storage-centric core-edgemodel:

Should be used in those data centers with the largest SAN sizerequirements

Provides a stable location for all storage ports

Model is full scalable using managed units and inter-fabricsrouting where needed

Two-tier host attachment provides:

Eliminates high bandwidth hosts from consuming ISL bandwidth

Reduces operational management overhead

Creates a more stable environment which increasesAVAILABILITY

A storage-centric flat model can be used in smaller sites


36/46



7. Design for storage transactions

Use storage subsystems with a sufficient number of target portsto meet the connectivity and throughput needs of the hosts

In large environments plan for and build large fabrics to simplifydata paths and eliminate the need for full time routing

For critical applications, consider localizing traffic within the

switching element by co-locating host and storage, and perhapseven within a blade or port group

LAN-specific requirements like any-to-any are not practical.What is needed is flexible host-to-storage connectivity in orderto efficiently utilize storage resources

Utilizing routing for transient requirements within the datacenter and for distance applications


37/46



8. Keep it Simple

It shouldnt be a science project to add hosts and storage orfigure out the path between an initiator and target

If the design is simple it will:

Be simpler to manage

Reduce the opportunity for mistakes

Maximize AVAILABILITY

Build out in Managed Units and use big managed units for bigSANs


38/46

Other SAN Design Considerations


39/46


Other SAN Design Considerations

Server trends, consolidation and virtualization

Distance solutions (replication, e-vaulting)

SAN-based storage virtualization

Tape SAN

Backup-to-disk (B2D)

S C lid i


40/46


Server Consolidation

Server consolidation in the data center is impacting the SAN designs

Partitioned hardware and server virtualization software

VMware

AIX MPARs and VIO servers

Bladed servers with embedded FC switches

May result in higher I/O workloads concentrated on fewer SANconnections

Host connections at the edge in a core-edge model will drive higherI/O workloads on average. This must be factored into the design,especially the bandwidth requirements between core and edgeswitches.

Isolation technologies (e.g. Brocade Access Gateway) are needed to

address embedded FC switch connectivity. These SAN switches areowned by the server group! Power consumption, cooling and cabling more important due to

density

Di t S l ti


41/46


Distance Solutions

Disk replication and e-vaulting must be accommodated

Storage subsystem-based replication still dominates themarket

Leveraging TCP/IP using Fibre Channel over IP (FCIP) is

the dominate communication method for open systems Director-based blades as well as standalone appliances

exist. Choose your poison!

Fast-write (and fast-read) functionality is required to

overcome WAN latency A separate fabric (or fabrics) may be justified depending

on: volume of data, technology used, compatibility, etc.

St Vi t li ti


42/46


Storage Virtualization

How many times have you heard This is the year of virtualized

storage?

Well, the jury is still out on this one

Everyone is jockeying for position

The standards dont exist

Whats the best way to do it (in-band, out-of-band, in the network,in the storage subsystem, etc.)?

The idea is to not be locked in to one storage vendor

But then you may be locked in to one software vendor

Who will win the software war? Do you want to place all your I/O

transactions into the hands of a little startup ISV? Who will win the platform war? Where will this software run? A

blade? An appliance? What vendor(s)?


43/46


The Tape SAN

Tape has a very different I/O profile (large block,

bandwidth intensive) than disk access (small block,transaction intensive)

Theres no real multi-pathing for fault tolerance (someexceptions). So, if you lose a component in the data

path you basically lose access to the device. The tape targets (drives) and initiators (host users)

consume lots of bandwidth for extended periods of time

Tape I/O operations DO NOT behave well during fabricevents (usually abort the mission)

Tape connectivity does not experience the same level ofconfiguration changes (zoning, LUN masking,adds/changes/deletions) as disk connectivity

Th T SAN


44/46


Tape SAN Design Rules

Separate tape (and virtual tape) from the disk SAN

This separates high bandwidth I/O workloads (tape and mediaservers) from IOPs sensitive workloads (disk)

Flat design model is best:

Minimize/eliminate over-subscription over ISLs

Most flexible drive sharing within a fabric Tape users should have a dedicated HBA into Tape SAN

Tape and disk I/O bandwidth peaks during backup job

Disk I/O is small block

Tape I/O is large block

Routing into Tape SANmay be done on a exception basis(locally) or for long distance requirements (e.g. e-vaulting)

In collapsed core designs, localize data paths

The Tape SAN(continued)

B k t Di k


45/46


Backup-to-Disk

Backup-to-disk trends are also impacting the SAN designs

B2D storage may be in the form of virtual tape or nativefile systems

Even though its disk the I/O profile is like tape (largeblock, bandwidth intensive) versus disk (small block,

transaction intensive) High bandwidth usage for long periods of time

The potential for congestion due to over-subscription isincreased

Tape and virtual tape do not support real HA and multi-pathing therefore a dual fabric design is notnecessarily needed

Should be part of the Tape SAN treat it like real tape


46/46

Thanks!

SAN101 Brocade

Documents

Transcript of SAN101 Brocade