How Microsoft IT Deployed Exchange 2010Ross Smith IVSenior Program Manager, Exchange ServerMicrosoft Corporation

SESSION CODE: UNC202

Kyryl PerederiySenior Systems Engineer, Business Online ServicesMicrosoft Corporation

Session ObjectivesProvide an overview of Microsoft IT Architecture and Design decisions for its Exchange 2010 deploymentTechnical drill down into messaging solutions deployed at Microsoft ITBest practices and lessons learned around Exchange 2010 and Exchange 2010 designs for the enterprise

Purpose of Microsoft IT

Be Microsoft’s First and Best Customer

Protect Microsoft Digital Assets

Run World-Class IT

Drive productivity for our customers, clients and partners

Dogfooding ConceptExchange 2000 – started 3 weeks before shipExchange 2003 – started 6 months before shipExchange 2007 – started 22 months before ship, migrations completed at RTM in Dec. 7, 2006Exchange 2010

In production environment since Feb. 2007Entire company transitioned before RTM (October 2009)40 handoffs, 30,000 crashes, and 4,000 unique bugs

Closeness to the Exchange PG is second to noneMicrosoft IT sign-off required before shipAlso dogfooded Windows 7 and Office 2010 at the same time!

Microsoft IT Exchange Environment OverviewMultiple Exchange organizations/forests (Corp, Dogfood, Windeploy, WinSE, Extranet)Mailbox Limits

Exchange 2003 timeframe: 200 MB Exchange 2007 timeframe: 500 MBExchange 2007 SP1 timeframe: 1 GBExchange 2010 timeframe: 5 GB

Mailboxes in Corp. forest – 180,000+Tracking several distinct user profiles since Exchange 2007 deployment

~100-150 messages/day – e.g. Full Time Employees~50-75 messages/day – e.g. Contingent Staff

Evolution of Exchange Deployment at Microsoft

SAN

CCR

Parallel SCSI

Reduced Cost• Greater server density per GB,

reduced power consumption, lower hosting costs

• Consolidation from 74 Exchange sites in 2000, to 4 Exchange sites today!

• Reduced storage cost due to cheaper technology (SANDASSATA)

• Backup/Data Protection optimizations

Year 1998 2000 2003 2007 2010

100

400

800

10,000Exchange 5.5• Standalone Server• 150 GB of storage• 50 MB mbx quotas

Exchange 2003• 7-Node Cluster • (4 x Active / 3 x Passive)• 1 TB of SAN storage• 200 MB mbx quotas

Exchange 2007Exchange Online• 2-Node CCR Cluster (Active/Passive) • 2x10TB of DAS storage• 500MB & 1GB mbx

quotas

Exchange 2000• 2-Node Cluster• (Active/Passive) • 500 GB of storage• 100 MB mbx quotas

Exchange 2010Exchange Online• DAG architecture• 16x35TB of JBOD storage• Scale out• 5GB mbx quotas

100%

65%

41%18%

$ / Seat (1998 =$‘s 100%)

Increased Availability• Protection from server-level

failures (Exchange 2000/2003)• Protection from Server & Storage

level failures (CCR – Exchange 2007 and DAG – Exchange 2010)

• Possibility of achieving geo-redundancy via native Exchange features (Exchange 2007/2010)

Increased Capability• Significant mailbox limits increase• Massive adoption of mobile

messaging scenarios (phone, outlook web access)

• Transitioning Unified Messaging from 3rd party platform to Exchange

• Instant Messaging/Presence integration

12%

ScaleServer Usable Capacity (GB)

35,000

Microsoft IT Exchange Deployment OverviewHighly Consolidated Exchange deployment – servicing Microsoft users from 4 Regional DatacentersExchange Servers

Exchange 2003 timeframe: ~30 active mailbox servers (SAN clusters)Exchange 2007 timeframe: ~34 active mailbox servers (CCR config)Exchange 2010 RTM: 77 mailbox servers (DAG config)

40,000

35,000100,000

2,000

Microsoft IT Exchange 2010 Deployment GoalsMaintain 99.99% availability goalIncrease the user mailbox size limits from 1 GB to 5 GBReduce hardware, storage costs and backup costsDiminish/eliminate impacts to user during mailbox migrations/moves while increasing migration velocityMinimize disruptions due to issues with single database (failover at the DB level instead of entire server node)Invest in flexible and scalable mail transport and client access services for all protocols

Microsoft IT Exchange 2010 Hardware Configurations

Example of server CPU core ratios (Redmond):Server Count: MBX=38; CAS=27; HUB=7Server Ratios: MBX:CAS=~4:3; MBX:HUB=~5:1

http://technet.microsoft.com/en-us/library/dd346701.aspx

Role Configuration

Hub Transport 2x4 cores, 16GB RAM

Client Access 2x4 cores, 16GB RAM

Unified Messaging 2x4 cores, 16GB RAM

Multi-Role (Hub/CAS/UM) 2x4 cores, 16GB RAM

Mailbox (Large DAGs: Redmond, Dublin, Singapore) 2x4 cores, 32GB RAM

Mailbox (4-node DAG: Sao Paulo – small scale) 1x4 cores, 16GB RAM

Microsoft IT Exchange 2010 Load Balancing & Fault Tolerance

Enterprise scenario – must build load balancing and fault tolerance provisions in the design from day one

Resilience to server and service level failuresTolerance to increased load due to planned server downtime and environmental conditions (spam attacks, “snow day”)

Component Load Balancing Fault ToleranceMailbox Server N/A DAG

Hub Transport Server MBX HUB: Built in

Internet Transport Exchange Hosted Services (EHS)Hardware Load Balancing

Client Access Server Internal: Hardware Load BalancingExternal: Network Load Balancing + UAG 2010 WPLB

Unified Messaging UM IP Gateway: Multiple IP Gateways per dial planIP Gateway UM: Built in (Round Robin between UM)

Microsoft IT Internal Transport Routing Topology

Singapore

Sao Paulo

Redmond

Dublin

Redmond-Exchange

AD Site Link

AD Site with Exchange Servers

AD Site without Exchange Servers

Microsoft IT Internal Transport Routing Topology

Sao Paulo

Redmond

Dublin

Singapore

Redmond-Exchange

AD Site Link

Exchange Routing

AD Site with Exchange Servers

AD Site without Exchange Servers

Microsoft IT Internal Transport Routing Topology

Dublin

Sao Paulo

Singapore

Redmond-Exchange

AD Site Link

Custom Site LinkExchangeCost=10ADCost=999

set-adsitelink Dublin-to-RedmondExchange -ExchangeCost 10set-adsitelink SaoPaulo-to-RedmondExchange -ExchangeCost 10set-adsitelink Singapore-to-RedmondExchange -ExchangeCost 10

Redmond

Microsoft IT External Transport Routing TopologyWindows

Corporate Forest

WinSE Exchange

Redmond

Hub Transport

Dublin

Hub Transport

Sao Paulo

Hub Transport

Singapore

Hub Transport

Hardware Load Balancers

Microsoft Exchange

Hosted ServicesEHS/FOPE

@microsoft.com@exchange.microsoft.com

@winse.microsoft.com

EHS/FOPE

Microsoft IT Mailbox Architecture Design GoalsMicrosoft IT had several goals when designing the Exchange 2010 mailbox server platform:

Scalability goal: Mailbox limits 5GBAvailability goal: 99.99%

multiple database copiesfaster failovers

Business goal: reduce costsTechnical goal: scalable and modular

Microsoft IT Exchange 2007 Mailbox Server Platform

Exchange 2007: Active – Passive 2 node cluster CCR architecture146GB DAS SFF disks; RAID 5: 10 TB of usable space per serverTarget Scale: 6000 mailboxes @ 1GB mailbox size limit

Passive NodeActive Node

SAS

Log Shipping

SAS SAS SAS

RAID RAID

Microsoft IT Exchange 2010 Mailbox Server Architecture

Scale out modelFor example, the 10-node DAG implementation has ~30,000 mailboxes @ 5GB quota; ~3,000 mailboxes per server node

JBOD disk array: 35x7.2K 3.5” nearline 1TB SAS disks connected directly to each serverEach server is allocated 35 dedicated disks; one LUN per disk with one database copy

Why 7.2K SAS instead of plain SATA?Nearline SAS disk drive shares the same low cost mechanics with 1TB 7.2K SATA counterpartMore robust interface20-30% better random IO throughput vs. SATA with only 5% higher priceMicrosoft IT sees ~2.75% AFR (Annualized Failure Rate) after one year in production

D01

: 1T

B

Server 1D

02: 1

TB

D03

: 1T

B

D35

: 1T

B

D34

: 1T

B

......SAS

D01

: 1T

B

Server 2

D02

: 1T

B

D03

: 1T

B

D35

: 1T

B

D34

: 1T

B

......

SAS

D01

: 1T

B

Server 3

D02

: 1T

B

D03

: 1T

B

D35

: 1T

B

D34

: 1T

B

......

SAS

D01

: 1T

B

Server ...

D02

: 1T

B

D03

: 1T

B

D35

: 1T

B

D34

: 1T

B

......

SAS

D01

: 1T

B

Server 10

D02

: 1T

B

D03

: 1T

B

D35

: 1T

B

D34

: 1T

B

......

SAS

Microsoft IT Mailbox Architecture – DAG Deployment Numbers

Location # DAGs /region

# nodes /DAG

# disks/server

# disks /DAG

# copies for each DB

Total DB Copies /DAG

# DBs /DAG

Actual Mailboxes

Supported # mailboxes@ 5GB

Redmond 2 11 Node DAG

35 385 4 (JBOD) 384 96 ~29000 / DAG

33000 / DAG

Redmond 1 10 Node DAG

35 350 4 (JBOD) 348 87 ~26000 30000

Dublin 1 16 Node DAG

35 560 4 (JBOD) 512 [560] 128 [140]

~33000 48000

Singapore 1 16 Node DAG

35 560 4 (JBOD) 512 [560] 128 [140]

~38000 48000

Sao Paulo 1 4 Node DAG

12 48 3 (JBOD) 48 16 ~2000 2500

N – used capacity; [M] – provisioned capacity

Microsoft IT Mailbox Server Architecture Failure ModelArchitecture is designed for a 3 server targeted failure model

Requires MaxActiveDatabases to be set on each serverConsider 10-node DAG

35 database copies / server~300 mailboxes per database

Requires operational maturity to ensure oversubscription does not occur

Number of Active Databases / Server

Number of Active Mailboxes / Server

Normal Runtime 9 2700

1st Server Failure 10 3000

2nd Server Failure 11 3300

3rd Server Failure 13 3900

Microsoft IT Mailbox Architecture – Thin Provisioning5GB mailbox limits – do I really need that much storage!?Microsoft IT thin provisioning approach:

Provisioning storage based on the expected utilization as opposed to based on maximum possible utilizationAllows balancing projected costs vs. committed costsHistorical data shows how mailboxes grow within MicrosoftAverage vs. maximum mailbox sizes are used in storage calculations

Thin provisioning requires sufficient operational maturity and processes to adjust/expand on the infrastructure as needed

Caution! You can burn lots of hours trying to control and get the solution back to operational stability if expected utilization is exceeded or trend changes drastically

Microsoft IT Mailbox Architecture – Storage ProvisioningDesign Parameters:

Target mailbox size: 2.5GB (quota at 5GB) with 150 message/day profileDeleted items retention time: 30 days1 Database and log stream per disk (JBOD)Daily mail traffic per mailbox: 30MB (average)I/O Profile: 0.33 IOPS / mailbox (E2K7 0.8 IOPS/mailbox) [Redmond]

3GB provisioned storage / mailboxThis includes overhead, content index, transaction logs

I/O RequirementsEach 7.2K SAS can sustain 100 IOPS100 / 0.33 = 303 mailboxes (per DB/disk)

Capacity RequirementsFormatting disk capacity 931GB931/3 = 310 mailboxes

Design is capacity and IO bound and is well balanced

Microsoft IT Mailbox Architecture – Migration VelocityData transfer rates:

Single mailbox move thread: 1.7 Mbytes/sec (5-7GB / hour)Multiple mailboxes: affected by many factors (MRS throttling policy, number of DBs and servers, CPU and disk performance, WAN bandwidth and latency)Item count does affect rate

Number of mailboxes moved (transitioned from Exchange 2007 to Exchange 2010):

In early beta phases, 200-300 per nightOn Beta 2, 1500-2000 per nightAt peak over 4000 per night

Microsoft IT Exchange Backup – E2007 Streaming ApproachLegacy Backup approach (Windows 2003 platform) – streaming from Active –

NTBACKUP basedBackup Window: 4 hoursSchedule: Full – weekly, Incremental – dailyLow cost SATA disks in RAID5 as backup target

Passive NodeActive Node Log Shipping

Backup StorageSATA RAID-5

BACKUP

Database Storage Replica Storage

SG1

Mon Tue Wed Thu Fri Sat Sun

SG2

SG3

SG4

SG5

SG6

SG7

Full

Full

Full

Full

Full

Full

Full

Inc Inc Inc Inc Inc Inc

Inc Inc Inc Inc Inc

Inc Inc Inc Inc Inc

Inc Inc Inc Inc

Inc Inc Inc Inc

Inc Inc IncInc Inc Inc

Inc Inc

Inc Inc

Inc

Inc

Inc Inc Inc Inc Inc Inc

Microsoft IT Exchange Backup – E2007 VSS Approach Based on System Center Data Protection Manager 2007VSS based approach from “Passive”“Express Full” backup technology (deltas)Incremental backup – every 15 minutes. Reduced RPOSame Low cost SATA disks in RAID5 as backup target

SAS SAS

Passive NodeActive Node

Passive NodeActive Node

DPM SRVER1GigE

1GigE

DPM Agent

DPM AgentDPM Agent

DPM Agent

Microsoft IT Exchange Backup – Lagged CopyCan we rethink backup strategy and accomplish the same data protection needs entirely within Exchange platform?How many copies of the data are sufficient and what are the necessary properties of each copy?

Leveraging Exchange 2007 SCR as “continues” backup:Log replay lag and truncation lag time matches backup retention window (14 days) – watch for log LUN utilization on the SCR backup nodeRestore operation involves log replay until target recovery point

Primary Secondary RemoteHistorical

Microsoft IT Exchange Backup – Exchange 2010 Approach

Accidentally Deleted Items

Long Term Data Retention

Exchange 2010Feature Set

Mailbox Resiliency

Single Item Recovery

Large Mailboxes + Retention

Policies

Fast recoveryData redundancy

Guaranteed item retention

Move away from PSTs

Feature Benefits

Fast

Rec

over

yD

ata

Rete

ntion

HW/SW Failures

Dumpster 2.0

Mailbox

Primary

Dumpster 2.0

Mailbox

Secondary

Dumpster 2.0

Mailbox

Tertiary

Dumpster 2.0

Mailbox

Quaternary

Microsoft IT Exchange Client AccessSignificant architectural differences in client access between Exchange 2007 and 2010

CAS Array

Outlook

Apps

Load Balancing

Layer

HTTPs: (OWA, EAS, EWS, OA)

RPC: (MoMT, DoMT)

Domain Controllers

OWA

MailboxServers

Exchange 2007

CAS Array

Outlook

HTTPs: (OWA, EAS, EWS, OA)

RPC: (MoMT, DoMT)

Domain Controllers

OWA

Apps

MailboxServers

Load Balancing

Layer

Exchange 2010

Microsoft IT Exchange Client Access – Internal Load BalancingRPC Client Access array namespace per datacenter, with host record located in internal DNS infrastructure only (e.g. outlook.internal.company.com)Microsoft IT example: Client Access for Redmond

rpc://outlook.redmond.corp.microsoft.com & https://msg.microsoft.com

Areas of significanceConnection Concurrency/RateBandwidthLoad balancing methodAffinity/session persistence

Protocol Affinity/Persistence Setting

RPC/TCP Source IP

RPC/HTTPs (OA) None* (original method Source IP)

OWA/ECP Cookie

EWS Cookie/SSL ID

EAS HTTP Authorization Header

Autodiscover none

OAB Source IP

CAS Array

Outlook

HTTPs: (OWA, EAS, EWS, OA)

RPC: (MoMT, DoMT)

Domain Controllers

OWA

Apps

MailboxServers

Load Balancing

Layer

Exchange 2010

Users in Redmond site: 100KObserved Load Balancer Footprint:RPC:

250K TCP connections1Gbps bandwidth

HTTPs:100K TCP connections300Mbps bandwidth

Microsoft IT Mobile Messaging Topology OverviewOWA, OA, EAS, EWS, OAB and Autodiscover servicesCommon URL namespace for mobile messaging clients within region (e.g. https://msg.microsoft.com) Split DNS infrastructure

Roaming clients (internal vs. external) continue to use the same URLsVirtualized URL architecture

Each region has AutodiscoverInternalURI set to Load Balanced FQDN (e.g. https://emeamsg.microsoft.com)Each region has InternalURL and ExternalURL set to Load Balanced FQDN (e.g. https://emeamsg.microsoft.com)

SAN certificates usedNo CAS FQDN values

Microsoft IT Mobile Messaging DesignSession State Management

Optimized per protocolClient load distribution

Loads for each protocol are treated differentlyUtilizes the least connections method, scoped per protocol

Load balancer awareness of healthHeath checks support the application building protocol specific awareness options

CAS Array

InternetCorpNet

HTTP

RPC

HLB

Microsoft IT Exchange Site Resilience EffortsBased on native Exchange 2010 functionality: continues replication to remote site

Overall architecture based on “N+M” approachIncremental deployment to Dublin datacenter

Additional ConsiderationsSingle IP subnet NOT required = Spans AD sites.Database mobility: failover as granular as one DB

Pilot scope: 1000+ users, 13 DBs @ 400GBObserved RTO: ~10 min, RPO: zero data loss (database mobility scenario)

ExchangeDomain

MBX

HUBCAS

Redmond Datacenter Dublin DataCenter

DC/GCDC/GC

EU-IE-DUBDC AD SiteNA-WA-EXCH AD Site

MBX MBX

CAS/HUB CAS/HUB

MBX MBX

WAN

https://backup.exchange.microsoft.comrpc://backupoutlook.exchange.microsoft.com

https://exchange.microsoft.comrpc://outlook.exchange.microsoft.com

Microsoft IT Exchange Site Resilience Efforts – Lessons LearnedData replication over the WAN can be a bottleneck

Latency greatly influences achievable throughputSufficient bandwidth is critical for replication and seeding operationsPlan for initial seeding and ongoing log replication/content indexing [proportionate to the number of remote copies]

Overall RTO (Recovery Time Objective) is dependent onTime to detect the failureTime to make the failover decisionTime to activate the remote copyTime for the environment to converge (AD, DNS replication)

Determine operational feasibility of RTO for your environment through planning and practice

Microsoft IT Exchange 2010 Deployment Lessons LearnedHardware is evolving faster than the user profile demands on isolated server roles

Modern processors are expected to increase megacycles by 30%Combining server roles is optimal in certain situations (HT/CAS)

Colder I/O footprint and native advanced HA features of Exchange 2010 enable use of SATA in JBOD configuration and increase GB density per ULarger mailboxes push more rigor around thin provisioning and capacity planningUse of cheaper/slower storage lowers the performance tolerance of the messaging ecosystem

User profiling and trending becomes important again for the mailbox role

Changes in client access patterns require advanced investments in the load balancing layerInternet mail management becomes a competency in itself. Move from on-premises investments in SMTP perimeter systems to EHS/FOPEFully embrace Exchange native replication framework as data protection mechanism

ConclusionITShowcase paper is expected later this monthTakeaways from the MSIT deployment of Exchange 2010:

Reduction in IOPS due to database optimizations = better performance and reduced storage costsExchange 2010 features do enable alternative strategies (JBOD, Exchange Native Data Protection) and they do work!Increased savings in storage costs and lowered TCOIncreased mailbox migration velocity which accelerates migration of the entire company more quicklyElimination of Backups saves MSIT $6 million+ per yearNew features in Exchange around high availability allow us to hit four 9’s and sometimes 5+!

ConclusionThere are many different design dimensions that have to be considered when designing for high availability and site resilience with Exchange 2010The choices you will make will determine the number of copies and hardware you deploy

Design choices should be based on customer requirementsExchange 2010 allows you to take advantage of new options which can lower costs

Resources

www.microsoft.com/teched

Sessions On-Demand & Community Microsoft Certification & Training Resources

Resources for IT Professionals Resources for Developers

www.microsoft.com/learning

http://microsoft.com/technet http://microsoft.com/msdn

Learning

Related Content

Breakout SessionsUNC301 – Microsoft Exchange Server 2010: Sizing and Performance - Get It Right the First Time – Thurs, 5pmUNC304 – Microsoft Exchange Server 2010: High Availability Deep Dive – Wed, 9:45amUNC305 – Microsoft Exchange Server 2010 High Availability Design Considerations – Tues, 8amUNC306 – Going Big! Deploying Large Mailboxes with Exchange 2010 without Breaking the Bank – Thurs, 3:15pm

Interactive SessionsUNC01-INT – Real-World Database Availability Group (DAG) Design – Tues, 1:30pmUNC02-INT – Busting Microsoft Exchange Server 2010 Storage Myths! – Tues, 3:15pmUNC05-INT – Deploying the E2010 CAS Role: Load Balancing & Certificates – Thurs, 1:30pm

Hands-on LabsUNC02-HOL – Microsoft Exchange Server 2010 High Availability and Storage Scenarios

Complete an evaluation on CommNet and enter to win!

© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

JUNE 7-10, 2010 | NEW ORLEANS, LA