3/23/2007 11:48 AM

UCM309 1

Exchange Server 2007Storage Changes and Design Considerations

Scott Schnollscott.schnoll@microsoft.comSenior Technical Writer, Exchange ServerMicrosoft Corporation

Patrocinadores

Microsoft Exchange 2007 Design Goals

Exchange 2007 Storage Features

ESE Database Fundamentals

Reducing Disk I/O

ESE Changes

Storage Design

Testing, Validating and Monitoring

3/23/2007 11:48 AM

UCM309 2

Leverage 64-bit Windows

Use large amounts of memory

Support large mailboxes (> 1 GB)

Provide fast search

Reduce storage I/O

Lower storage costs

Reduce complexity

Increase reliability

Large, low-cost mailboxes

Enabled by reduced I/O

I/O profile is changing

Features that allow you to use large mailboxes

Content indexing

Messaging Records Management

Fast recovery with VSS and continuous replication

14-day deleted item dumpster default

Dual phase commit remains

Phase 0: commit the user transaction in fast way

Sequential write of page changes (modification, deletion, insertion)

Phase 1: update the database in an atomic way

Balanced Trees with backward and forward links

Several trees in a database

Random access, fixed page size

Ability to coalesce I/O

Two fundamental use of memory

Cache: to keep in memory frequently used pages

Buffer: to keep track of transactions as they occur

Checkpoint depth

Size (in MB) of log files to keep in memory. Cached pages of a storage group‟s databases that is updated in RAM but not yet to disk

20 MB per storage group

Commit changes to disk after logging

Cache changes in buffer

3/23/2007 11:48 AM

UCM309 3

DB cache size “unlimited”

RAM rule of thumb: 2 GB + 5 MB per user

Increasing cache size reduces DB reads

50 databases in 50 storage groups

Databases mounted in parallel

Small RAM = small cache = more I/O

Large RAM = large cache = less I/O

Following sizing rule of thumb (2 GB + 2-5 MB/user) produces significant decrease of read operations

Increased page size to 8 KB (was 4 KB)

I/O coalescing up to 1 MB (was 64 KB)

Larger but fewer I/Os

EDB file only; no STM file

Take advantage of VLM

Maximum database cache from 900 MB to many GBs

More storage groups = more checkpoint depth

No more VM fragmentation

Log file generations up to billions

Log files 1 MB in size

More storage groups = more checkpoint depth

More checkpoint depth =

Going less often to disk

Accepting more changes in RAM directly before committing to disk

Optimize the number of physical writes by keeping the page dirty longer

Handling a greater volume of database changes

More transactions, more users

Database write is improved

3/23/2007 11:48 AM

UCM309 4

Mailbox Server Performance

0

500

1000

1500

2000

2500

3000

4GB 8GB 16GB 24GB 32GB

Read IO/Sec Write IO/sec

Hardware: 4 x dual core AMD 2.2 GHz

User profile: 4000 Outlook 2003 online users simulated with Exchange Load Generator, 100 MB mailbox size, 17 local deliveries/sec

Read, Overall IOPs/u Reductions

0

100

200

300

400

500

600

Exchange 2003 Exchange 2007

600

320275

240

Read IOPS

Writes/sec

x64 and baseline improvements around cache and I/O profile demonstrate improvements in 4 GB 4 GB testing

ProLiant DL385 2 Dual-Core CPU (2.2GHz), 4GB RAM, 1500MMB3 users, U320 SCSI 24 DB disks, 4 Logs. Search/Indexing=OFF

IOPS ReductionExchange 2003 v Exchange 2007

0

0,2

0,4

0,6

0,8

1

1,2

Exchange 2003 Exchange 2007

1,035

0,274

0,625

0,078

0,41

0,196

IOPS/User

Reads/User

Writes/User

Exchange 2003: 4GB

Exchange 2007: 22 GB

4000 user/250MB Mailbox1 IOPS Profile

1:1 read:write ratio is heavily dependent on clients using Cached Exchange Mode or clients in Online mode w/small mailboxes (100mb or less)

Large mailboxes in Online mode changes ratio to 2:1 (depending upon item counts in key folders)

BlackBerry devices also push up reads significantly (don‟t know how much)

ProLiant DL385 2 Dual-Core CPU (2.2GHz), 4GB RAM, 1500MMB3 users, U320 SCSI 24 DB disks, 4 Logs.Search/Indexing=OFF.

Read Write Ratio

Exchange 2003 69% 31% ~2:1

Exchange 2007 51% 49% ~1:1

0

100

200

300

400

500

600

Read IOPS Write IOPS

504

231175 168

Exchange 2003

Exchange 2007

3/23/2007 11:48 AM

UCM309 5

Reduction in database read I/O with larger database cache

Reduction in database write I/O with more aggregate checkpoint and I/O coalescing

DB write I/O:Log write I/O is 1:.5 to 1:1, but typically closer to 1:.5

User profile affects ratio

More options than FC shared storage

Understand the performance implications of other storage solutions

Cluster continuous replication (CCR)

Removes dependency on shared storage

But doubles storage requirements

Fast recovery is enabled with features such as

LCR/CCR

VSS

Database portability

Predict baseline IOPS using two primary factors: amount of database cache per user, and number of messages each user sends/receives each day

See „Predict Exchange 2007 Baseline IOPS‟ in Rob Quimby‟s blog

Use Profile Analyzer to determine profile

Enter gathered information into Storage Calculator

3/23/2007 11:48 AM

UCM309 6

Get the calculator from http://msexchangeteam.com/attachment/432207.ashx

Requires Excel 2007

Compatibility pack for Office XP and 2003 can be obtained here

The calculator does not make any recommendations toward storage design (RAID parity, number of disks, etc.) as the design depends on the storage being used

Feedback alias: strgcalc

Database

~15% overhead for larger dumpster [MSIT profile]

~ 5% overhead for content indexing

~10% overhead for database whitespace

Log

Recommended log LUN size

Move mailbox

Example:

1000 User, 250MB mailbox = 250 GB

CI 12.5 GB, Dumpster 37.5 GB, Whitespace 25 GB

Total = 325 GB

Backup, restore, and re-seed operations can require significant I/O particularly with larger databases

Maintenance and online defragmentation must run, is scheduled, and causes significant I/O

Messaging records management is a scheduled crawl of the database

Content Indexing

Outlook Cached mode

Sorts & searches performed by client

Outlook Online Mode

Sorts & searched performed by server

Initial index creation expensive

Place no more than 5000 items in a folder

Messaging Records Management (MRM)

Database crawl impacted by database size

MRM can be expensive, so do NOT run it with other tasks (e.g., backup, maintenance)

3/23/2007 11:48 AM

UCM309 7

Keep them small, and take into account

Streaming backup/restore

Offline defrag/repair

Online maintenance

Reseed time (1 database = 25 MB/sec)

Using CCR/LCR as the 1st line of defense mitigates only the first point

Without continuous replication: 100 GB

With continuous replication: 200 GB

SATA

Use enterprise SATA with full-time duty cycle

Use faster RPM with RAID5

RAID5 performance is worse in Exchange 2007 due to the 1:1 read:write ratio

SAS

SAS is the new SCSI with larger disk sizes

Use 2½” small form factor (SFF) for speed

Use 3½” for capacity

Balance RPM with I/O requirements

10k RPM may be enough

iSCSI

Isolate the iSCSI network

Use Gigabit – Jumbo Frames & Flow Control

MPIO (v2 initiator) for higher throughput & reliability

Make iSCSI LUNs persistent

MPIO can sustain high MB/sec for backup & checksum integrity (VSS)

Fiber Channel

Use MPIO

Use WHQL qualified firmware/drivers

Use storage vendor queue depth settings

Balance performance with capacity requirements

RAID10 has best reliability

Can lose more disks in a RAID group before data loss

Light performance decrease during failed disk/RAID rebuild

Log LUN should be RAID10 with a 100% write workload

RAID5 provides best capacity efficiency

Due to poor performance, the extra space is not often utilized

1:1 write ratio causes RAID5 LUNs to perform worse than in Exchange 2003

Heavy performance decrease during failed disk/RAID rebuild

RAID6 provides increased data protection over RAID5

Worse performance than RAID5 with less capacity efficiency

Heavy performance decrease during failed disk/RAID rebuild

3/23/2007 11:48 AM

UCM309 8

Performance varies based on solution

Customers should work with storage vendors to get RAID performance data based on chosen solution and it‟s configuration

Include failure and rebuild scenarios in design

RAID5 and RAID6 are discouraged because of performance reasons

If I/O requirements can be met with RAID1, deploy RAID1

If IO requirements can‟t be met with RAID1, deploy RAID10

CCR enables clustering with non-shared storage

Twice the storage for database redundancy calls for less expensive storage (DAS, SAS)

1 database per storage group

4 LUNs per storage group, 2 Log and 2 database (Active/Passive)

Logs and Database on isolated disk

Active and Passive on isolated storage

LCR - separate storage controller and PCI bus

Design Passive LUNs to match Active LUNs in both performance and capacity

Passive copy is first line of defense

Place passive LUNs on separate storage from the active LUNs

Passive I/O will not impact production

Ability to run VSS backup on passive does not impact active

3/23/2007 11:48 AM

UCM309 9

More transactional I/O than non-replicated

May require cache setting reevaluation

Recommend 25%:75% read:write on LOG LUN

Active Passive

Log Writes X CR

Log Reads CR CR

DB Writes X CR

DB Reads X CR

X - Traditional Exchange I/OCR - Addition I/O when using continuous replication

LCR/CCR – first line of defense

Copy of data; first recovery vehicle

Not a backup

Active copy of database

Legacy streaming to disk

Legacy streaming to tape

VSS snapshot – copy to disk or tape

VSS clone – copy to disk or tape

Passive copy of database

VSS snapshot – copy to disk or tape

Offloads backup I/O to passive LUN

LCR/CCR only

With CCR/LCR, use VSS to backup passive copy

Backup does not affect active LUN

Checksum integrity (eseutil) occurs on passive copy

Daily Full Backup

2000 user – 2 GB mailboxes is 4 TB of data

@ 175 GB/hr = ~ 23 hours

Weekly Full & Daily Incremental

Acceptable because LCR/CCR is the first line of defense

Stagger databases for full backups

Example: 14 DBs, 2 full per night, 12 incremental

2000 user – 2 GB Mailbox is 650 GB of data

@ 175 GB/hr = 3.7 hrs

Benefits:Enables hardware-based VSS at SG level, providing single SG backup and restore

Isolates performance between SGs when not sharing spindles between LUNs

Increased reliability; a capacity or corruption problem on a single LUN only affects one SG

Concerns:50 SGs with continuous replication could require 200 LUNs; this exceeds some storage array maximums. CCR solutions could have 100 LUNs on each node, while LCR could have all 200 LUNs presented to a single server

A separate LUN for each SG causes more LUNs per server increasing the administrative costs and complexity

3/23/2007 11:48 AM

UCM309 10

Performing a nightly full backup on 1/7th of the databases could reduce complexity by placing all SGs to be backed up on the same log and db LUN

Benefits:

Simplified storage administration; fewer LUNs to manage

Potentially reduce the number of backup jobs

Concerns:

Limits ability to do hardware-based VSS backup/restores

2 TB limit on MBR partition limits how far this scales

A capacity or corruption problem on a single LUN could affect more than one SG

Use single partition on each LUN

GUID partition table (GPT) partitions are available, but MBR remains best practice

Use GPT if you need or will need partitions larger than 2 TB or up to 128 primary partitions

Default allocation unit size is 4 KB

Use storage vendor recommended setting, or 64K

Diskpar(t) Blog:http://msexchangeteam.com/archive/2005/08/10/408950.aspx

Use when you have more LUNs than available drive letters

Common scenario in large, multi-node single copy clusters

Also recommended for faster activation of passive copy in an LCR environment

Transaction Logs (L:) Databases (P:)

L:\SG1LOG P:\SG1DB

L:\SG2LOG P:\SG2DB

L:\SG3LOG P:\SG3DB

L:\SG4LOG P:\SG4DB

3/23/2007 11:48 AM

UCM309 11

Storage is infinitely configurable; evaluate tested configurations and leverage the outlined best practices

ESRP facilitates 3rd party storage testing, and best practice solution publishing for Exchange

Exchange 2007 solutions on the way

Use latest version of Jetstress to validate Exchange 2003 solution to fit Exchange 2007 solution

http://www.microsoft.com/technet/prodtechnol/exchange/2003/esrp.mspx

Understand what success looks like

Test with as many servers attached to the storage as you will have in production

Test with large databases

Performance characteristics of storage change based on data set size

Determine that storage meets throughput requirements and then determine max throughput of solution (headroom)

Determine that storage meets backup throughput and I/O requirements to meet your backup and restore SLA

Jetstress

Simulates Exchange I/O characteristics

Easy to use

Accurate in simulating Exchange I/O

Test reliability with 24-hour burn-in

Validate that solution meets performance requirements

Test performance and reliability of end-to-end backup/restore solution

Use Load Generator to simulate clients

Updated for Exchange 2007

Support for 1:1 read:write ratio

Support for 50 storage groups

New backup and restore functionality

Wizard GUI interface similar to ExBPA

Available for 64-bit or 32-bit operating systems

Engine redesign – better tuning

Database create/copy speed improvements

3/23/2007 11:48 AM

UCM309 12

Catch hardware failure/warning conditions that could lead to database corruption

Catch storage performance issues before clients are affected

Exchange I/O characteristics can change over time (e.g., user profiles change, mailboxes grow, number of users increases, third-party application install, etc.)

Shared storage scenarios open up Exchange storage to other application I/O load

I/O changes over time

Use Extended ESE Performance Counters

RPC Average Latency: avg 50ms, max 100ms

Disk latency: avg 20ms, max 40ms

Log writes: <10ms

Database reads: <20ms

Database writes: <20ms

Database page faults should be <1

Catch issues before they affect host

Monitoring specific to each storage solution

Some storage vendors provide a MOM management pack

If not MOM, then most likely a SNMP based interface

General metrics include

Disk/Spindle % utilization

Read cache hit ratio

Write pending requests

Storage processor % utilization

Performance Monitor, Logs, and Alerts

Microsoft Operations Manager

Exchange Mgmt Pack for MOM

Exchange Best Practices Analyzer Tool

ExBPA management pack for MOM 2005

Event Viewer

Network Monitor 3.0

Exchange Performance Troubleshooting Analyzer (ExPTA)

3/23/2007 11:48 AM

UCM309 13

Reduce cost and complexity

Enable large mailboxes

VSS and CCR enable fast recovery

Content indexing and Messaging Records Management enable large mailboxes

Exchange 2007 storage solutions are different

Not just IOPS per user; it‟s message activity and amount of RAM in system

Ask-the-ExpertsObtenha Respostas às Suas Questões

Date, time

Date, time

Outros RecursosPara Profissionais de TI

TechNet Plus2 incidentes de suporte gratuito profissional

software exclusivo: Capacity Planner

software Microsoft para avaliação

actualizações de segurança e service packs

acesso privilegiado à knowledge base

formação gratuita

e muito mais.

www.microsoft.com/portugal/technet/subscricoes

3/23/2007 11:48 AM

UCM309 14

Questionário de AvaliaçãoPassatempo!

Complete o questionário de avaliação e devolva-o no balcão da recepção.

Habilite-se a ganhar uma Xbox 360 por dia!

UCM002

Exchange Server 2007Storage Changes and Design Considerations

© 2007 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.