Post on 06-Aug-2020
3/23/2007 11:48 AM
UCM309 1
Exchange Server 2007Storage Changes and Design Considerations
Scott Schnollscott.schnoll@microsoft.comSenior Technical Writer, Exchange ServerMicrosoft Corporation
Patrocinadores
Microsoft Exchange 2007 Design Goals
Exchange 2007 Storage Features
ESE Database Fundamentals
Reducing Disk I/O
ESE Changes
Storage Design
Testing, Validating and Monitoring
3/23/2007 11:48 AM
UCM309 2
Leverage 64-bit Windows
Use large amounts of memory
Support large mailboxes (> 1 GB)
Provide fast search
Reduce storage I/O
Lower storage costs
Reduce complexity
Increase reliability
Large, low-cost mailboxes
Enabled by reduced I/O
I/O profile is changing
Features that allow you to use large mailboxes
Content indexing
Messaging Records Management
Fast recovery with VSS and continuous replication
14-day deleted item dumpster default
Dual phase commit remains
Phase 0: commit the user transaction in fast way
Sequential write of page changes (modification, deletion, insertion)
Phase 1: update the database in an atomic way
Balanced Trees with backward and forward links
Several trees in a database
Random access, fixed page size
Ability to coalesce I/O
Two fundamental use of memory
Cache: to keep in memory frequently used pages
Buffer: to keep track of transactions as they occur
Checkpoint depth
Size (in MB) of log files to keep in memory. Cached pages of a storage group‟s databases that is updated in RAM but not yet to disk
20 MB per storage group
Commit changes to disk after logging
Cache changes in buffer
3/23/2007 11:48 AM
UCM309 3
DB cache size “unlimited”
RAM rule of thumb: 2 GB + 5 MB per user
Increasing cache size reduces DB reads
50 databases in 50 storage groups
Databases mounted in parallel
Small RAM = small cache = more I/O
Large RAM = large cache = less I/O
Following sizing rule of thumb (2 GB + 2-5 MB/user) produces significant decrease of read operations
Increased page size to 8 KB (was 4 KB)
I/O coalescing up to 1 MB (was 64 KB)
Larger but fewer I/Os
EDB file only; no STM file
Take advantage of VLM
Maximum database cache from 900 MB to many GBs
More storage groups = more checkpoint depth
No more VM fragmentation
Log file generations up to billions
Log files 1 MB in size
More storage groups = more checkpoint depth
More checkpoint depth =
Going less often to disk
Accepting more changes in RAM directly before committing to disk
Optimize the number of physical writes by keeping the page dirty longer
Handling a greater volume of database changes
More transactions, more users
Database write is improved
3/23/2007 11:48 AM
UCM309 4
Mailbox Server Performance
0
500
1000
1500
2000
2500
3000
4GB 8GB 16GB 24GB 32GB
Read IO/Sec Write IO/sec
Hardware: 4 x dual core AMD 2.2 GHz
User profile: 4000 Outlook 2003 online users simulated with Exchange Load Generator, 100 MB mailbox size, 17 local deliveries/sec
Read, Overall IOPs/u Reductions
0
100
200
300
400
500
600
Exchange 2003 Exchange 2007
600
320275
240
Read IOPS
Writes/sec
x64 and baseline improvements around cache and I/O profile demonstrate improvements in 4 GB 4 GB testing
ProLiant DL385 2 Dual-Core CPU (2.2GHz), 4GB RAM, 1500MMB3 users, U320 SCSI 24 DB disks, 4 Logs. Search/Indexing=OFF
IOPS ReductionExchange 2003 v Exchange 2007
0
0,2
0,4
0,6
0,8
1
1,2
Exchange 2003 Exchange 2007
1,035
0,274
0,625
0,078
0,41
0,196
IOPS/User
Reads/User
Writes/User
Exchange 2003: 4GB
Exchange 2007: 22 GB
4000 user/250MB Mailbox1 IOPS Profile
1:1 read:write ratio is heavily dependent on clients using Cached Exchange Mode or clients in Online mode w/small mailboxes (100mb or less)
Large mailboxes in Online mode changes ratio to 2:1 (depending upon item counts in key folders)
BlackBerry devices also push up reads significantly (don‟t know how much)
ProLiant DL385 2 Dual-Core CPU (2.2GHz), 4GB RAM, 1500MMB3 users, U320 SCSI 24 DB disks, 4 Logs.Search/Indexing=OFF.
Read Write Ratio
Exchange 2003 69% 31% ~2:1
Exchange 2007 51% 49% ~1:1
0
100
200
300
400
500
600
Read IOPS Write IOPS
504
231175 168
Exchange 2003
Exchange 2007
3/23/2007 11:48 AM
UCM309 5
Reduction in database read I/O with larger database cache
Reduction in database write I/O with more aggregate checkpoint and I/O coalescing
DB write I/O:Log write I/O is 1:.5 to 1:1, but typically closer to 1:.5
User profile affects ratio
More options than FC shared storage
Understand the performance implications of other storage solutions
Cluster continuous replication (CCR)
Removes dependency on shared storage
But doubles storage requirements
Fast recovery is enabled with features such as
LCR/CCR
VSS
Database portability
Predict baseline IOPS using two primary factors: amount of database cache per user, and number of messages each user sends/receives each day
See „Predict Exchange 2007 Baseline IOPS‟ in Rob Quimby‟s blog
Use Profile Analyzer to determine profile
Enter gathered information into Storage Calculator
3/23/2007 11:48 AM
UCM309 6
Get the calculator from http://msexchangeteam.com/attachment/432207.ashx
Requires Excel 2007
Compatibility pack for Office XP and 2003 can be obtained here
The calculator does not make any recommendations toward storage design (RAID parity, number of disks, etc.) as the design depends on the storage being used
Feedback alias: strgcalc
Database
~15% overhead for larger dumpster [MSIT profile]
~ 5% overhead for content indexing
~10% overhead for database whitespace
Log
Recommended log LUN size
Move mailbox
Example:
1000 User, 250MB mailbox = 250 GB
CI 12.5 GB, Dumpster 37.5 GB, Whitespace 25 GB
Total = 325 GB
Backup, restore, and re-seed operations can require significant I/O particularly with larger databases
Maintenance and online defragmentation must run, is scheduled, and causes significant I/O
Messaging records management is a scheduled crawl of the database
Content Indexing
Outlook Cached mode
Sorts & searches performed by client
Outlook Online Mode
Sorts & searched performed by server
Initial index creation expensive
Place no more than 5000 items in a folder
Messaging Records Management (MRM)
Database crawl impacted by database size
MRM can be expensive, so do NOT run it with other tasks (e.g., backup, maintenance)
3/23/2007 11:48 AM
UCM309 7
Keep them small, and take into account
Streaming backup/restore
Offline defrag/repair
Online maintenance
Reseed time (1 database = 25 MB/sec)
Using CCR/LCR as the 1st line of defense mitigates only the first point
Without continuous replication: 100 GB
With continuous replication: 200 GB
SATA
Use enterprise SATA with full-time duty cycle
Use faster RPM with RAID5
RAID5 performance is worse in Exchange 2007 due to the 1:1 read:write ratio
SAS
SAS is the new SCSI with larger disk sizes
Use 2½” small form factor (SFF) for speed
Use 3½” for capacity
Balance RPM with I/O requirements
10k RPM may be enough
iSCSI
Isolate the iSCSI network
Use Gigabit – Jumbo Frames & Flow Control
MPIO (v2 initiator) for higher throughput & reliability
Make iSCSI LUNs persistent
MPIO can sustain high MB/sec for backup & checksum integrity (VSS)
Fiber Channel
Use MPIO
Use WHQL qualified firmware/drivers
Use storage vendor queue depth settings
Balance performance with capacity requirements
RAID10 has best reliability
Can lose more disks in a RAID group before data loss
Light performance decrease during failed disk/RAID rebuild
Log LUN should be RAID10 with a 100% write workload
RAID5 provides best capacity efficiency
Due to poor performance, the extra space is not often utilized
1:1 write ratio causes RAID5 LUNs to perform worse than in Exchange 2003
Heavy performance decrease during failed disk/RAID rebuild
RAID6 provides increased data protection over RAID5
Worse performance than RAID5 with less capacity efficiency
Heavy performance decrease during failed disk/RAID rebuild
3/23/2007 11:48 AM
UCM309 8
Performance varies based on solution
Customers should work with storage vendors to get RAID performance data based on chosen solution and it‟s configuration
Include failure and rebuild scenarios in design
RAID5 and RAID6 are discouraged because of performance reasons
If I/O requirements can be met with RAID1, deploy RAID1
If IO requirements can‟t be met with RAID1, deploy RAID10
CCR enables clustering with non-shared storage
Twice the storage for database redundancy calls for less expensive storage (DAS, SAS)
1 database per storage group
4 LUNs per storage group, 2 Log and 2 database (Active/Passive)
Logs and Database on isolated disk
Active and Passive on isolated storage
LCR - separate storage controller and PCI bus
Design Passive LUNs to match Active LUNs in both performance and capacity
Passive copy is first line of defense
Place passive LUNs on separate storage from the active LUNs
Passive I/O will not impact production
Ability to run VSS backup on passive does not impact active
3/23/2007 11:48 AM
UCM309 9
More transactional I/O than non-replicated
May require cache setting reevaluation
Recommend 25%:75% read:write on LOG LUN
Active Passive
Log Writes X CR
Log Reads CR CR
DB Writes X CR
DB Reads X CR
X - Traditional Exchange I/OCR - Addition I/O when using continuous replication
LCR/CCR – first line of defense
Copy of data; first recovery vehicle
Not a backup
Active copy of database
Legacy streaming to disk
Legacy streaming to tape
VSS snapshot – copy to disk or tape
VSS clone – copy to disk or tape
Passive copy of database
VSS snapshot – copy to disk or tape
Offloads backup I/O to passive LUN
LCR/CCR only
With CCR/LCR, use VSS to backup passive copy
Backup does not affect active LUN
Checksum integrity (eseutil) occurs on passive copy
Daily Full Backup
2000 user – 2 GB mailboxes is 4 TB of data
@ 175 GB/hr = ~ 23 hours
Weekly Full & Daily Incremental
Acceptable because LCR/CCR is the first line of defense
Stagger databases for full backups
Example: 14 DBs, 2 full per night, 12 incremental
2000 user – 2 GB Mailbox is 650 GB of data
@ 175 GB/hr = 3.7 hrs
Benefits:Enables hardware-based VSS at SG level, providing single SG backup and restore
Isolates performance between SGs when not sharing spindles between LUNs
Increased reliability; a capacity or corruption problem on a single LUN only affects one SG
Concerns:50 SGs with continuous replication could require 200 LUNs; this exceeds some storage array maximums. CCR solutions could have 100 LUNs on each node, while LCR could have all 200 LUNs presented to a single server
A separate LUN for each SG causes more LUNs per server increasing the administrative costs and complexity
3/23/2007 11:48 AM
UCM309 10
Performing a nightly full backup on 1/7th of the databases could reduce complexity by placing all SGs to be backed up on the same log and db LUN
Benefits:
Simplified storage administration; fewer LUNs to manage
Potentially reduce the number of backup jobs
Concerns:
Limits ability to do hardware-based VSS backup/restores
2 TB limit on MBR partition limits how far this scales
A capacity or corruption problem on a single LUN could affect more than one SG
Use single partition on each LUN
GUID partition table (GPT) partitions are available, but MBR remains best practice
Use GPT if you need or will need partitions larger than 2 TB or up to 128 primary partitions
Default allocation unit size is 4 KB
Use storage vendor recommended setting, or 64K
Diskpar(t) Blog:http://msexchangeteam.com/archive/2005/08/10/408950.aspx
Use when you have more LUNs than available drive letters
Common scenario in large, multi-node single copy clusters
Also recommended for faster activation of passive copy in an LCR environment
Transaction Logs (L:) Databases (P:)
L:\SG1LOG P:\SG1DB
L:\SG2LOG P:\SG2DB
L:\SG3LOG P:\SG3DB
L:\SG4LOG P:\SG4DB
3/23/2007 11:48 AM
UCM309 11
Storage is infinitely configurable; evaluate tested configurations and leverage the outlined best practices
ESRP facilitates 3rd party storage testing, and best practice solution publishing for Exchange
Exchange 2007 solutions on the way
Use latest version of Jetstress to validate Exchange 2003 solution to fit Exchange 2007 solution
http://www.microsoft.com/technet/prodtechnol/exchange/2003/esrp.mspx
Understand what success looks like
Test with as many servers attached to the storage as you will have in production
Test with large databases
Performance characteristics of storage change based on data set size
Determine that storage meets throughput requirements and then determine max throughput of solution (headroom)
Determine that storage meets backup throughput and I/O requirements to meet your backup and restore SLA
Jetstress
Simulates Exchange I/O characteristics
Easy to use
Accurate in simulating Exchange I/O
Test reliability with 24-hour burn-in
Validate that solution meets performance requirements
Test performance and reliability of end-to-end backup/restore solution
Use Load Generator to simulate clients
Updated for Exchange 2007
Support for 1:1 read:write ratio
Support for 50 storage groups
New backup and restore functionality
Wizard GUI interface similar to ExBPA
Available for 64-bit or 32-bit operating systems
Engine redesign – better tuning
Database create/copy speed improvements
3/23/2007 11:48 AM
UCM309 12
Catch hardware failure/warning conditions that could lead to database corruption
Catch storage performance issues before clients are affected
Exchange I/O characteristics can change over time (e.g., user profiles change, mailboxes grow, number of users increases, third-party application install, etc.)
Shared storage scenarios open up Exchange storage to other application I/O load
I/O changes over time
Use Extended ESE Performance Counters
RPC Average Latency: avg 50ms, max 100ms
Disk latency: avg 20ms, max 40ms
Log writes: <10ms
Database reads: <20ms
Database writes: <20ms
Database page faults should be <1
Catch issues before they affect host
Monitoring specific to each storage solution
Some storage vendors provide a MOM management pack
If not MOM, then most likely a SNMP based interface
General metrics include
Disk/Spindle % utilization
Read cache hit ratio
Write pending requests
Storage processor % utilization
Performance Monitor, Logs, and Alerts
Microsoft Operations Manager
Exchange Mgmt Pack for MOM
Exchange Best Practices Analyzer Tool
ExBPA management pack for MOM 2005
Event Viewer
Network Monitor 3.0
Exchange Performance Troubleshooting Analyzer (ExPTA)
3/23/2007 11:48 AM
UCM309 13
Reduce cost and complexity
Enable large mailboxes
VSS and CCR enable fast recovery
Content indexing and Messaging Records Management enable large mailboxes
Exchange 2007 storage solutions are different
Not just IOPS per user; it‟s message activity and amount of RAM in system
Ask-the-ExpertsObtenha Respostas às Suas Questões
Date, time
Date, time
Outros RecursosPara Profissionais de TI
TechNet Plus2 incidentes de suporte gratuito profissional
software exclusivo: Capacity Planner
software Microsoft para avaliação
actualizações de segurança e service packs
acesso privilegiado à knowledge base
formação gratuita
e muito mais.
www.microsoft.com/portugal/technet/subscricoes
3/23/2007 11:48 AM
UCM309 14
Questionário de AvaliaçãoPassatempo!
Complete o questionário de avaliação e devolva-o no balcão da recepção.
Habilite-se a ganhar uma Xbox 360 por dia!
UCM002
Exchange Server 2007Storage Changes and Design Considerations
© 2007 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.