Mastering Microsoft SQL Server consolidation planning

Mastering Microsoft SQL Server consolidation planningebook | Jani K. Savolainen

About the author

Jani K. Savolainen is an IT entrepreneur, founder & CTO of

DB Pro, and an inventor of several software products, including the SQL Governor. He has been working with

Microsoft SQL Server technology and software development since 2001. Jani’s core competencies in

the field of SQL Server are full stack code optimization, database design,

DW/BI, data science, and capacity planning of large scale SQL Server

platforms. During his free time Jani designs crypto graphical algorithms, solves brain puzzles, plays chess and Texas holdem, composes electronical

music, and writes poems. Jani is also the chairman of PASS Helsinki,

the Finnish local chapter of the Professional Association for SQL

Server Users.

3Mastering Microsoft SQL Server consolidation planning

www.sqlgovernor.com

Contents4 Introduction Microsoft SQL Server consolidation planning Are you planning to migrate your Microsoft SQL Server system? MS SQL Server and savings? Problems in traditional SQL Server consolidation planning methods MS SQL – what next?

8 The phases of a consolidation planning project SQL Server consolidation project Choosing a consolidation planning strategy Consolidation planning project Consolidation planning – Requirements specification Consolidation planning – Detailed specification SQL Governor consolidation planning solution

12 How to monitor the existing environment SQL Server performance counters to monitor Selecting the right time span and grain for SQL Server monitoring Missing time series in SQL Server monitoring Monitoring SQL Server service times Adding SLAs in SQL Server capacity planning

16 SQL Server diagnostics and health check Disk allocation unit size Tempdb file configuration SQL Server trace flags 1117 and 1118 SQL Server instance configuration properties Wait statistics User database file groups Unused indexes Missing indexes Fragmented indexes Foreign keys without supporting index Page life expectancy

22 Requirementsspecification When to consolidate Microsoft SQL Server Data center environment prestudy Architecture prestudy Non-functional requirements in consolidation planning Risk analysis during the consolidation planning project Analysis of external dependencies Defining SQL Server target platform architecture alternatives Setting the metadata for capacity planning SQL Server platform cost model prestudy Calculating an optimal capacity distribution for SQL Server instances and servers Select desired SQL Server target architecture

www.sqlgovernor.com


Introduction


www.sqlgovernor.com

Microsoft SQL Server consolidation planningIn this book, which is based on my previously released blog posts from 2016 - 2017, I will help my readers to overcome the challenges in Microsoft SQL Server consolidation planning.

Are you planning to migrate your Microsoft SQL Server system? A significant proportion of IT investments in companies is allocated to the replacement, updating and expansion of existing information systems. The most common reasons for system replacement are:

◆ outdated software

◆ obsolete server infrastructure

◆ company merger

◆ SQL Server licence renewal

◆ seeking cost savings

The increase in application use and ballooning quantities of data mean that there’s always a pressure to expand.

MS SQL Server and savings?The Microsoft SQL Server database platform (database servers with their software) at the back end of business-critical applications is often an extremely important element from the perspective of information system replacement and expansion projects. Optimal design and implementation of larger systems can create savings that are well into the double-digit percentages in comparison with the previous database platform investment. They also create a foundation for good performance and scalability of applications. In contrast, a poorly implemented modernization project can lead to substantial over-investments or cause performance problems with the system, along with a shorter than planned service life.

www.sqlgovernor.com


In the design of a new environment, it is essential to understand the complex whole composed of the various business requirements, licensing and budget-related requirements, the architecture, the environment’s non-functionality-related special features and capacity requirements. This understanding forms the basis for selecting the combination that can best serve the business.

Problems in traditional SQL Server consolidation planning methodsTraditional data platform capacity planning contains a lot of time consuming manual work and is often based on rough estimates. In larger SQL Server environments, consolidation planning often takes tremendous time and binds a great deal of human resources, which makes it expensive. Time-to-solution is also hard to predict. It is not rare to see consolidation planning projects containing only a few dozens of source servers but lasting easily six months or more.

It is easy to understand how difficult it is to synchronize SQL Server license renewal with consolidation implementation, especially when deadlines are hard to predict. However, most of the work effort could be shortened from months to even days with an efficient consolidation process and intelligent planning automation.

Another typical caveat is that servers are often replaced by another server in a 1-to-1 manner, which means that there are no consolidation benefits to be gained. The idea of consolidation is to reduce the overall amount of servers by grouping several source SQL Server instances or servers into fewer target host servers than existed in the source system. This typically reduces software license costs and the overall number of CPU cores needed. If consolidation occurs in a traditional migration project, it is rarely based on comprehensive monitoring combined with a mathematical planning method. Consolidation planning is seen more as a task for the data platform architect than a mathematical challenge. The truth is: It is both.

In cases where the source system’s data is monitored, it is often only implemented on the server level, which means there is no true understanding of the underlying DBMS instances and databases. Instead of long-term source system monitoring only snapshots of peak values are taken. This in turn leads to a situation where capacity is estimated on a peak basis only, meaning that the long-term instance-level usage data cannot be distributed equally between the target servers.


www.sqlgovernor.com

MS SQL – what next?All in all, in larger production environments of up to thousands of SQL Servers, without a proper method there are just too many variables in order to manage everything manually.

Define the business need

Requirements specification Planning work Detailed

specification

Migration implementa-

tion

Consolidation project phases

The most common reasons driving a data platform renewal project‒and great opportunities for platform consolidation

and optimization

• outdated software• obsolete server infrastructure• company merger• SQL Server licence renewal• seeking cost savings

www.sqlgovernor.com


The phases of a consolidation planning project


www.sqlgovernor.com

SQL Server consolidation projectA consolidation project entails transferring a larger environment comprising several SQL servers into a new environment. The initial system consists of a number of applications and servers scattered across various locations. It may include several SQL Server versions with thousands or more databases. High-availability solutions may be in use.

It is often sensible to centralize the services with a single data center and simplify the structures and versions of the SQL Server services. This means reducing the number of servers to an optimal level. Determining the required CPU, disk, and memory capacity is extremely important. Addressing this issue, our SQL Governor® product automates the process and improves its efficiency.

The consolidation process for an environment with several SQL Servers resembles the somewhat less complicated consolidation of a single server but requires iteration and paying attention to consolidation needs and possibilities.

Choosing the consolidation planning strategyThere are three different types of consolidation planning strategies: server-centric-, instance-centric and database-centric. All the strategies typically support server virtualization. In server-centric approach, the idea is to consolidate source servers as virtual machines under the target host server by reducing overall target host server count compared to the count of source servers. In an instance-centric approach, the DBMS instances are the building blocks of the new architecture, whereas a database-centric approach separates databases from their hosting instances.

Our approaches are server-centric and instance-centric consolidation, which are ideal choices for environments with a large amount of host servers, instances and databases. A database-centric approach functions well in smaller environments. However, database-centric approach often demands extra work and may become too complex and time consuming to implement in large production environments. This is because the SQL Server Agent jobs, Services, Maintenance plans, User accounts, Proxies, SSIS packages, Database Mail, Resource Governor, etc. reside on the SQL Server instance level and services like SSRS, SSAS, SSIS, DQS and MDS reside on server level.

Another problematic aspect in a database-centric approach is the growing uncertainty of individual user database performance forecasting. How will a given user database perform in the new environment with other user databases? There is no simple answer because tempdb performance is hard to predict. It interoperates with user databases that never existed under same DBMS instance before. This generates a different kind of overall workload that is not trivial to trace.

https://www.sqlgovernor.com/en/governor-a-tool-for-sql-server-optimisation/?utm_source=Ebook&utm_medium=pdf&utm_campaign=Mastering%20consolidation%20planning

www.sqlgovernor.com


Consolidation planning project SQL Server consolidation planning is an iterative process. It needs input data from various sources such as the architecture, servers, software, licensing, services, versions, and source system monitoring data. All this affects the final design of the consolidation model. It is a good idea to divide the project into two subprojects: the requirements specification and the detailed specification. This lowers the risks in the implementation phase and improves the overall quality of the project delivery with a more optimal TCO.

Consolidation planning – Requirements specificationThe consolidation planning process requirements specification is described below. Some of the phases of the project are done in parallel and some of the phases use feedback information from each other. The purpose of the requirements specification is to gather the high-level requirements for the overall consolidation project by answering the question: what. In our method, what is different from traditional capacity planning, is that already in the end of the requirements specification the actual implementation of capacity calculation is done. I will go through each of these phases later in this book.

Set metadataAnalysis of

existing environment

Data center environment

prestudyLifecycle planning

Non-functional requirements

SQL Server diagnostics and

health check

Architecture prestudy

Analysis of external

dependencies

Risk analysis

Defining architecture alternatives

Calculating capacity

distribution

Cost model prestudy

Select desired end

architecture


www.sqlgovernor.com

Consolidation planning – Detailed specificationYou can find a diagram of the process for producing the detailed specification below. Suffice to say here, the purpose of the detailed specification is to describe how exactly the consolidation implementation will be done. We will get back to these phases later in this book.

SQL Governor migration planning solutionWith our patent pending method and our SQL Governor® software it is possible to improve the consolidation planning process through significant overall cost savings. SQL Governor® is a unique, easy-to-use, domain-independent solution for efficient database consolidation capacity planning, monitoring, diagnostics, alerting and capacity management in a Microsoft SQL Server DBMS environment. With SQL Governor® and our unique method it is possible to:

◆ Calculate an optimal server topology for your target architecture based on efficient time series analysis and machine learning.

◆ Save up to 50% in license and server investment costs compared to your existing data platform.

Compatibility check

Planning for high availability

Planning of phasing

Planning of the update methods

Specification of service

interruptions

Planning of testing

Allocation of resources

Recovery procedures

https://www.sqlgovernor.com/en/governor-a-tool-for-sql-server-optimisation/?utm_source=Ebook&utm_medium=pdf&utm_campaign=Mastering%20consolidation%20planning

www.sqlgovernor.com


How to monitor the existing environment


www.sqlgovernor.com

SQL Server performance counters to monitorSQL Server monitoring is crucial for a successful consolidation project. Without the right performance counters and metrics, it is basically impossible to forecast the future capacity needs of the system. There are important metrics at three levels of the system: server level, instance level and database level.

Good server level performance counters to monitor are: ◆ peaking and average CPU usage

◆ RAM usage

Good instance level performance counters to monitor are: ◆ peaking and average CPU usage

◆ RAM usage

◆ tempdb allocated and used size

◆ tempdb read/write iops

◆ tempdb read/write mb/s

◆ templog allocated and used size

◆ templog write iops

◆ templog write mb/s

If database level consolidation occurs, good database level performance counters to monitor are:

◆ disk read/write iops

◆ disk read/write mb/s

◆ database files allocated and reserved size

◆ log files allocated and reserved size

◆ CPU time ms

Note! In addition to these performance counters there are many other useful counters as well. Some of those additional counters give more information about the health of the system rather than performance, for example, counters like read latency, buffer pool size, buffer cache hit ratio, processor queue length, deadlock count etc. It is a good idea to cover these counters in the “SQL Server diagnostics and health check” phase of the project.

www.sqlgovernor.com


Selecting the right time span and grain for SQL Server monitoringTypically, there should be at least 3 months of concurrent monitoring during the peak months to get any idea of overall resource consumption. The problem with shorter periods is the inability to predict long-term trends and therefore to fit the appropriate time series together. In more challenging environments 6, 12 or even 24 months may be required in order to forecast performance metrics with a high level of accuracy. Ideally, there is performance data from the whole lifecycle of the environment to generate an hour-based time series. That is one good reason why it is so important to have a SQL Server monitoring software such as SQL Governor® running all the time in the production environment. Performance data is the most valuable asset for a SQL Server consolidation project.

Missing time series in SQL Server monitoringSometimes it happens that there is no performance data for a certain time span. Many reasons can cause this, including a server, service, hardware or network failure or a planned service break. It is noteworthy that a missing time series may affect the


www.sqlgovernor.com

actual capacity calculations. Missing time series data should be investigated and analyzed to decide whether the information is critical for capacity planning. If the answer is yes, then it is good idea to continue planning work with extra monitoring. Generating actual time series is preferable instead of summing up existing data to maintain accuracy and comparability over time between the DBMS instances in capacity planning.

Monitoring SQL Server service timesDifferent servers, instances, databases, customers and weekdays may have different service times. You need a mechanism to keep track of service time windows in your existing system in order to get a better understanding of your production environment capacity needs over time. This is because you may want to filter out those members from your result set which lie outside service times. Once again, this makes the actual capacity calculation more precise and practical. Our SQL Governor® software allows you to set service times for each individual performance object (such as a server) to be monitored.

Adding SLAs in SQL Server capacity planningA Service Level Agreement (SLA) tells us how well a certain performance counter is behaving at each moment. For each SLA you need to define the target percentage that the SLA must be equal to or exceed. One typical example of an SLA is server uptime: If the SLA criteria (in this case “the server is up and running”) is met for more than 99.9999% of all members in the time series, the target SLA is met; otherwise not. This makes it possible to fine-tune the performance metrics for capacity planning. SQL Governor® software includes a comprehensive set of monitoring SLAs for this purpose.

www.sqlgovernor.com


SQL Server diagnostics and health check


www.sqlgovernor.com

Disk allocation unit sizeSQL Server prefers a 64 KB allocation unit size for database files. The default setting of 4 KB is not optimal for storing SQL Server databases. This is because SQL Server does IO in chunks of 8 x 8 KB pages.

Tempdb file configurationThe tempdb data and log file sizes should be large enough. In addition, ever since the SQL Server’s architectural change in version 2008, it is able to dedicate multiple threads on tempdb data file processing. The amount of tempdb data files should match the dedicated CPU core count on current instance, up to maximum of 8 data files.

Tempdb is the cornerstone of SQL Server’s performance because temporary data objects such as user objects, internal objects and row versions are processed there. In earlier versions of SQL Server this was not all that relevant. Today, you should create as many files as needed to maximize your disk bandwidth because it reduces tempdb storage contention and improves scalability.

Note! Do not create too many files because that can reduce performance and increase management overhead.

SQL Server trace flags 1117 and 1118Trace flags 1117 and 1118 are introduced as default server trace flags for SQL Server 2016. You should consider turning these trace flags on in the earlier SQL Server versions. These trace flags boost the SQL Server data file processing performance.

Trace flag 1117 causes all files in a file group to grow equally. This trace flag helps in cases where there is heavy tempdb latch wait pressure. Another way to solve this issue is to pre-allocate tempdb data files into their maximum allowed capacity instead of using the trace flag. This is sometimes preferable because trace flag 1117 affects all user databases as well and may cause too much concurrent page allocations, especially on instances with a large number of big databases.

Trace flag 1118 favors full extents instead of mixed extents. Every new allocated object for each database on the instance gets its private 64KB of data. This is good because non-harmonized tempdb files can cause contention on some pages.

www.sqlgovernor.com


SQL Server instance configuration propertiesThere are a few SQL Server instance configuration properties that should be checked. These are max degree of parallelism, cost threshold for parallelism and fill factor (%). Max degree of parallelism sets the maximum amount of parallel CPU cores to dedicate to each individual statement during parallel plan execution. A value of 0 (and a value equal to the CPU count) means that all the CPU cores up to 64 may be used. A value of 1 means that only one core should be used. SQL Server will evaluate parallel execution plans for queries, index DDL operations, as well as static and keyset-driven cursor population.

If the estimated execution plan duration is equal to or greater than the cost threshold for parallelism value for a serial plan in seconds, then the max degree of parallelism number of cores is allowed to process the statement in parallel plan mode.

Thefillfactorsettingfine-tunestheSQLServerindexperformance.Every time when index is being created or rebuilt, its fill factor determines how many percent of the index leaf-level pages are left empty for allocating new index entries in the future. The default fill factor of 100% equals fill factor values of 0 and 100.

Fill factor means how full the index pages will be. The default of 100% is rarely optimal in OLTP database use. A better default value is 95% to avoid maximum page splitting when creating new indexes without defining a fill factor.

A good fill factor value can reduce hypothetical page splits by providing enough space for expanding the index as data is added to the table. On the other hand, a fill factor that is too low will cause poor read performance on the index because more index pages need to be read. You should always define an index fill factor explicitly when creating a new index. A good rule of thumb is to use lower fill factors for “hot” indexes, i.e. indexes that are under constant change.

Wait statisticsWait statistics contain essential information of the overall health status of the instance. At most, the CPU signal waits vs other resource waits ratio should be 20 vs. 80, respectively, with CPU waits not exceeding 20%. Anything over 20% indicates increased CPU pressure.


www.sqlgovernor.com

User database file groupsThe bigger user databases should contain more file groups in order to maximize disk performance. There are different prerequisites for different SAN systems, but there are still some certain general principles, especially for dedicated storage devices. It is a good idea to dedicate the PRIMARY file group only for system objects and references to secondary data files (.ndf). If the database contains big data tables, there should be a file group of its own for them, as well as file groups for ordinary data tables and for non-clustered indexes. Each of these file groups should contain one or more data files, depending on the storage system / LUN configuration. This scales up the storage system performance. Set the ordinary DATA file group as DEFAULT.

Note! Do not use PRIMARY file group as DEFAULT. We want only system objects to be stored in the larger databases.

www.sqlgovernor.com


Unused indexesUnused indexes are indexes that exist in a table but have not been used after the last SQL Server service restart. Keep in mind that there may be a lot of unused indexes, if the service was just restarted. However, if the service has already been running a long time (e.g. for several months) and still some indexes are unused, it is a good idea to check whether they really are unnecessary. If so, the indexes should be dropped. This is because excessive indexes cause table insert, update, and delete operations perform slower.

Missing indexesMissing indexes are indexes that should exist but don’t. There are often queries that don’t use indexing effectively. These indexes can be found for example in the SQL Server Management Studio’s estimated query plans.

Fragmented indexesPhysical and logical Index fragmentation causes indexes to perform slower. That’s why it is important to plan index fill factors well, and check index fragmentation on a regular basis. This should be a standard routine of every well-designed index maintenance run. Indexes with a fragmentation level of over 30% should be rebuilt and indexes over 5% fragmentation reorganized with a full scan. You should update all index statistics regularly.

Foreign keys without supporting indexForeign keys without a supporting index will slow down the query performance. It is a best practice to always create an index for each foreign key column.


www.sqlgovernor.com

Page life expectancyPage life expectancy tells us the amount of time, in seconds, the data page will be kept in the buffer pool without references. This kind of memory pressure is good to check on the instance level. A decent formula to calculate the lowest acceptable value for this counter is: maximum memory in GB / 4 * 300. If page life expectancy is too low, the instance needs more RAM in order to prevent too many physical reads from the disk system.

Our SQL Governor® software already supports many of the features that were defined in this chapter.

www.sqlgovernor.com


Requirements specification


www.sqlgovernor.com

When to consolidate Microsoft SQL ServerIt is inevitable that the 5-year-old server is not performing as well, as new servers. Also, the older the server gets, the more vulnerable it will be to hardware related problems. Production environment server stability is inversely correlated with the age of the current deployment.

Another concrete reason for renewing the server infrastructure is the SQL Server licensing. After the current licensing period ends, it is a good time for a SQL Server version upgrade, hardware renewal and SQL server consolidation.

It is recommended to identify any servers that can be left out of the consolidation project scope, for example servers that are:

◆ relatively new servers; or

◆ near to the end of their planned lifecycle.

Another consolidation constraint for a certain standalone server or server cluster may be

◆ the amount of estimated consolidation work; and

◆ its external dependencies.

Sometimes there are a few servers with such an old version of SQL Server with some legacy applications running that there is no sense or possibility to upgrade to the current SQL Server version.

According to my experience, existing SQL Server infrastructures are typically not as consolidated as they should be. If no consolidation is applied for the next SQL Server licensing period, it can easily lead into extra costs of up to 60% for that period. It would be ideal to start the consolidation planning at least one year before the actual end of the licensing period, but often lesser time like 3 to 6 months is just enough to collect some continuous monitoring data, and to proceed consolidation requirements specification and capacity planning for environments with up to 100 source servers.

www.sqlgovernor.com


Data center environment prestudyIn this phase, we get an overall understanding of the existing production environment. It is important to have enough knowledge of the actual production environment, and whether it is going to change in any way.

The main tasks are: ◆ data center environment analysis

◆ network and domain model analysis

◆ hardware analysis

◆ SQL Server architecture and storage analysis

◆ licensing, service model and SLA analysis

◆ upgrade path analysis

These tasks cover issues such as the logical architecture in the data center, and which kind of networks, servers, and hardware it’s running. It is good to know the data systems, clients and data movement between them because this could actually affect how the existing instances are stacked on the new servers. The domain model and how the user rights are managed is important as well. Knowing your existing hardware is very important because it makes it possible to benchmark existing servers and compare them to planned ones.

The upgrade path for SQL Server versions and editions is very important because it may generate some recessive extra work and testing effort for the actual consolidation implementation. It is actually often one of the recognized risks for the success of the overall consolidation project. Upgrading SQL Server version may also bring in some potential performance issues because the engine is optimized in a different way for each version of the SQL Server. Communication with third parties is important in order to recognize which instances have to be and are able to be upgraded, and which SQL Server versions are still under Microsoft support.

Another task is to find out who owns the servers, other hardware, and licenses, and which kind of hosting service model you have. This makes the budget of the project and saving potential easier to manage and understand.


www.sqlgovernor.com

Architecture prestudyIt is important to comprehend the criticality level for each server and SQL Server instance. In addition to this, to make the target system architecture planning and capacity calculation more effective, there must be a clear understanding of service level agreements for uptime, CPU usage, and such, for outlier management of the monitoring performance counter time series data.

There is no better way to plan a new target architecture than learning the pros and cons from the old one. The SQL Server architecture and storage analysis is a review of the current implementation model of SQL Server installations: the Always On clusters, Active – Active, Active – Passive, and Standalone server configurations, and which storage tiers are being used with them. It is important to streamline storage tiers on SQL Server instance basis in order to make overall instance performing steadily, not to forget having the best storage tier available for tempdb usage. Remember to find out which kind of SQL Server maintenance job routines and backup policies you have, to understand the wholeness of SQL Server resource consumption over time.

The main tasks in data center environment prestudy

• Data center environment analysis• Network and domain model analysis• Hardware analysis• SQL Server architecture and storage analysis• Licensing, service model and SLA analysis• Upgrade path analysis

www.sqlgovernor.com


Non-functional requirements in consolidation planningThe most essential non-functional requirements of consolidation planning are:

◆ performance

◆ scalability

◆ availability

◆ maintainability

◆ security

Typically, performance is measured with SLA’s such as “CPU usage should be 80% or lower 99.9% of the time”. Another way to set performance goals is to define certain wait statistics limitations in the target environment. You can also monitor certain performance counters both in the source and target environments and compare the results after consolidation. Examples of such performance counters are:

◆ buffer cache hit ratio

◆ page life expectancy

◆ processor queue length

◆ CPU waits

◆ disk waits

◆ deadlocks, etc.

Proper index maintenance is the backbone of good DBMS performance. Remember to pay attention to this and check the current situation of index maintenance. It should be part of the diagnostics phase.

When it comes to the scalability of the system, it is a good idea to determine which of the servers and clusters:

◆ are dedicated and do not need to scale

◆ will be able to scale up

◆ will be able to scale out

In addition, it is necessary to define the lifecycle for each target server and DBMS instance.


www.sqlgovernor.com

Availability is very important. The criticality level of each DBMS instance should be determined in order to understand which kinds of target server and High Availability options (Always On Failover Clustering, Always On Availability Groups, virtualization HA options, etc.) are needed.

Consolidation and maintainability go hand in hand: The fewer clusters and servers you have, the easier it is to maintain them. Technical documentation is also crucial. During this phase, there should be a top-level plan to write documentation about system configuration, maintenance plans, backup routines, and disaster recovery.

On the security side, it is critical to go through any policies on authentication modes, logins, passwords, users, database encryption, and such.

Risk analysis during the consolidation planning projectRisk analysis plays an important role in consolidation planning. The earlier we acknowledge the pitfalls of the project, the higher the chance to avoid them – and save time and money. Risk analysis should start in the beginning of the project and should be an iterative process during the whole project lifecycle. Always pay attention to possible risks in anything you work on. Consolidation planning projects are the big budget type of projects you don’t want to mess up.

It is a good idea to divide project risks into several categories, for example: ◆ high-level risks

◆ medium-level risks

◆ low-level risks Every customer and project have its special characteristics, but it is a good idea to define some meaningful explanation for each risk level in co-operation with the customer.

A good example of a high-level risk is one that: ◆ prevents the project from being finished

◆ increases project costs significantly (e.g. by 100%)

◆ causes a significant project delay (e.g. of 100%)

◆ causes a serious negative impact on the target system’s functionality

www.sqlgovernor.com


A good example of a medium-level risk is one that: ◆ increases project costs by 50%

◆ causes a severe project delay of 50%

◆ causes a negative impact on the target system’s functionality

A good example of a low-level risk is one that: ◆ increases project costs by 25%

◆ cause a project delay of 25%

◆ causes some negative impact on the target system’s functionality

In addition to the description and status of the risk, it is useful to follow risk status during the project and to define a preliminary description of how to avoid the risk. Of course, some of the risks may be unavoidable.

Some typical risks of a consolidation planning project are: ◆ there is no data / metadata about all the servers and instances available

◆ there is no clear picture of current service and / or licensing costs

◆ there are no source hardware specifications available

◆ capacity planning is done with less monitoring data than needed

◆ there is uncertainty about certain servers / instances / databases and how fast they will grow in the future due the possible business environment changes

◆ hosting company (if there is one) allows only certain series and models of host servers to be used

◆ no proper DBMS diagnostics is done before the consolidation project

◆ same DBMS instances are using very differently performing SAN tiers

◆ bad maintenance plans


www.sqlgovernor.com

Analysis of external dependenciesIt is important to understand which kind of data systems, processes and applications are integrated into the SQL Server architecture. Similarly, we should understand how external data systems affect existing SQL Server deployments. Some of the aspects of an SQL Server deployment are:

◆ How do applications connect to servers/databases: using a connection string, SQL Server alias, DNS alias, configuration files, registry settings, hard coded values, or something else?

◆ Do SQL Server and applications use mixed or integrated security mode?

◆ Do applications use a single technical user name to connect to SQL Server?

◆ Are there firewall rules and are SQL Server instances using fixed TCP ports?

◆ Is there point-to-point interoperability between SQL Server systems, Linked Servers, Analysis Services, Reporting Services and Integration Services?

◆ Is there any interoperability between SQL Server systems and external systems?

◆ Is the interoperability message-based, ETL-based, or both?

◆ When does this interoperability occur?

◆ What are the criticality levels of these systems – are there any SLA’s defined for the dependencies?

◆ Where would a malfunction affect the system and how badly?

It is a good idea to draw a graph of the data flows between the systems and write down the main points. It is also useful to draw a process chart with a schedule for each data flow in order to understand the business logic behind the scenes.So why are these external dependencies so important? It is common that the whole backbone of the business information data flow is built on these external dependencies: If one node fails, the rest will fail too.

www.sqlgovernor.com


Defining SQL Server target platform architecture alternativesIn this phase we investigate the existing SQL Server system architecture. First, we need to figure out which kind of installations there are: standalone instances, failover cluster / Always On availability groups, replicated instances, etc. Second, we have to determine how well the current model has worked. Third, we should find out whether there is anything that needs to be changed or cannot be changed in the installations. Finally, we should identify the non-functional requirements of the installations and any issues that have arisen during the diagnostics and health check phase.

When defining the target system architecture, the main driver should be how the source system architecture is built. It is wise to discuss the functionality and demands of the current architecture with project stakeholders in order to see what should be kept and what should be changed. A good rule of thumb is to favor simple architectures such as standalone and failover cluster installations or Always On availability groups. Keep in mind that for consolidation purposes the simpler the architecture, the better it suits for consolidation. An active-active cluster architecture has the disadvantage of needing extra “empty space” in terms of CPU processing power and RAM usage if the same service level is required after a failover. For an active-passive cluster configuration there is no such drawback.

Our international patent pending SQL Governor® product uses a configurable, automatic mechanism for defining the logical topology of the target architecture when it comes to instance level consolidation. This makes capacity planning more flexible and allows you to fulfill different architecture needs.

It is a good idea to define a couple of different target architecture scenarios. The cost model prestudy will show which is the most cost-effective solution. All in all, pay attention to simplifying the overall architecture if possible – the key is to reduce the number of existing servers, not to increase them.


www.sqlgovernor.com

Setting the metadata for capacity planningThere are two types of metadata in the source system: one that is updated automatically, and one that is updated manually. There is a lot of different kinds of metadata that describes the special characteristics of the source system data more precisely. It would be a good idea to keep track of at least some of the following metadata:

◆ Which DBMS instances should be dedicated to one specific server (means server level right-sizing or server level consolidation)?

◆ Which servers have additional services like SSIS, SSAS or SSRS running on them (means server level right-sizing or server level consolidation)?

◆ What is the planned consolidation date for each server or instance?

◆ Which servers are production servers, and which are dedicated for testing or development purposes?

◆ What is the service, customer, and owner for each server, instance, and database?

◆ What is the criticality level of each server or instance?

◆ What are the generic virtualization parameters such as whether hyper-threading is used on the server?

◆ What is the metadata about index maintenance, database backups, and so on?

Our SQL Governor® product contains lots of automatically and manually updated metadata fields. The same metadata can be then used in capacity planning projects, which makes grouping of source system DBMS instances much easier.

www.sqlgovernor.com


SQL Server platform cost model prestudyIn this business-driven phase of the project we will match all the SQL Server and OS licensing, hardware, service, and capacity costs against competing architecture models. This phase is iterated together with the “Calculating optimal capacity distribution” phase of the project. This iteration may produce one or more target architecture models that we let the project stakeholders to select from.

The best way to do a cost model prestudy is to iterate competing target architecture models and to see the cost differences between them. Often, even when the licensing costs are minimal in some architecture models, the hardware and service costs negate this benefit compared to some other architecture options, so pay attention to all the factors.

Calculating an optimal capacity distribution for SQL Server instances and serversIn this phase, we calculate an optimal capacity distribution of SQL Server source instances and servers over the desired target architecture models. Our SQL Governor®

software uses an international patent pending calculation method based on the metadata, benchmark data, and time series data collected in the “Monitoring existing environment” and “Setting the metadata” phases. This phase is iterated together with the “Cost model prestudy” phase.

Our instance-level calculation method uses so-called templates in the capacity planning. Each of these templates is bound to a certain type of target system architecture, server hardware, and some other input parameters. A template can then generate as many target servers that a certain set of source DBMS instances needs for future use. This makes capacity planning more flexible to fulfill different architecture needs. The planning criteria consists of all the key inputs which need to be defined and set in SQL Governor® in order to automatically generate the optimized capacity plan. The planning criteria covers e.g. the following: the targeted lifespan of the new platform, the server hardware specifications for the new platform, threshold limits for target system RAM and CPU time series, their respective target SLA’s, and so forth.


www.sqlgovernor.com

Once the monitoring period is completed and the planning criteria has been set, SQL Governor® automatically generates an optimized capacity plan, which is based on the benchmark and trend-extrapolated time series data. It calculates the saving potential in terms of CPU cores by comparing the number of cores in the source platform and the new platform. SQL Governor® uses advanced time series analysis and machine learning in order to find out the optimal result out of the analysis for the instance-level consolidation. The server-level consolidation is a less automated process but can deliver great savings as well for the customer.

Select desired SQL Server target architectureIn this phase, the project stakeholder evaluates the costs and the pros and cons of the calculated architecture solutions we have presented and selects the best fit for their business case.

Epilogue

There are many things that need to be taken into account when it comes to SQL Server capacity planning. The more business critical your servers are, the more precise methodology and software is needed, in addition to the various DBA skills. Whether your data platform is small or large, before the actual capacity planning it is a good idea to diagnose the existing servers. Also you should gather the monitoring data from all the servers on server, instance, and database level with the minimum granularity of one hour, in order to get good baselines. It is also necessary to categorize the servers into different planning groups based on various business, criticality, and technical constraints. In the actual planning phase, remember to pay attention in the outlier data, such as service breaks, and eliminate those from the actual calculations. I hope you have found this book useful for your right-sizing and consolidation purposes. Rely on facts, not guesstimates!

www.sqlgovernor.com


Test our ROI calculatorEstimate how much you could save on your

SQL Server licensing costs!

www.sqlgovernor.com | [email protected]

https://www.sqlgovernor.com/en/calculator/?utm_source=ebook&utm_medium=PDF&utm_campaign=SQL%20Server%20migration%20planning



https://www.sqlgovernor.com/?utm_source=ebook&utm_medium=PDF&utm_campaign=SQL%20Server%20migration%20planning

mailto:sales%40sqlgovernor.com?subject=

Mastering Microsoft SQL Server consolidation planning

Documents

Transcript of Mastering Microsoft SQL Server consolidation planning