Limitations, Performance and Suggested Workarounds...

© Copyright Metalogix International GmbH, 2012 – 2014 of 14 Metalogix Software Corporation. All rights reserved. metalogix.com

Limitations, Performance and Suggested

Workarounds using Content Matrix for migrations

to SharePoint Online (Office 365)

Product Version: 7.2.0.1

Document Release Date: 12/18/2014


Contents Overview ............................................................................................................................................................. 3

Limitations in SPO Migrations .................................................................................................................... 3

Limited Application Programming Interfaces available for SPO .............................................. 3

Limitations and Factors that Impact Performance of Migrations to SPO ............................. 5

Performance Expectations ............................................................................................................................ 7

Recommendations .......................................................................................................................................... 8

Use Azure Based Virtual Machines ....................................................................................................... 8

To achieve performance levels of 1GB/hour or higher: ............................................................. 12

Gradual Migrations .................................................................................................................................. 13

Disabling the ‘Following Content’ Feature ...................................................................................... 13

Mapping Domain users to Claims Based User Naming Conventions .................................. 14

Improvements and Additions to this Document .......................................................................... 14


Overview

Migrations to Office 365 (multi-tenant) SharePoint Online (referenced as SPO in this

document), have a number of significant limitations when compared to migrations from

on-premises to on-premises SharePoint instances (whether across a LAN or a WAN), or

when compared to migration to other dedicated hosted offerings such as Office 365-D

(dedicated) SharePoint Online targets, Azure, Amazon Web Services, Rackspace, and

others.

This document explains why those limitations exist, and helps prepare Content Matrix

clients to plan a migration of over 100GB to SPO. This includes:

Listing and explaining the limitations and their significance on migrations

Ensuring that customers allocate the appropriate time and consideration

required when migrating their on-premises farms to SPO.

Discussing workarounds and methodologies to overcome those limitations.

This document is essential reading for anyone migrating more than 100GB to SPO.

Limitations in SPO Migrations

Limited Application Programming Interfaces available for SPO

Content Matrix uses a number of SharePoint Application Programming Interfaces (API’s)

to optimize migrations based on the migration scenario involved. Migrations to on-

premises and private cloud (e.g. Azure/Amazon/O365-D) implementations of SharePoint

support the use of the full SharePoint Server Object Model, the richest API available for

SharePoint, or a thin web services layer that exposes the SharePoint Server Object Model

(Metalogix Extensions Web Services/MEWS), for both reading, and writing SharePoint

Content.


However, due to the multi-tenant nature of SPO, Microsoft has to ensure that no single

tenant can cause other tenants’ environments to break. In light of that, Microsoft cannot

expose the full SharePoint Object Model in SPO. They currently expose three, relatively

limited API’s that are useful for migrations:

The Client Side Object Model (CSOM)

The Native Web Services (NWS) API

The REST based interfaces to SharePoint Online

SharePoint Online Remote PowerShell

Office 365 Remote PowerShell

Each of these has several limitations when compared to the full SharePoint Server Object

Model, and these limitations have an impact not only on what is possible with migrations

of on-premises SharePoint and other content to SPO, but also on performance of

migrations to SPO. Microsoft continues to expand these APIs, and it is likely that many

but not all of these limitations will be removed over time.

The most important of the functional limitations that impact migrations are:

Limitation Impact on migrations

Inability to preserve Item IDs in lists 1. Migrations typically allow end users to continue working on the source farm until the migration is complete. Once that is complete, Admins put the source farm in read-only mode, and can create an Incremental Copy, to copy only deltas since the full migration, and then launch the new version of SharePoint.

2. Lists with dependencies on other lists such as Lookup columns rely on item IDs in the Lookup lists.

3. Because the CSOM does not support retaining the Item ID of list items, Content Matrix has to use an ID preservation mechanism known as ID Stuffing that is time consuming (between 17% and 24% impact on performance, and does not work for Blog Site entries because these by default have 1 item already in place on Office 365.

Copying MySites does not yet include

User Profile information (CSOM and

1. With Content Matrix 7.2, Admins will have to connect to each OneDrive (For Business) Site Collection separately after MySites are migrated to OneDrive for


Content Matrix). This will also be resolved

in Content Matrix 7.3

Business. This will be resolved in a 7.2 Hotfix.

2. Admins cannot copy MySite profile information. This will be resolved in either a Content Matrix 7.2 Hotfix, or Content Matrix 7.3.

Versioning limitations in the CSOM API

1. Authorship information for rejected versions in a document library with approval is lost.

2. Authorship/last modified information for minor versions changes to that of migrating user

Nintex workflows cannot yet be migrated

to Office 365 since Nintex does not yet

have an SPO offering that is equivalent to

their on-premises offering due to the

same CSOM limitations.

1. Nintex workflows have to be recreated using Nintex’ SPO offering to the best extent possible.

Limitations and Factors that Impact Performance of Migrations to

SPO

SPO is on the open Internet, and is a multi-tenant environment in which multiple

Microsoft clients (up to 10,000) are hosted on a single SharePoint Server Farm. In light of

that, Microsoft uses a number of mechanisms to protect SPO customers’ environments,

and the integrity of these Server Farms.

Based on Metalogix performance benchmarking, there is an impact of between 40% and

45% on the performance of migrations to SPO because of some of these necessary

protection mechanisms. Our discussions with the SPO product group, and our

experience with numerous customers who have migrated large farms to SPO have shown

that the following are some of the known factors that impact this performance:


Mechanism Used Impact on migrations

User and Tenant based throttling – this

ensures that any single user or tenant

cannot perform so many simultaneous

operations on their SPO environment,

that these would cause performance

issues for other tenants. This article while

written for SP2010, still applies to SP2013

Large or complex migration jobs can be

cut off mid-migration requiring Content

Matrix to pause jobs, and then continue

once the throttling has been reduced.

Farm based throttling – If a Microsoft

SPO farm becomes unhealthy due to

extreme levels of activity on that farm,

Microsoft will be throttling migrations to

SPO, and will not permit migrations to

continue until the farm’s health returns to

normal.

Any migration job can be throttled at any

given time of the day (usually during peak

hours for the region of that farm) if the

farm is too busy, causing Content Matrix

to have to pause jobs, and then continue

once the throttling has been reduced.

This makes migration performance

extremely unpredictable, and variable

based on the time of day, day of the

week, and other variables completely

beyond Metalogix or other migration

vendors’ control.

Virus scanning SPO requires stringent virus scanning in

order to ensure that all tenants on the

shared server farms are protected. This

virus scanning slows down migrations as

each document migrated has to be

scanned.

Hardware based load balancing -

determines which Web Front End (WFE)

Server in the SharePoint Server Farm to

route incoming traffic to, based on how

busy any given WFE is at that time.

While this routes Content Matrix traffic to

the WFE with the lowest workload when a

migration begins, Content Matrix is

locked to that WFE for the duration of a

migration job. If the Server Health Score

of that server increases (higher is worse),

to 7 (0-10 range), Content Matrix pauses

jobs, and waits for that Server’s health

score to get back to 4.

This can slow down migrations that are

large (large amount of content), or

complex (many items with multiple

metadata fields, and/or many versions).

http://blogs.msdn.com/b/besidethepoint/archive/2010/09/13/http-request-throttling-in-sharepoint-2010.aspx


Third-party commercial denial-of-

service monitoring platform for

monitoring and throttling capabilities (see

Microsoft’s Security/Trust page for Office

365).

While we have no indication that this has

throttled migrations, it is conceivable that

a migration will appear to SPO as a Denial

of Service attack, requiring users to restart

the job, or break the job up into smaller

parts.

Microsoft is working towards ways of allowing migration vendors to improve

performance of migrations, and is actively advising vendors on techniques to use to

reduce throttling. In addition, Microsoft is working on figuring out workarounds that

could potentially allow migration vendors to improve the performance of migrations

when clients using those vendors indicate that they are in the process of migrating to

SPO. However, at this time, Microsoft has not provided a date for the release of these

workarounds.

Performance Expectations

For comparison purposes, benchmarking work that we have carried out for on-premises

migrations, including data from large scale migrations carried out by our customers,

results in a range of migration speeds from 2GB/hour to 100GB/hour depending on

hardware used, network latency, complexity of the workload and other factors. WAN

based migrations tend to be significantly slower than LAN based migrations, but still

average from 2GB to 5GB/hour.

Both our internal benchmarking and data from customers has shown that for SPO

migrations with a single machine averaged between 200MB and 550MB/hour

depending on the workload, the time of day, how busy the farm is, and numerous

other factors that are outside of our control. Production migrations by customers

from Azure, with multiple parallel machines, and multiple usernames and passwords

range from 400MB/hour to 10GB/hour. Achieving 10GB/hour requires the use many

machines on Azure (in the same region of the country as the SPO farm), and this rate

cannot be achieved consistently at this stage. This technique works in some SPO farms,

but not in others. Still, with parallel migration operations from Azure VMs, speeds can

get to 2GB to 4GB/hour.

http://office.microsoft.com/en-us/business/microsoft-business-security-solutions-FX103045813.aspx


Recommendations

Use Azure Based Virtual Machines

As mentioned above, we have found that migrations from Azure Virtual Machines (VM)

in the same region as the SPO instance being migrated to, increase performance of the

migration. The process that we recommend for any migration other than those from

SP2013 to SP2013 Online is:

1. Determine what region your Office 365 tenancy is in. You can do this by

contacting your Microsoft TAM (Technical Account Manager) and asking what

region it is in, or by first pinging that location:

In the ping results, you can see that ping gives you a prodnet number (17-37 in

the above screenshot). If you don’t have a TAM, MS Support should be able to

tell you what region (East, Central, West, etc. your tenancy is in based on this

number. If not, please contact Metalogix).

2. Create an Azure VM in the same region as your O365 tenancy with Windows

Server 2008 or 2012, and SQL Server 2008 or 2012.

3. Using SQL Server Management Studio Back up your source Content DBs to .bak

files:


4. Use ftp, or other mechanism to copy the .bak file to your Azure VM. You can ship

a physical hard disk to the Azure team if needed. For more information how to

ship a physical hard disk, please see Microsoft’s guidance here.

5. In your Azure VM, restore the .bak files to a new databases:

http://www.windowsazure.com/en-us/documentation/articles/storage-import-export-service/


6. Once these databases have been restored, give your Azure VM admin dbo access

to each of these databases

7. Install Content Matrix SharePoint Edition Console on the Azure VM (you can

install the console on multiple other Azure VMs that do not have SQL Server on

them, but must be in the same internal network in order to enable parallel, multi

username actions).

8. In Content Matrix, use the Connect to SharePoint Database option:

9. Enter the name of the database server (at this point most likely (local))

10. Choose the Content DB you just restored, and the Site Collection within that

Content DB that you would like to connect to.

11. Migrate content as you normally would with Content Matrix

12. For any operations that require farm level connections (such as copying the

Managed Metadata Services Term Store), migrate that from the source farm

instead of Azure.

13. For the final Incremental Copy, there is no need to re-backup your Content DBs

and restore them in Azure. You can perform and incremental copy from the

actual source as long as you selected ‘Preserve Item ID’ in all List copy or Site

copy operations that include lists:


14. Anticipate throttling – SPO will throttle migration jobs on occasion whether it’s

because of your migration activity, or other activity on the farm in which your

tenancy is located. Content Matrix monitors the Server Health Score returned

from SPO. This health score has now been modified to include not only CPU and

RAM resources for the WFE you’re connecting to, but also the overall farm health.

15. Content Matrix will pause jobs gracefully once the Server Health Score reaches 9,

and restart the job once the health score returns to 4 (a health score of 0 is best,

with 10 being enough to have a job suddenly cut off).

16. If you would like to see health score logging or modify current values, open the

file C:\Users\<username> \AppData\Roaming\Metalogix\UserSettings.xml.


17. In that file, you should see entries such as:

<XmlableEntry>

<Key>ServerHealthLoggingInterval</Key>

<Value>300000</Value>

</XmlableEntry>

<XmlableEntry>

<Key>ServerHealthLoggingEnabled</Key>

<Value>False</Value>

</XmlableEntry>

<XmlableEntry>

<Key>ServerHealthScorePauseValue</Key>

<Value>9</Value>

</XmlableEntry>

</XmlableTable>

18. If you modify the element ServerHealthLoggingEnabled to True, that will log the

server health score in your job log.

19. If you would like to modify the value at which migration restarts, modify (or add)

the parameter ServerHealthScoreRestartValue. You cannot restart at a value

higher than 5 since this is known to cause migration issues.

To achieve performance levels of 1GB/hour or higher:

1. Create multiple migration users in your SPO tenancy, each with Global Admin

permissions.

2. Spin up multiple Azure VMs in the same domain as the SQL Server mentioned

above.

3. Ensure that the SQL Server VM has sufficient RAM to deal with multiple

simultaneous jobs.

4. Plan your migration so that each VM is migrating several jobs at a time, with

larger Site Collections or Sites being migrated on separate machines (ie. Balance

the workload across multiple VMs).

5. To achieve optimal performance, first migrate the structure of your SharePoint

farm, and then the content.

6. Use PowerShell to control migration jobs. This allows you to programmatically

control migrations, check against an Excel file, DB, or SharePoint list for a list of

jobs that are still required, and perform migrations in the order you would like

without manning the consoles frequently.

7. PowerShell also uses far fewer UI resources, and therefore results in faster

migrations.


Gradual Migrations

Due to the migration performance, Metalogix recommends taking a gradual migration

approach, which may involve migrating one division at a time, and going live with that

division. This can be very beneficial as it allows you to assess the impact on your business

users and your helpdesk after moving one group of people to a new user interface, and

use focus groups to determine which features you would like to implement as you

gradually migrate the business to SPO.

Disabling the ‘Following Content’ Feature

SPO sites (but not site collections), have a feature called: ‘Following Content’

This feature is enabled by default when a new site is created. When it is disabled,

performance of Site copies improves by approximately 20% to 25% for document

heavy workloads.

Since Content Matrix does not currently give you the option of turning off only a

particular feature on the target site, the only way to achieve this currently is to

specify in the Site Options tab of the Paste Configuration screen to ‘Clear Default

Features on Target’:


Future versions of Content Matrix will support the selection feature by feature of

which source and which target features you would like enabled on the target

after migration.

Mapping Domain users to Claims Based User Naming Conventions

Content Matrix automatically maps domain users from source to target domains if

domain mapping is in place. For example, if the source domain is metalogix1, and the

target domain is metalogix2, setting up a Domain Mapping in the ‘Edit Global Mappings’

screen accessible from the Settings menu (see our online help on this), allows used to

setup a domain mapping between metalogix1 and metalogix2.

Once that has been setup, Content Matrix will automatically map metalogix1\jdoe to

metalogix2\jdoe, and so on. In addition, for migrations to Office 365 or other claims

based authentication providers, if the source domain is the same name as the email

address format of usernames in Office 365 scenarios using Active Directory Federation

Services (ADFS), a typical source domain name such as metalogix1\jdoe would be

automatically mapped to i:0#.f|membership|[email protected].

However, if you have multiple domains on your source farm, or domains that are not

named the same as the @<companyname>.com email address that the user would use

to login to Office 365, you can still create the appropriate mapping.

There are two ways to address this. These are described in detail including a sample csv

file and PowerShell scripts that can be used to make the process easy in a document

titled “Content Matrix User Mapping from AD to Claims including O365”, which is

included in the installer zip file for Content Matrix. This document is also available in the

Metalogix Support Knowledge Base.

Improvements and Additions to this Document

Metalogix will be adding more detailed examples to this document over time.

http://www.metalogix.com/help/Content%20Matrix%20Console/SharePoint%20Edition/003_Configuration/010_GlobalUserMappings.htm

http://www.metalogix.com/Support/Knowledge-Base.aspx

Limitations, Performance and Suggested Workarounds...

Documents

Transcript of Limitations, Performance and Suggested Workarounds...