Introduction to the Data Migration Framework in Microsoft...

1 AX 2012 Data Migration Framework

Introduction to the Data Migration Framework in Microsoft Dynamics

by Ruben Barron

Senior Consultant - Blue Horseshoe Solutions.


Table of Contents Intro to the Data Migration Framework (DMF) ........................................................................................ 3

DMF Process Overview ............................................................................................................................. 3

Prepare Source Data ............................................................................................................................. 3

Define type of AX entity to be used ...................................................................................................... 3

Fields in the Source file are mapped from the source file to the staging table .................................... 4

Transferring Data to the Staging Table ................................................................................................. 4

Data transferred from the Staging tables to the final AX Target destination. ...................................... 4

DMF Components ..................................................................................................................................... 4

Accessing the DMF tool. ........................................................................................................................ 4

Set Up – Target Entities ......................................................................................................................... 5

Set Up - Source Data Format ................................................................................................................ 6

Set Up – File Location (data migration framework parameters) .......................................................... 8

Processing Groups ..................................................................................................................................... 9

Entities .................................................................................................................................................. 9

Moving Data from Source Files to Staging Tables............................................................................... 14

Moving Data from staging table to the Final AX destination. ............................................................. 16


Intro to the Data Migration Framework (DMF) For any organization embarking into a new MS Dynamics AX 2012(AX 2012) implementation loading

legacy data from difference sources into AX2012 is a critical component to have a good start and a

successful implementation. Microsoft introduced the new Data Migration Framework (DMF) as an

extension of AX 2012 to address the challenge of loading data to AX 2012. We consider that the DMF

tool represents a great step forward for data upload and with great potential that can be used to

complete data migration for the most common data entities needed for an implementation as well as a

framework that can be customized to load other required entities.

DMF Process Overview The DMF process can be summarized in the following diagram and includes preparing source data,

defining the type and order of AX entities to load, mapping the source file to the staging table,

transferring from the source file to the staging table and finally transferring the data to the final

destination in AX.

Data Migration Process

Prepare Source Data

Define type of AX entity to be used

Mapping and Validation of

Source File fields to Staging AX fields

Transferring Data from Source file to the Staging Table

Transfer of data from Staging to

target AX destination.

· Prepare source files to be uploaded.

· Define AX entity(entities) to be used for upload.

· Example of entities include Customers, Vendors, Sales Orders, Journal Entries, etc.

· Mapping and validation before data is transferred from source file to staging

· Transfer data from source file to Staging table.

· Data can be validated at this stage too.

· Transfer data from Staging to final target AX destination.

· AX records are created or updated

DMF Process Overview

Prepare Source Data

During this step, source data files are created and prepared to be uploaded to the AX application. A

typical source file will have a header row with field names identifying each data column. The number of

fields for each file will depend on the requirements for data upload. Thus it is important to note that

some of these field names are required for each entity. The DMF version only supports flat files. Future

versions of the DMF tool will support data from ODBC and AX destinations.

Define type of AX entity to be used

During this step we define the entity or entities that will be loaded to AX. These DMF entities are

predefined relationships that come in the application and include common AX entities such as

customers, vendors, products, etc. and available AX fields.


Fields in the Source file are mapped from the source file to the staging table

During the mapping stage, the DMF tool will allow you to map each field from the source file to an AX

field in the in the staging table for the corresponding DMF entity. The DMF tool uses the field name to

map them against the available fields in the AX Staging table. Users have the ability to review and

manually map these fields.

Transferring Data to the Staging Table

Once fields from the source file have been mapped to AX fields in the staging table data should be ready

to be transferred to the staging table. This staging table acts as a temporary area and intermediate

location before data is transferred to the final AX destination. While the data resides on the staging

table, the DMF can apply some basic validation for referential data integrity.

Data transferred from the Staging tables to the final AX Target destination.

Once data has been reviewed and validated in the stating table, the data is then moved to the target AX

entity and AX records are created. If data is reloaded the DMF tool will try to update the AX fields.

DMF Components

Accessing the DMF tool.

After installation, the DMF tool can be accessed from the AX client and shows as a module in the AX

application. You will see a common, setup and periodic areas as part of the DMF module.

Data Migration Framework in AX 2012


Set Up – Target Entities

There is some setup that needs to be configured prior to use the DMF tool.

Target Entities - The DMF tool comes with a set of predefined entities that can be used to load the most

common entities in AX. The following list includes the entities supported out of the box:

· Bill of materials (BOM)

· Contacts

· Contact addresses

· Customer

· Customer addresses

· Dimension

· Employees

· Employee addresses

· Inventory journal

· Ledger balances

· Main account

· Open invoices (accounts payable)

· Open invoices (accounts receivable)

· Open sales orders

· Open purchase orders

· Price disc (agreements)

· Product

· Project

· Unit of measure

· Vendor

· Vendor address

The list of available entities can be accessed from the DMF> setup > Entities area. These DMF entities

come already preconfigured and can be defined as a mapping relationship between the fields in the

staging table and the target destination. These DMF entities also include procedures to move data

from the staging table to the final destination creating objects and fields in AX. It is noteworthy to say

that these entities can be used to create new custom entities with additional fields or logic. The DMF

tool also comes with a custom entity wizard to create a DMF entity from scratch.


Set up – Data Entity Sample

Set Up - Source Data Format

Source format allow the DMF tool to identity the type of source data that will be imported into AX. In

the source format, a user specifies the different parameters that define the source file to be imported.

Some of the parameters that can be specified include the type of source(file, ODBC, AX table), file

format, first row header, row column delimiters and few other described in the following table. A user

can define more than one source format to be used in the DMF upload process based on specific needs.

Example a source format can be created for flat files that are comma delimited. Another source format

can be created for flat files with fixed length and a different source format can be created for data that

is planned to be imported from an ODBC connection.


Setup - Source Data Formats

FIELD DESCRIPTION

GENERAL TAB

Source Name Identifies the different source that can be configured in the application.

Type Types include File, AX and ODBC. DMF currently supports only Flat files. Future versions of the DMF will support AX tables and ODBC connections.

File Format Options include character delimited or fixed length. Character delimited includes characters such as commas, semicolon, pipes, etc.

First Row Header Flag to indicate if firs row of the file includes a header row.

Row Column Delimiter Indicates which characters will be used to determine end of the line for the source file. Defaults to {CR}{LF)

Text Qualifier Indicates character to encase text and commas. Example “8230 E Main Ave, Suite A“

Regional Settings Regional settings for file. We normally default this to code 1252 for English. A very large selection of regional setting come with the application to cover all world regions/languages depending on customer requirements. It also includes list for Unicode and other specific formats.


Multiple Value Separator Delimited used to indicate multiple entries. Example multiple addresses, multiple email, multiple phones 317-509-90900 ; 508-999-1212

Table 1.1 Setup – Source Format General Setup

There is an Application tab that is part of the Source Format configuration and that include several

setting to configure dimensions and how data is going to be processed.

Setup – Source format – Application tab

Set Up – File Location (data migration framework parameters)

There is a required configuration that includes defining a shared folder in which files will be located for

the DMF tool. The DMF admin user needs to have admin access to this folder in order for the DMF to

properly work.

Setup – DMF Parameters (shared directory)


Processing Groups

Processing Groups are used in DMF as the main driver that will allow users to configure entities, move

data to the stating table, validate data and copy data from the staging to the final AX destination. These

processing groups can be reused to reimport data and also keep a history log of transactions and errors.

Processing Group – Creating a Processing Group

Entities

After creating a processing group, the first thing to configure is an entity. These entities are used to

indicates what DMF entity to be used( example Vendor, customer) for processing, order in which

entities will be processed if more than one is selected, data source to use, source file to use for

processing and if AX business logic will be taking into consideration when uploading. The user will also

use this entity to map and validate the source fields against the staging AX fields.

Settings for Entity configuration

New Entity: A user will select the New menu from the processing group to create a new entity or

entities.

Sequence: Indicates the order in which entity or entities will be processed to load data from source to

staging area. You can have more than one entity to be processed. An example could be to load

vendors, products and customers in one single load. While this is option is available, we prefer to keep

it simple and only run one entity per processing group to keep things simple to debug and also allow the

processing group to be reused for future loads.

Entity Name: The entity name is used to select the DMF entity that will be used for processing data.

This entity contains the mapping relationship with all the available fields that will be mapped from the

source file to the staging tables.


Source Format: This source data formats are configured in the setup and specifies the type of source

data that will be loaded.

Run Business Logic: There are two setting that can be used to enable the check for AX business logic.

Sample File Path: This path is used to indicate the location of the source file to be used for the mapping

and validation of source data. Note: If a large file is used you can use just a sample of data to do

perform the mapping and validation.

Processing Groups – Entity Configuration

Generate Source Mapping: The next step is to map the source file against the staging table. If you use

AX column names as your header the DMF tool will automatically try to do the mapping. If a different

name is used, a user will have to manually configure the mappings. It is probably recommended to

generate the source template file using the DMF tool tp speed up the mapping process. Still the user

will need to review and approve the mapping. Use the “Generate Source File” to create a template with

AX fields.


Modify Source Mapping: During this stage the user can review and modify the mappings from the

source to the staging area. If any of the fields on the Source file is not mapped, a user has the option to

manually link the field to desired field on the staging table using the graphical interface or on the table

view.

Processing Group – Entity Mapping Source to Staging


Validate: After the mappings have been reviewed, the user can then validate. The validation process

ensures that all fields in the source file are correctly mapped to the staging table. If an error is

encountered a user can go back to the mappings and make corrections in the mappings.

Processing Group – Validation

View Target Mapping: This option can be used to validate the mappings from the staging to final Ax

target destination. Please note that the staging only contains fields from the source file. Fields that

are not used are nor displayed and will not be used in your migration.


Processing Group – View Target Mapping

Preview Source File: Once mappings have been validated, a user has the option to preview the source

file before the file is copied to the Staging Table. This is a good check that show how data aligns with

each AX field after mapping. Please note that is not data validation during this stage..

Processing Group – Preview data source file.


Moving Data from Source Files to Staging Tables

Once data has been mapped and validated for the Entities, the data is ready to be transferred from the

source file to the staging table.

Get Staging Tables: Data is ready to be copied to the staging table. A user selects the Get Staging Table

menu option to start the process. During this stage the user has the option to run the job on demand or

batch it. This batch option is probably

useful when processing large files.

Processing Groups – Moving data to Staging table


You will see a message indicating that the data has been copied to the staging table. If errors were

encountered during the load, these errors can be viewed in the log file in the execution history of the

processing group menu.

Processing Group – Copying data to Staging


Moving Data from staging table to the Final AX destination.

Once data has been transferred to the staging table and validated, the data is ready to be transferred to

the final destination in AX by selecting the “Copy data to target.” A message will be displayed informing

of the processing and creating the AX records if the processing was successful. If records are reloaded,

the AX records will be updated.

Processing Group – Copying Data from Staging Table to Target


Processing Group – Created Vendor Records.

Introduction to the Data Migration Framework in Microsoft...

Documents

Transcript of Introduction to the Data Migration Framework in Microsoft...