Migration from CRM 4 / CRM 2011 On-premise to CRM 2013 Online

We have been recently involved in a number of migrations of CRM 4 and CRM 2011 on-premise installations to CRM 2013 Online.
While we were responsible for customization migration and data migration before and had already performed migrations from CRM 4 to CRM 2011 in the past we tried to find the best way how to perform these migrations now for CRM 2013 and this post summarizes our experience. It outlines the tasks that need to be executed and refers to the tools and useful articles that can be utilized to accomplish these tasks.

Introduction in subject

First of all I’d like to provide some introduction in the migration subject for people who have not been involved in it. The whole migration can logically be split into 2 stages: customization migration and data migration.

Customization in MS CRM represents the database structure (what tables are used for data storage) and application code responsible for implementation of some business logic. If the business logic needs to work on the server then it is implemented in server plug-ins, custom workflow activities or workflows. If something should be done with user interface (ex. Some controls should be made hidden depending on certain conditions) then such logic is implemented using JavaScript.

While CRM online does not provide direct access to underlying database the entities customization (logically matched to tables in database) is usually easily migrated to CRM online by means of standard import / export processes even for different versions of CRM. However changes in API provided by Microsoft CRM SDK for different CRM versions vary substantially and real efforts are needed to make CRM code for CRM 4 working for CRM 2013.

Nevertheless after the customization has been migrated and all code was updated to work with the new API the next task is to move the data from the old database to the new located in cloud CRM online.
Below both these stages and the tasks that need to be accomplished are considered in details.

Migration process at a glance

1. Validation of CRM customization including JavaScript forms, plugins & custom activities code
2. Recreation of customer CRM in a test environment
3. Customization migration to CRM 2013 without any data
4. Basic data migration using CRM connector or third-party migration tool (including WaveAccess & C4CRM script engine)
5. Migration of standard entities not migrated by Microsoft CRM connector using our scripts and the script engine
6. Development of scripts for custom entities and migration of their data (In case of using CRM connector on the step 4)
7. Running data comparison reports between CRM 2011 and CRM 2013 to make sure that everything was migrated as required
8. Manual testing of the migrated system

Migration process in details

Validation of CRM customization including JavaScript forms,plug-ins & custom activities code

Here we use Custom Code Validation Tool for Microsoft Dynamics CRM 2011 and 2013 to validate client-side JavaScript code and Microsoft Dynamics CRM SDK Legacy Feature Check Tool mostly for quick validation if CRM4 SDK was used in the way that “old” endpoints were used and “old” interfaces were implemented by plugins and there were references to the “old” assemblies.

Note: In case of migration from CRM 2011 to CRM 2013 often the client-side code will be working in Internet Explorer but will not work in other browsers. However it might be not a problem for a customer and in this case after the validation is done, issues are identified and discussed and minimal corrections are done, we move to the next step.

Note: Detailed coverage of Custom Code Validation Tool for Microsoft Dynamics CRM 2011 and 2013 is done in this post.

Note: What is checked inside of Microsoft Dynamics CRM SDK Legacy Feature Check Tool can be found in this post.

Note: This first step can be performed on one of the customer installations (ideally not on the production, however it has no impact on the system and in case the customer does not have other environments, it can be installed on production). This step may be considered as a prerequisite for the project because it has a vital value for estimating efforts and budget respectively. After this step is done, a final decision has to be done about moving or not moving forward with the project.

The validation process takes several hours and the estimation process takes about 4-8 hours depending on the project size.

Recreation of customer CRM in a test environment

After the decision on the previous step is made, we recreate the customer environment locally. Having an environment close to identical to the customer’s one allows us to perform all preparation work without any CRM users affected. Here we request a CRM 2011 database for the appropriate CRM organization that we need to migrate. We restore the database into a new organization and make sure that functionality is working before we start making any upgrade related changes.

In case of it is necessary to migrate CRM 4 to CRM 2013 we make upgrade in 2 stages: first of all we upgrade it to CRM 2011 and after that follow to the process described in this post.

Customization migration to CRM 2013 without any data

After CRM functions correctly in our local environment we start eliminating issues identified on the first step.
This step includes the following activities:

1. Rewriting plug-ins code to use the new SDK API
2. Getting rid of old CRM Forms API
3. Providing JavaScript code that will repeat existing functionality but for other than Internet Explorer browsers (ex: XML document processing)
4. Dealing with Auto-save event that was introduced in new forms

The useful material about replacing typical “old” code snippets with the new one can be found here.
In respect to the Auto-save event it provides users with an experience similar to what Word type of editors do, so even if a user forgets to save data they will be saved automatically. In real life scenarios it creates problems for implementation partner who has to insert custom logic triggered by the save event. Right now there is no good answer for all situations. Basically there are two recommendations: the first one is to turn off the Auto-save option at all for the whole CRM organization on configuration level (http://blogs.msdn.com/b/crm/archive/2013/10/15/manage-auto-save.aspx) or to insert a piece of JavaScript code in each save event handler to programmatically cancel AutoSave event using new CRM forms API (http://blogs.msdn.com/b/crm/archive/2013/11/18/how-to-manage-auto-save-for-individual-forms-of-entity.aspx)

By the way at this step it might be possible to replace some of existing JavaScript code with business rules feature introduced in CRM 2013 that allows manipulating controls on the form via configuration instead of programming. You can find an overview of business rules usage here.

Also some plug-ins now might be replaced with synchronous real time workflows that simplify implementation and further maintenance of the up-to-date business logic by business users.

Both business rules and real time workflows are nice features but using them instead of an existing code is completely optional and can be skipped on this step.

The main deliverable of this step is a new environment with customer customization up and running.

Data migration options

Right now there are the following available options for data migrations

1. Download, configure and move substantial part of your data using free Microsoft Dynamics CRM 2011 Instance Adapter and move the rest part by writing custom utilities using SDK.
2. Purchase a 60 days license of Scribe Dynamics CRM Adapter
   (https://openmind.scribesoftware.com/releases/dynamics-crm-adapter-55) configure mapping between source and destination CRM entities in required sequence and move required data.At the end of November 2013 the license price is 1800 USD. Time required to configure mapping and to run the migration depends on the customization complexity and previous experience with Scribe and it might take from 20 up to 40 hours. A very brief overview can be found in our post here.
3. Engage Synchronicity that after receiving your database will move the data to your target CRM online organization using their own product Echo http://www.synchronicity-systems.co.uk/Echo/FAQs.
4. Engage WaveAccess that will perform migration utilizing Microsoft Dynamics CRM 2011 Instance Adapter and custom script engine that will process all entities and specific cases not supported by the adapter.

Basic data migration using Microsoft Dynamics CRM 2011 Instance Adapter

Microsoft Dynamics CRM 2011 Instance Adapter formally supports data migration from CRM 2011 to CRM 2013. http://www.microsoft.com/en-us/download/details.aspx?id=35385

By the way it is claimed that it supports even more than that such as permanent data synchronization. While usage of this tool is described in our separate post here for now I’d like to say that in order to use it you need to establish mapping between entities in source and destination CRM organizations using provided user interface and you should have all required users created in both organizations. The import of entities should respect the logical dependency between entities in the way that parent entities should be imported first and the child entities afterwards. This is not required, but it’s recommended to avoid extra time for additional passes.

However inevitably there are cyclic references and from our experience they are not processed completely correctly. You might face with a situation that the tool tries to reestablish relationships between entities 7 times and then it stops processing it.

Nevertheless using this tool you can migrate the data for the most of standard and custom entities. However currently the connector has issues with migrating marketing lists, activities and in particularly there are problems with activity parties as well as with attachments. An issue with many to many relationships is reported often as well.

Also we faced with a number of situations when the tool was able to import 90% of records for a specific entity and did not provide any reasonable explanation why it could not migrate the rest 10% of records.

Migration of standard entities not migrated by Microsoft CRM connector using custom engine scripts

As it was said after using the Microsoft Dynamics CRM 2011 Instance Adapter you might have imported most of your entities but not all of them and then there is an open question how to move the rest of data. One of the options is to explicitly write C# code that will retrieve data from the source CRM 2011 via CRM 2011 webservice calls and will insert data into the target CRM 2013, but it is a time consuming operation. Having faced with such a task a number years ago for migration from and to different versions of CRM we created a script engine that is able to parse special XML scripts having FetchXML like syntax and translate them into calls to CRM web services.

Below is an example of the script

<?xml version="1.0" encoding="utf-8" ?>
<operations>
<!-- Retrieve all marketing list members and save them into a local collection records2011. Here the datasource is specified in the “in” parameter, Entity attribute contains the required entity type Var attribute is used for defining a variable that will contain retrieved data. -->
<read in="crm2011" entity="listmember" var="records2011">
<!-- List attributes of the list members that will be retrieved from CRM 2011-->
<attr name="listmemberid" /><attr name="entityid" />
<attr name="entitytype" />
<attr name="listid" />
<attr name="ownerid" />
</read>
<!-- Save the whole number of retrieved records in the log file -->
<log>Total listmembers - {records2011.count}</log>
<!-- Iterate over the retrieved from CRM 2011 -->
<for var="record2011" in="records2011">
<sandbox>
<log>Listmember {record5.entitytype} : {record5.entityid.Id} : {record5.listid.Id}</log>
<!-- For each list member record in CRM 2011 try to find a matched list member record in CRM 2013 (with the same object ID, entity type and for the same marketing list-->
<read in="crm2013" entity="listmember" var="records2013">
<criteria>
<condition attr="entityid">{recors2011.entityid.Id}</condition/>
<condition attr="entitytype">{record2011.entitytype}</condition/>
<condition attr="listid">{record2011.listid.Id}</condition/>
</criteria>
</read>
<!-- If such a record is not found then we will create a new one for this list member and marketing list. Here there is a demonstration of the script engine to call .NET object methods that accept parameters from the script context-->
<if condition="{records2013.count eq 0}">
<log>Record not found. Adding a new listmember</log>
<set var="">{Crm5Utils.AddListMember('crm2013',record2011.listid.Id,record2011.entityid.Id)}</set>
</if>
<if condition="{records2013.count gt 0}">
<log>Record found</log>
</if>
</sandbox>
</for>
</operations>

Development of scripts for custom entities and migration of their data

While in many cases the mentioned CRM adapter will allow you to migrate the custom entities data as well there are some situations when custom logic needs to be implemented after retrieval from the source entity and before insert data into the target entity. In this case our scripts engine will allow us to implement this business logic. As it was shown above some of the implementation may be done as a piece of C# code when all the rest can be accomplished via script itself.

Running data comparison reports between CRM 4 & CRM 2011 and CRM 2013 to make sure that everything was migrated as required

After migration is done by any tool it is necessary to make sure that the number of records for each entity type is the same in source and target CRM. If there is a difference then it is necessary to determine the reason and to eliminate it.

Again our scripts engine performs queries to all entity types and produces appropriate comparison table containing for each entity type for each user the number of records owned by this user in CRM 2011 and the number of appropriate records owned in CRM 2013. In case differences are found, the same script finds the records that make this difference and after that it is possible to understand the reason, eliminate it and synchronize the data again.

Manual testing of the migrated system

After migration is complete it is still vitally important to go through all forms and to make sure that all screens functions properly. In case multiple browsers support is required the testing needs to be done under all browsers. To make sure that everything is covered, prepared test plans should be created and followed.

Summary

CRM migration is a rather complicated job. It includes a number of steps and requires knowledge of several tools. Not all tools are perfect and mature enough. However everything is doable. The customization migration requires more efforts and is less predictable than data migration. Still the data migration also might be a challenge for large CRM installations with big data volumes. However there are a number of companies who provide either products to make part of the job or even full-packaged services to have things done.
If you are an IT company and need to migrate CRM for some of your customers and do not want to spend your resources on migration tasks please consider engagement of WaveAccess as an outsourcing option.
If you are not an IT company and would like to have a local partner please contact Consulting4CRM http://www.proserv4crm.com

Let us know if you have any questions and we will try to help you!