Data migration is widely recognized as a complex process that demands meticulous planning from the outset. This article delves into the critical areas that require careful consideration during planning and execution of data migration.
In this article, we will thoroughly explore the key aspects that demand attention while undergoing data migration. By addressing these aspects systematically, organizations can enhance the efficiency and success of their data migration efforts.
Need for Migration: Frequent requirements around Data Migration involve replacing the legacy system into a new application.
Tools & Technologies used: There are several options available for Data Migration, like a console utility built on C#, using SSIS packages, using Azure Data Factory pipelines. There are some tools available in market like Kingsway Soft and Scribe which provide a wrapper around the ETL to perform migration by following some easy steps.
Considerations for Data Migration Approach:
| Components | Focus Area |
| Data Migration – Complexity Assessment | You can begin by asking a few questions to analyze the complexity of the migration, such as: – What will be the size of the overall migration? – How many tables and columns will be migrated? – What will be the mode of data transfer? – How will you access and connect to the source data source? – How the data extraction will work? – Will there be any staging in between, or do you plan to map the source to the destination directly? – What level of data quality do you expect for data transfer, and will it require any cleanup or transformation algorithms? – What Integration Platform you are suggesting make sure, that chosen platform seamlessly integrate with the existing architecture? |
| Mapping | Begin the data migration process by using a mapping sheet between the source and destination tables and columns. Pay special attention to the data type and size of each column as migration failures frequently occur due to discrepancies in these aspects. Additionally, consider the data type and size of the source data in different environments such as Dev, Test, and Prod. Mismatches in source database environments may arise from poor design decisions or direct changes made to instances. To address these potential challenges, it is strongly recommended to ensure that the data type and size in the mapping sheet aligns with those used in the Production instance. By following this recommendation and matching the data type and size from the source file, you can significantly reduce the risk of migration failures caused by discrepancies. |
| Identify Table Sequence | The sequence of tables can be determined by analyzing the relationships and dependencies within the database. In case of Dynamics 365 CE, a helpful article by Kingsway-Soft provides insights for data migration, guiding the order in which tables should be migrated to ensure a smooth process. |
| Identify Record Sequence – Parent Records (Self Relationships) | To optimize the data migration process, in case of self-relationships, focus on the particular table by prioritizing the creation of parent records first, followed by child records. In the first iteration, ensure a full load without setting up parent. After completing the first iteration, update the parent relationship/field. |
| Record Ownerships | When migrating data from Dynamics 365 CE, it is essential to determine the ownership of each record. In Dynamics, records can be owned by either a “user” or a “team.” During the migration process, you must set the value for ownership. However, if you are migrating data from a different source into Dynamics 365 CE, owner column may not be present in the source table. In such cases, it is crucial to collaborate with business users and establish a strategy for assigning ownership of the records. The assignment of ownership should be driven by the Business Unit and Security Model structure within Dynamics, you can check here for details. |
| Unknown Owners | During the ownership assignment process, there might be cases where owners are unavailable due to inactive user accounts or previous owners leaving the organization. In such situations, identify these inactive owners, and collaborate with business stakeholders to define new suitable owners. If reassigning ownership is not an immediate priority, consider assigning such records to the Admin user or another placeholder user temporarily. This temporary assignment allows you to proceed with the migration smoothly. Later, you can use this designated placeholder user to reassign ownership to the appropriate owners when this information becomes available. |
| Duplicate Records | While conducting data migration, it is advisable to implement a duplicate detection policy to ensure that irrelevant or junk data is not migrated from the source system. It is essential to carefully formulate the duplicate detection process, particularly for core entities, to avoid duplication of data in the destination system. |
| Master Data, Data Transformation and Change Tracking | In each application, there will be some master data to ensure consistency, such as country, city, departments, etc. When designing the migration process, it is crucial to discuss a well-defined plan for identifying and mapping these columns. Sometimes, transformations may be necessary between the source and destination, like mapping “United States” to “USA.” These transformations should be implemented within the Migration Platform. Additionally, it is essential to establish a Change Management to ensure that any changes made to the master data in either the source or destination systems are synchronized between the two systems. This synchronization helps maintain data integrity and consistency throughout the migration process |
| SQL Server Logs | It has been noticed that during migration in a low-space database server with enabled SQL logs, the process can encounter issues in the Production environment. To mitigate this problem, it is recommended to disable the SQL log while performing the migration. |
| GUIDs retainment | If retaining the GUIDs is not restricted, consider the option of maintaining similar GUIDs for the new environment. This approach can help prevent lookup-related issues in Dynamics 365 CE during Data Migration |
| Cleaning up Columns | Though not critical, it is recommended to examine the source database for underutilized or nominal columns. Once identified, engage stakeholders in discussions to consider skipping such data during migration to avoid unnecessary or junk data |
| Auditing | Engage stakeholders to understand auditing requirements during the migration in Production. Generally, keep auditing disabled for large volumes. Enable auditing selectively for specific tables and columns to meet compliance needs |
| Custom Components like Plugins and Workflows | When executing migration on large volumes, it is essential to determine which custom code and components, such as Plugins, workflows, and Power Automate, should be active during the migration process. To optimize performance, enable only specific components that are necessary for the migration, while disabling others that may not be crucial for the data transfer |
| Cleanup Approach | Engage stakeholders in discussions regarding the possibility of data cleanup during the migration process from Dynamics 365 CE to Dynamics 365 CE. This collaborative approach aims to identify and eliminate junk and unnecessary data, ensuring a streamlined and efficient data migration. read here for detail approach regarding cleanup. |
| Configurable exception logging | Create the integration in a way that allows you to have control over exception and warning messages. Provide the option to turn off or on these messages as needed. This flexibility will help improve the overall migration process and reduce migration time by enabling you to manage exceptions and warnings according to specific requirements. |
| Data Validation & Testing | Make sure, you allocate dedicated time for data validations and testing, since Data Migration testing can be complex, you can think of automating migration testing by creating utility which can be configurable to check and map specific volumes or loads. |
| Monitor Load Speed and Predict Production Load | Analyze the load speed, for example per minute record loading and plan for Production load timeline. |
| Downtime | Estimate the downtime on business operations during the migration process. You can plan for off-office hours migration if the application performance is low during migration. |
By proactively addressing these aspects, the data migration can be executed smoothly and with minimal disruption to the applications.
Additionally, you can explore valuable tips for optimizing data migration time here.
decodeDynamics 
Leave a comment