Data Migrations - plan to maintain data integrity
Data migration is the process of transferring data from one system to another.
Common events that may trigger a requirement for data migration include:
The shift to cloud
Application modernisation and re-platforming
Data centre consolidation or migration
In many cases, project teams focus on functional requirements only, with data requirements parked until go-live.
This oversight can mean significant cost and time overruns as data issues often appear at an advanced stage in many projects, forcing redesigns and rework.
The data stream should receive equal attention and priority to all other project streams.
PLANNING FOR YOUR DATA MIGRATION
1. Assess your data.
A common mistake is to assume that existing data is well understood and also that it will be compatible with the data structures of the intended target environment. Before migrating data we recommend verifying any assumptions to ensure that you have a clear understanding of what you will be migrating, potential risks that must be managed before they become issues, and what functional changes may need to be planned for to ensure that data will fit into the target system.
Assessments should look at three key areas:
Data Structure: Confirm actual table and column structures (of source and target systems) to confirm data mappings and identify functional disconnects. Tools such as Data360 and Safyr can add tremendous value at this stage
Data Flows: Understand the business logic that may be built into the data layer of your source system, to ensure that this can be planned for in the target. MANTA's unified data lineage platform is a good choice to get this understanding.
Data Content: Profiling the data can provide clarity on issues such as missing, duplicated or misfielded data, identify disconnects between source and target data types, and give an indication of where manual data remediation tasks may be required.
In our experience, an iterative development approach is most effective.
Using the inputs from the assessment we:
- Define and develop a staging environment for the source data
- Design and define a landing environment that will be compatible with the target.
- Develop data integration and data quality processes to move the source data to the landing environment, addressing data risks and issues along the way. Data360 allows us to build agile data pipelines to quickly connect, transform and move data, and repeat the process to keep source data synchronised with our landing area.
- Generate exception reports where manual data remediation is required in the source e.g. for missing data.
Following the initial development phase, we repeat the process focussing our attention on the new landing area. This means that we are identifying data risks that were not addressed during earlier phases. We can repeat this process as often as required until approval is given to migrate the clean data into the target.
This approach is designed to manage data risks that may otherwise derail your entire project, and to ensure that data migrated is, if possible, of higher quality than the original source.
Reuse your investment
The last thing you need is to invest millions in new systems only to find that the data quality issues that plagued you previously have resurfaced within months of your go-live date.
Our approach allows new systems to reuse business processes and validations built to migrate data, ensuring you get the business returns you expect from your system.