3 Essentials For Migrating Your Data Integration (ETL) Platform

1) Does Migrating the Data Integration or ETL Platform to Something Different Add Value?

When looking at migrating your Data Integration or ETL platform to another, you need to really look at the WHY behind it.

It’s not just the platform that adds or takes away value, it’s the environment it operates in that influences the platform largely.

It could be that the custom legacy code to update four different databases is insufficient to handle your increased data warehouse (DW) & business intelligence (BI) needs. Or you might be missing new platform feature sets like Data Profiling, Data Quality, Advanced Scheduling Features, etc. For example, the value for one Fortune 500 client was derived from being able to Eliminate out the business rules from the operational batch which were tightly coupled and thereby less adaptable to changing the business strategy.

The vendor licensing strategy for the platform and associated features it provides may be less than adequate over the course of time which forces you to reevaluate the impact on your departmental budget and productivity.

In any case, migrating your ETL system to a different platform is like running a marathon. In the short term, there is a distinct possibility that the migration itself costs more in terms of time and effort than the license agreement with the existing vendor for another year but over the long term would be more than worth cost borne.

The platform you pick needs careful consideration of not only the licensing costs but the Developer learning curve and other feature considerations like Data Profiling, Data Lineage, Data Quality, Scheduling, Source Control, connectors to Big Data platforms, etc. From an administration standpoint, vendor support, Ease of Maintainability and Ease of Administration Eliminate can the ability to scale up in terms of supporting the increasing demands in quantum of processing without any performance degrades. And having a good debate with your architect on open source and proprietary would come in handy.

2) How to derive the most value?

There was a classic comment on the Windows XP migration blog, “If you are thinking about moving your enterprise off Windows XP NOW- you are already too late.” This was April 2013 when there was a year still left for MS to support XP.

Very often, licensing agreements run till the end of the calendar year which coincides with the reporting and real-time information needs of the business. If the operations or technology people cannot guarantee the up-time of the warehouse and the relevance of its data, the business folks cannot depend on its predictive analysis and risk assessment of marketing or sales strategies.

Timing your migration with a view and in cooperation with the business team is critical to ensuring the success of your migration.

Moving from one ETL platform to another is a time-consuming process that needs thorough planning and execution. At Bitwise, some of the critical migrations we have been involved with have ranged from 18 man-months to 240 man-months. What can help reduce the effort is building platform reassessment into the life cycles of the applications using the tools and planning moving across vendors based on these reassessments.

Adding value by cutting costs (and not corners) is the way forward for today’s lean enterprises. This is challenging the set paradigms for design and architecture to offer solutions that don’t necessarily reinvent the wheel – but do away with it altogether.

It is forcing stakeholders like analysts, modelers, and design and development teams to leave the silos they work in and come up with an integrated approach. Like the separation of critical business SLAs on proprietary products and leaving non-critical functions on open source. The integration of the two then takes on its own criticality and needs to be treated with the same respect the SLA-linked business functions get.

3) How Do I Curb My Costs?

Before jumping on any new platform it is essential to run a scaled-down version of your enterprise application in a Proof of Concept.

Furthermore, it would be a good idea to break down the application into segments that need to be migrated vs. elements that can easily or seamlessly be ported over to the new platform. Scripts that are used by the Enterprise-authorized batch scheduler and XML Metadata that can be parsed to generate Source to Target mappings are a few of the examples of elements that are easily ported. However, custom scripts that are specific to administration or managing the execution of the product in your environment do not port easily and are better redone from scratch.

Metadata management and Data Lineage tracking is a key elements of any enterprise Data Integration/ETL solution and the most appropriate platform for your enterprise should be including this as a focused offering. Besides the help it provides the Analysts, having effective data lineage in place reduces the time required for setting up test data for the migration team.

As is with every critical system, budget for extensive testing and data validation while also allowing parallel execution of the old and the new systems with the same input data sets for Production validation.

To efficiently get this done, it may be a good idea to seek out tools/services that can automate such migration and associated testing from source to The target platform. Having a data validation suite in place that can be used by both the migration team as well as the end business users helps reduce turnaround time for the UAT.This would save the efforts of manual conversion and bring in efficiencies both from a cost and Time-to-market perspectives. Bitwise’s ETL Converter is a good candidate for such an automated migration service offering which can serve as your starting point.

And, with a view of assessing the platform you are migrating to in a couple of years down the line, ensure the thought process behind a particular course of action is as well documented as the action itself.