Data Warehouse Modernization
Converting DataStage ETL to SSIS for Leading Insurance Company
Bitwise helped a NASDAQ listed, $6 Billion US insurance company, successfully migrate their Data Integration (ETL) platform from DataStage to Microsoft SSIS. The first and primary step in this migration was to complete the inventory analysis, and then scope the exact ETL jobs to be migrated.
The approach included a collaborative effort with client teams and the Source ETL Analyzer utility (an automated assessment tool put together by Bitwise), which helped in laying the foundation for the ETL Converter by analyzing the
- Complexity definition of each ETL job and identification of common features across all the ETL jobs.
- Identification of common features across all the ETL jobs
Client Challenges and Requirements
The client had to move their entire data integration platform from IBM DataStage to Microsoft SSIS, covering the following aspects:
- Development of the SSIS ETLs after analyzing the original DataStage jobs
- Unit testing, which involved the extensive field level comparison of DataStage outputs against those generated by newly developed SSIS ETLs
- System integration and performance testing
- Deployment to stage environment followed by Production environment
All of the above aspects had to be covered for about 700 Datastage jobs spread across 22 subject areas within the fixed timeline of 4 months to avoid Datastage license renewal. The jobs were of varied complexity with 42% jobs in the Medium-to-Complex category.
Bitwise Solution
The Bitwise ETL Converter is customized based on the information captured that makes the automation highly cost-effective in terms of reducing the overall development efforts. The converter is additionally bundled with Data Validation automation that enormously reduces the turn-around time to productionalize the new solution.
The Bitwise ETL Converter enables the automation of the migration process, thereby minimizing the time-to-production and brings in better quality within the new Data Integration platform. Its support for various ETL Tools and easy customization, based on the client’s needs, enables it to lead the market in offering a dynamic and complete migration toolset.
Automated assessment with Dark Data Discovery framework
Phased migration approach
Consistency and standardization across the ETL Conversion jobs
Automated data validation
Repeatable process
Tools & Technologies We Used
IBM DataStage
SQL Server Integration Services (SSIS)
Bitwise Source ETL Analyzer
Bitwise ETL Converter
Bitwise ETL Validation Framework
Key Results
22% Reduction during Assessment Phase and Inventory Cleanup
55% Reduction in Development Effort
25% Effort savings in Data Validation
65% Effort savings in Defect cycle efficiencies brought automation consistencies
