ETL Migration
Migrate Legacy Informatica ETL Code to AWS Glue
A national full-service restaurant company needed to migrate its ETL from on-premise Informatica application to AWS Glue to maintain and enhance its EDW, reporting and analytics applications in order to provide accurate and timely data to users and ensure data availability for Reporting and BI functions.
Client Challenges and Requirements
- Modernization of legacy ETL code from Informatica BDM to AWS Services within defined timeframe to retire Informatica
- Over 100 Informatica workflows and 150+ mappings to convert to AWS Glue
- 350+ shell scripts to be converted into Python
Bitwise Solution
- Bitwise followed standard migration framework and AWS cloud native purpose-built services for end-to-end data solution implementation to accelerate migration timeline within budget requirements
- Identification of all Informatica jobs with complexities
- Informatica mappings converted into AWS Glue jobs, Informatica workflows converted into Step function jobs, Unix scripts converted into Python scripts and SQL files to run as is using Python scripts
- Utilized fully managed AWS workflow orchestration service (AWS Step function) and leveraged centralized, secure, hierarchical storage for configuration data management (parameters) and secrets management
- Consistency and standardization across the data integration jobs
- Support migrated systems during all testing phases and fixed reported issues with minimal turnaround time
Tools & Technologies We Used
Informatica BDM
AWS Glue Studio
PySpark & Python
AAWS Step Functions
AWS Lambda
AWS Secret Manager, AWS System Manger
AWS CloudWatch, EventBridge
Redshift
Key Results
Effective analysis of Informatica jobs resulting in 30% reduction in Assessment efforts
Existing Python scripts executed using EC2 system manager with minimal changes
Fully converted and unit tested code delivered with increased time efficiency
Completed migration within timeframe to retire Informatica and avoid renewal costs
