Cloud Data Migration
On-premise to Cloud Data Replication with AWS Glue
A Global Research and Advisory Company intended to modernize its on-premises ‘Enterprise Data Quality’ application by migrating it to the AWS cloud. Bitwise spearheaded the initiative and replicated the on-premises Oracle data to PostgreSQL on AWS.
Client Challenges and Requirements
- Replicating data from on-premise to AWS cloud
- Scrapping Materialized View for data replication and creating a process that is Cloud native, scalable, secure, and open for modifications
- Build a metadata-driven, scalable data replication system using AWS services
- Trigger based on source system availability (online/offline)
- Orchestrate data replication processes
Bitwise Solution
- Developed a metadata-driven AWS-based data replication process that is re-runnable, fault-tolerant, and triggered by Rest API linked to the existing scheduler
- API triggers the step function that orchestrates all other services
- Lambda: Monitors database health. System up/down status. Reads Metadata table, returns the Staging table name, and metadata query to the Step function
- Map: Receives the details of all the tables in a particular workflow and creates ‘n’ number of virtual instances if there are ‘n’ number of tables present
- Virtual instances contain the glue job and a Lambda. Virtual instances run in parallel ensuring all the tables in the workflow are replicated into staging parallelly
- Glue: Connections are created and used for on-prem cloud databases and unloading takes place as truncate and load in the staging table in PostgreSQL
- Lambda: Loads the data from the staging table to the target table in PostgreSQL
Tools & Technologies We Used
AWS Glue
AWS Lambda
PostgreSQL (AWS RDS)
AWS Step-Function
Rest API
SQL
Key Results
A cloud native application that is reliable, scalable and secure
Metadata driven process which has re-runabilty, fault tolerance and is open for modifications
Fully operational and unit-tested code delivered with ~40% overall time efficiency
