Data Migration
Cloud Data Migration from Netezza to Google BigQuery
A major payment processing firm based out of USA wishes to build a cloud based big data strategy that the enterprise as well as external users can leverage to build better analytics and better business decisions. One of the key components and challenge in the build process was migration of on premise data to cloud.
Client Challenges and Requirements
- Migration scope included 23TB of data, 1500 tables and 1200 plus jobs running on Netezza
- Maintaining Netezza batch process performance (ensuring compliance with defined SLA) so as to have a balance between data migration load and regular batch process load
Bitwise Solution
- Design and develop a generic process to extract data, generate delimited files and push the files to designated cloud buckets
- Generic tokenization process to tokenize PCI data before pushing to cloud
- Generic de-tokenization process to include actual value in output extracts
- Auto generation of BigQuery YAML (for table creation) using Netezza metadata
- Automatic validation of historical data migration
- Build the cloud migration architecture and solution design
- Compile the phased migration plan aligned with business priority and timeline
- Adoption of PaaS model – managed service pipeline for processing and storage
- Cloud native services for designing ETL workflows and scheduling
- Integrated Data Quality Framework
- Build a multi layered Data Lake for quick and controlled access to data layers (raw, transformed, aggregated)
Tools & Technologies We Used
Netezza
Ab Initio
Shell Script
Python
Control M
GCS
Dataflow
BigQuery
Cloud SQL
Airflow
Key Results
Seamless migration of the DW on to GCP
Successful migration of 23TB of historical data ensuring data integrity
Automated Data Validation resulting in 25% effort saving
Auto generation of BQ YAML from Netezza DDL resulted in 85% effort savings associated with DDL creation
