Data Migration
Data Migration from Teradata to Snowflake
A Direct Banking and Payment Services company needed to migrate their on-premise data warehouse residing on Teradata to cloud-based data warehouse Snowflake. The process involved migrating tables, DDL including historical and incremental data with support for cut over to on premise.
Client Challenges and Requirements
- Unavailability of any tool for migrating Teradata DDL to Snowflake DDL with constraints
- Time consuming manual process for migrating large number of tables and high volume of data
- Manual validation of migrated data resulting in inaccuracy and inefficiency of the process
- Challenges to migrate incremental data due to lack of CDC feature in Teradata
- Migration of sensitive data with Security
- No provision to handle pipeline failures and restartability
Bitwise Solution
Bitwise leveraged its data migration framework consisting of the following features:
Schema Mapping and Auto conversion: Mapped the schema of Teradata with Snowflake and auto convert the DDL
Data Validation: Developed a process/service to validate the historical as well as CDC data with provision of sample and full data after migration.
High Performance, Availability and Scalability: Implemented containerization to execute jobs for extraction, load and Validation
Auto Scheduling: Automatic scheduling of migration process for data extraction, load and validation
Tokenization: Implemented tokenization process to handle sensitive data
Monitoring, Logging and Reporting: Ability to monitor the data migration process to check the progress/status with user friendly dashboards and reports.
Tools & Technologies We Used
Teradata
Snowflake
Bitwise Cloud Data Migration Framework Using Java, Apache Spark
Containerization using Docker and Kubernetes
Key Results
Fast processing of data migration and validation
100% accuracy of migrated data with sample and full validation
High scalability and availability with reduced manual intervention
