ETL Modernization with ADF and Microsoft Fabric (PySpark)

A global leader in oil and gas drilling services needed to modernize its legacy ETL platform to improve efficiency, scalability, and performance. The objective was to enhance their data management capabilities with Azure Data Factory (ADF) and Microsoft Fabric in support of a company-wide digital transformation initiative.

Client Challenges and Requirements

• Assess and document legacy platform for both Enterprise and Commercial applications.
• Migrate Informatica PowerCenter mappings into Azure Data Factory and Microsoft Fabric (PySpark).
• Address limited stakeholder awareness of Azure Data Factory and Microsoft Fabric.
• Reduce high licensing costs associated with the legacy platform.
• Resolve performance issues with existing ETL processes.

Bitwise Solution

• Migrate the Enterprise Application to Azure Data Factory while retaining the on-premises Oracle database. Transition the Commercial Application to Microsoft Fabric (PySpark) utilizing Fabric Lakehouse.
• Leverage a Generic Replication framework for the Enterprise Application, incorporating key activities like configurable lookup and copy operations to facilitate secure data extraction from on-premises sources to the Azure landing layer.
• Replicate or load Oracle and PostgreSQL sources into Lakehouse for Commercial Applications, providing centralized storage and scalability.
• Perform a feasibility assessment of the ADF native scheduler, identifying it as a promising alternative to the Informatica scheduler for optimized scheduling and execution.

Tools & Technologies We Used

Key Results

Assessment of the entire inventory, including highly complex mappings

20% increase in performance and cost optimization across the platform

Accelerated project timeline with 50% reduction in Assessment efforts using Bitwise Source ETL Analyzer