Modernizing DataStage to Databricks for a Fortune 500 Insurer's Property & Casualty Commercial Line

A top-tier insurance provider needed to upgrade the core calculation engine behind its Property & Casualty commercial line of business. The engine processes complex financial data across earnings, premium tax, deferred costs, currency conversion, and multiple accounting standards, generating billions of derived records. The legacy DataStage platform limited throughput, kept each operating unit in its own database, and made compliance hard to demonstrate.

Client Challenges and Requirements

Complex financial calculations: Logic spanned earnings, premium tax, deferred costs, currency conversion, and multiple accounting standards.
Massive data volumes: Billions of derived records were generated from permutations of transactional data.
Scalability bottlenecks: The legacy DataStage platform limited processing throughput.
Siloed databases: One database per operating unit limited integration and agility.
Limited auditability: Compliance and transparency were difficult to demonstrate.

Bitwise Solution

Built scalable, modular calculation pipelines using Databricks Lakeflow Jobs, notebooks, and Python.
Used ADLS Gen2 and Databricks Volumes for transient storage, with Unity Catalog providing secure, governed access.
Replaced legacy schedulers and scripts with Lakeflow orchestration, using serverless compute and YAML/widget parameterization.
Integrated Azure Communication Services for notifications and activity tracking.
Consolidated 65 siloed databases into a single multi-tenant, cloud-native environment.
Added detailed logging and audit trails to improve observability of calculation logic.

Tools & Technologies We Used

Databricks Lakeflow Jobs

Notebooks and Python

Unity Catalog

ADLS Gen2

Azure Communication Services

Databricks Volumes

Serverless Compute

Multi-Tenant Architecture

Audit Logging

Migration from Legacy DataStage

Key Results

Achieved 98% data accuracy in calculation engine output across all operating units.

Unified 65 operating units onto a single multi-tenant Databricks platform, replacing 65 siloed databases.

Processed billions of records reliably at scale through Lakeflow pipelines.

Delivered faster, more scalable processing than the legacy DataStage platform for complex financial calculations.

Strengthened compliance and transparency with a full audit trail on calculation logic.

Download Case Study

Required. I consent to Bitwise processing my personal data to deliver the requested resource and for related follow-up communication. My data may be stored in our CRM (Zoho) and shared with Bitwise affiliates and service providers as described in the Privacy Policy. Read our Privacy Policy. Questions: DPO@bitwiseglobal.com.I would like to receive newsletters and marketing communications from Bitwise. I understand I can unsubscribe at any time.I consent to a Bitwise representative contacting me about relevant services and offers.

Related Insights

Automating Regulatory Data Call Query Migration with AI-Driven SQL Mapping on Databricks

Case Study

Zero Defect Modernization Embedding Shift-Left Quality into a Leading Payment Processor's Oracle

Case Study

Powering 24×7 Operational Resilience Across a Top-Tier U.S. Grocery & Retail Chain

Case Study

Unifying Fragmented Merchant Applications for a Leading Payment Processor Through Cloud-Native Platform Modernization

Case Study

Automating Regulatory Data Call Query Migration with AI-Driven SQL Mapping on Databricks

Case Study

Zero Defect Modernization Embedding Shift-Left Quality into a Leading Payment Processor's Oracle

Case Study

Powering 24×7 Operational Resilience Across a Top-Tier U.S. Grocery & Retail Chain

Case Study

Unifying Fragmented Merchant Applications for a Leading Payment Processor Through Cloud-Native Platform Modernization

Case Study

Let's Engineer Your AI Advantage

GET IN TOUCH

Let's Engineer Your AI Advantage

GET IN TOUCH