Automating Regulatory Data Call Query Migration with AI-Driven SQL Mapping on Databricks

Client Challenges and Requirements

• Connectivity gaps: Legacy SQL Server systems could not easily connect to the Databricks cloud environment.
• Mapping ambiguity: Multi-valued, delimiter-based fields caused partial matches.
• Missing cross-references: Legacy field descriptions had no link to the target Helix BDW descriptions.
• Incomplete migration: Certain Data Call attributes had never been fully loaded into BDW.
• AI model limitations: Complex, lengthy SQL scripts caused models to lose context continuity.

Bitwise Solution

• Applied AI/LLM automation on Databricks to parse, map, and generate SQL across 100+ regulatory data calls.
• Ingested legacy data into Unity Catalog, with an LLM SQL parser extracting table and field relationships.
• Matched fields against Unity Catalog target tables through a destination search engine, using multiple methods and confidence scoring.
• Extracted data securely from SQL Server into Databricks Delta tables over JDBC, authenticated via Azure AD.
• Added human-in-the-loop (HITL) validation of low-confidence mappings before generating production-ready SQL.
• Evaluated multiple LLMs on Databricks, including Llama 3.1, Claude Sonnet 4.5, Claude Opus 4.1, and Gemini 3 Pro.
• Automated the flow end to end:
  • Data ingestion into Unity Catalog volumes, then LLM SQL parsing.
  • Data sampling and destination search, using view hints to suggest target schema locations.
  • Matching strategies with a brute-force catalog search fallback, then confidence scoring.
  • HITL preparation and validation, then SQL generation and data validation.

Tools & Technologies We Used