Job Description
• Design, build, and maintain data ingestion pipelines from SQL Server into Azure Databricks using CDC-based and batch patterns
• Implement and manage medallion architecture (Bronze, Silver, Gold) using Delta Lake and Unity Catalog
• Write and optimize complex T-SQL queries, stored procedures, and CDC configurations on Microsoft SQL Server
• Develop PySpark and Spark SQL transformations for large-scale data processing and curated analytical layers
• Build and maintain vector database pipelines that transform structured and unstructured source data into embeddings for downstream AI and search applications
• Collaborate with BI and analytics teams to deliver curated data models, dashboards, and AI/BI Genie experiences
• Configure and troubleshoot connectivity between SQL Server, Databricks, and third-party connector applications
• Monitor pipeline health, implement alerting, and resolve data quality and performance issues proactively
• Maintain ...