Join VLink as a Senior Databricks Data Engineer in Connecticut and transform how we manage data. Work on cutting-edge technologies to facilitate real-time and batch data processing.
In this senior-level role, you will manage end-to-end data pipelines using Apache Kafka and lead the architectural design of Databricks Lakehouse models. Collaborating with various teams, your ability to optimize large-scale data processing and ensure compliance will support key business decisions and risk management activities.
Key Responsibilities:
• Manage real-time data ingestion with Apache Kafka
• Design scalable ETL/ELT workflows utilizing PySpark
• Architect Databricks Lakehouse models for energy markets
• Collaborate across departments for reporting solutions
• Implement data quality monitoring and governance
Requirements:
• Extensive experience with Databricks and ETL processes
• Proficient in SQL and data modeling techniques
• Strong understanding of data governan...