Data Pipeline Development & Operations . Design, build, and operate scalable and reliable data pipelines on the Databricks platform . Develop end-to-end data workflows from ingestion through transformation to consumption . Implement robust error handling, monitoring, and alerting mechanisms . Ensure data pipeline reliability, performance, and maintainability . Optimize pipeline performance through efficient Spark job design and cluster configuration . Manage and orchestrate complex data workflows using Databricks Jobs and workflows
Legacy Code Modernization . Refactor legacy code and data pipelines to PySpark for improved performance and scalability . Migrate traditional ETL processes to modern ELT patterns on Databricks . Assess existing codebases and identify opportunities for optimization and modernization . Ensure backward compatibility and data integrity during migration processes . Document refactoring approaches and create migration playbooks . Collaborate with stakeholders t...