Databricks Platform Developer

We are looking for a Senior Databricks Developer with strong hands-on experience in designing, developing, implementing and optimizing enterprise-scale data solutions on the Databricks Lakehouse platform.
The ideal candidate will have strong development expertise in Python, PySpark, SQL, Delta Lake and Databricks, along with experience building automated workflows, CI/CD pipelines, infrastructure-as-code and AI-enabled solutions.
This role will focus on developing reusable platform components, engineering automation, optimizing data processing workloads and enabling scalable, secure and cost-efficient data solutions. The candidate will work closely with data engineering, data science, cloud infrastructure and security teams to improve platform capabilities and developer productivity.
Key Responsibilities
Design, develop and implement scalable data engineering solutions using Databricks, Python, PySpark and SQL.
Develop and optimize notebooks, workflows, jobs, Delta Live Tables and Lakeflow pipelines.
Build reusable frameworks, libraries, templates and automation components for enterprise data engineering.
Implement and optimize Delta Lake solutions, including partitioning, liquid clustering, Z-Ordering, OPTIMIZE and VACUUM.
Develop solutions using Unity Catalog, including catalogs, schemas, permissions and governed data access.
Work with Databricks Workflows, job clusters, SQL warehouses and serverless compute.
Troubleshoot and optimize application performance, Spark execution, query processing and resource utilization.
2. CI/CD & Development Automation
Design and implement CI/CD pipelines for Databricks applications using Azure DevOps or GitHub Actions.
Develop and deploy Databricks solutions using Databricks Asset Bundles, Git integration and Terraform.
Implement automated testing, code quality checks, deployment validation and release management.
Develop reusable infrastructure-as-code modules and deployment templates.
Automate environment provisioning, configuration management and application deployment across development, test and production.
Implement version control, branching strategies and deployment rollback mechanisms.
3. AI & Agentic Workflow Development
Design and develop AI-powered automation and agentic workflows for platform operations and engineering processes.
Build AI agents using Databricks Mosaic AI Agent Framework, Model Serving and Vector Search.
Integrate AI agents with enterprise applications and services using APIs and Model Context Protocol (MCP).
Develop intelligent workflows for job failure analysis, root-cause investigation, operational monitoring and remediation recommendations.
Leverage Generative AI tools, including GitHub Copilot and Anthropic models, to accelerate development, testing and documentation.
Implement AI governance, evaluation, observability and human-in-the-loop controls.
4. Platform Performance & Cost Optimization
Analyze and optimize Databricks workloads, Spark jobs, SQL queries and compute utilization.
Implement cluster optimization strategies, autoscaling, auto-termination and workload management.
Monitor DBU consumption, billing usage, compute utilization and query performance.
Develop dashboards and automation to identify cost anomalies, idle resources and optimization opportunities.
Apply Photon, serverless compute and workload consolidation strategies where appropriate.
Implement cost governance and resource optimization practices.
5. Platform Integration & Governance
Develop solutions leveraging Unity Catalog, Delta Lake and Databricks Lakehouse architecture.
Implement secure data access, service principals, identity integration and role-based access controls.
Work with cloud storage, networking, secrets management and enterprise security integrations.
Support the development and deployment of enterprise ML, GenAI and RAG workloads.
Collaborate with cross-functional engineering teams to establish development standards and reusable engineering practices.
Required Skills
Strong hands-on experience with Databricks development and engineering.
Advanced programming skills in Python, PySpark and SQL.
Strong experience with Apache Spark, Delta Lake and Lakehouse architecture.
Experience developing Databricks notebooks, jobs, workflows and data pipelines.
Hands-on experience with Databricks Asset Bundles and CI/CD automation.
Experience with Azure DevOps or GitHub Actions.
Strong understanding of Terraform and Infrastructure as Code.
Experience with Unity Catalog and Databricks security/governance.
Knowledge of Spark performance tuning and compute optimization.
Experience with Git, automated testing and deployment practices.
Good to Have
Experience with Databricks Mosaic AI, Agent Framework and Model Serving.
Knowledge of Vector Search, RAG and MCP integrations.
Experience with Generative AI and agentic workflow development.
Exposure to Databricks system tables, FinOps and DBU cost optimization.
Experience with cloud infrastructure and automation.
Databricks or Azure certifications .
We are committed to fostering an inclusive workplace. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability status, protected veteran status, or any other characteristic protected by applicable law.
#J-18808-Ljbffr