Databricks Data Platform Engineer

Job Title: Senior Databricks Data Platform Engineer
Location: Canada, Remote
Seniority on the skill/s required on this requirement: Senior Level
Estimated Duration: 1 year contract with possible extension
Work authorization: Candidate must be authorized to work in Canada.
Job SummaryWe are looking for a highly experienced Senior Databricks Data Platform Engineer to own the engineering, automation, administration, and optimization of an enterprise Databricks Lakehouse platform. The ideal candidate should have strong hands‑on experience in Databricks platform administration, Unity Catalog, Terraform, CI/CD automation, platform cost optimization (FinOps), Delta Lake, and cloud infrastructure.
This is a platform engineering role rather than a traditional data pipeline development position. The candidate will be responsible for building a secure, scalable, reliable, and cost‑efficient Databricks platform while enabling engineering and data science teams.
Key Responsibilities
Databricks Platform Engineering & Administration
Own Databricks environments across development, testing, and production.
Administer Unity Catalog, including metastores, catalogs, schemas, external locations, and access controls.
Manage compute resources, cluster policies, SQL warehouses, and serverless compute.
Manage Databricks Runtime upgrades and platform configurations.
Develop reusable platform templates and self-service capabilities.
CI/CD & Platform Automation
Build CI/CD pipelines using Databricks Asset Bundles, Git folders, Azure DevOps, or GitHub Actions.
Implement Terraform for Databricks and cloud infrastructure.
Automate testing, security scanning, and deployment workflows.
Establish release, branching, and rollback standards.
Automate SCIM provisioning, secret management, token rotation, and environment drift detection.
Agentic Workflows & AI-Enabled Operations
Develop AI-driven workflows for job failure analysis, root-cause investigation, and cost anomaly detection.
Build AI agents using Mosaic AI Agent Framework, Model Serving, and Vector Search.
Integrate enterprise systems through Model Context Protocol (MCP).
Implement AI governance, observability, evaluation, and human-in-the-loop controls.
Support GenAI, RAG, and ML workloads.
Platform Cost Optimization (FinOps)
Monitor Databricks usage and costs using system tables.
Optimize cluster sizing, autoscaling, auto-termination, Photon, and serverless compute.
Implement cost governance, budget alerts, and tagging standards.
Develop cost dashboards and anomaly detection alerts.
Support DBU forecasting and capacity planning.
Storage & Performance Optimization
Implement Delta Lake storage optimization standards.
Manage OPTIMIZE, VACUUM, liquid clustering, Z-ordering, and partitioning.
Optimize ADLS Gen2 / S3 storage and retention policies.
Tune Spark and SQL workloads.
Identify and resolve small-file issues, redundant data, and storage inefficiencies.
Governance, Security & Compliance
Implement secure platform controls and least-privilege access.
Manage Unity Catalog governance, lineage, and audit logging.
Implement data masking, classification, and row/column-level security.
Partner with security and compliance teams.
Reliability & Operational Excellence
Define platform SLOs and monitoring standards.
Lead incident management and root-cause analysis.
Develop operational runbooks and platform documentation.
Mentor engineers and promote Databricks best practices.
Required Qualifications
7+ years of experience in data engineering, data platform, or DevOps roles.
3+ years of enterprise Databricks administration and engineering experience.
Strong expertise in Unity Catalog, workspace administration, cluster policies, and Databricks Workflows.
Hands‑on experience with Databricks Asset Bundles and Terraform.
Experience building CI/CD pipelines using Azure DevOps, GitHub Actions, or similar tools.
Demonstrated experience in Databricks cost optimization and FinOps.
Strong knowledge of Delta Lake internals and storage optimization.
Strong programming skills in Python and SQL, with working knowledge of PySpark.
Experience with Azure, AWS, or GCP.
Experience implementing security and governance for sensitive data.
Preferred Qualifications
Experience with Agentic AI and GenAI frameworks such as Mosaic AI, LangChain, and LangGraph.
Practical knowledge of MCP.
Familiarity with GitHub Copilot and Anthropic models.
Databricks Data Engineer Professional or Platform Administrator certifications.
Experience with Databricks system tables and FinOps dashboards.
Knowledge of Lakeflow, Delta Live Tables, Databricks SQL, and MLOps.
Familiarity with Docker and Kubernetes.
Experience migrating workloads from Hadoop, Snowflake, or legacy platforms to Databricks.
Personal Attributes
Strong automation-first mindset.
Cost‑conscious and data‑driven approach.
Excellent ownership and troubleshooting skills.
Strong communication and stakeholder management skills.
Ability to mentor engineers and drive platform engineering best practices.
Mindtris, a minority women‑owned enterprise, is at the forefront of digital transformation, technology excellence, and business growth solutions. Specializing in talent mobilization and innovation, we are dedicated to enhancing customer experiences across diverse sectors such as Information Technology, Telecommunications, Healthcare, Engineering, and the Public sector. With a focus on deploying top‑tier talent and fostering innovation, we empower businesses to thrive and excel in a rapidly evolving digital landscape, helping them reach new heights of success.
Mindtris is committed to fostering workforce diversity and is proud to be an equal opportunity employer.
#J-18808-Ljbffr