Senior Databricks Data Platform Engineer (Pyspark)
Enterprise Solutions Inc.
Location: Montreal, QC (REMOTE)
Type: Contract
Role Overview
As a Senior Databricks Data Platform Engineer you will own the engineering automation and day to day health of our enterprise Databricks Lakehouse platform This is a platform-first role rather than building individual business pipelines you will build the foundation that every data engineer analyst and data scientist at Client depends on including workspaces compute Unity Catalog governance deployment pipelines and the guardrails that keep the platform secure reliable and cost-efficient
You will treat the platform as a product That means codifying infrastructure and configuration delivering changes through fully automated CICD introducing agentic AI workflows that reduce manual operational effort and continuously driving down compute and storage cost while improving performance You will combine deep hands-on Databricks expertise with a FinOps and DevOps mindset and you will measure your success in platform reliability developer productivity and dollars saved
This is a highly collaborative role You will partner with data engineering data science security cloud infrastructure and finance teams to set platform standards, guide adoption of new Databricks capabilities and mentor engineers across the organization
About the role
Own the architecture configuration and lifecycle of Client Databricks environments across development test and production, including workspaces, account-level settings, networking and identity integration
Administer and evolve Unity Catalog including metastores catalogs schemas, external locations storage credentials and fine-grained access controls
Design and enforce compute strategy across all-purpose clusters job clusters, serverless compute SQL warehouses instance pools and cluster policies
Manage platform upgrades Databricks Runtime adoption feature previews and deprecations with minimal disruption to consuming teams
Build reusable platform templates starter projects and self-service capabilities that let engineering teams onboard quickly and consistently
CICD Platform Automation
Design build and maintain CICD pipelines for Databricks code and configuration using Databricks Asset Bundles Git folders and Azure DevOps or GitHub Actions
Codify platform infrastructure with Terraform Databricks and cloud providers ensuring every workspace cluster policy, permission and Unity Catalog object is version-controlled and reproducible
Implement automated testing code quality gates security scanning and promotion workflows from development through production
Establish branching release and rollback standards for notebooks jobs Delta Live Tables, Lakeflow pipelines and ML assets
Automate routine administration such as user and group provisioning SCIM secret management token rotation and environment drift detection
Agentic Workflows AI-Enabled Operations
Design and deploy agentic workflows that automate platform operations such as job failure triage rootcause analysis cost anomaly investigation and remediation recommendations
Build AI agents and tools on Databricks Mosaic AI Agent Framework Model Serving Vector Search and integrate them with enterprise systems through Model Context Protocol MCP
Use Copilot and Anthropic models to accelerate infrastructure-as-code pipeline development code review and documentation
Establish guardrails evaluation, observability and human-in-the-loop controls so agentic automation is safe auditable and trusted
Partner with data science and engineering teams to provide a governed platform foundation for GenAI RAG and ML workloads
Platform Cost Optimization FinOps
Own Databricks cost visibility and accountability using system tables billing usage compute and query history tagging standards and chargeback or showback reporting
Identify and deliver savings through cluster rightsizing, auto-termination, autoscaling tuning spot and reserved capacity strategies Photon and serverless evaluation and workload consolidation
Enforce cost guardrails through cluster policies budget s and compute governance that prevent waste without slowing teams down
Build dashboards and automated s that surface cost anomalies idle resources and optimization opportunities to engineering and finance stakeholders
Partner with finance and procurement on DBU commitments forecasting and capacity planning
Storage Performance Optimization
Define and enforce Delta Lake storage standards including file sizing liquid clustering Zordering partitioning strategy and deletion vectors
Automate table maintenance using predictive optimization OPTIMIZE VACUUM and retention policies to control storage growth and improve query
SkillsMandatory Skills :
Databricks
Good to Have Skills :
Azure Databricks
We are an Equal Opportunity Employer. We celebrate diversity and are committed to creating an inclusive environment for all employees
#J-18808-Ljbffr
Type: Contract
Role Overview
As a Senior Databricks Data Platform Engineer you will own the engineering automation and day to day health of our enterprise Databricks Lakehouse platform This is a platform-first role rather than building individual business pipelines you will build the foundation that every data engineer analyst and data scientist at Client depends on including workspaces compute Unity Catalog governance deployment pipelines and the guardrails that keep the platform secure reliable and cost-efficient
You will treat the platform as a product That means codifying infrastructure and configuration delivering changes through fully automated CICD introducing agentic AI workflows that reduce manual operational effort and continuously driving down compute and storage cost while improving performance You will combine deep hands-on Databricks expertise with a FinOps and DevOps mindset and you will measure your success in platform reliability developer productivity and dollars saved
This is a highly collaborative role You will partner with data engineering data science security cloud infrastructure and finance teams to set platform standards, guide adoption of new Databricks capabilities and mentor engineers across the organization
About the role
Own the architecture configuration and lifecycle of Client Databricks environments across development test and production, including workspaces, account-level settings, networking and identity integration
Administer and evolve Unity Catalog including metastores catalogs schemas, external locations storage credentials and fine-grained access controls
Design and enforce compute strategy across all-purpose clusters job clusters, serverless compute SQL warehouses instance pools and cluster policies
Manage platform upgrades Databricks Runtime adoption feature previews and deprecations with minimal disruption to consuming teams
Build reusable platform templates starter projects and self-service capabilities that let engineering teams onboard quickly and consistently
CICD Platform Automation
Design build and maintain CICD pipelines for Databricks code and configuration using Databricks Asset Bundles Git folders and Azure DevOps or GitHub Actions
Codify platform infrastructure with Terraform Databricks and cloud providers ensuring every workspace cluster policy, permission and Unity Catalog object is version-controlled and reproducible
Implement automated testing code quality gates security scanning and promotion workflows from development through production
Establish branching release and rollback standards for notebooks jobs Delta Live Tables, Lakeflow pipelines and ML assets
Automate routine administration such as user and group provisioning SCIM secret management token rotation and environment drift detection
Agentic Workflows AI-Enabled Operations
Design and deploy agentic workflows that automate platform operations such as job failure triage rootcause analysis cost anomaly investigation and remediation recommendations
Build AI agents and tools on Databricks Mosaic AI Agent Framework Model Serving Vector Search and integrate them with enterprise systems through Model Context Protocol MCP
Use Copilot and Anthropic models to accelerate infrastructure-as-code pipeline development code review and documentation
Establish guardrails evaluation, observability and human-in-the-loop controls so agentic automation is safe auditable and trusted
Partner with data science and engineering teams to provide a governed platform foundation for GenAI RAG and ML workloads
Platform Cost Optimization FinOps
Own Databricks cost visibility and accountability using system tables billing usage compute and query history tagging standards and chargeback or showback reporting
Identify and deliver savings through cluster rightsizing, auto-termination, autoscaling tuning spot and reserved capacity strategies Photon and serverless evaluation and workload consolidation
Enforce cost guardrails through cluster policies budget s and compute governance that prevent waste without slowing teams down
Build dashboards and automated s that surface cost anomalies idle resources and optimization opportunities to engineering and finance stakeholders
Partner with finance and procurement on DBU commitments forecasting and capacity planning
Storage Performance Optimization
Define and enforce Delta Lake storage standards including file sizing liquid clustering Zordering partitioning strategy and deletion vectors
Automate table maintenance using predictive optimization OPTIMIZE VACUUM and retention policies to control storage growth and improve query
SkillsMandatory Skills :
Databricks
Good to Have Skills :
Azure Databricks
We are an Equal Opportunity Employer. We celebrate diversity and are committed to creating an inclusive environment for all employees
#J-18808-Ljbffr
Vacancy posted 30 minutes ago
Similar jobs that could be interesting for youBased on the Senior Databricks Data Platform Engineer (Pyspark) in Winnipeg, MB vacancy
- A tech solutions company is looking for an experienced Python/PySpark Developer to design and optimize data-driven applications. This role involves developing applications, utilizing Databricks, and integrating data pipelines. The ideal candidate will possess strong skills...DatabricksSeniorRemote work
- Job Title: Senior Databricks Data Platform Engineer Location: Canada, Remote Seniority on the skill/s required on this requirement: Senior Level Estimated Duration: 1 year contract with possible extension Work authorization: Candidate must be authorized to work...DatabricksSeniorContract workRemote work
- Lead Databricks data platform architecture at ShyftLabs from our downtown Toronto office three days weekly, using Spark and Python.As a Senior / Lead Data Engineer, you will guide enterprise-scale platforms for Fortune 500 clients. Apply Databricks, Apache Spark, Python, SQL...DatabricksSeniorWork at office
- ...Luxoft in Toronto is seeking a senior data engineer to support the Analytical Work Bench integration, including AWS-based data pipelines and Jupyter... ...with standard testing practices. Join a team that leverages PySpark, PyTest, and Pylint, while integrating with CI/CD tooling...Senior
- Senior Databricks Data Engineer Location: Toronto (Hybrid - minimum 2 days per week onsite at the client's downtown Toronto office) Contract Length: 2 Months Key ResponsibilitiesDatabricks Engineering and Migration Migrate and refactor existing ETL/ELT pipelines,...DatabricksSeniorContract workWork at office2 days per week
- Validate enterprise data pipelines with Databricks and PySpark as an ETL QA Engineer at Kumaran Systems. Apply AI-assisted testing expertise.This ETL QA Engineer validates large-scale data pipelines and ensures data quality. The work centers on Databricks, PySpark, ETL/ELT concepts...DatabricksWork at office
- Montreal-based Senior Databricks Platform Engineer specializing in Lakehouse automation, Unity Catalog governance, and reliable remote platform operations.This contract role owns enterprise Databricks environments across development, test, and production. You will build reusable...DatabricksSeniorContract workRemote work
- Lead enterprise Data and AI engineering with Databricks and AWS in Vanguard’s hybrid working model, building scalable platforms for analytics.Vanguard seeks an experienced Senior Lead Data and AI Engineer with 9–12+ years in Data Engineering and modern cloud data platforms....DatabricksSenior
- Build advertising data infrastructure remotely across Canada with Databricks, Apache Spark, and AI-native engineering practices. Own pipelines powering campaign analytics.This Senior Data Engineer role focuses on advertising analytics, campaign performance, and business intelligence...DatabricksSeniorRemote work
- Lead Azure Databricks platform engineering at Canada Life in a hybrid Toronto or London role, using cloud administration and operational support expertise.As a Senior Data Platform Engineer, you will design, adapt, and maintain cloud solutions supporting Canada Life business...DatabricksSenior
- ...position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Data Engineer – Data Platform based in Canada.As a Senior Data Engineer – Data Platform, you will play a key role in developing and operating a...DatabricksSeniorContract workFixed term contractImmediate startRemote workWorldwide
- ...this job. Posted 19 Oct 2026.About this job Overview of the FunctionWe are seeking an experienced and forward-thinking Senior Data Platform Engineer to manage, optimize, and evolve the cloud-native platforms used by the Data Office. The role combines deep data engineering...DatabricksSeniorManual laborWork at officeLocal area
- ...Rodan Energy Solutions is seeking a hands-on Data Engineer to shape the technical direction of a modern data platform. You will own bronze/silver/gold data layers, extend ingestion pipelines, and work with Databricks, PySpark, and SQL to deliver analytics-ready data for energy...Databricks
- ## Job Description# Databricks Data EngineerApply**Job#: 3057559****Job Description:****Databricks Data Engineer****Overview:**Apex Systems is seeking a Databricks Data Engineer in... ...experience, including SQL, Python/PySpark, data migration and ETL development, with experience...DatabricksHourly payContract workManual labor
- Data Engineer - Databricks (Remote)We work with global enterprises to design and build data platforms that power digital products used by millions of users. Our projects involve large-scale datasets, complex distributed pipelines, and real business impact. We’re looking...DatabricksSeniorFull timeRemote workMonday to fridayFlexible hours
- Principal Data Engineer – Databricks | Spark | Delta Lake | PySpark | Data Lakehouse | AWS/AzureJob DescriptionLocation: Toronto Work Model: Onsite (4 days/week) Key Requirements 12–18 years of overall Data Engineering experience. 8+ years of experience with Enterprise...DatabricksSenior
- Miratech Group is seeking a Senior Data Architect to design end-to-end data platforms for APEX, integrating Genesys A3S Snowflake and Verint feeds. You will own... ...office. Ideal candidates have 10+ years in data warehousing and hands-on Databricks expertise. #J-18808-LjbffrDatabricksSeniorWork at office2 days per week
- Lead enterprise data platforms with Databricks and Apache Spark at ShyftLabs, working three days weekly from its downtown Toronto office. Shape cloud solutions for Fortune 500 clients.As a Senior / Lead Data Engineer, you will own delivery from discovery through production...DatabricksSeniorWork at office
$84k - $128k per year
Senior Data Engineer (Databricks Focused)This site uses cookies to provide you with a more responsive and personalised service. By using this site you agree to our use of cookies. Please read our privacy statement for more information on the cookies we use and how to delete...DatabricksSeniorFull timeManual laborWork at officeLocal areaFlexible hours- ~12–18 years of overall Data Engineering experience ~8+ years of experience with enterprise Data Warehouse and Data Lake platforms ~5+ years of hands‑on experience with Databricks and Apache Spark at scale ~ Strong experience modernizing legacy Cloudera platforms including...DatabricksSenior
- Are you ready to lead reliability engineering in a complex cloud environment? Working with one of... ...clients, this role calls for aCloud Platform Engineer-Databricks to support a new project focused on processing critical data. This is a contract opportunity offering the...DatabricksSeniorContract work
- Build data pipelines with Python, PySpark, and Snowflake while transforming large-scale datasets through distributed processing. Apply deep SQL expertise to reliable engineering work.This Data Engineer role focuses on Python programming, PySpark transformations, and Snowflake...Internship
$120k - $147k per year
We're searching for Senior Data Engineer, Data Platform & Operations. Are you searching for new possibilities? Westland Insurance Group Ltd. is growing and has a new opportunity for a Senior Data Engineer, Data Platform & Operations. This role is a hybrid...DatabricksSeniorInternshipWork at officeFlexible hours- Develop LLM workflows and data pipelines for AXIS Capital’s enterprise AI platforms using Databricks, PySpark, and Python. The work supports business teams.This Senior AI Platform Engineer builds reusable capabilities for underwriting, claims, operations, and strategic functions...DatabricksSenior
- Engineer secure, scalable Azure data platforms with Databricks, Spark, and Delta Lake for BDO Digital’s Technology Advisory Services practice.This senior engineering role combines platform development with technical leadership across client engagements. Build reliable pipelines...DatabricksSenior
- ...Compunnel, Inc. is seeking an Azure Cloud Data Engineer responsible for designing and maintaining scalable cloud-based data platforms. Your role includes developing high-performance... ...management. Candidates with expertise in Databricks and experience in handling large datasets...Databricks
- Azure Data Engineers with Databricks expertise will build scalable pipelines using SQL, Python, and cloud ETL tools across Azure environments.This position... ..., and performance. You will transform datasets with PySpark and Spark, integrate Azure data sources, and prepare information...Databricks
- NTT DATA seeks a Senior Data Engineer to join our Toronto, Ontario team. The role focuses on designing, building, and optimizing scalable data solutions... ...workflows. Ideal candidates will have 8+ years with Databricks, PySpark, Python and SQL, plus strong Azure data services...DatabricksSenior
- ...Permanent Full Time Are you a Senior Data Platform Engineer professional focused on platform administration, operations and support, have strong Leadership and Communication skills who is looking for a challenging and rewarding opportunity? The Canada Data Team is looking...DatabricksSeniorPermanent employmentFull timeImmediate start
- ...The Data Engineer will design, develop, and maintain scalable data pipelines and workflows using Python, PySpark, and cloud technologies. This role involves managing data storage solutions... ...Kubernetes. Familiarity with cloud platforms (Azure, AWS, GCP). Understanding of...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Databricks Data Platform Engineer (Pyspark). Be the first to apply!
Related searches
- data integration developer Winnipeg, MB
- data centre engineer Winnipeg, MB
- data engineer Winnipeg, MB
- python data engineer Winnipeg, MB
- big data engineer Winnipeg, MB
- gcp data engineer Winnipeg, MB
- remote data engineer Winnipeg, MB
- data cabling engineer Winnipeg, MB
- data engineering intern summer Winnipeg, MB
- junior data engineer Winnipeg, MB

