Senior Databricks Data Engineer - Oil & Gas

We are looking for a Senior Databricks Data Engineer to join our team.

This role is ideal for someone who enjoys building end-to-end solutions and wants to work on real-world industrial use cases that comes from an Engineering background, and has experience in Oil & Gas.

In this position, you’ll contribute to a variety of impactful client projects, including:

• Building AI-powered anomaly detection systems for operational and industrial data
• Modernizing asset management workflows
• Developing natural language interfaces for querying enterprise and operational data
• Implementing Azure Databricks foundations and pipelines with best practices
• Creating scalable data pipelines and ML workflows in cloud environments
• Enabling real-time and batch data processing for analytics and AI use cases
• Working through complex and challenging data conditions, including incremental processing, late-arriving or changing records, complex business and engineering rules, and reconciliation of results across processing runs

The ideal candidate combines strong data engineering experience with an engineering or applied-science background. Direct experience industrial time-series data, scientific measurements, telemetry, financial reconciliation, or other datasets where accuracy, traceability, and incremental recalculation are critical.

Key Responsibilities

• Design, build, and optimize pipelines for sensor and related operational data.
• Develop complex transformation and calculation logic based on engineering requirements.
• Implement robust incremental-processing patterns for high-volume and continuously changing datasets.
• Design, build, and deploy end-to-end AI/ML solutions in production environments
• Develop robust backend systems and APIs to support AI-driven applications
• Build and maintain data pipelines and feature engineering workflows
• Implement and operationalize machine learning models (training, deployment, monitoring)
• Work with modern AI tooling (LLMs, agents, orchestration frameworks)
• Collaborate with clients to translate business problems into technical solutions
• Contribute to architecture decisions and best practices across projects
• Mentor client team members and contribute to internal capability building
• Work directly with engineering and operational SMEs to understand physical processes and translate their knowledge into technical requirements.
• Make engineering calculations and data transformations explainable, traceable, testable, and auditable.
• Document data lineage, calculation logic, assumptions, dependencies, and exception-handling rules.

Ideal Background

• Senior-level experience designing and developing production data pipelines with Azure Databricks including strong experience with complex SQL, Python, Spark, or comparable data-processing technologies.
• Complex Excel and CSV integration experience
• Experience with Databricks dashboards and Genie
• Strong requirements gathering experience with Field Engineers and Technical SMEs
• Demonstrated experience with incremental processing, change detection, reconciliation, and pipeline design
• Experience handling time-series, telemetry, sensor, operational, scientific or industrial data
• Medallion architecture and modelling
• Strong analytical and investigative skills, with the patience to work through detailed logic and difficult data-quality problems.
• Degree or professional background in petroleum, reservoir, chemical, mechanical, geological, geophysical, or another relevant engineering or applied-science discipline is strongly preferred.
• Experience in upstream oil and gas, thermal operations, SAGD, well surveillance, production engineering, or subsurface data would be a significant asset.

This role presents an exciting opportunity to work on practical, high-impact AI use cases - not just prototypes, shape how AI is applied to client environments, and change the game on traditional processes and platforms. Come join a growing organization helping clients take a new, lean and value-driven approach to data and engineering!