Principal Data Engineer - Databricks
Astra-North Infoteck Inc.
Principal Data Engineer – Databricks | Spark | Delta Lake | PySpark | Data Lakehouse | AWS/AzureJob DescriptionLocation: Toronto
Work Model: Onsite (4 days/week)
Key Requirements
12–18 years of overall Data Engineering experience.
8+ years of experience with Enterprise Data Warehouse and Data Lake platforms.
5+ years of hands‑on experience with Databricks and Apache Spark at scale.
Strong experience modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.
Experience redesigning ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake.
Experience refactoring legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines.
Experience ensuring data reconciliation, audit integrity, and consistency during migration.
Experience designing and governing Enterprise Data Warehouse and Data Lake/Lakehouse architectures.
Experience implementing layered architectures including:
Curated/Conformed Layer
Experience modernizing traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures.
Strong experience with finance and risk data models, including:
Financial Hierarchies
Credit Risk Models
Liquidity Risk Models
Experience enabling reporting use cases including aggregation, drill-down, and drill-back capabilities.
Experience building and managing semantic/consumption layers for BI, reporting, and analytics.
Ability to define business metrics, dimensions, hierarchies, and KPIs.
Experience with Databricks SQL, Delta Tables, and dbt or similar frameworks.
Strong experience developing and optimizing large-scale data pipelines using:
Experience optimizing workloads using Z-ORDER, OPTIMIZE, caching, and cluster configurations.
Experience implementing data governance, data quality frameworks, reconciliation controls, and exception handling.
Experience establishing data lineage and metadata management.
Knowledge of data security, access control, and compliance standards.
Experience with cloud platforms such as AWS or Azure.
Experience with CI/CD pipelines using:
Familiarity with orchestration tools such as:
Experience with dbt is a plus.
Ability to act as a technical authority and lead architecture decisions.
Experience mentoring senior engineers and establishing engineering standards.
Strong stakeholder management skills with finance, risk, analytics, and governance teams.
Ability to translate complex data structures into business-ready insights.
Nice to Have
Experience in Banking, Financial Services, Insurance (BFSI), Capital Markets, or regulatory reporting.
Oracle Financials
Experience supporting AI/ML workloads.
Databricks or cloud certifications.
Key Responsibilities
Lead Cloudera to Databricks transformation initiatives.
Design and implement enterprise Data Lakehouse and Data Warehouse solutions.
Build scalable, high-performance data pipelines and modern data architectures.
Drive data modernization, governance, quality, and security initiatives.
Support regulatory, management, and analytical reporting platforms.
Provide technical leadership, mentor engineering teams, and establish best practices.
#J-18808-Ljbffr
Work Model: Onsite (4 days/week)
Key Requirements
12–18 years of overall Data Engineering experience.
8+ years of experience with Enterprise Data Warehouse and Data Lake platforms.
5+ years of hands‑on experience with Databricks and Apache Spark at scale.
Strong experience modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.
Experience redesigning ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake.
Experience refactoring legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines.
Experience ensuring data reconciliation, audit integrity, and consistency during migration.
Experience designing and governing Enterprise Data Warehouse and Data Lake/Lakehouse architectures.
Experience implementing layered architectures including:
Curated/Conformed Layer
Experience modernizing traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures.
Strong experience with finance and risk data models, including:
Financial Hierarchies
Credit Risk Models
Liquidity Risk Models
Experience enabling reporting use cases including aggregation, drill-down, and drill-back capabilities.
Experience building and managing semantic/consumption layers for BI, reporting, and analytics.
Ability to define business metrics, dimensions, hierarchies, and KPIs.
Experience with Databricks SQL, Delta Tables, and dbt or similar frameworks.
Strong experience developing and optimizing large-scale data pipelines using:
Experience optimizing workloads using Z-ORDER, OPTIMIZE, caching, and cluster configurations.
Experience implementing data governance, data quality frameworks, reconciliation controls, and exception handling.
Experience establishing data lineage and metadata management.
Knowledge of data security, access control, and compliance standards.
Experience with cloud platforms such as AWS or Azure.
Experience with CI/CD pipelines using:
Familiarity with orchestration tools such as:
Experience with dbt is a plus.
Ability to act as a technical authority and lead architecture decisions.
Experience mentoring senior engineers and establishing engineering standards.
Strong stakeholder management skills with finance, risk, analytics, and governance teams.
Ability to translate complex data structures into business-ready insights.
Nice to Have
Experience in Banking, Financial Services, Insurance (BFSI), Capital Markets, or regulatory reporting.
Oracle Financials
Experience supporting AI/ML workloads.
Databricks or cloud certifications.
Key Responsibilities
Lead Cloudera to Databricks transformation initiatives.
Design and implement enterprise Data Lakehouse and Data Warehouse solutions.
Build scalable, high-performance data pipelines and modern data architectures.
Drive data modernization, governance, quality, and security initiatives.
Support regulatory, management, and analytical reporting platforms.
Provide technical leadership, mentor engineering teams, and establish best practices.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Principal Data Engineer - Databricks in Winnipeg, MB vacancy
- ~12–18 years of overall Data Engineering experience ~8+ years of experience with enterprise Data Warehouse and Data Lake platforms ~5+ years of hands‑on experience with Databricks and Apache Spark at scale ~ Strong experience modernizing legacy Cloudera platforms including...Databricks
- Principal Data Engineer work in Toronto centers on Databricks, Apache Spark, and Lakehouse modernization. This onsite role requires four days weekly.This principal-level position leads enterprise data engineering initiatives across Databricks, Spark, Delta Lake, PySpark, and...DatabricksWork at office
- BDO Canada, au sein de BDO Digital, recherche un ingénieur(e) principal(e) de données spécialisé Databricks. Vous dirigerez des mandats clients et concevrez des pipelines de données évolutifs et sécurisés sur Azure et Databricks, en assurant l’alignement avec les objectifs...Databricks
- ## Job Description# Databricks Data EngineerApply**Job#: 3057559****Job Description:****Databricks Data Engineer****Overview:**Apex Systems is seeking a Databricks Data Engineer in Toronto, ON to support a large-scale data migration and centralization initiative into an enterprise...DatabricksHourly payContract workManual labor
- Data Engineer - Databricks (Remote)We work with global enterprises to design and build data platforms that power digital products used by millions of users. Our projects involve large-scale datasets, complex distributed pipelines, and real business impact. We’re looking for...DatabricksFull timeRemote workMonday to fridayFlexible hours
- Azure Data Engineers with Databricks expertise will build scalable pipelines using SQL, Python, and cloud ETL tools across Azure environments.This position focuses on designing, implementing, and optimizing large-scale data pipelines for scalability, reliability, and performance...Databricks
- Senior Databricks Data Engineer Location: Toronto (Hybrid - minimum 2 days per week onsite at the client's downtown Toronto office) Contract Length: 2 Months Key ResponsibilitiesDatabricks Engineering and Migration Migrate and refactor existing ETL/ELT pipelines,...DatabricksContract workWork at office2 days per week
- Job Title: Senior Databricks Data Platform Engineer Location: Canada, Remote Seniority on the skill/s required on this requirement: Senior Level Estimated Duration: 1 year contract with possible extension Work authorization: Candidate must be authorized to work...DatabricksContract workRemote work
- Parkland Corporation is expanding its Enterprise BI team with a BI Developer focused on data engineering in an Azure Databricks environment. You will design, build, test, and support trusted data products and pipelines, leveraging Delta Lake, Unity Catalog, and related Databricks...Databricks
- Databricks and Apache Spark specialists can build scalable data solutions for a six-month contract supporting a top financial client. This hands-on role focuses on... ...ETL, and governance expertise to enterprise data engineering for a financial client. #J-18808-LjbffrDatabricksContract work
- Build scalable data solutions as a Databricks Developer using Apache Spark and advanced data engineering practices with a top financial client. This six-month contract focuses on enterprise-grade pipelines.Design, develop, and optimise Databricks solutions across large-scale...DatabricksContract work
- Own Databricks infrastructure, Terraform deployments, and streaming reliability at Homebase, with required Toronto hub attendance... .... Build the platform beneath AI-powered products.As a Data and ML Platform Engineer, you will control how data enters, runs through, and returns...DatabricksWork from home
- Enhance your data engineering skills as a Databricks Engineer focused on Azure cloud solutions. Utilize your extensive Spark experience to optimize and deliver large-scale data pipelines. This role demands over 10 years of expertise in data engineering, particularly within...Databricks
- Lead Databricks data platform architecture at ShyftLabs from our downtown Toronto office three days weekly, using Spark and Python.As a Senior / Lead Data Engineer, you will guide enterprise-scale platforms for Fortune 500 clients. Apply Databricks, Apache Spark, Python, SQL...DatabricksWork at office
- Lead enterprise data platforms with Databricks and Apache Spark at ShyftLabs, working three days weekly from its downtown Toronto office. Shape cloud... ...solutions for Fortune 500 clients.As a Senior / Lead Data Engineer, you will own delivery from discovery through production...DatabricksWork at office
$84k - $128k per year
Senior Data Engineer (Databricks Focused)This site uses cookies to provide you with a more responsive and personalised service. By using this site you agree to our use of cookies. Please read our privacy statement for more information on the cookies we use and how to delete...DatabricksFull timeManual laborWork at officeLocal areaFlexible hours- Gestalte als Data & Analytics Engineer mit SQL und Python die zentrale Datenplattform der dpa in Berlin. Die Vollzeitstelle ist unbefristet und... ...Analytics Engineer. Du erschließt redaktionelle Datenquellen auf Databricks und AWS, integrierst Content-, Nutzungs- und Business-Daten...Databricks
- Lead enterprise Data and AI engineering with Databricks and AWS in Vanguard’s hybrid working model, building scalable platforms for analytics.Vanguard seeks an experienced Senior Lead Data and AI Engineer with 9–12+ years in Data Engineering and modern cloud data platforms....Databricks
- Build advertising data infrastructure remotely across Canada with Databricks, Apache Spark, and AI-native engineering practices. Own pipelines powering campaign analytics.This Senior Data Engineer role focuses on advertising analytics, campaign performance, and business intelligence...DatabricksRemote work
- Lead Azure Databricks engineering at Seaspan in a hands-on team leadership role focused on scalable data solutions and reliable delivery. Direct Data Engineers while shaping Lakehouse architecture, pipelines, and enterprise data products.As Team Lead, Data Engineering, you...Databricks
- ...About the Role Our client, a rapidly expanding telecommunications firm, is seeking a highly experienced Principal Data Engineer to establish and lead their data infrastructure initiatives. This role offers the flexibility of remote work and the opportunity to build robust...Remote workHome office
- ...solutions company is looking for an experienced Python/PySpark Developer to design and optimize data-driven applications. This role involves developing applications, utilizing Databricks, and integrating data pipelines. The ideal candidate will possess strong skills in Python,...DatabricksRemote work
- Senior AWS Data Engineer (Strong Databricks) We are looking for a Senior AWS Data Engineer to design and lead scalable, secure, and high-performance cloud data platforms across AWS/Azure. Mississauga, ON (Hybrid).Location: Mississauga, ON (Hybrid)Contract: Full Time/CTHExperience...DatabricksFull timeContract work
- ...1984, we started out as a team of three engineers. Today, we have grown to become a global... ...locations - Remote About The Role The Data Fabric team is a cross-functional team... ...-tenant, multi-cloud data platform on Databricks (GCP & Azure) to connect, govern,...DatabricksLong term contractWork at officeRemote workFlexible hours
- Location: Montreal, QC (REMOTE) Type: Contract Role Overview As a Senior Databricks Data Platform Engineer you will own the engineering automation and day to day health of our enterprise Databricks Lakehouse platform This is a platform-first role rather than building...DatabricksContract workRemote work
- The Databricks Engineer will design, develop, and optimize large-scale data pipelines using Databricks and Azure cloud services.This role requires advanced expertise in Spark/PySpark, Delta Lake technologies, data governance, and cloud-based data engineering.The engineer will...Databricks
- A leading data solutions company is seeking a Senior Data Engineer to join their fully remote team. In this role, you will design and build scalable ETL/ELT pipelines using Databricks on AWS, ensuring data quality and collaborating closely with cross-functional teams. The ideal...DatabricksRemote workFlexible hours
- NTT DATA seeks a Senior Data Engineer to join our Toronto, Ontario team. The role focuses on designing, building, and optimizing scalable data solutions... ...workflows. Ideal candidates will have 8+ years with Databricks, PySpark, Python and SQL, plus strong Azure data services...Databricks
- ...Triton Digital Canada Inc. is seeking an MLOps Data Engineer to bridge data science and production systems. You will design and automate... ...optimize large-scale data processing with Apache Spark, and leverage Databricks to deliver reliable ML solutions. Your work will enable...DatabricksRemote work
$85k - $90k per year
...Description Design and maintain scalable data pipelines that collect, transform and... ...Degree or Diploma in Computer Science, Data Engineering, Information Systems, or related field.... ...Azure data services (e.g., Data Factory, Databricks, Synapse, Fabric, Purview). · Experience...DatabricksPermanent employment
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal Data Engineer - Databricks. Be the first to apply!
Related searches
- project engineer assistant project manager Winnipeg, MB
- chief engineer Winnipeg, MB
- director software engineering Winnipeg, MB
- engineering project manager Winnipeg, MB
- mechanical engineering project manager Winnipeg, MB
- mechanical engineering director Winnipeg, MB
- chief power engineer Winnipeg, MB
- directeur ingénierie Winnipeg, MB
- general engineer Winnipeg, MB
- data integration developer Winnipeg, MB

