Senior Databricks Data Engineer

Senior Databricks Data Engineer
Location: Toronto (Hybrid - minimum 2 days per week onsite at the client's downtown Toronto office)
Contract Length: 2 Months
Key ResponsibilitiesDatabricks Engineering and Migration
Migrate and refactor existing ETL/ELT pipelines, scripts and processing workloads into Databricks.
Develop production-quality data pipelines using Python, PySpark and SQL.
Translate existing processing and business logic into scalable Databricks-native implementations.
Develop and maintain ingestion, transformation and metric-calculation logic across multiple enterprise data sources.
Modernize existing workflow and orchestration processes, including scheduled jobs and Airflow/DAG-based processing.
Implement reusable engineering patterns to accelerate migration of additional workloads.
Data Integration
Integrate data from enterprise applications, APIs, databases, files and other source systems.
Work with source-system SMEs to understand existing data structures, interfaces and processing logic.
Implement source-to-target mappings based on approved requirements and target data models.
Support integration with enterprise platforms such as ServiceNow and cybersecurity data sources.
Troubleshoot connectivity, ingestion, schema and data-quality issues.
Data Quality, Testing and Reconciliation
Develop automated developer tests and data-validation routines.
Validate source-to-target transformations and data movement through bronze, silver and gold processing layers where applicable.
Perform reconciliation between legacy processing outputs and Databricks-generated results.
Identify, document and resolve data defects, processing exceptions and reconciliation discrepancies.
Produce clear validation evidence to support business and technical acceptance.
Engineering Quality and DevOps
Develop maintainable, reusable and well-documented production code.
Follow enterprise coding, security, architecture and release-management standards.
Use Git-based repositories and established CI/CD and deployment processes.
Participate in peer reviews and contribute to improving engineering standards and reusable patterns.
Support code check-in, deployment readiness, defect remediation and production-transition activities.
Monitor and troubleshoot pipeline execution and processing failures.
Agile Delivery and Collaboration
Work within an Agile delivery team executing against two-week sprint commitments.
Participate in sprint planning, backlog refinement, technical design sessions, defect triage and sprint reviews.
Estimate engineering effort and identify technical dependencies, blockers and risks.
Work closely with architects and technical leads to ensure alignment with enterprise architecture and Databricks standards.
Collaborate directly with client engineering teams and transfer knowledge throughout the engagement.
Produce technical documentation, source-to-target mappings, implementation notes and handover materials.
Required Skills and Experience
7+ years of experience in data engineering, software engineering or data-platform development.
Strong hands-on experience building production workloads on Databricks.
Advanced development skills in:
Python
PySpark / Apache Spark
SQL
Experience designing and implementing ETL/ELT and large-scale data-processing pipelines.
Experience migrating legacy data pipelines or processing applications to modern cloud data platforms.
Strong understanding of data ingestion, transformation, modelling, source-to-target mapping, data quality and reconciliation.
Experience with workflow orchestration technologies such as Apache Airflow or equivalent.
Experience integrating data through APIs, databases, files and enterprise applications.
Experience with Git, CI/CD and enterprise software-development practices.
Strong troubleshooting skills across data, code, integration and environment issues.
Ability to understand existing code and processing logic and efficiently refactor it into a modern architecture.
Experience working within structured Agile/Scrum delivery environments.
Strong written and verbal communication skills with the ability to work directly with client technical teams.
Must be based in the Greater Toronto Area and available to work onsite in downtown Toronto a minimum of two days per week.
Preferred Qualifications
Databricks certification such as Databricks Certified Data Engineer Associate or Professional.
Experience delivering data solutions within financial services, banking or another highly regulated industry.
Experience with cybersecurity, IT controls, compliance or operational-risk datasets.
Experience integrating platforms such as ServiceNow, CrowdStrike, BeyondTrust, FireEye, CyberArk, Tenable or similar enterprise security platforms.
Experience with cloud platforms such as Azure or AWS.
Experience modernizing Airflow DAGs or other legacy orchestration frameworks.
Familiarity with enterprise data-governance, metadata, lineage and auditability requirements.
Experience working in environments with strict security, architecture and release-governance processes.
#J-18808-Ljbffr