Senior Data Bricks/Scala/Spark Engineering Specialist (Saint-Prosper)

Job Title

Senior Data Bricks/Scala/Spark Engineering Specialist

Location

Montreal, QC, Onsite

Employment Type

Contract

Duration

12 Months

Responsibilities

We're migrating complex on‑prem regulatory reporting pipelines from a legacy ETL + Autosys + SQL + Teradata stack to a modern Databricks + Snowflake platform on Azure. The role is hands‑on: design, implement, test, and reconcile production pipelines feeding regulatory reports under strict parity requirements.

Must‑Have Skills

• Scala / Spark — production experience writing Spark applications in Scala (not just notebooks); comfortable with the DataFrame API, joins, window functions, partitioning, and performance tuning
• Databricks — Serverless compute, Unity Catalog, Asset Bundles, Databricks CLI
• SQL fluency — comfortable writing, analyzing and extracting requirements from complex SQL scripts
• Snowflake — schema design, performance, Spark‑Snowflake connector
• Azure — ADLS, networking basics, secrets/identity (Entra ID / managed identities)
• Orchestration — Airflow (DAG authoring, sensors, retries, SLAs)
• CI/CD — Artifactory, GitHub Actions pipelines: build, sharded test matrices, artifact promotion through dev to QA to UAT to prod
• Testing — Experience in TDD, writing unit tests (ScalaTest, AnyFlatSpec) and BDD (Concordion or equivalent)
• Data quality & reconciliation — building automated parity checks against legacy outputs, drift detection, row‑level reconciliation tooling
• Large‑scale migrations — proven track record migrating legacy ETL (Autosys/Informatica/etc.) to cloud data platforms, including dependency mapping and cutover planning
• Up‑to‑date data engineering practices - medallion architecture (Bronze/Silver/Gold), idempotent pipelines, schema evolution, lineage, observability

Nice‑to‑Have

• Financial services / regulatory reporting domain
• Python (Databricks utilities, tooling)
• Spec‑driven development workflows (specs to plans to tasks to implementation)
• Gradle (composite builds) and JVM tooling

Employment Practices

EEO, ADA, FMLA compliant. VLink is an equal opportunity employer committed to fostering an inclusive environment where diversity is celebrated. All qualified applicants will be considered for employment without regard to race, colour, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status. Employment is contingent upon successful completion of a background check. Applicant information will be handled in accordance with VLink's privacy policy.