Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Big Data Engineer (Pyspark / Spark / Kafka)

Astra-North Infoteck Inc.

Work Mode:
Hybrid – 3 Days a Week in Office
Job Overview
We are seeking an experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should have hands-on experience in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, and Apache NiFi, along with a strong understanding of data ingestion, transformation, and real-time processing frameworks.
Key Responsibilities
Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi.
Build and maintain batch and real-time data processing solutions.
Develop and support Kafka-based streaming applications and event-driven architectures.
Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets.
Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting.
Implement data ingestion solutions from databases, APIs, files, and streaming sources.
Monitor, troubleshoot, and enhance the performance of Spark jobs and data pipelines.
Collaborate with architects, business analysts, and development teams to deliver high-quality data solutions.
Support platform upgrades, deployments, testing, certification, and production releases.
Ensure data quality, governance, security, and operational excellence across data platforms.
Mandatory Skills
PySpark
Apache Spark (Spark SQL, DataFrames)
Hadoop Ecosystem (HDFS, Hive, YARN)
SQL
Python
Preferred Skills
Spark Streaming
Hive
Scala
Jenkins, Bitbucket, Git
JIRA, Confluence
Data Warehousing Concepts and Dimensional Modeling
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Big Data Engineer (Pyspark / Spark / Kafka) in Winnipeg, MB vacancy
  • Astra-North Infoteck Inc. in Toronto seeks an experienced Data Engineer with deep Big Data expertise to design, develop, and support enterprise-scale data platforms. The role focuses on PySpark, Apache Spark, Kafka, Hadoop ecosystem, and NiFi, covering ingestion, transformation... 
    Big data

    Astra-North Infoteck Inc.

    Winnipeg, MB
    1 hour ago
  • Wipro Limited in Mississauga, Canada is seeking an experienced data engineer to design and implement large-scale data pipelines using Python, PySpark and SQL. You will work with Spark, Hive and Hadoop in a Big Data ecosystem to optimize data quality, profiling and ETL/ELT processes... 
    Big data

    Wipro Ltd.

    Winnipeg, MB
    1 hour ago
  •  ...Tata Consultancy Services Canada Inc. is seeking a Big Data Engineer to design and maintain robust data pipelines using Apache Spark, Scala, and Java for enterprise analytics. You will ingest, transform, and optimize large datasets, focusing on performance and cost efficiency... 
    Big data

    Tata Consultancy Services Limited

    Winnipeg, MB
    54 minutes ago
  • Join LTIMindtree as a Big Data Specialist, focusing on Hadoop, Spark, and Scala. Utilize your expertise in data application development to create scalable solutions for large data sets. With 3 to 5 years of experience, you will play a critical role in designing and developing... 
    Big data

    LTM

    Winnipeg, MB
    55 minutes ago
  • A leading engineering consultancy in Canada is looking for a Senior Software Engineer specialized in Java and Spark. In this role, you will work on a large social media platform processing petabytes of data. The position involves optimizing data systems, designing solutions,... 
    Big data

    Capgemini Engineering

    Winnipeg, MB
    1 hour ago
  •  ...The Data Engineer will design, develop, and maintain scalable data pipelines and workflows using Python, PySpark, and cloud technologies. This role involves managing data storage solutions...  ...Qualifications Experience with big data tools (e.g., Kafka, Hadoop). Knowledge... 
    Big data

    Compunnel Inc.

    Winnipeg, MB
    58 minutes ago
  •  ...leading IT Services firm in Toronto is seeking a Mid-Senior Level Data Tester to join their team. You will be responsible for testing GCP...  ...have 8-10 years of experience in Data testing, proficient in GCP Big Query, Python, and SQL, and possess excellent communication skills... 
    Big data
    Full time

    Astra-North Infoteck Inc.

    Winnipeg, MB
    1 hour ago
  • Build data pipelines with Python, PySpark, and Snowflake while transforming large-scale datasets through distributed processing. Apply deep SQL expertise to reliable engineering work.This Data Engineer role focuses on Python programming, PySpark transformations, and Snowflake... 
    Internship

    Lancesoft

    Winnipeg, MB
    1 hour ago
  •  ...metParticipate in team meetings to discuss project updates and progress8-10 years of overall experience with 3-4 years in Data testingKnowledge of GCP Big Query, Python and PysparkExperience in ETL and DB testing, Data migration, Report validationsExpert in writing... 
    Big data

    Astra-North Infoteck Inc.

    Winnipeg, MB
    54 minutes ago
  • Build GCP data migration solutions with PySpark and Salesforce APIs, focusing on Dataproc, BigQuery, and scalable ETL pipelines.This Data Migration Engineer role requires four to five years minimum of experience...  .... You will use Python, Apache Spark, BigQuery, GCS, and Salesforce... 

    Astra-North Infoteck Inc.

    Winnipeg, MB
    1 hour ago
  • Databricks and Apache Spark specialists can build scalable data solutions for a six-month contract supporting a top financial client. This hands-on role...  ...Databricks, Spark, streaming ETL, and governance expertise to enterprise data engineering for a financial client. #J-18808-Ljbffr
    Contract work

    BeachHead

    Winnipeg, MB
    1 hour ago
  • Build scalable data solutions as a Databricks Developer using Apache Spark and advanced data engineering practices with a top financial client. This six-month contract focuses on enterprise-grade pipelines.Design, develop, and optimise Databricks solutions across large-scale... 
    Contract work

    BeachHead

    Winnipeg, MB
    56 minutes ago
  •  ...Responsibilities Design and build realtime and batch data pipelines for fraud detection and AML...  ...streaming transaction data using Apache Spark and Kafka. Develop ETLELT workflows supporting...  ...expansive, high-performing recruitment engine operating across North America and Asia... 
    Big data

    Cynet Systems

    Winnipeg, MB
    59 minutes ago
  • Senior Software Engineer specializing in Core Java, Apache Spark, and Python develops scalable server-side applications and analytical data systems.This senior engineering role focuses on high-performance applications and large-scale data processing. You will use Core Java,... 
    Big data

    Astra-North Infoteck Inc.

    Winnipeg, MB
    55 minutes ago
  • GCP Data Migration Engineer – PySpark & Salesforce Data MigrationGCP Data Migration Engineer – PySpark & Salesforce Data Migration Required Skills...  ...Strong hands-on experience with GCP Dataproc and Apache Spark (PySpark preferred) Python scripting for custom data transformations... 

    Astra-North Infoteck Inc.

    Winnipeg, MB
    54 minutes ago
  • Sensor Tower is seeking a Senior Data Engineer to build and own end-to-end data products. You will work with a global team of engineers and...  ...scientists to deliver impactful data solutions. You will develop PySpark-based data programs, perform data mining, and contribute to... 

    Sensor Tower

    Winnipeg, MB
    55 minutes ago
  •  ...Samsara is seeking a Data Engineer in Canada to join our Data team. You will design and maintain data pipelines in a central data lake, using SparkSQL and PySpark to ingest and transform IoT device data and software product data into our core data model for analytics and dashboards... 
    Remote work

    Samsara

    Winnipeg, MB
    55 minutes ago
  • $90k - $110k per year

     ...empowerment. Position OverviewBuild and maintain robust big data processing solutions using Apache Spark, Scala, and Java. Develop scalable applications to...  ...skills. Preferred Qualifications Experience with Kafka, Flink, or other real-time streaming platforms. Familiarity... 
    Big data
    Long term contract

    Tata Consultancy Services Limited

    Winnipeg, MB
    56 minutes ago
  •  ...in Kimberley, is seeking a highly experienced Senior Data Engineer to build and manage their sophisticated big data infrastructure. This is a critical, office-...  ...distributed data processing frameworks like Apache Spark, Hadoop, or Flink. Expertise in SQL and experience with... 
    Big data
    Work at office

    Placements24

    Winnipeg, MB
    59 minutes ago
  • We are looking for a Data Engineer with significant experience (9+ years) in Python and Spark for a full-time onsite position, five days a week. This role is crucial for...  ...Data Engineer, you will utilize your Python and Pyspark skills to design software solutions and conduct... 
    Full time

    Talentola

    Winnipeg, MB
    59 minutes ago
  • $136k - $160k per year

     ...Unchain Data in Toronto is seeking a Software Developer on our Data Engineering team to build core datasets powering analytics, experimentation, and ML across our platform...  ...-to-end data pipelines, Python production code, Spark or Flink, and SQL. The role offers bonus opportunities... 

    Unchain Data

    Winnipeg, MB
    1 hour ago
  • Technology|Analytics - Packages|Python - Big Data, Technology|Cloud Integration|Azure Data...  ...an experienced Azure Databricks Data Engineer to design, develop, and optimize scalable...  ...in Azure data services, Databricks, PySpark, and modern data engineering practices to... 
    Big data
    Relocation

    Infosys Limited

    Winnipeg, MB
    1 hour ago
  • LTM in Mississauga, Ontario is seeking a Specialist with 5 to 7 years of experience in Scala and Apache Spark to design, develop, and optimize scalable big data processing solutions within the Hadoop Spark SparkSQL ecosystem. You will design, implement, and optimize data pipelines... 
    Big data

    LTM

    Winnipeg, MB
    55 minutes ago
  • Build large-scale analytical systems with Apache Spark, Core Java, Python, SQL, and Linux in this experienced Software Engineer opportunity.This Software Engineer role centers...  ...application development and analytical data processing. The position calls for senior hands... 
    Big data

    PeoplePilot

    Winnipeg, MB
    1 hour ago
  • $111k - $160k per year

    Title and Summary Senior AI Engineer, Big Data & AI Solutions Our Purpose Mastercard powers economies and empowers people in 200+ countries...  ...practical experience will also be considered.Proficiency in Python, PySpark, SQL, and distributed data processing.Hands-on experience with... 
    Big data
    Worldwide

    Mastercard Inc.

    Winnipeg, MB
    54 minutes ago
  • Apply Big Data and Spark expertise to Amazon Ads systems in Toronto, helping process billions of behavioral signals through scalable services.As a Software Development Engineer with MADS, you will build cloud-based services for secure, privacy-safe advertiser analytics. Your... 
    Big data
    Internship

    Amazon Inc.

    Winnipeg, MB
    1 hour ago
  • Location: Montreal, QC (REMOTE) Type: Contract Role Overview As a Senior Databricks Data Platform Engineer you will own the engineering automation and day to day health of our enterprise Databricks Lakehouse platform This is a platform-first role rather than building... 
    Contract work
    Remote work

    Enterprise Solutions Inc.

    Winnipeg, MB
    1 hour ago
  • A tech solutions company is looking for an experienced Python/PySpark Developer to design and optimize data-driven applications. This role involves developing applications, utilizing Databricks, and integrating data pipelines. The ideal candidate will possess strong skills... 
    Remote work

    Compunnel, Inc.

    Winnipeg, MB
    1 hour ago
  •  ...Samsara Inc. in Canada is seeking a Data Engineer to join our Data team. You will design and maintain data pipelines using SparkSQL and PySpark in our central data lake, transforming data from IoT devices into our core data model for analytics and dashboards. This remote... 
    Remote work
    Flexible hours

    Samsara Inc.

    Winnipeg, MB
    57 minutes ago
  •  ...Triton Digital Canada Inc. is seeking an MLOps Data Engineer to bridge data science and production systems. You will design and automate CI...  ...CD pipelines, optimize large-scale data processing with Apache Spark, and leverage Databricks to deliver reliable ML solutions. Your... 
    Remote work

    Triton Digital Canada Inc.

    Winnipeg, MB
    1 hour ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Big Data Engineer (Pyspark / Spark / Kafka). Be the first to apply!