Big Data Engineer (Pyspark / Spark / Kafka)
Astra-North Infoteck Inc.
Work Mode:
Hybrid – 3 Days a Week in Office
Job Overview
We are seeking an experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should have hands-on experience in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, and Apache NiFi, along with a strong understanding of data ingestion, transformation, and real-time processing frameworks.
Key Responsibilities
Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi.
Build and maintain batch and real-time data processing solutions.
Develop and support Kafka-based streaming applications and event-driven architectures.
Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets.
Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting.
Implement data ingestion solutions from databases, APIs, files, and streaming sources.
Monitor, troubleshoot, and enhance the performance of Spark jobs and data pipelines.
Collaborate with architects, business analysts, and development teams to deliver high-quality data solutions.
Support platform upgrades, deployments, testing, certification, and production releases.
Ensure data quality, governance, security, and operational excellence across data platforms.
Mandatory Skills
PySpark
Apache Spark (Spark SQL, DataFrames)
Hadoop Ecosystem (HDFS, Hive, YARN)
SQL
Python
Preferred Skills
Spark Streaming
Hive
Scala
Jenkins, Bitbucket, Git
JIRA, Confluence
Data Warehousing Concepts and Dimensional Modeling
#J-18808-Ljbffr
Hybrid – 3 Days a Week in Office
Job Overview
We are seeking an experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should have hands-on experience in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, and Apache NiFi, along with a strong understanding of data ingestion, transformation, and real-time processing frameworks.
Key Responsibilities
Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi.
Build and maintain batch and real-time data processing solutions.
Develop and support Kafka-based streaming applications and event-driven architectures.
Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets.
Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting.
Implement data ingestion solutions from databases, APIs, files, and streaming sources.
Monitor, troubleshoot, and enhance the performance of Spark jobs and data pipelines.
Collaborate with architects, business analysts, and development teams to deliver high-quality data solutions.
Support platform upgrades, deployments, testing, certification, and production releases.
Ensure data quality, governance, security, and operational excellence across data platforms.
Mandatory Skills
PySpark
Apache Spark (Spark SQL, DataFrames)
Hadoop Ecosystem (HDFS, Hive, YARN)
SQL
Python
Preferred Skills
Spark Streaming
Hive
Scala
Jenkins, Bitbucket, Git
JIRA, Confluence
Data Warehousing Concepts and Dimensional Modeling
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Big Data Engineer (Pyspark / Spark / Kafka) in Winnipeg, MB vacancy
- Astra-North Infoteck Inc. in Toronto seeks an experienced Data Engineer with deep Big Data expertise to design, develop, and support enterprise-scale data platforms. The role focuses on PySpark, Apache Spark, Kafka, Hadoop ecosystem, and NiFi, covering ingestion, transformation...Big data
- Wipro Limited in Mississauga, Canada is seeking an experienced data engineer to design and implement large-scale data pipelines using Python, PySpark and SQL. You will work with Spark, Hive and Hadoop in a Big Data ecosystem to optimize data quality, profiling and ETL/ELT processes...Big data
- Join LTIMindtree as a Big Data Specialist, focusing on Hadoop, Spark, and Scala. Utilize your expertise in data application development to create scalable solutions for large data sets. With 3 to 5 years of experience, you will play a critical role in designing and developing...Big data
- A leading engineering consultancy in Canada is looking for a Senior Software Engineer specialized in Java and Spark. In this role, you will work on a large social media platform processing petabytes of data. The position involves optimizing data systems, designing solutions,...Big data
- ...Tata Consultancy Services Canada Inc. is seeking a Big Data Engineer to design and maintain robust data pipelines using Apache Spark, Scala, and Java for enterprise analytics. You will ingest, transform, and optimize large datasets, focusing on performance and cost efficiency...Big data
- ...The Data Engineer will design, develop, and maintain scalable data pipelines and workflows using Python, PySpark, and cloud technologies. This role involves managing data storage solutions... ...Qualifications Experience with big data tools (e.g., Kafka, Hadoop). Knowledge...Big data
- ...leading IT Services firm in Toronto is seeking a Mid-Senior Level Data Tester to join their team. You will be responsible for testing GCP... ...have 8-10 years of experience in Data testing, proficient in GCP Big Query, Python, and SQL, and possess excellent communication skills...Big dataFull time
- Build GCP data migration solutions with PySpark and Salesforce APIs, focusing on Dataproc, BigQuery, and scalable ETL pipelines.This Data Migration Engineer role requires four to five years minimum of experience... .... You will use Python, Apache Spark, BigQuery, GCS, and Salesforce...
- Build data pipelines with Python, PySpark, and Snowflake while transforming large-scale datasets through distributed processing. Apply deep SQL expertise to reliable engineering work.This Data Engineer role focuses on Python programming, PySpark transformations, and Snowflake...Internship
- ...Responsibilities Design and build realtime and batch data pipelines for fraud detection and AML... ...streaming transaction data using Apache Spark and Kafka. Develop ETLELT workflows supporting... ...expansive, high-performing recruitment engine operating across North America and Asia...Big data
- Databricks and Apache Spark specialists can build scalable data solutions for a six-month contract supporting a top financial client. This hands-on role... ...Databricks, Spark, streaming ETL, and governance expertise to enterprise data engineering for a financial client. #J-18808-LjbffrContract work
- ...metParticipate in team meetings to discuss project updates and progress8-10 years of overall experience with 3-4 years in Data testingKnowledge of GCP Big Query, Python and PysparkExperience in ETL and DB testing, Data migration, Report validationsExpert in writing...Big data
- Build scalable data solutions as a Databricks Developer using Apache Spark and advanced data engineering practices with a top financial client. This six-month contract focuses on enterprise-grade pipelines.Design, develop, and optimise Databricks solutions across large-scale...Contract work
- ...Samsara is seeking a Data Engineer in Canada to join our Data team. You will design and maintain data pipelines in a central data lake, using SparkSQL and PySpark to ingest and transform IoT device data and software product data into our core data model for analytics and dashboards...Remote work
- Senior Software Engineer specializing in Core Java, Apache Spark, and Python develops scalable server-side applications and analytical data systems.This senior engineering role focuses on high-performance applications and large-scale data processing. You will use Core Java,...Big data
$90k - $110k per year
...empowerment. Position OverviewBuild and maintain robust big data processing solutions using Apache Spark, Scala, and Java. Develop scalable applications to... ...skills. Preferred Qualifications Experience with Kafka, Flink, or other real-time streaming platforms. Familiarity...Big dataLong term contract- ...in Kimberley, is seeking a highly experienced Senior Data Engineer to build and manage their sophisticated big data infrastructure. This is a critical, office-... ...distributed data processing frameworks like Apache Spark, Hadoop, or Flink. Expertise in SQL and experience with...Big dataWork at office
- We are looking for a Data Engineer with significant experience (9+ years) in Python and Spark for a full-time onsite position, five days a week. This role is crucial for... ...Data Engineer, you will utilize your Python and Pyspark skills to design software solutions and conduct...Full time
$136k - $160k per year
...Unchain Data in Toronto is seeking a Software Developer on our Data Engineering team to build core datasets powering analytics, experimentation, and ML across our platform... ...-to-end data pipelines, Python production code, Spark or Flink, and SQL. The role offers bonus opportunities...- GCP Data Migration Engineer – PySpark & Salesforce Data MigrationGCP Data Migration Engineer – PySpark & Salesforce Data Migration Required Skills... ...Strong hands-on experience with GCP Dataproc and Apache Spark (PySpark preferred) Python scripting for custom data transformations...
- Sensor Tower is seeking a Senior Data Engineer to build and own end-to-end data products. You will work with a global team of engineers and... ...scientists to deliver impactful data solutions. You will develop PySpark-based data programs, perform data mining, and contribute to...
- LTM in Mississauga, Ontario is seeking a Specialist with 5 to 7 years of experience in Scala and Apache Spark to design, develop, and optimize scalable big data processing solutions within the Hadoop Spark SparkSQL ecosystem. You will design, implement, and optimize data pipelines...Big data
$111k - $160k per year
Title and Summary Senior AI Engineer, Big Data & AI Solutions Our Purpose Mastercard powers economies and empowers people in 200+ countries... ...practical experience will also be considered.Proficiency in Python, PySpark, SQL, and distributed data processing.Hands-on experience with...Big dataWorldwide- Technology|Analytics - Packages|Python - Big Data, Technology|Cloud Integration|Azure Data... ...an experienced Azure Databricks Data Engineer to design, develop, and optimize scalable... ...in Azure data services, Databricks, PySpark, and modern data engineering practices to...Big dataRelocation
- A tech solutions company is looking for an experienced Python/PySpark Developer to design and optimize data-driven applications. This role involves developing applications, utilizing Databricks, and integrating data pipelines. The ideal candidate will possess strong skills...Remote work
- Apply Big Data and Spark expertise to Amazon Ads systems in Toronto, helping process billions of behavioral signals through scalable services.As a Software Development Engineer with MADS, you will build cloud-based services for secure, privacy-safe advertiser analytics. Your...Big dataInternship
- Build large-scale analytical systems with Apache Spark, Core Java, Python, SQL, and Linux in this experienced Software Engineer opportunity.This Software Engineer role centers... ...application development and analytical data processing. The position calls for senior hands...Big data
- ...Samsara Inc. in Canada is seeking a Data Engineer to join our Data team. You will design and maintain data pipelines using SparkSQL and PySpark in our central data lake, transforming data from IoT devices into our core data model for analytics and dashboards. This remote...Remote workFlexible hours
$85k - $90k per year
...Design and maintain scalable data pipelines that collect, transform... ...in Computer Science, Data Engineering, Information Systems, or related... ...role. · Proficiency in Python, PySpark, data modeling, ETL processes... ...data pipeline engineering and big data design principles....Big dataPermanent employment- Location: Montreal, QC (REMOTE) Type: Contract Role Overview As a Senior Databricks Data Platform Engineer you will own the engineering automation and day to day health of our enterprise Databricks Lakehouse platform This is a platform-first role rather than building...Contract workRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Big Data Engineer (Pyspark / Spark / Kafka). Be the first to apply!
Related searches
- data integration developer Winnipeg, MB
- data centre engineer Winnipeg, MB
- data engineer Winnipeg, MB
- python data engineer Winnipeg, MB
- big data engineer Winnipeg, MB
- gcp data engineer Winnipeg, MB
- remote data engineer Winnipeg, MB
- data cabling engineer Winnipeg, MB
- data engineering intern summer Winnipeg, MB
- junior data engineer Winnipeg, MB

