Big Data Engineer (Pyspark / Spark / Kafka)
Astra-North Infoteck Inc.
Work Mode:
Hybrid – 3 Days a Week in Office
Job Overview
We are seeking an experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should have hands-on experience in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, and Apache NiFi, along with a strong understanding of data ingestion, transformation, and real-time processing frameworks.
Key Responsibilities
Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi.
Build and maintain batch and real-time data processing solutions.
Develop and support Kafka-based streaming applications and event-driven architectures.
Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets.
Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting.
Implement data ingestion solutions from databases, APIs, files, and streaming sources.
Monitor, troubleshoot, and enhance the performance of Spark jobs and data pipelines.
Collaborate with architects, business analysts, and development teams to deliver high-quality data solutions.
Support platform upgrades, deployments, testing, certification, and production releases.
Ensure data quality, governance, security, and operational excellence across data platforms.
Mandatory Skills
PySpark
Apache Spark (Spark SQL, DataFrames)
Hadoop Ecosystem (HDFS, Hive, YARN)
SQL
Python
Preferred Skills
Spark Streaming
Hive
Scala
Jenkins, Bitbucket, Git
JIRA, Confluence
Data Warehousing Concepts and Dimensional Modeling
#J-18808-Ljbffr
Hybrid – 3 Days a Week in Office
Job Overview
We are seeking an experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should have hands-on experience in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, and Apache NiFi, along with a strong understanding of data ingestion, transformation, and real-time processing frameworks.
Key Responsibilities
Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi.
Build and maintain batch and real-time data processing solutions.
Develop and support Kafka-based streaming applications and event-driven architectures.
Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets.
Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting.
Implement data ingestion solutions from databases, APIs, files, and streaming sources.
Monitor, troubleshoot, and enhance the performance of Spark jobs and data pipelines.
Collaborate with architects, business analysts, and development teams to deliver high-quality data solutions.
Support platform upgrades, deployments, testing, certification, and production releases.
Ensure data quality, governance, security, and operational excellence across data platforms.
Mandatory Skills
PySpark
Apache Spark (Spark SQL, DataFrames)
Hadoop Ecosystem (HDFS, Hive, YARN)
SQL
Python
Preferred Skills
Spark Streaming
Hive
Scala
Jenkins, Bitbucket, Git
JIRA, Confluence
Data Warehousing Concepts and Dimensional Modeling
#J-18808-Ljbffr
Vacancy posted more than 2 months ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Big Data Engineer (Pyspark / Spark / Kafka). Be the first to apply!
Related searches
- data integration developer Winnipeg, MB
- data cabling engineer Winnipeg, MB
- python data engineer Winnipeg, MB
- remote data engineer Winnipeg, MB
- junior data engineer Winnipeg, MB
- data centre engineer Winnipeg, MB
- gcp data engineer Winnipeg, MB
- data engineer Winnipeg, MB
- big data engineer Winnipeg, MB
- data engineering intern summer Winnipeg, MB
