Big Data Engineer (PySpark / Spark / Kafka)
Full-time
Astra North Infoteck Inc.
Job Description: Big Data Engineer (PySpark / Spark / Kafka)
Work Mode: Hybrid – 3 Days a Week in Office
Job Overview
We are seeking an experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should have hands-on experience in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, and Apache NiFi, along with a strong understanding of data ingestion, transformation, and real-time processing frameworks.
Key Responsibilities
- Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi.
- Build and maintain batch and real-time data processing solutions.
- Develop and support Kafka-based streaming applications and event-driven architectures.
- Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets.
- Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting.
- Implement data ingestion solutions from databases, APIs, files, and streaming sources.
- Monitor, troubleshoot, and enhance the performance of Spark jobs and data pipelines.
- Collaborate with architects, business analysts, and development teams to deliver high-quality data solutions.
- Support platform upgrades, deployments, testing, certification, and production releases.
- Ensure data quality, governance, security, and operational excellence across data platforms.
Mandatory Skills
- PySpark
- Apache Spark (Spark SQL, DataFrames)
- Apache Kafka
- Hadoop Ecosystem (HDFS, Hive, YARN)
- Apache NiFi
- SQL
- Python
Preferred Skills
- Spark Streaming
- Airflow / Oozie
- Hive
- Scala
- Jenkins, Bitbucket, Git
- JIRA, Confluence
- Cloud Platforms (GCP/AWS/Azure)
- Data Warehousing Concepts and Dimensional Modeling
Vacancy posted 13 hours ago
Similar jobs that could be interesting for youBased on the Big Data Engineer (PySpark / Spark / Kafka) in Toronto, ON vacancy
$73.9k - $109.36k per year
...Challenge We are looking for a dynamic Data Developer to join a team involved in designing... ...comfortable with ambiguity, able to think big (while paying careful attention to detail)... ...Science, Information Systems, Statistics, Engineering, or similar. Great to Have ~...SuggestedFull timeCasual work- ...Job Description Data Engineer - Python, PySpark, SQL, Snowflake & API Integration Job Description : We are seeking a skilled Data Engineer with strong hands-on experience in Python, PySpark, SQL, Snowflake, and API integrations to build scalable data pipelines...SuggestedContract work
$90k - $125k per year
...trusted by 500 million people. About The Role As a Data Engineer you’ll be tasked with designing, building, and... ...SQL, Python, or related languages. Experience with big data technologies and concepts (Spark, Hadoop, Kafka). Excellent analytical, troubleshooting...Big dataFull timeWork from homeFlexible hours- ...At Electric Mind, Engineering is where strategy meets action. Our team... ...developer to help build and maintain data pipelines on Databricks for a... ...bronze → silver → gold) using PySpark, and publishes curated data to... ...via file exports and Kafka. You'll work alongside a senior...SuggestedPermanent employmentFull timeContract work
- The project involves the data engineering organization’s Analytical Work Bench , which is an integrated... ...Solid expertise in Big Data solutions utilizing AWS EMR.... ...Advanced proficiency in Python Core and PySpark, with over 8 years of hands-on experience...Big data
$180k - $220k per year
...solving. BitGo is seeking a skilled Senior Data Engineer to join our Data Engineering team and... ...best practices (Snowflake, BigQuery, Spark, Presto, Druid, etc) ~ Strong experience... ...systems design and event driven systems (Kafka) ~ A self-starter capable of adapting quickly...Full timeWork at officeWorldwide- ...consultants bring deep expertise in Data Science, Machine Learning and... ...in the world. The Data Engineer will be responsible for... ...proficiency in Python, SQL, and PySpark. ~ Creating and optimizing complex... ...frameworks like Apache Spark. ~ Understanding of Datawarehouse...Local area
- ...Data Engineer (Azure) Job Type: Contract Contract Duration: 6 Months Location: Downtown Toronto, ON (Hybrid - 3 Days Onsite) What You'll Be... ...data transformation solutions using Mapping Data Flows, T-SQL, PySpark, and Python. Design and maintain enterprise data warehouse and...Long term contractContract work
$160k - $220k per year
...let's talk. About the Team The Data Platform team is responsible for... .... We expect great things from our engineers and reward them with stimulating new... ...systems such as Kinesis, Kafka ~ Data processing systems like Flink, Spark, Beam ~ Storage & Compute systems...Local areaWorldwideFlexible hours- ...intelligence. Leading infrastructure vendors, engineering firms, and construction companies use the... ...seamlessly with CRMs, ERPs, and data lakes, embedding actionable insights directly... ...Teamwork Makes the Dream Work as we tackle big challenges and celebrate wins together....Full timeWork at officeImmediate start3 days per week
- ...Responsibilities Design, develop, and own data pipelines, models, and products supporting Product,... ...tools, data marts, and data services. Use Scala, Spark, Airflow, Java, SQL, and Python across the team’s data engineering work. Use LLMs and agents at scale to produce...Full time
- ...Technology (I&T) team, a critical piece of the engine that keeps us in motion. From enterprise... .... Join us! Job Summary The Expert, Data Engineering is responsible to lead/mentor... ...Python, Scala, SQL, Java ~ Experience with Spark Streaming with Java or Scala is a MUST...Full timeWork at officeRemote workMonday to friday
$102.7k - $137k per year
...Position Title: Engineering Manager, Data AI & Analytics Position Type: Regular - Full-Time Requisition ID: 42289 In every role, McCainers... ...in languages such as Python, SQL, Java, Scala, Spark, or PySpark Experience with modern data platforms such as Databricks...Full timeWork from home$95k - $145k per year
Azure Data Engineering Specialist-Databricks Position Description CGI is looking for a skilled and collaborative Azure Data Engineering... ...Azure SQL. • Build batch and streaming data pipelines with Spark/PySpark on Azure Databricks. • Work with large-scale, complex...Big data$108.6k - $155.1k per year
...About the Role: As a Senior Data Engineer at Tubi, you'll be instrumental in developing and... ...organization. Each product team presents unique big data challenges, requiring you to be... ...-to-use, and high quality datasets using Spark and DBT Track down data quality issues...Big dataRemplacementLong term contractFull timeTemporary workWork at officeLocal areaFlexible hoursDay shift2 days per week- ...important work of your career. About the team Product and Risk Data Engineering is Stripe's single source of truth data engineering layer for... ..., is deeply knowledgeable of key tools including Airflow/Spark/Kafka/Flink, is empathetic, excels at building strong relationships...
$80 - $100 per hour
...at a company committed to its people, find out more about what Aviso has to offer at . The Opportunity: We’re looking for a Data Engineer to join our Data and Analytics Group within the Technology Services Partners (TSP) department for 12 months. Reporting to the...Hourly payLong term contractTemporary workInternshipWork at officeRemote work$115k - $130k per year
...intelligence for life sciences - unifying data, accelerating results, and helping commercial... ...OFFER As a Senior Data Implementation Engineer, reporting to the Manager, Technical... ...solutions Utilize modern data tools like PySpark and Snowflake to explore and troubleshoot...Full timeInternshipWork at officeLocal areaImmediate startRemote workWork from homeFlexible hours$110.5k - $143k per year
...live, work and play. It’s an exciting time to join the team at Hydro One! Job Function: Develop and execute comprehensive data governance strategy to ensure effective data management across organizational domains. Collaborate with cross-functional teams to define...$160k - $200k per year
...Traditional handoffs between product, design, and engineering are giving way to smaller, faster teams... ...in days instead of months. As a Senior Data Engineer, you'll own the data platform... ...streaming or event-driven data (e.g., Kafka, Kinesis). Experience supporting data science...Long term contractPermanent employmentFull timeInternship3 days per week$75k - $105k per year
...offering customers affordable monthly payment options for their next big home improvement, vehicle or retail purchase. We are small... ...across a wide range of functions to implement and solve data engineering challenges? Are you equally skilled at delivering simple solutions...Full timeInternshipWork at officeFlexible hours- Job Responsibility: Job Summary What is the opportunity? As a Lead Portfolio Data Engineer, you will support the development and maintenance of scalable data stores that supply big data in forms needed for business analysis. You will apply extensive, in-depth knowledge...Big dataFull timeFlexible hours
- ...We are looking for a Senior Data Architect to design the end-to... ...architectural trade-offs to both engineers and business stakeholders. Working... ...Genesys Cloud experience is a big plus. Define ingestion... ...transformation logic in T-SQL, Spark SQL and Python/PySpark. Document...Hourly payFull timeInternshipWork at officeRemote workRelocation2 days per week
$120k - $130k per year
...our culture. About this Role As a Data Engineer on the Data Science team at OpenTable, you... ...data engineering stack components such as PySpark, Databricks, Snowflake, Airflow,... ...storage and infrastructure solutions such as Kafka, Snowflake, Delta Lake, or vector/NoSQL databases...Work at officeLocal area2 days per week$130k - $160k per year
...million people. About The Role As a Data Architect you’ll be tasked with... ...analysts, scientists, and other engineers to ensure reliable, secure, and efficient... ...solutions. ~ Experience with big data technologies and concepts (Spark, Hadoop, Kafka). ~ Excellent analytical...Big dataWork from homeFlexible hours- Job Responsibility: Cloud Data Engineers Toronto, Hybrid Experience and qualifications: 1. Must have working knowledge in designing and implementing data pipelines on any of the cloud providers (AWS is preferred). Must be able to work with large volumes of data coming from...Full timeShift work
$120k - $150k per year
...consolidation and enrichment of any data, DataVisor's fraud and anti-... ..., powerful decision engine, and investigation tools work together... ...team of world-class experts in big data, machine learning, security... ...billions of events) using tools like Spark, SQL, and our proprietary AI...Big dataFull time$95k - $145k per year
Senior Data Platform Engineering Specialist Position Description Location: Toronto/Montreal Preferred... ...delivering Data Engineering, DataOps, Big Data, or Analytics solutions.... ...tools, including: Databricks / Apache Spark Snowflake Apache Airflow, dbt (data...Big dataLong term contractPermanent employmentFull time- ...undertaking significant investment in its enterprise data and cloud capabilities. They are looking for an experienced GCP Data Engineer to support the design, development, and... ...concepts. Experience with Apache Beam, Airflow, Spark, or comparable data-processing and...Contract work
$120k - $140k per year
...Requisition Number: 106522 Data Engineer Sr Location ~ This role is hybrid, with the onsite locations in Montreal, QC, Canada or... ...Experience with distributed data processing technologies such as Spark or equivalent frameworks. ~ Hands-on experience with one or...ApprenticeshipWork at office
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Big Data Engineer (PySpark / Spark / Kafka). Be the first to apply!
Related searches
- remote data engineer Toronto, ON
- data centre engineer Toronto, ON
- data cabling engineer Toronto, ON
- data engineer Toronto, ON
- data integration developer Toronto, ON
- python data engineer Toronto, ON
- junior data engineer Toronto, ON
- data engineering intern summer Toronto, ON
- big data engineer Toronto, ON
- gcp data engineer Toronto, ON