Data Engineer Python/Pyspark/Openshift
Compunnel Inc.
The Data Engineer will design, develop, and maintain scalable data pipelines and workflows using Python, PySpark, and cloud technologies.
This role involves managing data storage solutions, orchestrating ETL processes, and deploying containerized applications to support enterprise data initiatives.
Key Responsibilities
Develop, construct, test, and maintain scalable data pipelines and ETL/ELT workflows using Python, PySpark, and SQL.
Manage and optimize data storage solutions in Snowflake and Data Lakes.
Integrate data from multiple sources while ensuring data quality, integrity, and security.
Design and implement automated workflows using Azure Data Factory or similar orchestration tools.
Deploy and manage containerized applications and services on OpenShift.
Collaborate with Data Scientists, Analysts, and stakeholders to understand data requirements.
Monitor and troubleshoot data pipelines, resolving performance bottlenecks.
Maintain documentation of data architecture, workflows, and best practices.
Ensure compliance with data governance and security policies.
Required Qualifications
Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
Minimum 10 years of overall experience in data engineering or similar roles.
Strong proficiency in Python and PySpark.
Expertise in SQL and experience with Snowflake Data Warehouse.
Hands-on experience with data orchestration tools such as Azure Data Factory.
Knowledge of Data Lakes architecture and management.
Experience deploying and managing containerized applications on OpenShift or Kubernetes.
Familiarity with cloud platforms (Azure, AWS, GCP).
Understanding of data security, governance, and compliance best practices.
Strong problem-solving skills and attention to detail.
Excellent communication and teamwork abilities.
Preferred Qualifications
Experience with big data tools (e.G., Kafka, Hadoop).
Knowledge of CI/CD pipelines and DevOps practices.
Familiarity with additional cloud data services and tools.
#J-18808-Ljbffr
This role involves managing data storage solutions, orchestrating ETL processes, and deploying containerized applications to support enterprise data initiatives.
Key Responsibilities
Develop, construct, test, and maintain scalable data pipelines and ETL/ELT workflows using Python, PySpark, and SQL.
Manage and optimize data storage solutions in Snowflake and Data Lakes.
Integrate data from multiple sources while ensuring data quality, integrity, and security.
Design and implement automated workflows using Azure Data Factory or similar orchestration tools.
Deploy and manage containerized applications and services on OpenShift.
Collaborate with Data Scientists, Analysts, and stakeholders to understand data requirements.
Monitor and troubleshoot data pipelines, resolving performance bottlenecks.
Maintain documentation of data architecture, workflows, and best practices.
Ensure compliance with data governance and security policies.
Required Qualifications
Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
Minimum 10 years of overall experience in data engineering or similar roles.
Strong proficiency in Python and PySpark.
Expertise in SQL and experience with Snowflake Data Warehouse.
Hands-on experience with data orchestration tools such as Azure Data Factory.
Knowledge of Data Lakes architecture and management.
Experience deploying and managing containerized applications on OpenShift or Kubernetes.
Familiarity with cloud platforms (Azure, AWS, GCP).
Understanding of data security, governance, and compliance best practices.
Strong problem-solving skills and attention to detail.
Excellent communication and teamwork abilities.
Preferred Qualifications
Experience with big data tools (e.G., Kafka, Hadoop).
Knowledge of CI/CD pipelines and DevOps practices.
Familiarity with additional cloud data services and tools.
#J-18808-Ljbffr
Vacancy posted 47 minutes ago
Similar jobs that could be interesting for youBased on the Data Engineer Python/Pyspark/Openshift in Winnipeg, MB vacancy
- A leading data engineering firm is seeking a Data Engineer to design and maintain scalable data pipelines using Python and PySpark. This role requires managing data storage solutions, orchestrating ETL processes, and deploying containerized applications. The ideal candidate...Suggested
- Build scalable data solutions with Python 3.X, PySpark, and SQL while working across large-scale transformation workflows. Apply distributed computing and cloud platform experience to modern data engineering.This Python PySpark Data Engineer position focuses on reusable Python...Suggested
- A tech solutions company is looking for an experienced Python/PySpark Developer to design and optimize data-driven applications. This role involves developing applications, utilizing Databricks, and integrating data pipelines. The ideal candidate will possess strong skills in...SuggestedRemote work
- Luxoft in Toronto is seeking a senior data engineer to support the Analytical Work Bench integration, including AWS-based data pipelines and... ...quality with standard testing practices. Join a team that leverages PySpark, PyTest, and Pylint, while integrating with CI/CD tooling such...Suggested
- Strong programming experience in Python 3.x , including OOP, data structures, exception handling, and reusable modules. Hands-on experience with PySpark, Apache Spark, Spark SQL, and Spark DataFrames . Strong expertise in SQL, query optimization, and relational databases...Suggested
- Build scalable data integration solutions with Databricks and PySpark at Teqfocus, using SQL and Python to process enterprise data. Work hands-on across Delta Lake and master data... ...ingestion workflows.Teqfocus seeks a Data Engineer with five or more years of professional experience...
- ...We are seeking an experienced Python/PySpark Developer to design, develop, and optimize data-driven applications and solutions. The ideal candidate will have strong expertise in Python, PySpark, and Databricks, along with experience in Azure DevOps and enterprise data integration...Full time
- # Forward Deployed Engineer (Python/Pyspark Developer)Apptoza Inc. · Toronto, Ontario, CABack to resultsForward Deployed Engineer (Python/Pyspark Developer... ...Engineer who combines production-grade software engineering, data engineering, AI application development, and client-facing...
- Build GCP data migration solutions with PySpark and Salesforce APIs, focusing on Dataproc, BigQuery, and scalable ETL pipelines.This Data Migration Engineer role requires four to five years minimum of experience. You will use Python, Apache Spark, BigQuery, GCS, and Salesforce...
- Utilisez Python et PySpark pour transformer les données chez BMR, dans un rôle axé sur l’intégration, la modélisation et les solutions BI. Collaborez avec les analystes d’affaires.Comme Développeur de données, vous soutiendrez l’évolution de l’architecture et la transformation...
- Advanced Python and PySpark define this full-time permanent role with Citi in Mississauga, Canada... ...development experience to large-volume data processing.The Python Developer will apply... ...and advanced Python skills across data engineering and ETL development. The position...Permanent employmentFull time
- GCP Data Migration Engineer – PySpark & Salesforce Data MigrationGCP Data Migration Engineer – PySpark & Salesforce Data Migration Required Skills... ...with GCP Dataproc and Apache Spark (PySpark preferred) Python scripting for custom data transformations and business logic...
- Apply your advanced skills in a role as an Expert Python/PySpark Developer, dedicated to enhancing data-driven solutions. This position offers flexible modes of teamwork and application development. We are in search of an experienced Python/PySpark Developer, enriched with...Flexible hours
- Build analytical data solutions with Python, PySpark, and AWS in downtown Toronto, working onsite four days weekly on a hybrid contract.This senior development assignment supports a global IT services firm serving a leading investment and wealth management services firm. You...Contract workWork at office
- ...Overview We are seeking an experienced Data Engineer with strong expertise in Big Data... ...candidate should have hands-on experience in PySpark, Apache Spark, Kafka, Hadoop ecosystem components... ...Ecosystem (HDFS, Hive, YARN) SQL Python Preferred Skills Spark Streaming...Work at office3 days per week
- ...Project description The project involves the data engineering organization’s Analytical Work Bench , which is an integrated development... ...solutions utilizing AWS EMR. Advanced proficiency in Python Core and PySpark, with over 8 years of hands-on experience. Practical...
- Location: Montreal, QC (REMOTE) Type: Contract Role Overview As a Senior Databricks Data Platform Engineer you will own the engineering automation and day to day health of our enterprise Databricks Lakehouse platform This is a platform-first role rather than building...Contract workRemote work
- Design enterprise AWS data platforms with Python, SQL, and ETL expertise as a Senior AWS Data Engineer on a twelve-month contract.This senior individual contributor role requires... ...-grade cloud solutions. Apply AWS Glue, PySpark, Lambda, S3, Redshift, and streaming technologies...Contract work
- Build reliable data pipelines with SparkSQL and PySpark as a remote Data Engineer at Samsara, serving candidates across Canada. Transform IoT data into actionable models... ...large-volume ETL pipelinesExpertise in SQL, Python, REST APIs, and Spark platformsApply data modeling...Remote work
$85k - $90k per year
...Description Design and maintain scalable data pipelines that collect, transform and... ...or Diploma in Computer Science, Data Engineering, Information Systems, or related field.... ...Engineer or similar role. · Proficiency in Python, PySpark, data modeling, ETL processes and data...Permanent employment- Build scalable data infrastructure at Nasdaq using Python, cloud platforms, and distributed pipelines. Support a global data marketplace serving hundreds of thousands of professionals.As a Data Engineering Specialist, reporting to the Senior Director of Data Engineering, you...
- Step into a senior Data Engineer role focusing on developing data pipelines with Python and SQL. This position is ideal for experts in cloud technologies. You'll be responsible... ...and quality, while managing deployments on OpenShift and utilizing Azure Data Factory for process...
- Astra-North Infoteck Inc. in Toronto seeks an experienced Data Engineer with deep Big Data expertise to design, develop, and support enterprise-scale data platforms. The role focuses on PySpark, Apache Spark, Kafka, Hadoop ecosystem, and NiFi, covering ingestion, transformation...
- ...professionals, and our annual growth rate exceeds 25%. Job Description Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare client. You will play a key role in developing and enhancing an enterprise AI...Full timeInternshipRemote workRelocation
- ...OpportunityOpticca Consulting is seeking a highly experienced Senior OpenShift Engineer to design, deploy, secure, operate, and continuously... ...Cluster Management, Advanced Cluster Security, OpenShift Data Foundation, Quay, Service Mesh, GitOps, or Pipelines....Flexible hours
- Build Python tooling and APIs as a Data Platform Engineer at BRP, improving platform efficiency through automation, CI/CD, and scalable operations.Working in BRP’s Data Solutions group, you will extend data platform capabilities and improve efficiency. You will develop custom...
- Build complex data pipelines with Python and SQL Server while working onsite five days weekly alongside operational teams. Maintain extremely accurate reporting solutions and data systems.As a Data Engineer, you will develop, test, deploy, and maintain ELT pipelines from varied...
- Build scalable data pipelines with SQL and Python as an in-office Data and Software Engineer at General Bank of Canada in Edmonton. Develop reliable data solutions for enterprise banking operations.This permanent full-time role supports data integration, transformation, modeling...Permanent employmentFull timeWork at office
- Engineer monitored, governed data flows with Python and Spark for Croesus's wealth management platforms. This hybrid position requires two weekly office days in Laval or Montréal.The Data Engineering Expert develops and operates modern Databricks Lakehouse pipelines supporting...Work at office
- Build Heylist’s data foundation with Python and SQL as a full-time Data Engineer working from a flexible location. Shape reliable systems for creator, campaign, and social data.Heylist seeks a rigorous, product-minded Data Engineer with five-plus years of experience. You will...Full timeFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Data Engineer Python/Pyspark/Openshift. Be the first to apply!
Related searches
- data integration developer Winnipeg, MB
- data cabling engineer Winnipeg, MB
- python data engineer Winnipeg, MB
- remote data engineer Winnipeg, MB
- junior data engineer Winnipeg, MB
- data centre engineer Winnipeg, MB
- gcp data engineer Winnipeg, MB
- data engineer Winnipeg, MB
- big data engineer Winnipeg, MB
- data engineering intern summer Winnipeg, MB

