Data Engineer Python/Pyspark/Openshift
Compunnel Inc.
The Data Engineer will design, develop, and maintain scalable data pipelines and workflows using Python, PySpark, and cloud technologies.
This role involves managing data storage solutions, orchestrating ETL processes, and deploying containerized applications to support enterprise data initiatives.
Key Responsibilities
Develop, construct, test, and maintain scalable data pipelines and ETL/ELT workflows using Python, PySpark, and SQL.
Manage and optimize data storage solutions in Snowflake and Data Lakes.
Integrate data from multiple sources while ensuring data quality, integrity, and security.
Design and implement automated workflows using Azure Data Factory or similar orchestration tools.
Deploy and manage containerized applications and services on OpenShift.
Collaborate with Data Scientists, Analysts, and stakeholders to understand data requirements.
Monitor and troubleshoot data pipelines, resolving performance bottlenecks.
Maintain documentation of data architecture, workflows, and best practices.
Ensure compliance with data governance and security policies.
Required Qualifications
Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
Minimum 10 years of overall experience in data engineering or similar roles.
Strong proficiency in Python and PySpark.
Expertise in SQL and experience with Snowflake Data Warehouse.
Hands-on experience with data orchestration tools such as Azure Data Factory.
Knowledge of Data Lakes architecture and management.
Experience deploying and managing containerized applications on OpenShift or Kubernetes.
Familiarity with cloud platforms (Azure, AWS, GCP).
Understanding of data security, governance, and compliance best practices.
Strong problem-solving skills and attention to detail.
Excellent communication and teamwork abilities.
Preferred Qualifications
Experience with big data tools (e.G., Kafka, Hadoop).
Knowledge of CI/CD pipelines and DevOps practices.
Familiarity with additional cloud data services and tools.
#J-18808-Ljbffr
This role involves managing data storage solutions, orchestrating ETL processes, and deploying containerized applications to support enterprise data initiatives.
Key Responsibilities
Develop, construct, test, and maintain scalable data pipelines and ETL/ELT workflows using Python, PySpark, and SQL.
Manage and optimize data storage solutions in Snowflake and Data Lakes.
Integrate data from multiple sources while ensuring data quality, integrity, and security.
Design and implement automated workflows using Azure Data Factory or similar orchestration tools.
Deploy and manage containerized applications and services on OpenShift.
Collaborate with Data Scientists, Analysts, and stakeholders to understand data requirements.
Monitor and troubleshoot data pipelines, resolving performance bottlenecks.
Maintain documentation of data architecture, workflows, and best practices.
Ensure compliance with data governance and security policies.
Required Qualifications
Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
Minimum 10 years of overall experience in data engineering or similar roles.
Strong proficiency in Python and PySpark.
Expertise in SQL and experience with Snowflake Data Warehouse.
Hands-on experience with data orchestration tools such as Azure Data Factory.
Knowledge of Data Lakes architecture and management.
Experience deploying and managing containerized applications on OpenShift or Kubernetes.
Familiarity with cloud platforms (Azure, AWS, GCP).
Understanding of data security, governance, and compliance best practices.
Strong problem-solving skills and attention to detail.
Excellent communication and teamwork abilities.
Preferred Qualifications
Experience with big data tools (e.G., Kafka, Hadoop).
Knowledge of CI/CD pipelines and DevOps practices.
Familiarity with additional cloud data services and tools.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Data Engineer Python/Pyspark/Openshift in Winnipeg, MB vacancy
- Build data pipelines with Python, PySpark, and Snowflake while transforming large-scale datasets through distributed processing. Apply deep SQL expertise to reliable engineering work.This Data Engineer role focuses on Python programming, PySpark transformations, and Snowflake...SuggestedInternship
LanceSoft, Inc.
Winnipeg, MB1 hour ago - A tech solutions company is looking for an experienced Python/PySpark Developer to design and optimize data-driven applications. This role involves developing applications, utilizing Databricks, and integrating data pipelines. The ideal candidate will possess strong skills in...SuggestedRemote work
Compunnel, Inc.
Winnipeg, MB1 hour ago - ...Luxoft in Toronto is seeking a senior data engineer to support the Analytical Work Bench integration, including AWS-based data pipelines and... ...quality with standard testing practices. Join a team that leverages PySpark, PyTest, and Pylint, while integrating with CI/CD tooling such...Suggested
Luxoft (old)
Winnipeg, MB1 hour ago - ...We are seeking an experienced Python/PySpark Developer to design, develop, and optimize data-driven applications and solutions. The ideal candidate will have strong expertise in Python, PySpark, and Databricks, along with experience in Azure DevOps and enterprise data integration...SuggestedFull time
Compunnel Inc.
Winnipeg, MB1 hour ago - # Forward Deployed Engineer (Python/Pyspark Developer)Apptoza Inc. · Toronto, Ontario, CABack to resultsForward Deployed Engineer (Python/Pyspark Developer... ...Engineer who combines production-grade software engineering, data engineering, AI application development, and client-facing...Suggested
Linuxassociation
Winnipeg, MB1 hour ago - Apply your advanced skills in a role as an Expert Python/PySpark Developer, dedicated to enhancing data-driven solutions. This position offers flexible modes of teamwork and application development. We are in search of an experienced Python/PySpark Developer, enriched with...Flexible hours
Compunnel Inc.
Winnipeg, MB1 hour ago - Build analytical data solutions with Python, PySpark, and AWS in downtown Toronto, working onsite four days weekly on a hybrid contract.This senior development assignment supports a global IT services firm serving a leading investment and wealth management services firm. You...Contract workWork at office
C.G. Consulting Group
Winnipeg, MB1 hour ago $85k - $90k per year
...Description Design and maintain scalable data pipelines that collect, transform and... ...or Diploma in Computer Science, Data Engineering, Information Systems, or related field.... ...Engineer or similar role. · Proficiency in Python, PySpark, data modeling, ETL processes and data...Permanent employment$75 - $100 per hour
Senior Python Developer with PySpark and AWSThis is with a global IT services firm for a leading investment/wealth management services firm.... ...100hr on B2B Project descriptionThe project involves the data engineering organization’s Analytical Work Bench, which is an integrated...Contract workWork at officeC.G. Consulting Group
Winnipeg, MB1 hour ago- Build Python systems for data lineage, code parsing, and metadata analysis with I8IS INC. in Mississauga... ..., Ontario. Develop accurate, scalable engineering solutions.This full-time developer... ...transformation.Key Responsibilities: Design engines parsing programming and query...Full time
Linuxassociation
Winnipeg, MB1 hour ago - Use Python, Spark, and Airflow to create scalable pipelines for Robinhood’s data lake. Work in Toronto on foundational financial datasets.This Senior Data Engineer role focuses on reliable data infrastructure supporting Robinhood’s metrics and products. You will develop application...
Robinhood
Winnipeg, MB1 hour ago - Build trusted data foundations at Foresight using Python and advanced SQL. Work from New York, San Francisco, or Montreal, with occasional travel.As a Senior Data Engineer, you will develop production pipelines for complex project data. Your work supports predictions, AI recommendations...
Foresight Works
Winnipeg, MB1 hour ago - Python data engineering leads this intermediate position in Toronto, with backend services, analytics, and hybrid work two days onsite weekly.Randstad Canada presents an intermediate Software Engineer opportunity with a leading SaaS company. The work centers on backend development...Work at office
Linuxassociation
Winnipeg, MB1 hour ago - Step into a senior Data Engineer role focusing on developing data pipelines with Python and SQL. This position is ideal for experts in cloud technologies. You'll be responsible... ...and quality, while managing deployments on OpenShift and utilizing Azure Data Factory for process...
Compunnel Inc.
Winnipeg, MB1 hour ago - ...professionals, and our annual growth rate exceeds 25%. Job Description Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare client. You will play a key role in developing and enhancing an enterprise AI...Full timeInternshipRemote workRelocation
- Build Python tooling and APIs as a Data Platform Engineer at BRP, improving platform efficiency through automation, CI/CD, and scalable operations.Working in BRP’s Data Solutions group, you will extend data platform capabilities and improve efficiency. You will develop custom...
BRP
Winnipeg, MB1 hour ago - Engineer monitored, governed data flows with Python and Spark for Croesus's wealth management platforms. This hybrid position requires two weekly office days in Laval or Montréal.The Data Engineering Expert develops and operates modern Databricks Lakehouse pipelines supporting...Work at office
Croesus
Winnipeg, MB1 hour ago - Build complex data pipelines with Python and SQL Server while working onsite five days weekly alongside operational teams. Maintain extremely accurate reporting solutions and data systems.As a Data Engineer, you will develop, test, deploy, and maintain ELT pipelines from varied...
Petroglyph Development Group
Winnipeg, MB1 hour ago - Build scalable data solutions in Markham, ON, using advanced SQL and Python with Pandas in a hybrid, three-days-on-site schedule.This Senior Data Engineer, Senior BI, or Analytics Engineering Lead designs reliable solutions for business intelligence, reporting, and advanced...
Altis
Winnipeg, MB1 hour ago - Software Engineer, Data Systems (Python)Remote - Canada About NorthbeamNorthbeam is building the world's most advanced marketing intelligence platform, providing top eCommerce brands a unified view of their business data through powerful attribution modeling and customizable...Remote workWork from homeFlexible hours
Northbeam LLC
Winnipeg, MB1 hour ago - Build reliable data systems remotely at Intrinsic Development using Python, SQL, and modern ETL pipelines. Daily AI use shapes this engineering work.As a Data and AI Engineer, you will build and operate data platforms in a product-led environment. The work includes ETL and ELT...Remote work
Y-Axis
Winnipeg, MB1 hour ago - OpenShift and Kubernetes expertise define this hands-on DevOps Engineer role in Toronto, with hybrid work requiring four days onsite weekly.Support and maintain applications... ...and container technologiesShell, Bash, Python, or similar scripting experienceAnalytical, self-...
Epsilon Solutions Ltd.
Winnipeg, MB1 hour ago - ...Price Café offers free Tea and Coffee. Data Steward Price Electronics, a division... ...Winnipeg, Manitoba DEPARTMENT Software Engineering LOCATION Winnipeg, MB (on-site)... ...build repeatable checks in Excel, SQL or Python; and create exception reports and quality...Long term contractPermanent employmentFull timeContract workWork at office
- ...leading IT Services firm in Toronto is seeking a Mid-Senior Level Data Tester to join their team. You will be responsible for testing... ...years of experience in Data testing, proficient in GCP Big Query, Python, and SQL, and possess excellent communication skills. The position...Full time
Astra-North Infoteck Inc.
Winnipeg, MB1 hour ago - ...Scotiabank is seeking a Software Engineer (Python) in Toronto to design, develop, and support enterprise applications and data platforms. The role emphasizes Python, SQL, APIs, and scalable data pipelines, collaborating with business and technology teams to deliver high-performance...
Scotiabank
Winnipeg, MB1 hour ago - US Tech Solutions is seeking an Application Developer - Sr Python Engineer for a project-based vendor integration initiative in Toronto. The role... ...and APIs, wiring systems via REST/gRPC, and modernizing data platforms across Azure Databricks, Delta Lake, ADLS, Azure Data...
Us Tech Solutions
Winnipeg, MB1 hour ago - Validate enterprise data pipelines with Databricks and PySpark as an ETL QA Engineer at Kumaran Systems. Apply AI-assisted testing expertise.This ETL QA Engineer validates large-scale data pipelines and ensures data quality. The work centers on Databricks, PySpark, ETL/ELT concepts...Work at office
KUMARAN SYSTEMS
Winnipeg, MB1 hour ago - ...We are looking for a senior Forward Deployed Engineer who combines production-grade software engineering, data engineering, AI application development, and client... .... Profile unfamiliar data using Python, pandas, PySpark, and SQL to assess coverage, quality, volumes,...
Astra-North Infoteck Inc.
Winnipeg, MB1 hour ago - ...radio inventory. The listener data team is at the heart of that... ...The RoleAs our MLOps Data Engineer, you’ll be the bridge between... ...processing pipelines in Apache Spark (PySpark and Scala) for large-scale,... ...Strong programming skills in Python, with hands-on experience in Apache...Remote work
Triton Digital Canada Inc.
Winnipeg, MB1 hour ago - ...HRbrain Inc. in Toronto is seeking a Senior Data Engineer to join the Data & Analytics team within the Data & Technology group, reporting directly... ...requires broad technical experience (Informatica IICS, Snowflake, Python, AWS) and a focus on batched reporting. #J-18808-Ljbffr
HRbrain Inc.
Winnipeg, MB1 hour ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Data Engineer Python/Pyspark/Openshift. Be the first to apply!
Related searches
- data integration developer Winnipeg, MB
- data centre engineer Winnipeg, MB
- data engineer Winnipeg, MB
- python data engineer Winnipeg, MB
- big data engineer Winnipeg, MB
- gcp data engineer Winnipeg, MB
- remote data engineer Winnipeg, MB
- data cabling engineer Winnipeg, MB
- data engineering intern summer Winnipeg, MB
- junior data engineer Winnipeg, MB