Sr. Data Engineer

On behalf of our public sector client, Affinity is seeking a Sr. Data Engineer.

Responsibilities

Data Engineering:

• Design, build, and maintain scalable data pipelines across on-premises and cloud platforms (Azure, Databricks, Microsoft Fabric, GCP, AWS) to ingest, transform, and store diverse datasets in support of enterprise business use cases.
• Develop, optimize, and maintain data models, including dimensional models (star and snowflake schemas), to improve query performance, scalability, and usability for analytics and reporting.
• Integrate data from a variety of sources, including relational databases, NoSQL platforms, APIs, and files, applying AI-enabled data integration techniques such as intelligent data mapping, schema discovery, metadata enrichment, and automated data quality validation to improve accuracy and efficiency.
• Enhance ETL/ELT processes through optimization, automation, and performance tuning to improve scalability, reduce bottlenecks, and support high-volume data processing.
• Develop and operate end-to-end ETL/ELT workflows using tools such as SSIS, Azure/Fabric Data Factory, Dataflows, and Notebooks, incorporating data validation, error handling, logging, monitoring, and scheduling to ensure reliable data operations.
• Automate data pipeline deployment and operations through CI/CD practices, including automated testing, release management, and monitoring to enable faster and more reliable delivery.
• Support the management and governance of enterprise data platforms, including data lakes, data warehouses, security controls, and access management.
• Partner with architects, developers, and stakeholders to translate requirements into solutions, and prepare curated data marts and fact/dimension tables to support analytics.

Data Analytics

• Analyze datasets to identify trends, patterns, and anomalies. Use statistical methods, DAX, Python, and R to generate insights that inform business strategies.
• Develop interactive Power BI dashboards and reports, leveraging DAX to create calculated columns and measures, monitor key performance indicators, deliver service dashboards, and communicate results effectively to stakeholders.
• Build predictive or descriptive models using statistical, Python, or R-based machine learning methods. Design and integrate data models to improve service delivery.
• Present findings to non-technical audiences in clear, actionable terms. Translate complex data into business-focused insights and recommendations.
• Deliver analytics solutions iteratively in an Agile environment. Mentor teams to enhance analytics fluency and support self-service capabilities.
• Provide data-driven analysis, visualizations, and AI-enabled insights to support corporate priorities, strategic initiatives, and informed decision-making.

Requirements

• Bachelor’s degree in Computer Science, Information Technology, Engineering, Mathematics, Statistics, Data Science, or a related field.
• 5+ years of experience working as a Data Engineer and/or Data Analyst.
• 5+ years of experience using Python (including PySpark) and SQL to develop, orchestrate, and optimize enterprise-grade ETL/ELT workflows in large-scale cloud environments.
• 4+ years of experience in Business Intelligence and Executive Reporting, including developing executive dashboards, KPI reporting, self-service BI solutions, and business performance reporting.
• 4+ years of experience with cloud or hybrid data platforms, including cloud modernization and/or hybrid/cloud data platform implementations.
• 4+ years of experience with data warehouse and lakehouse design, including enterprise data warehouses or lakehouses using star/snowflake schemas and fact/dimension modelling.
• 4+ years of experience designing and implementing ETL processes and data pipelines, including transforming and loading data from multiple sources into data warehouses.
• 2+ years of experience with data migration and modernization, including planning, executing, validating, and supporting migrations across on-premises, cloud, and cross-database environments.
• 2+ years of experience with DevOps, CI/CD, and Infrastructure as Code, including designing, implementing, or maintaining automated deployment pipelines and infrastructure management practices for cloud-based data platforms and services.
• 2+ years of experience supporting enterprise-scale public-sector applications or working in public-sector or mixed delivery environments.
• 1+ year of experience using AI-assisted development tools and practices to improve productivity, code quality, documentation, testing, and data engineering workflows, with appropriate review and quality controls.
• 1+ year of experience with modern data technologies, such as Microsoft Fabric, Databricks, Apache Spark, Delta Lake, or similar lakehouse and big data platforms.

Affinity Earn

Know someone who’s great for this, or any of our open roles? Earn up to $4,000/year for each successful referral through Affinity Earn. You can also earn up to $50,000 for helping us find new clients. Learn about our referral program at or browse our jobs & follow us at

About Affinity

Affinity Group is a technology and business consulting and services company. We believe in creating long term relationships between clients and consultants that foster a mutually beneficial partnership. Affinity is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All employment is decided on the basis of qualifications, merit and business need.

For more information on Affinity, please visit

Job Number: 13815