Kafka Platform Administrator

calgary, AB based, working from office 5 days a week

Role Summary:

We are looking for an experienced Kafka Platform Administrator / Kafka Developer with strong expertise in Apache Kafka, Kafka on-premises and cloud environments, and Red Hat OpenShift Service on AWS (ROSA). The ideal candidate should have hands-on experience in Kafka platform administration, cluster management, topic administration, connector development, and ongoing maintenance of existing integrations. The role requires a combination of platform engineering, development, operational support, and monitoring skills to ensure reliable, scalable, and highly available event-driven integration solutions.

Responsibilities

Administer, configure, and maintain Kafka platforms across on-premises, cloud, and Red Hat OpenShift Service on AWS (ROSA) environments. Manage Kafka clusters, brokers, topics, partitions, replication factors, retention policies, consumer groups, and access controls. Perform Kafka platform administration, including cluster health monitoring, capacity planning, performance tuning, and availability management. Deploy, configure, and support Kafka components and services across multiple environments, including development, testing, staging, and production. Develop, configure, deploy, and maintain Kafka connectors using Kafka Connect and related integration frameworks. Create new source and sink connectors to integrate Kafka with databases, enterprise applications, cloud services, and other external systems. Maintain and enhance existing connectors, troubleshoot connectivity and data ingestion issues, and implement fixes, upgrades, and performance improvements. Support connector lifecycle management, including configuration, deployment, validation, error handling, monitoring, and recovery. Manage Kafka topics and schemas, including topic creation, configuration changes, schema compatibility, and data retention requirements. Work with Kafka Connect, connector plugins, serialization formats, and data transformation mechanisms to support reliable data movement. Use Kafka CLI commands and administrative tools to manage topics, consumer groups, configurations, offsets, ACLs, and cluster operations. Troubleshoot producer and consumer issues, consumer lag, broker failures, connector task failures, rebalancing, and message delivery problems. Configure and maintain monitoring, alerting, and dashboards using Datadog to track Kafka cluster health, throughput, latency, consumer lag, connector status, and infrastructure utilization. Develop and maintain Datadog monitors, alerts, dashboards, and incident notifications to proactively identify platform and integration issues. Use GitHub for source control, connector configuration management, code reviews, versioning, and collaboration across development and operations teams. Support CI/CD pipelines and automated deployment processes for Kafka configurations, connector changes, and application releases. Provide L2/L3 production support, participate in incident management, perform root cause analysis (RCA), and implement preventive measures. Maintain operational documentation, runbooks, standard operating procedures, and platform administration guidelines. Collaborate with application developers, infrastructure teams, architects, security teams, and business stakeholders to deliver secure and reliable event-driven integration solutions. Follow best practices for Kafka security, platform governance, high availability, disaster recovery, and operational excellence.

Required Skills

Minimum 8 years of experience in middleware, messaging platforms, streaming technologies, platform administration, or integration development, with strong hands-on Kafka experience. Strong expertise in Apache Kafka administration and development across on-premises and cloud-based environments. Hands-on experience with Kafka on Red Hat OpenShift Service on AWS (ROSA), including deployment, configuration, operations, troubleshooting, and platform support. Experience administering Kafka clusters, brokers, topics, partitions, replication, retention settings, consumer groups, and access control lists (ACLs).Strong understanding of Kafka architecture, including producers, consumers, brokers, partitions, offsets, replication, and consumer group management. Practical experience with Kafka Connect, including connector deployment, configuration, troubleshooting, and lifecycle management. Experience developing or customizing Kafka source and sink connectors and maintaining existing connectors in production environments. Strong knowledge of connector configuration parameters, plugin management, connector tasks, error handling, retries, dead-letter queues, and offset management. Proficiency in Kafka CLI commands and administrative utilities for topic management, consumer group inspection, configuration changes, ACL management, and troubleshooting. Experience with command-line tools and scripting for platform administration, automation, and operational support. Hands-on experience with GitHub, including repositories, branching strategies, pull requests, code reviews, version control, and configuration management. Experience with Datadog monitoring, including dashboards, monitors, alerting, log analysis, metrics, and troubleshooting production incidents. Strong understanding of Kafka security, authentication, authorization, encryption, TLS, SASL, and access management. Experience troubleshooting Kafka performance issues, broker availability, connector failures, consumer lag, throughput bottlenecks, and message processing errors. Knowledge of REST APIs, JSON, Avro, XML, and other data serialization or integration formats. Experience working with relational databases, SQL, and enterprise systems for source/sink integrations. Understanding of Linux/Unix environments, networking fundamentals, DNS, certificates, and connectivity troubleshooting. Experience with CI/CD processes, deployment automation, configuration management, and environment promotion. Strong analytical, debugging, problem-solving, documentation, and communication skills.

Good to Have

Experience with Red Hat OpenShift, Kubernetes, containerized workloads, and cloud-native platform operations. Exposure to Red Hat Streams for Apache Kafka, Strimzi, or managed Kafka offerings such as Amazon MSK and Confluent Cloud. Experience with Schema Registry, schema evolution, and compatibility management. Knowledge of Kafka ecosystem tools such as MirrorMaker 2, Kafka REST Proxy, and Kafka UI or equivalent administration tools. Experience integrating Kafka with Oracle, SQL Server, PostgreSQL, Salesforce, SAP, or other enterprise applications. Experience with Debezium, Change Data Capture (CDC), JDBC connectors, and database replication patterns. Knowledge of Python, Java, or shell scripting for connector development, automation, and troubleshooting. Experience with Jenkins, GitHub Actions, Azure DevOps, or other CI/CD tools. Exposure to AWS services, IAM, networking, secrets management, and cloud observability. Experience with disaster recovery, multi-cluster replication, high availability, and business continuity planning. Experience in enterprise production support, on-call operations, incident response, and service-level management. Kafka-related certifications or relevant Red Hat, cloud, or streaming platform certifications.

Experience

8+ years of experience in middleware, messaging, streaming platforms, integration engineering, or platform operations, including substantial hands-on Kafka experience. Proven experience administering and supporting Kafka environments across on-premises and cloud platforms, particularly ROSA/OpenShift-based deployments. Demonstrated ability to develop new Kafka connectors and maintain, enhance, and troubleshoot existing connectors. Strong hands-on experience with Kafka CLI commands, topic administration, connector lifecycle management, GitHub workflows, and Datadog monitoring. Experience across the full operational lifecycle, including platform configuration, development, deployment, monitoring, production support, incident resolution, and continuous improvement. Ability to independently manage platform administration activities while contributing to Kafka development, integration delivery, automation, and operational reliability. Experience working in large enterprise environments with high-volume event streaming, stringent availability requirements, and established production support processes.