The organisation is operating an Apache Kafka platform that supports event-driven microservices and change data capture (CDC). During a three-month consulting assignment, the senior Kafka consultant will assess and optimise the existing infrastructure, formalise governance for its operation, and improve observability with Grafana while addressing concrete platform maintenance issues.
The mission
The platform acts as a data backbone for distributed applications, with work spanning Kafka brokers, KRaft, Schema Registry, Kafka Streams, and Kafka Connect. The assignment covers data pipelines, topic configuration, partitioning, retention policies, connector architecture, and secure data exchange through TLS, SASL, and ACLs. This work will make the Kafka platform easier to operate and support consistent availability across its clusters. It also establishes documented procedures for monitoring, troubleshooting, and platform governance.
You will assess the current Kafka architecture and translate your findings into practical improvements for production operations. Your day-to-day scope will include optimising pipelines, reviewing Kafka Connect integrations, implementing Grafana monitoring for Kafka clusters, and creating repeatable troubleshooting procedures. You will connect architecture decisions with operational requirements for microservices and CDC use cases. Documentation and knowledge transfer will help the technical team maintain the platform after the assignment.
Your responsibilities
- Assess the existing Kafka infrastructure and define improvements to its architecture, configuration, and operational model.
- Design Kafka architecture for event-driven microservices and CDC data flows, including topics, partitions, retention, and deployment requirements.
- Optimise data pipelines and Kafka Connect integrations to improve reliability, maintainability, and production readiness.
- Implement monitoring with Grafana and establish operational indicators for brokers, connectors, topics, and cluster health.
- Secure data exchanges using TLS, SASL, and ACLs, while supporting appropriate Kafka governance procedures.
- Document troubleshooting procedures and transfer knowledge to the technical team.
Your profile
Essential skills
- Bring significant experience as a Kafka consultant or senior Kafka engineer, including the production deployment and optimisation of Kafka platforms.
- Demonstrate advanced knowledge of the Apache Kafka ecosystem, including brokers, KRaft, and Schema Registry.
- Apply Kafka Streams and Kafka Connect to build and operate event-driven data flows.
- Understand distributed systems, microservices, CDC, and event-driven architecture principles.
- Configure and optimise topics, partitions, retention mechanisms, and data pipelines.
- Evaluate connector architectures and improve their monitoring and operational support.
- Use Grafana to implement meaningful monitoring for Kafka platforms and clusters.
- Apply IT architecture principles and contribute to governance for platform operationalisation.
- Secure Kafka communications with TLS, SASL, and ACLs.
- Work with containerisation technologies such as Strimzi or Confluent, and automate operational procedures with Ansible.
- Analyse production issues methodically, develop troubleshooting approaches, and communicate technical decisions clearly.
Preferred skills
- Familiarity with streaming data platforms such as Apache Iceberg, Apache Spark, or Apache Flink.