A Belgian federal public safety organisation is building an event-driven data architecture to support modernised processes and more data-driven operational services. This Elastic Data Engineer role focuses on Apache Kafka as the ingestion and processing layer, Elasticsearch as the data lake for structured and unstructured data, and the APIs and microservices that make this information available to other systems.
The mission
The project establishes a scalable architecture for real-time data integration across multiple operational data flows. Kafka is used for buffering and event processing through Kafka Streams, Kafka Connect, Schema Registry, Avro, and ksqlDB, while Elasticsearch provides storage, search, and analysis capabilities through Kibana, ingest pipelines, and index lifecycle management. Data consistency, indexing strategies, observability, and reliable error handling are important across the complete path from ingestion to consumption.
You will design, implement, configure, and document the components that connect these technologies. Your scope includes building a data service layer with REST APIs and microservices, deploying services through Docker, Kubernetes, and OpenShift, and supporting delivery through GitLab CI pipelines. You will work in a multidisciplinary team of internal and external specialists, with your output supporting analysis, monitoring, and integration with other public-sector systems.
Your responsibilities
- Design and maintain Kafka and Elasticsearch data pipelines that remain consistent and scalable as data volumes and use cases evolve.
- Configure Kafka topics, schemas, serialization, and consumer or producer flows using Schema Registry, Avro, Kafka Streams, Kafka Connect, and ksqlDB.
- Develop the data service layer with REST APIs and microservices that expose Elasticsearch data for analysis, monitoring, and system integration.
- Implement Elasticsearch indexing strategies, ingest pipelines, Kibana views, and index lifecycle management for structured and unstructured data.
- Deploy and operate pipeline components in Docker, Kubernetes, and OpenShift environments, using GitLab CI for repeatable delivery.
- Investigate failures, improve pipeline performance, automate recurring tasks with Python, and document technical decisions, configurations, and data flows.
Your profile
Essential skills
- Professional experience as a data engineer working with event-driven systems and real-time data integration.
- Practical expertise with Apache Kafka, including Kafka Streams, Kafka Connect, Schema Registry, Avro, ksqlDB, schema management, and topic administration.
- Experience operating Elasticsearch with Kibana, ingest pipelines, index lifecycle management, and robust indexing strategies.
- Ability to protect data consistency, scalability, and observability across Kafka and Elasticsearch data pipelines.
- Experience developing REST APIs and microservices as part of a data service layer.
- Working knowledge of Docker, Kubernetes, OpenShift, GitLab CI, PostgreSQL, and Python.
- Capability to analyse pipeline failures, implement error handling, and optimise data processing performance.
- Methodical and precise approach to configuration, testing, documentation, and technical analysis.
- Ability to work autonomously, communicate clearly with technical specialists and user representatives, and contribute effectively within a multidisciplinary team.
- A practical, service-oriented approach to prioritising work across multiple data pipelines and technical dependencies.
Education
- Degree in computer science, data engineering, information technology, or equivalent professional experience.