A public-sector defence organisation in Belgium is building an on-premise Data Exchange Platform to support secure, sovereign data use across multiple domains. As a Senior Data Engineer in Brussels, you will develop Python and SQL pipelines with Dagster, dbt and Delta Lake, from heterogeneous source ingestion through governed data products and BI exposure. The programme is delivered by a small team in the Belgian defence sector, working with a Data Architect, Solution Architect and BI Developer.
The mission
The platform runs on-premise because the organisation requires strict controls over security, data governance and sovereignty. Its lakehouse design uses Bronze, Silver and Gold layers, Delta Lake tables, ACID transactions and time travel, with dbt and DuckDB supporting analytical transformation. Sources include files, APIs and relational databases, with incremental and streaming ingestion across domains.
You will work in the data engineering stream alongside the Data Architect, Solution Architect and BI Developer. Your scope covers reliable ingestion, transformation, orchestration, metadata management and exposure of the Gold layer to reporting tools. You will implement Dagster assets, monitor runs, maintain end-to-end lineage in DataHub and support BI consumption through Power BI. The role connects production pipeline quality with governance documents, technical decisions and usable datasets for controlled-access users.
Your responsibilities
- Design and deliver idempotent data acquisition pipelines using ETL and ELT patterns for files, APIs and relational databases, including Kafka-based streaming flows.
- Build Bronze, Silver and Gold transformations with dbt, applying SCD2, upsert and append-only patterns for accurate data history.
- Maintain Delta Lake tables and exports, preserving ACID consistency and reliable access to governed data products.
- Orchestrate Dagster assets, monitor production runs, handle failures and automate alerts so pipelines remain traceable.
- Publish technical and business metadata to DataHub, maintain end-to-end lineage and expose Gold datasets for Power BI reporting.
Your profile
Essential skills
- Apply Python, SQL, Pandas and Apache Spark from a data engineering perspective to build and troubleshoot production pipelines.
- Demonstrate hands-on dbt-core capability across models, snapshots, macros, tests and profiles.
- Understand data architecture, lakehouse and datalake patterns, including Bronze, Silver and Gold design.
- Use Kafka for event-driven ingestion and work with relational platforms such as MSSQL and PostgreSQL, including advanced SQL and CDC.
- Work with Delta Lake features including ACID transactions, time travel, OPTIMIZE and VACUUM.
- Use Dagster or equivalent orchestration concepts and work comfortably with Linux, OpenShift and PowerShell development environments.
- Apply data governance and metadata management practices with DataHub or equivalent tools, including lineage and technical documentation.
- Communicate technical decisions clearly, work autonomously on complex multi-domain pipelines and collaborate effectively with architecture and BI colleagues.
- Hold Belgian nationality, which is required for this programme.