Ingestion at volume
Pipelines that keep up with the source material day after day, handle the malformed record without stopping, and fail loudly instead of quietly when the source changes shape.
02 — Service
Pipelines that survive real volume, and a team that can run them.
Ingestion, transformation and reporting built to hold up when the data grows — and the mentorship that means the system does not depend on me being available. Building the platform and building the capability are the same engagement.
What is included
What this covers
PostgreSQL performance
Schemas designed around the queries that matter, indexes that earn their write cost, and materialized views where a report should not re-compute the world on every load.
Orchestration
Apache Airflow for scheduling and dependencies, with retries, alerting and backfills that work — so a failed overnight run is a notification, not a discovery.
Distributed transformation
PySpark where the data genuinely outgrows a single machine, and deliberately not before — most workloads that are called big are a well-indexed database away from being fast.
Mentorship & capacity
Structured technical mentoring in Python, data engineering, cloud solutions, application security and CI/CD — for people who need real capability, not a certificate.
AI-assisted development
Working with an AI pair programmer day to day — VS Code and Claude in practice: what to hand off, what to review line by line, and how it changes the pace of building and debugging a pipeline.
Capacity building, at programme scale
Teaching data engineering to a Mastercard Foundation programme, with the Government of Rwanda.
Alongside client engagements, this service has run as structured technical training for a data-and-decisions capacity-building programme delivered in partnership with the Mastercard Foundation and the Government of Rwanda — teaching public-sector stakeholders how to use data for decision-making, and teaching the data engineering concepts underneath it.
The training moved from foundations to practice: containerising a workload with Docker, orchestrating it with Apache Airflow, and running it as a team through GitHub — pull requests, reviews, CI — rather than as scripts one person runs alone. It also covered building with an AI pair programmer, VS Code and Claude specifically — what to hand off, what to review line by line, and how it changes the pace of writing and debugging a pipeline. The slides used are below.
Training materials
What the sessions actually cover.
Slides used in the programme itself — open any one, or step through them here first.
Capstone project
An open-source data pipeline, built end to end.
Every topic in the training shows up together in one running system: a self-hosted, containerised ETL platform built entirely on open-source tools. It moves data from source databases and flat files into a target database, documents the structure of every table it loads, and makes that data available for analytics through a web interface — no manual scripting required to add a new pipeline.
- Apache Airflow: orchestrates and schedules every pipeline run
- PostgreSQL: the target database and the metadata store
- Redis & Celery: the task queue behind Airflow’s workers
- Flask: a web UI for building pipelines and browsing the data catalogue
- Docker Compose: the whole stack starts with a single command
Process
How the work runs.
-
01
Audit
We look at what data exists, where it comes from, what breaks today and what decisions it is meant to support.
-
02
Model
The data model and pipeline boundaries are designed before anything is built, because a schema is the hardest thing to change later.
-
03
Build
Pipelines land in version control, orchestrated and monitored, with the reporting layer the business actually reads.
-
04
Transfer
Runbooks, paired sessions and training, so your team operates and extends the platform themselves.
What you end up with
- Pipelines under version control and orchestration
- A documented data model and runbooks
- Monitoring and alerting on every scheduled run
- Training sessions recorded and tailored to your team
Questions
Asked often enough to answer here.
Our data lives in spreadsheets. Is that too early?
No, and it is a good moment to start. Moving to a database and a scheduled pipeline is far cheaper before a decade of conventions have accumulated in the sheets.
Can you work with the warehouse we already have?
Yes. Most engagements add pipelines and tuning around an existing PostgreSQL or cloud warehouse rather than replacing it.
Can you train our team instead of building it for us?
That is the point of combining the two. Mentorship can be the whole engagement, or it can run alongside a build so your team inherits something they helped construct.
We have no data engineer. Does this still work?
Yes. The build is designed to be operable by the people you have, and the capacity-building side exists to close exactly that gap.
Do you run training programmes, not just single-company engagements?
Yes — the Rwanda programme is exactly that: structured, multi-session training for a stakeholder group, not a single team. Ask about programme-scale delivery specifically.
Tell me what you are building.
The enquiry form arrives with Data Engineering & Capacity Building already selected.