Data platforms · AI engineering
Technical Lead Data Engineer

AI amplifies judgment. The engineer is the edge.

I run a global logistics company's Databricks lakehouse end to end, and I build with AI agents that carry the half of the work that doesn't need me.

Email Teams Power BI Web Fabric Wrike Databricks GitLab ROUX THE EDGE

My judgment, built into each workflow.

650
tables
~700
GB / day
900+
workloads
1 019
steps, longest unattended run
  • Cloudflare
  • Azure Databricks
  • Microsoft Azure
  • Claude
  • Claude Code
  • Power BI
  • Astro
  • GitLab
  • Microsoft Fabric
  • Resend
  • Unity Catalog
  • Playwright
  • VS Code
  • Python
Shipped

Built, tested, shipped.

Two systems on different stacks, built the same way. Each number below comes from code, logs or direct measurement.

The company's data platform

~700 GB a weekday across 900+ workloads

Exhibit A
A global freight and logistics company

The company's central data platform (a "lakehouse"), where every number comes from, plus the reports built on top of it. A delivery partner stood it up; I built it into what the company runs on today.

Every table loads the same way: one control row per load, one generic loader per layer, nothing hand-written per table. Twelve scheduled jobs run it every day.

927 active control rows across 8 control tables; 718.9 GB mean per weekday (702–784), measured on the live cluster

  • Took over a platform a delivery partner had stood up and built it into the one the company runs on: the framework, the jobs, the structure, the rules. More than 900 metadata-driven workloads move about 700 GB a weekday across 16 source schemas and 650 tables.
  • Moved the analytics workloads off SQL Server onto the lakehouse. Phases one and two of the round trip are in production, with 1,253 legacy objects specified for retirement. Each layer reads only the layers before it; a validator in CI rejects the rest.
  • Two workspaces under one metastore, production bound read-only to test. Every change goes through GitLab CI, where a validator rejects a layer reading upward, and secrets live in Key Vault, never in code.
  • The reports on top are the small part. A 34-page operations report was rebuilt as eleven thin ones in eight working days; refresh went from 92.9 minutes to 4.1.

Azure Databricks · Unity Catalog · Fabric · Power BI · GitLab CI

konradvanstaden.com, the homepage

konradvanstaden.com

19 pages, self-hosted end to end

Exhibit B
Scope
author's site rebuilt with booking
Scale
19 pages, one for each of 18 books
Result
19 static pages, fonts and scripts all self-hosted
Author's website

An author's website, rebuilt from the ground up with a booking system behind it.

The old site was an ageing Joomla install. The new one is nineteen light pages with every font, script and image served from his own domain. A content change is one rebuild and one scripted deploy.

19 static pages, self-hosted assets; the only outside call on load is the host's analytics beacon

  • A light homepage plus a page for each of his 18 books, and nothing on them is fetched from anyone else's servers apart from the host's own analytics ping. Rebuilt from a legacy Joomla site into 19 static pages. Self-hosted fonts and scripts; social embeds load only on click.
  • The booking system went live quietly. Until its last piece was in place the site kept using the old email link, so nothing ever broke for a visitor. Booking backend with no flag day: ships dark, falls back to yesterday's mailto until its key lands.
  • The switch to the new site was scripted, with a five-minute way back if anything went wrong. Checked on the real address before calling it done.

Astro · Cloudflare Pages · Pages Functions · Resend

    The work splits in two. One half runs to the end without me. The other half comes back to me in ordinary words before it ships.

    The way I work is the product.

    The machine’s half

    What runs without me.

    The longest single run went 1 019 steps before it needed me again. 326 unattended runs of 50+ steps, 114 of 100+, 29 of 200+. Counted from every transcript, 2026-07-02 to 2026-09-03.

    What can you do without me?

    I split the work into what needs me and what does not, then let it run the second pile to the end without stopping.

    Training wheels, then autonomy

    New automations run with me watching. After a few clean runs they run alone. Anything touching money, sending to the public, or deleting never runs alone.

    Bull in a china shop, then a diamond

    Get the rough version working fast, then cut it down until it is small and hard to break.

    The same setup does the dull work too. It sorts mail, handles tickets and writes up meetings. One session got into my home router, worked out why the line was bad, and filed the fault with the internet provider.

    My half

    What has to come through me.

    The gate is plain language. If it cannot be said in ordinary words, it does not ship. 437 of 5 011 typed turns across every transcript asking for it in plain terms, in one sentence, in dummy terms. Counted 2026-07-02 to 2026-09-03.

    Replay it back first

    Before anything gets built, the AI says the plan back to me in plain words. If the plan is wrong it dies there, before any code exists.

    Check the result, not the message.

    I don't trust "deploy succeeded". I open the live site and check the version on screen is the one I just shipped. Never trust a name, a dashboard or a green status. Read the rule back. Assert the deployed commit hash. Hit the live site with a fresh, uncached request.

    Audits never return zero

    A check that finds nothing is suspect. One or two small things per run is normal. Five new things every run means the thing being checked is changing under it.

    Two strikes before a rule

    The first time something goes wrong, I note it. The second time, it becomes a written rule, and the tools enforce it so I don't have to remember.

    One route
    contact@kosieroux.com

    Kosie Roux · Technical Lead Data Engineer · The machine runs what doesn’t need me. I check what does.

    UNITY CATALOG one metastore · two workspaces · production bound read-only to test SOURCES ERPWMSSQL ServerFiles & APIs two instances · JDBC3 versions · PostgreSQLon-prem and Azure SQLSharePoint · SFTP · REST 16 source schemas SILVER WAREHOUSE GOLD typed mirror of every source 16 schemas 650 tables star schema surrogate keys every row keyed to its source tables and views read by BI every number starts here READ BY Power BIFabric via SQL warehousesDirect Lake over a mirror CONTROL PLANE 8 control tables · 900+ active rows · 12 scheduled jobs · for-each fan-out one control row per load · one generic loader per layer · nothing hand-written per table each layer reads only the layers before it, never upward: a validator in CI rejects the rest

    Sources → silver → warehouse → gold, under Unity Catalog; read by Power BI and Fabric; a control plane of eight tables drives every load. Sanitised: no names, only counts already on this page.