Notes from building a distributed, petabyte-scale analytical query engine.
Along the way: system architecture, lakehouse storage, and what data systems need as AI agents become their users. What I tried, what held up, and what I'd do differently.
-
From one terabyte to ten
A distributed query engine that works at one terabyte has proved very little about ten. The lessons from this autumn, minus the numbers.
-
Indexes in an analytical engine: how they work and when they pay
In an engine that reads files from object storage, most indexes exist to rule data out; a few point straight at rows. How each kind works, when it pays, where Iceberg's Puffin files fit, and what building them taught me.
-
Simulating the commit path: deterministic tests for streaming writes to Iceberg
A streaming writer commits to an Iceberg table every few seconds, retries on conflict, and shares the table with compaction and garbage collection. I borrowed SlateDB's approach to deterministic simulation to test it. It caught less than I hoped, and taught me more.
-
An Iceberg catalog with no database: Lakekeeper on SlateDB
I wrote a backend that keeps an Iceberg REST catalog's state in SlateDB, a key-value store that lives entirely in object storage. It worked, and it kept the catalog's state out of a database. This note is mostly about what it cost: the guarantees I rebuilt, the ones I didn't, and what I'd do differently.
-
A metadata tree that grows with the change
The draft Iceberg v4 spec replaces the manifest list with a root manifest. It sounds like a file rename. It changes what a small commit costs.
-
Manifests are data too
Iceberg's manifests have always been row-oriented Avro. The draft version 4 spec makes them Parquet. Everything I'd learned about not reading data turned out to apply to metadata as well.
-
Porting an idea, not the code
Bringing dynamic indexes from a Java engine to one built on Rust and DataFusion. What carried over was a handful of rules. Almost none of the code did.
-
Random order is only useful if you can replay it
In 2017 I removed a flag that ran Gauge specs in random order. In 2025 I added it back. The difference was a seed, and a place to keep it.
-
Spot instances and the query that must not die
Spot capacity is the cheapest compute a cloud will sell you, and it comes with a short warning before it's taken back. Deciding where in the stack to absorb that warning was the interesting part.
-
One query, many clouds
When the data lives in more than one place, you can move the data to the query or move the query to the data. I spent a good amount of time learning what it takes to do the second.
-
Blitz: a file format for vectors on object storage
Vector search wants a handful of vectors out of millions, at full precision, from storage that charges you per request. None of the formats I tried did that well in my setup, so I wrote a small one that did.
-
What I tried and threw away
Three experiments that didn't make it: virtual threads for the executor, a vector database library, and product quantization. Each one taught me something the successes didn't.
-
Vectors inside a SQL engine
Most teams I talked to were standing up a separate vector database. I tried making similarity search just another thing our query engine could do.
-
Let the queries build the index
Most Parquet files don't carry bloom filters, and we didn't control the writers. So the engine started building its own indexes from data it was reading anyway.
-
Bloom filters at the scan
Parquet already knows how to say "this value isn't here". Getting an engine to ask cheaply, and to never trust a wrong answer, was the actual work.
-
The cheapest row is the one you never read
Before indexes, vectors or distributed anything, I had to learn where our query engine actually spent its time. The answer: reading things it didn't need.
-
Preparing an open-source project to lose its sponsor
When a company stops funding the team behind a project, the code keeps working. Everything around it is what needs preparing: the licence, the release, the contributor terms, and the things that quietly expire.
-
Loading someone else's code: AppDomains, then AssemblyLoadContext
A test runner has to load the user's assemblies into its own process. On .NET Framework I used an AppDomain for that. On .NET Core there are none, and the replacement taught me that isolation, file locks and native dependencies pull in different directions.
-
Benchmarks you can't trust, used for the one thing they're good at
Shared CI machines are terrible benchmark hardware. I put benchmarks on every Gauge pull request anyway, because the question wasn't how fast Gauge is but whether this change made it slower.
-
Once per suite means once per process, until it doesn't
Gauge's BeforeSuite hook ran once per parallel stream, not once per run. The fix was a few hundred lines. The decision behind it was that every language runner had to support threads.
-
Killing plugins politely: from a fixed timeout to keep-alives
Gauge gave every plugin four seconds to exit after a run, then killed it. On large projects that meant killing the HTML report halfway through writing it. The fix was to let the plugin say it was still working, and it came with a limit I'd point out more loudly today.
-
Your protocol's worst message is the summary
Gauge sent its report plugins one message with the entire run in it. On a big enough suite, protobuf refused to encode it. The fix was to cut the result at a boundary the format already had, and to let each plugin opt in.
-
Shipping a risky language feature behind a flag, and what the flag cost
Table-driven scenarios went into Gauge behind an experimental flag in 2018. The flag did its job: it let the core ship while everything around it caught up. It also stayed for seven years, and that part was not the plan.
-
Deprecate, don't delete: the price of backward compatibility
Gauge's plugins were written in half a dozen languages, and all of them shared one set of protocol buffer definitions. A small rename taught me that a shared schema has more than one kind of compatibility. The rule I set afterwards still holds, and eight years on, so does its bill.
-
IDE support, before and after the Language Server Protocol
I wrote Gauge's Visual Studio plugin in 2014, when every editor needed its own. Three years later the project moved to a language server. What moved into the server, and what still didn't fit in the protocol.
-
Telemetry we added, then deleted
In 2017 I wired anonymous usage telemetry into Gauge. Two years later we took it all out. Adding it took a few weeks; most of what I learned was about the parts around the ping.