# hack-n-learn > Notes by Srikanth Venugopalan on building a distributed, petabyte-scale analytical query engine, and on the architecture around it: query execution, indexing, vector search, lakehouse storage (Apache Iceberg, Parquet), distributed and multi-cloud execution, and what data systems need as AI agents become their users. Also notes from 2017 to 2025 on Gauge, the open-source test automation framework: wire protocols, plugin supervision, loading user code in .NET, telemetry, and keeping a project alive after its sponsor stepped back. The author works on OLAP query engines at e6data. Each note is written from first-hand engineering work and dated to when the work happened. Earlier writing (2012 to 2013) is also on the site. Every note is available as plain markdown by replacing the trailing slash of its URL with `.md`, and all of them together at https://sriv.github.io/llms-full.txt. ## Notes - [From one terabyte to ten](https://sriv.github.io/notes/from-one-terabyte-to-ten.md): 16 October 2026. A distributed query engine that works at one terabyte has proved very little about ten. The lessons from this autumn, minus the numbers. - [Indexes in an analytical engine: how they work and when they pay](https://sriv.github.io/notes/skipping-indexes.md): 23 August 2026. In an engine that reads files from object storage, most indexes exist to rule data out; a few point straight at rows. How each kind works, when it pays, where Iceberg's Puffin files fit, and what building them taught me. - [Simulating the commit path: deterministic tests for streaming writes to Iceberg](https://sriv.github.io/notes/simulating-the-commit-path.md): 13 August 2026. A streaming writer commits to an Iceberg table every few seconds, retries on conflict, and shares the table with compaction and garbage collection. I borrowed SlateDB's approach to deterministic simulation to test it. It caught less than I hoped, and taught me more. - [An Iceberg catalog with no database: Lakekeeper on SlateDB](https://sriv.github.io/notes/a-catalog-on-object-storage.md): 17 June 2026. I wrote a backend that keeps an Iceberg REST catalog's state in SlateDB, a key-value store that lives entirely in object storage. It worked, and it kept the catalog's state out of a database. This note is mostly about what it cost: the guarantees I rebuilt, the ones I didn't, and what I'd do differently. - [A metadata tree that grows with the change](https://sriv.github.io/notes/a-metadata-tree-that-grows-with-the-change.md): 5 June 2026. The draft Iceberg v4 spec replaces the manifest list with a root manifest. It sounds like a file rename. It changes what a small commit costs. - [Manifests are data too](https://sriv.github.io/notes/manifests-are-data-too.md): 13 May 2026. Iceberg's manifests have always been row-oriented Avro. The draft version 4 spec makes them Parquet. Everything I'd learned about not reading data turned out to apply to metadata as well. - [Porting an idea, not the code](https://sriv.github.io/notes/porting-an-idea-not-the-code.md): 18 December 2025. Bringing dynamic indexes from a Java engine to one built on Rust and DataFusion. What carried over was a handful of rules. Almost none of the code did. - [Random order is only useful if you can replay it](https://sriv.github.io/notes/gauge-reproducible-randomness.md): 6 November 2025. In 2017 I removed a flag that ran Gauge specs in random order. In 2025 I added it back. The difference was a seed, and a place to keep it. - [Spot instances and the query that must not die](https://sriv.github.io/notes/spot-instances-and-the-query-that-must-not-die.md): 5 October 2025. Spot capacity is the cheapest compute a cloud will sell you, and it comes with a short warning before it's taken back. Deciding where in the stack to absorb that warning was the interesting part. - [One query, many clouds](https://sriv.github.io/notes/one-query-many-clouds.md): 3 October 2025. When the data lives in more than one place, you can move the data to the query or move the query to the data. I spent a good amount of time learning what it takes to do the second. - [Blitz: a file format for vectors on object storage](https://sriv.github.io/notes/blitz-a-file-format-for-vectors.md): 25 February 2025. Vector search wants a handful of vectors out of millions, at full precision, from storage that charges you per request. None of the formats I tried did that well in my setup, so I wrote a small one that did. - [What I tried and threw away](https://sriv.github.io/notes/what-we-tried-and-threw-away.md): 3 February 2025. Three experiments that didn't make it: virtual threads for the executor, a vector database library, and product quantization. Each one taught me something the successes didn't. - [Vectors inside a SQL engine](https://sriv.github.io/notes/vectors-inside-a-sql-engine.md): 17 December 2024. Most teams I talked to were standing up a separate vector database. I tried making similarity search just another thing our query engine could do. - [Let the queries build the index](https://sriv.github.io/notes/let-the-queries-build-the-index.md): 4 December 2024. Most Parquet files don't carry bloom filters, and we didn't control the writers. So the engine started building its own indexes from data it was reading anyway. - [Bloom filters at the scan](https://sriv.github.io/notes/bloom-filters-at-the-scan.md): 20 July 2024. Parquet already knows how to say \"this value isn't here\". Getting an engine to ask cheaply, and to never trust a wrong answer, was the actual work. - [The cheapest row is the one you never read](https://sriv.github.io/notes/the-cheapest-row-is-the-one-you-never-read.md): 31 May 2024. Before indexes, vectors or distributed anything, I had to learn where our query engine actually spent its time. The answer: reading things it didn't need. - [Preparing an open-source project to lose its sponsor](https://sriv.github.io/notes/gauge-preparing-a-project-to-lose-its-sponsor.md): 1 September 2020. When a company stops funding the team behind a project, the code keeps working. Everything around it is what needs preparing: the licence, the release, the contributor terms, and the things that quietly expire. - [Loading someone else's code: AppDomains, then AssemblyLoadContext](https://sriv.github.io/notes/gauge-assembly-load-contexts.md): 14 August 2020. A test runner has to load the user's assemblies into its own process. On .NET Framework I used an AppDomain for that. On .NET Core there are none, and the replacement taught me that isolation, file locks and native dependencies pull in different directions. - [Benchmarks you can't trust, used for the one thing they're good at](https://sriv.github.io/notes/gauge-benchmarks-in-ci.md): 14 July 2020. Shared CI machines are terrible benchmark hardware. I put benchmarks on every Gauge pull request anyway, because the question wasn't how fast Gauge is but whether this change made it slower. - [Once per suite means once per process, until it doesn't](https://sriv.github.io/notes/gauge-suite-hooks-under-parallelism.md): 14 May 2020. Gauge's BeforeSuite hook ran once per parallel stream, not once per run. The fix was a few hundred lines. The decision behind it was that every language runner had to support threads. - [Killing plugins politely: from a fixed timeout to keep-alives](https://sriv.github.io/notes/gauge-killing-plugins-politely.md): 18 January 2019. Gauge gave every plugin four seconds to exit after a run, then killed it. On large projects that meant killing the HTML report halfway through writing it. The fix was to let the plugin say it was still working, and it came with a limit I'd point out more loudly today. - [Your protocol's worst message is the summary](https://sriv.github.io/notes/gauge-the-worst-message-is-the-summary.md): 20 November 2018. Gauge sent its report plugins one message with the entire run in it. On a big enough suite, protobuf refused to encode it. The fix was to cut the result at a boundary the format already had, and to let each plugin opt in. - [Shipping a risky language feature behind a flag, and what the flag cost](https://sriv.github.io/notes/gauge-a-risky-feature-behind-a-flag.md): 28 September 2018. Table-driven scenarios went into Gauge behind an experimental flag in 2018. The flag did its job: it let the core ship while everything around it caught up. It also stayed for seven years, and that part was not the plan. - [Deprecate, don't delete: the price of backward compatibility](https://sriv.github.io/notes/gauge-deprecate-dont-delete.md): 21 March 2018. Gauge's plugins were written in half a dozen languages, and all of them shared one set of protocol buffer definitions. A small rename taught me that a shared schema has more than one kind of compatibility. The rule I set afterwards still holds, and eight years on, so does its bill. - [IDE support, before and after the Language Server Protocol](https://sriv.github.io/notes/gauge-ide-support-before-and-after-lsp.md): 29 November 2017. I wrote Gauge's Visual Studio plugin in 2014, when every editor needed its own. Three years later the project moved to a language server. What moved into the server, and what still didn't fit in the protocol. - [Telemetry we added, then deleted](https://sriv.github.io/notes/gauge-telemetry-we-added-then-deleted.md): 5 June 2017. In 2017 I wired anonymous usage telemetry into Gauge. Two years later we took it all out. Adding it took a few weeks; most of what I learned was about the parts around the ping. ## About - [About the author](https://sriv.github.io/about/): background, current focus, open-source work (lead developer of Gauge, 2015 to 2021). - [Atom feed](https://sriv.github.io/atom.xml) - [GitHub](https://github.com/sriv) - [LinkedIn](https://www.linkedin.com/in/srikanth-v-6393b922/)