The Context Lake

What a context lake is

A data lake stores files. A context lake stores what happened: the inputs an analysis or an agent read, the instructions it was given, what it produced, and what was decided next, each kept as an immutable, versioned revision that anyone can open later. The point is not storage. The point is that the organization can look back at any result and see exactly what it rests on.

Three properties make a store a context lake rather than a folder of files:

• Immutable revisions. Every change is a new, content-addressed version. Nothing is overwritten, so a result can always be traced to the exact inputs that produced it.
• Provenance on write. Each artifact carries where it came from and what made it, so the next reader can make their own trust decision. This is the “R” of the FAIR Contexts Maturity Model.
• Tool-agnostic. The same artifacts are read and written by people, scripts, and any AI model or harness. No single vendor sits between you and your record.

In practice a context lake is a set of versioned packages in your own cloud storage. It is a pattern, not a product; the sections below show how to build one with open-source tools and, if you want search, governance and per-user access on top, how the commercial Quilt Platform deploys and exposes it.

Build it with open-source tools

The open-source core is a Python library, quilt3, and a Rust library with a command-line tool, quilt-rs and quilt-cli. Both are Apache-2.0. Both write the same package format, and both work against a plain S3 bucket with your own AWS credentials. No server, no catalog and no account with us are needed. The API and CLI are aimed at scripts and coding agents, so this is the path to show an agent.

From the command line

Install the CLI (a prebuilt binary via cargo-binstall, or cargo install quilt-cli to build from source):

cargo binstall quilt-cli

Make a directory with a file in it, create a package from it, change the file in the package’s working copy (under ~/QuiltSync by default), see what changed, and commit a new immutable revision. The hash printed by commit is the version.

mkdir -p ~/plate-exports
printf 'sample,od600\nA1,0.42\n' > ~/plate-exports/plate1.csv

quilt create -n lab/assays -s ~/plate-exports -m "first run"
echo "A2,0.38" >> ~/QuiltSync/lab/assays/plate1.csv
quilt status -n lab/assays
quilt commit -n lab/assays -m "add well A2"

Push it to a bucket you own. With no --host, the push uses the AWS credentials on this computer and needs no Quilt stack:

quilt push -n lab/assays --bucket my-lab-data

Turn on S3 Versioning on that bucket first. A revision’s manifest pins each file by its S3 version id, so with versioning on, a later push that changes a file cannot alter what an earlier revision points at. Without it, S3 keeps only the latest bytes under each key and the history is only as immutable as the bucket.

From Python

pip install quilt3

Set up a run to package: the instructions it ran with, and what it produced. Then run Python from the same directory.

mkdir -p ~/summary-run/outputs && cd ~/summary-run
printf 'Summarize each plate in one line.\n' > prompt.md
printf 'plate1: A1 0.42, A2 0.38\n' > outputs/summary.txt

import quilt3

pkg = quilt3.Package()
pkg.set_dir("outputs", "outputs")                  # what this run produced
pkg.set("prompt.md", "prompt.md")                  # the instructions it ran with
pkg.set_meta({
    "inputs": "quilt+s3://my-lab-data#package=lab/assays@9d0bd491",   # the revision it read
    "model": "us.anthropic.claude-haiku-5-5",
    "decision": "accepted; see review.md",
})

top_hash = pkg.build("lab/summaries")              # a local, immutable revision
pkg.push("lab/summaries", registry="s3://my-lab-data")   # the same revision, in your (versioned) bucket

Quilt versions the files and gives the set a content hash. Your workflow has to record the relationships: which input revision it read, which instructions and model it ran, and what was decided about the result. Put those in the package too, as files or metadata, and link them by revision. A hash on its own says what was saved, not what it was made from.

Every revision is identified by its content hash, so a result that names its package revision names its exact inputs. That is the whole mechanism. The rest of this page is about who else can read it.

More on the open tools, including the QuiltSync desktop app, is on the open source page.

A worked example: fine-tuning a small model, in public

In autumn 2026 we post-trained NVIDIA Nemotron 3.5 Lightning, a 30-billion-parameter open-weight model, to pull the key facts out of scientific documents, and ran the whole project inside a context lake. Two tasks, in order: the opening pages of scientific files, judged against Claude Opus 5.5; then the abstracts of papers, judged against Claude Haiku 5.5. The code and method are public at github.com/quiltdata/fine-tuning. The full record of the second task (data, every run, every grader verdict) is a public package anyone can open: models/nemotron-lightning, with the blind judging sheets alongside. The first task’s method and results are written up in METHODOLOGY.md; its run files are not public. Four things about how it was run matter more than the model.

Each round started with a written target

We versioned the target and the acceptance criteria before each training round: the test set, the metric, the comparison model, the graders, and the bar. When results led us to narrow the task or revise how we judged it, we recorded that decision before the next round and kept the earlier plans and verdicts visible. The bar the final models were read against is not the bar the first round was read against, and the record shows every step between them.

The misses stayed on the record

The first round missed the quoting bar outright. The second passed it, but bought quoting at the cost of fact coverage, and the record shows that too. The model we finally served on the first task still missed the quoting bar, by four files out of 262, and the miss is on the record next to the result. Nothing was deleted or quietly re-run; each result is a revision in the package, next to the plan written before it. The public write-up keeps a list of the corrections that changed a conclusion, with the earlier numbers left in place.

The plan changed, and the change is traceable

The answer key listed three to five facts per file, but only the first two are what a reader deciding whether to open a file needs; the rest were one model’s opinion about what mattered next. So we drew the line at two, wrote that down as a new plan before training again, and kept the old metric alongside. The next round closed the gap to Opus because the teacher now wrote its answers with those two facts in hand. Readers can follow the chain: result, decision, next plan, result.

The work passed between models and people

An open-weight teacher wrote every training target, and code checked them. Two frontier models graded the outputs; from the fourth reward round one of them also chose which of the student’s own answers to train on, and the record says so. A coding agent ran the jobs, a harness recorded every model’s outputs as package revisions, and one person who had never trained a model made the calls. The people and agents handed off through numbered entries, each answering the last, caveats included. One such handoff: the entry that reported the first task’s results listed what might be tried next; the entry written the next day picked one of those items, recorded the owner’s decision to narrow the key, and kept the earlier numbers beside it.

The outcome. The first model, served in 4-bit on one GPU, is level with a properly prompted Claude Opus 5.5 on the first 8,000 characters of a scientific file, at about 1% of Opus’s cost; read whole, Opus is about ten points ahead, and the served model misses our own quoting bar by four files. The second model, the abstract model, meets our recall and faithfulness bar under the primary grader, level with Claude Haiku 5.5 at 42% of Haiku’s cost; but when an abstract reports no result it still often presents the aim or method as a finding (47 of 74 such abstracts, against 9 for Haiku), and the second grader finds a faithfulness gap of about two points. The cost figures assume an endpoint kept busy; an idle GPU costs the same. All of this, and the rest of the shortfalls, is written next to the results in the two write-ups, and the second task’s record can be checked file by file.