▮▮▮▯ BC / Annotation
← back to project page

The last mile is annotation.

Long-form companion · ICML 2026 submission · ~12 min read

Connectomics — the project of mapping every neuron and every synapse in a brain — has a dirty secret. The bottleneck is not the microscope. It is not the segmentation network. It is not even the storage cluster. It is people, with mice and keyboards, fixing errors. FlyWire's full Drosophila brain reconstruction documents 33 person-years of crowdsourced proofreading spread over 139,255 neurons. That number is not an outlier. It is the field's actual unit of scale.

This story is not specific to brains. Across scientific imaging — galaxies, X-rays, fungal cell walls, plumes in hyperspectral aerial photos, road network in satellite imagery — the same pattern repeats. Acquisition gets cheaper. Automated models get better. Proofreaders stay expensive. The last mile of every pipeline is a human staring at a screen, clicking.

We do not want to replace proofreaders. We want to make one proofreader produce ten.

The bet: learn from process, not outcomes

There is already a small body of work on automating proofreading. Almost all of it learns from annotation outcomes: the final segmentation, the cleaned-up skeleton, the corrected bounding box. That is what filesystems store, so that is what gets used.

We make a different bet. Instead of learning from the answer, learn from the trace of how a human got there — the screen recording, the cursor path, the mistakes that were placed and then undone two seconds later. Borrow from a decade of behavioral cloning literature: VPT for Minecraft, Mind2Web for browsers, Decision Transformer for sequence modelling of trajectories. In that framing, an annotator at a GUI is exactly an embodied agent — and embodied agents we know how to imitate.

Why might this be a better bet? Three reasons.

The problem: nobody has the right data

The cleanest way to make this bet is to record every screen, every click, every keystroke of an annotation session — and then train a model on millions of those sessions. But of course nobody has that data. Connectomics labs ship segmentations, not screen recordings. Pathology departments ship diagnoses, not mouse paths.

So we synthesize it. We build nine self-contained annotation environments in HTML and JS. We render them to screenshots with Playwright. We write Python code that plays the role of the human annotator — code policies, with realistic mistake rates and undo behavior. Then we let those code-annotators run, headlessly, for millions of frames.

Nine tasks. One agent. No task-specific heads.
Pixels in, click coordinates out.

What the nine tasks look like

We did not want toy tasks. We wanted each environment to feel like a credible synthetic analog of a real annotation workflow that some lab actually pays for. So we drew a list from neuroscience, behavior science, and remote sensing, and built:

Each task carries about 3-4 million training frames. The model never sees an explicit task label. It just sees the GUI screenshot and predicts the next click.

The architecture (boring, on purpose)

The agent is deliberately simple. A DINOv2-B/14 vision encoder turns each screenshot into a grid of patch tokens. An 8-layer transformer with full attention inside each image and causal attention across time produces a distribution over discrete x and y bins. About 95 million parameters total. No memory, no recurrence, no fancy heads.

Two design choices matter. One: we use a pretrained DINOv2 encoder and fine-tune it. Without pretraining, the model plateaus at terrible performance — the annotation domain is too far from natural image statistics to learn the visual front end from scratch on 30M frames. Two: we predict x and y as independent categorical distributions over pixel bins. That avoids the soft-argmax failure mode (averaging bimodal heatmaps into the gap between modes) and means we can use vanilla cross-entropy loss.

What the agent learns

A few results we keep coming back to.

1. Mistakes are linearly decodable

Our synthetic annotator makes mistakes on purpose (about 10% of placements are deliberate misclicks followed by an undo) so that the model sees recovery behavior. We never give the model an explicit "is_mistake" label. But if you take the layer-6 or layer-8 activations of the trained model and fit a linear probe, you can predict whether the model is about to make a mistake at ROC AUC 0.92.

The agent represents "I am about to do something wrong" as a linear direction in its hidden state. We did not ask it to. It emerged because the training data couples mistakes to their corrections.

2. The error-correction paradox

Here is a thing we did not expect. The training data has roughly a 10% mistake-and-undo rate. The model learns to imitate the recovery shape — but it commits the initial mistake far less often than its teacher.

In a head-to-head with four real human annotators, the model's correction rate is 0.8%, against human rates of 2.4-7.6%. It is, in this narrow sense, cleaner than its teachers. We have a few hypotheses (the cross-entropy loss averages out idiosyncratic individual-annotator mistakes; the implicit prior in DINOv2 is "behave canonically") but no definitive answer yet.

3. Behaviorally indistinguishable from a competent human

Four humans, five colored-dot-tracking tasks each, with minimal instructions. Three of four produced fractions of navigation/placement/MIP-toggle/undo that matched our virtual annotator within the human spread. The trained model, in turn, matches the most cautious of the three almost exactly: 88 vs. 87 navigation steps, 110 vs. 111 total steps, 15.6 vs. 16.6 placements.

We thought this would be the part of the paper that needed the most defending. It turned out to be the easiest section to write, because the action histograms just stacked on top of each other.

4. Behavioral cloning beats frontier VLMs on pixel precision

We ran Gemini 3 Flash and Qwen3-VL-32B (≈100-300× our parameter count) on the same teacher-forced placement-accuracy benchmark. Our 95M-parameter BC model wins on every placement task we tested, often by a wide margin: 97.4% vs. 80% / 25% on colored dots, 66.5% vs. 28% / 4% on neuron tracking.

The lesson: annotation is mostly a fine-motor problem. VLMs trained primarily on screenshot Q&A can reason about what to do, but they cannot click a 5-pixel target. Behavioral cloning, even at small scale, can.

5. ICL is zero

We were optimistic that a behavioral cloner trained on 9 tasks might generalize in-context to a 10th. We added a shape-matching task — different from any of the 9 — and evaluated zero-shot, prefix (2 GT steps as context), and few-shot (3 demo sequences) settings.

Accuracy: 0-1% across all three. Fine-tuning the same model on 5,000 steps of the new task recovers ~77%. Multi-task pretraining gave us motor skills, not goal-recognition. Like VPT (Video PreTraining for Minecraft) before us, BC is a great prior — it is not AGI in a bottle.

6. Smaller wins at this data scale

We trained four model sizes (25M, 28M, 95M, 320M) to matched training loss and asked which generalizes best on real evaluation. The 28M model wins 7 / 9 tasks. The 320M model wins only 1.

A 32M-frame dataset is not yet the regime where scale dominates. The implication is concrete: before training larger BC models for science, we need a lot more annotation traces, ideally from real annotators on real tools. Building that data pipeline is the natural next research direction.

The synthetic-to-real transfer

The point of the whole exercise — proving that synthetic pretraining is a useful foundation for real annotation — is in the closing section of the paper. We took the multitask checkpoint and fine-tuned it on two real connectomics datasets:

Closed-loop autoregressive tracing — where the model has to identify a neuron at z=0 and follow it through 50 z-slices, clicking the cross-section at each — reaches 95.1% skeleton accuracy on H01 and 89.4% on C. elegans. The done rate is 100% on both. The first-node-identification accuracy is 100%.

Most of the learning happens in the first ~20K fine-tuning steps. The pretrained model already achieves 99% action-validity and 43% canvas @5px on H01 before any fine-tuning at all — the GUI fluency transfers, even when the visual domain shifts from synthetic dots to real EM.

Things that did not work

Worth saying out loud:

Where this goes next

The framework's value is not in any single result — it is in being a meeting ground between two communities that talk past each other. Domain scientists know exactly what their annotation pain points are; ML researchers have the techniques to attack them. Without a shared benchmark of synthetic environments, the conversation never gets concrete.

Three things we are doing next:

  1. Build the data pipeline. Collect real screen recordings from real proofreaders on a few partner tools. Process them with our pipeline. See whether synthetic-pretrained models transfer with thousands rather than millions of frames.
  2. Get past pure BC. RL on top of BC, especially for the long-horizon tasks where DAgger failed. The synthetic environments are an unusually clean place to do that because we have perfect ground truth.
  3. More tasks. We left building-footprint annotation, shadow-to-building tagging, and foveated visual search on the floor; they need to come in. We also want a chromosome-tracing task and a galaxy-morphology task to cover more domains.

Try it

The code, configs, and the full task suite are public. The model checkpoints, evaluation data, and a sandbox web app are all linked from the project page. We would love for someone to come at this from the domain-science side and tell us what task we should add next.

The last mile of scientific data analysis
doesn't have to be human-mile-by-human-mile.

← back to project page