▮▮▮▯ BC / Annotation
ICML 2026 · under review 95M params · 9 tasks · 1 model

A Systematic Study of Behavioral Cloning for Scientific Data Annotation

What does it look like to teach a neural network to annotate like a scientist? We build a framework of synthetic annotation environments and train a 95M-parameter agent end-to-end on raw GUI pixels — then push it onto real electron microscopy.

Author list withheld during anonymous review. Affiliations + names appear on the camera-ready.
Paper (PDF) arXiv Code Data Blog post
TL;DR

The last mile of scientific data analysis is annotation. Connectomics burns 33 person-years on a single fly brain. Astronomy, pathology, behavior, remote sensing — all paying the same tax.

We attack the problem head-on: build 9 synthetic GUI annotation environments (colored-dot tracing, cell lineage, animal behavior, hyperspectral plumes, road graphs, 3D exploration, …), simulate the human as a code policy with realistic mistakes, and train one model on raw pixels of the GUI.

Across single-task and multi-task regimes the agent learns the workflow, represents its own mistakes linearly, matches a competent human annotator, and — after fine-tuning — successfully traces neurons through real electron microscopy of H01 human cortex and C. elegans.

01 Headline findings

95M parameters

One model, nine tasks, end-to-end pixels

A 95M-parameter DINOv2 + transformer agent learns all 9 GUI annotation tasks from raw screenshots. No task-specific heads. No segmentation hints. Just (image → click).

95.1% skeleton acc · H01

Transfers from synthetic to real EM

Fine-tuning the pretrained model on real neuron-tracing data — H01 human cortex and Witvliet C. elegans — yields autoregressive skeleton accuracy of 95.1% (human) and 89.4% (worm). The synthetic pretraining → real fine-tuning paradigm works.

0.92 ROC AUC

Mistakes are linearly decodable

A linear probe on the model's layer-6/8 activations distinguishes intentional clicks from mistakes-it-will-correct at ROC AUC 0.92. The agent represents 'I was wrong' even though no explicit mistake supervision was given.

0.8% model · vs. human 2.4-7.6%

The error-correction paradox

Training data contains ~10% deliberate mistakes-then-undos. The model imitates the recovery shape but corrects far less often than humans — 0.8% vs. 2.4-7.6%. Cleaner than its teachers.

88 / 110 nav / total · agent

Indistinguishable from humans

Across navigation/placement/MIP/undo fractions, the trained agent matches the most cautious of four human annotators almost perfectly: 88 vs. 87 navigation steps, 110 vs. 111 total.

0% ICL · 77% after FT

In-context learning is ~0

Pretrained behavioral cloners do not pick up a new task format from a template or a few demos. ICL accuracy is 0-1% across zero-shot, prefix, and few-shot evaluations. Fine-tuning recovers ~77%.

28M > 320M 7 / 9 tasks

Smaller wins at this data scale

At matched training loss, a 28M-parameter model beats the 320M model on 7 / 9 tasks. The 32M-frame dataset is not yet in the regime where bigger helps.

97.4% vs. 25-80% colored dot · BC / VLM

BC beats frontier VLMs on pixel precision

On teacher-forced placement @5px, our 95M BC model outperforms both Gemini 3 Flash and Qwen3-VL-32B across all 5 placement tasks — despite being 100-300× smaller. Annotation is mostly a fine-motor problem.

02 Nine synthetic annotation tasks

Each task ships an HTML+JS GUI rendered headlessly by Playwright, a code-defined "virtual annotator" that produces realistic action sequences with mistakes-then-undos, and a screenshot+(x,y)-click dataset of millions of frames. One architecture; nine tasks; no task-specific heads.

Colored Dot Tracking sequence
Neuroscience · 3D tracing

Colored Dot Tracking

Trace a path through a stack of z-slices, using only the GUI.

Details

A continuous trajectory of colored Gaussian dots is hidden inside a 16-slice z-stack. The annotator navigates with +z/-z, toggles MIP, and clicks each dot in order from blue (start) to red (end). This is the synthetic analog of neuron tracing.

Challenge. 3D navigation with sparse evidence per slice; the path crosses itself in projection.

Actions: +z -z MIP place done
Screen: 432 × 448 BC @5px: 97.4%

Neuron Tracking sequence
Neuroscience · multi-object

Neuron Tracking

Re-identify 6-10 neurons across 10 frames with no color cues.

Details

All neurons look identical (green blobs). Identity is preserved only through spatial continuity under elastic deformation and accumulating rotation. The annotator works in batches of 3-4 neurons through all 10 frames.

Challenge. Identity tracking without appearance cues, under non-stationary rotation.

Actions: select id place next prev cancel done
Screen: 432 × 368 BC @5px: 66.5%

Cell Lineage Tracking sequence
Bio · time-lapse

Cell Lineage Tracking

Follow each cell across divisions in a Voronoi tissue.

Details

Cells divide between frames. The annotator marks the root, then propagates each cell forward — handling divisions by placing markers on both daughters. Auto-propagation for non-splitting cells keeps the workload tractable.

Challenge. Tree-structured annotation with auto-propagation logic and division detection.

Actions: place select next done
Screen: 368 × 288 BC @5px: 61.2%

Animal Behavioral Tracking sequence
Behavior · pose

Animal Behavioral Tracking

Mark head and tail of 4-8 animals across 10 frames.

Details

Each video has a 'species' shared across frames — same body plan, limbs, optional tail. The annotator places a front (head) and back (tail) marker per animal, working through groups of 1-4.

Challenge. Orientation tracking on small, articulated targets without color or markers.

Actions: place front place back next done
Screen: 528 × 416

Animal Limb Tracking sequence
Behavior · keypoints

Animal Limb Tracking

Find implicit keypoints on a bird, insect, spider, or snake.

Details

Animals are rendered as overlapping ellipses and lines — there are no visible markers, the annotator must infer keypoint locations from the shape. A tabbed GUI groups body / antennae / legs / wings / tail.

Challenge. Implicit landmark inference on high-articulation, low-temporal-correlation motion.

Actions: tab nav place next done
Screen: 416 × 304 BC @5px: 37.1%

Multichannel Image Alignment sequence
Imaging · registration

Multichannel Image Alignment

Mark corresponding landmarks across visually distinct channels.

Details

Two channels of an image are warped by affine + elastic transforms. The annotator places matching landmarks across channels — Shi-Tomasi corners are the GT, but appearance differs sharply (e.g., RGB vs. edge-channel projection).

Challenge. Correspondence across appearance shift; 50% natural + 50% synthetic DGPs.

Actions: switch channel place done
Screen: 432 × 352 BC @5px: 42.9%

Spectral Plume Finding sequence
Remote sensing · detection

Spectral Plume Finding

Find blobs that appear in exactly 3 of 5 spectral bands.

Details

Among confounders that appear in 1, 2, or all 5 bands, real plumes appear in exactly 3. The annotator toggles bands to count, places confirm markers, then draws polygon boundaries (16-24 points).

Challenge. Multi-step reasoning: explore → count → confirm → draw polygon.

Actions: b1-b5 confirm draw cancel done
Screen: 496 × 400

Road Network Construction sequence
Remote sensing · graph

Road Network Construction

Build a road graph: place nodes, connect edges.

Details

Synthetic aerial maps with MST + Delaunay road networks. The annotator places nodes at intersections in writing order, then connects roads by path-following from the last placed node.

Challenge. Graph annotation with structural constraints (degree ≥ 3, min 40° angles).

Actions: place node place edge remove undo done
Screen: 336 × 272

3D Exploration & Classification sequence
Spatial reasoning

3D Exploration & Classification

Rotate a 3D object, then pick 1 of 9 classes.

Details

Five colored spheres arranged into one of 9 topologies — line, plus, methane, square pyramid, bowtie, etc. The annotator rotates 3-10 times before classifying, with random SO(3) initial orientations forcing real exploration.

Challenge. Active exploration policy: when have you seen enough to decide?

Actions: rot ±X rot ±Y classify 1-9 confirm
Screen: 384 × 336

03 Synthetic → real electron microscopy

The synthetic-pretrain → real-finetune paradigm is the actual deliverable. Below: closed-loop autoregressive neuron tracing on two real EM datasets, fine-tuned from the same multitask base checkpoint.

H01 · Human cortex

1,013 myelinated axon skeletons, 6.5M frames of fine-tuning. Trace each axon by clicking it through 50 z-slices.

H01 · Human cortex 3D reconstruction
3D oblique view · agent-traced neuron
H01 · Human cortex tracing GIF
Closed-loop tracing through z-slices
Skeleton accuracy (autoreg)95.1%
Canvas @5px95.1%
Canvas @10px97.5%
Done rate100%
First node accuracy100%

C. elegans · Witvliet nerve ring

Dense neuropil, 30.5% segmented voxels. 19,872 fine-tuning episodes on the synthetic-pretrained checkpoint.

C. elegans · Witvliet nerve ring 3D reconstruction
3D oblique view · agent-traced neuron
C. elegans · Witvliet nerve ring tracing GIF
Closed-loop tracing through z-slices
Skeleton accuracy (autoreg)89.4%
Canvas @5px78.0%
Canvas @10px83.9%
Done rate100%
First node accuracy100%

04 Behavioral cloning vs. frontier VLMs

On pixel-precise placement, behavioral cloning at 95M parameters beats both frontier VLMs across every placement task we ran.

Task BC · 95M Gemini 3 Flash Qwen3-VL-32B
Colored Dot Tracking
97.4%
80.0%
25.0%
Neuron Tracking
66.5%
28.0%
4.0%
Cell Lineage Tracking
61.2%
32.4%
0.0%
Multichannel Alignment
42.9%
18.5%
12.9%
Animal Limb Tracking
37.1%
24.0%
8.3%

Teacher-forced placement accuracy @5px, n=64.

05 Indistinguishable from a competent human

Four human annotators ran 5 colored-dot-tracking instances each. The virtual annotator's action distribution falls inside the human envelope — and the trained model is closest to the most cautious human.

Annotator 1 · cautious
Annotator 1 · cautious
Annotator 4 · efficient
Annotator 4 · efficient
Trained model
Trained model
Synthetic GT
Synthetic GT
VirtualAnn. 1Ann. 3Ann. 4Model
Navigation % 73.878.170.066.6—
Placement % 14.415.421.122.9—
MIP toggle % 10.04.96.48.5—
Total steps 1181118871110
Correction % 6.64.67.62.40.8

06 Read the long version

blog post

Annotation is the last mile.

A long-form walk through the why, the how, and what we learned — written for the bench scientist and the ML researcher in the same room.

Read →
“Most existing work learns from annotation outcomes. We learn from annotation process — the screen recording, the cursor path, the mistakes-then-undos.”

07 Cite

@inproceedings{bcsda2026,
  title     = {A Systematic Study of Behavioral Cloning for Scientific Data Annotation},
  author    = {Anonymous},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
  series    = {ICML 2026},
  year      = {2026},
}