Calibration corpus & review-12b
9,279 reviews assembled from eLife, preprints.ai self-generated, and PREreview, used to train a calibration model that maps the 11-agent panel output to a final grade. The model is in training and not yet wired into the production pipeline.
Pipeline today
The shipping production pipeline is described on the methodology page. Eleven specialist roles review routed manuscript context; a calibrated judge produces the Evidence letter subject to deterministic hard gates, while Trust is rule-based and Novelty is corpus-binned with literature grounding when available.
An earlier deterministic label lookup was built from roughly 800 historical eLife reviews. That mapping is useful historical calibration material, but it is not a gold-standard validation of the current three-axis production score.
What review-12b is
review-12b is an in-training calibration model whose only job will be to map the structured output of the 11-agent panel to the same grade scale, learned end-to-end against the 9,279-review corpus rather than via the hand-tuned lookup. It is a shape-matching layer, not a replacement reviewer.
- 6,411 eLife reviews. Public, structured, with explicit evidence-strength and significance labels. The largest single source.
- 1,570 historical preprints.ai model outputs. Useful only for self-consistency analysis; they are not expert ground truth and must not validate the system against itself.
- 1,298 PREreview reviews. Harvested from the Zenodo community
prereview-reviews(1,593 total records; 1,298 retained after dedup and quality filtering). Released under CC-BY 4.0.
Status
review-12b is not integrated into the live grade pipeline. Current Evidence grades use the calibrated judge and hard gates described on the methodology page. If review-12b is evaluated for production, we will publish agreement, calibration error, grade-shift histograms, and a human-audited disagreement set before rollout.
Caveats — what this doesn't measure
- The eLife slice is biased toward life-sciences manuscripts that survived editorial triage. It is not a random sample of the preprint universe.
- PREreview reviews are CC-BY 4.0, but reviewers self-select onto the platform; their reviews may differ systematically from journal peer review.
- The 1,570 self-generated reviews exist only to test whether the model can reproduce its own consensus stably across repeats. They are excluded from any held-out evaluation set.
- Agreement with eLife's public-review labels is not the same as agreement with the eLife reviewers' underlying judgement about quality. We measure label match, not truth.
Code & attribution
Pipeline orchestration: agents/agentic_review.py. PREreview reviews are reproduced under CC-BY 4.0; eLife public reviews are reproduced under their open licensing.