Methodology
Machine-generated screening signals with visible provenance—three independent axes, explicit limitations, and no claim of scientific verification.
For papers with sufficient inputs, the pipeline produces a compact [Evidence][Trust][Novelty] screening profile—for example BC6. It is a triage aid, not a verdict. Each axis has its own provenance requirements and may be marked available, limited, or unavailable; missing execution metadata is never treated as proof that a stage ran.
Product contract: these outputs are machine-generated screening signals with visible provenance. They are not peer review, scientific verification, publication recommendations, or probabilities that a paper is correct. Human interpretation is required.
The three axes
A rubric-guided model judge reviews routed manuscript sections, bounded context, panel outputs, and automated checks. The letter summarises machine-detected methodology, consistency, and claim-support signals. It is not a validated probability of correctness.
- A Few material concerns detected under the current rubric
- B Generally strong signal with minor detected limitations
- C Mixed signal with notable limitations to inspect
- D Substantial methodological or claim-support concerns detected
- E Critical concerns detected under the current rubric
A rule-based marker signal covering applicable open-science and reporting checks. No language model directly chooses this letter, but extraction quality and heuristic applicability affect what is counted. Missing Layer 1 execution or a degraded critical check makes this axis unavailable.
- A Excellent — meets nearly every applicable check
- B Strong — most checks met
- C Moderate — some checks met
- D Limited — few checks met
- E Minimal — essentially no trust signals
What it weighs (each counted only when it applies): transparency markers — data & code availability, conflict-of-interest disclosure, funding, author contributions, and (for human/animal-subject work) an ethics statement — plus reproducibility, statistical rigor, and citation health. Downward caps: the fraud / error checks (paper-mill and fabrication scans, statcheck statistical-consistency, retracted-reference detection) force the grade down regardless of the rest.
A 1–10 corpus-relative model signal, informed by related literature when retrieval was available. It is not an exhaustive prior-art search. Every displayed value carries a basis label: Literature-grounded, Literature-assisted · reconstructed provenance, or LLM-derived. Novelty is withheld only when the value or its generating origin is unknown, or the input run was insufficient.
| 1–3 | Landmark · Major breakthrough · Fundamental |
| 4–6 | Significant · Important (median) · Notable |
| 7–10 | Valuable · Incremental · Marginal · Minimal |
Availability and run status
- Complete screening run — positive evidence records full text, the expected eleven roles, non-empty Layer 1 and literature outputs, the rubric judge, valid score contracts, and no critical degradation.
- Limited screening signal — at least one axis is usable but a required stage, provider, or provenance field is incomplete. Each affected axis is labelled separately.
- Screening signal withheld — the input was not recorded as full text, the pipeline was insufficient, or no axis has enough provenance to display.
Run completeness describes execution only. A complete run can still be wrong, and an apparently strong letter is not scientific verification. The axes must not be collapsed into a publication verdict or universal “best paper” ordering.
The pipeline
Deterministic checks (Layer 1)
Before model reviewers run, the paper is checked by deterministic modules. Their structured outputs contribute reporting markers and potential fraud/error flags. A failed or skipped safety-critical module is recorded as degraded, not interpreted as a clean result.
- Trust-marker detection — data / code availability, COI, ethics, funding, author contributions, preregistration, and ORCID, with source excerpts where the module records them.
- Paper-mill signals — template / fingerprint cross-reference.
- Retracted references — extracted citation identifiers are matched against available retraction records; coverage depends on extraction and source databases.
- Statistical consistency — GRIM and statcheck-style p-value checks.
- Text-capture gate — new runs below the extraction threshold are deferred or withheld. Historical rows without positive full-text provenance remain labelled limited.
Eleven model-reviewer roles (Layer 2)
Eleven expected roles receive role-specific manuscript sections and bounded context from full-text extraction, then produce structured machine outputs. Section routing improves coverage but does not guarantee that every role saw every claim, figure, table, or supplement:
- Methodologist — design, controls, confounders.
- Statistician — tests, effect sizes, multiple-testing correction, sample size.
- Scientific validity — do the claims follow from the evidence?
- Ethics & transparency — ethics, funding, COI, contributions.
- Domain primary / clinical / methods / literature / reproducibility — five specialist voices, selected per paper.
- Devil's advocate — adversarial critic stressing the consensus.
- Positive evidence — counter-weight that surfaces what the paper does well, against a 10-axis rubric.
Models may differ by role and deployment. Each role emits structured strengths, concerns, internal scores, and questions. The report should be read as a collection of machine observations to investigate, not as eleven peer reviews.
A note on independence: model reviewers share training data, prompts, inputs, and reasoning patterns. Agreement reflects correlated model behaviour as well as corroboration; it is not equivalent to independent human agreement.
Scoring and derivation (Layer 3)
The screening profile combines model outputs with explicit rules:
- Trust signal from applicable Layer-1 checks—transparency, reproducibility, statistical reporting, and citation health—with downward safety gates (
core/trust_v2.py). - Novelty 1–10 by percentile-binning the panel's significance into the corpus bell curve (
core/novelty_scale.py). - Evidence letter from a rubric-guided model judge, mapped by fixed internal score thresholds and constrained by deterministic gates (
core/calibrated_judge.py). - Legacy editorial/publication fields remain in some API responses for backwards compatibility but are not displayed as product outputs and are not validated predictions.
- Claim screening extracts candidate major claims and machine-classifies them against recorded concerns. Those classifications require checking against the manuscript.
Arbitration (Layer 4)
The panel is advisory. The rubric-guided judge supplies the Evidence letter when its execution is positively recorded; a fallback makes the Evidence axis limited. Deterministic hard gates can only move a screening signal downward or withhold it. These gates reduce silent success but do not prove that a flagged issue is real or that an unflagged paper is sound.
Calibration & validation
The current letters and numbers are operational screening bands, not externally validated probabilities. Production calibration remains incomplete.
- Historical eLife mapping — an earlier label lookup was compared with historical eLife review labels. Its very high lookup agreement measures implementation of that retired mapping, not scientific truth and not validation of the current three-axis production score.
- Self-refreshing novelty bins — the 1–10 cutpoints are recomputed from the live assessed corpus. This maintains a distribution; it does not validate novelty against expert judgement.
- Small venue-proxy check — a held-out set of roughly 50 papers was used as a smoke test for grade separation. Publication venue is confounded and is not a gold-standard quality label.
- Expert validation set required — a blinded, field-stratified human-reviewer baseline is still needed; we do not report a current human-agreement or scientific-accuracy figure.
Limitations
- This is a machine-generated screening signal with visible provenance, not peer review or scientific verification.
- The pipeline cannot replicate experiments — fabricated data that reads plausibly will score higher than it should.
- Evidence is a model judgement; Trust is rule-based marker coverage; Novelty depends on literature retrieval. None is a direct measurement of truth.
- Reviewer and judge contexts are bounded. The system routes sections but does not yet provide claim-complete coverage of every figure, table, and supplement.
- “Complete run” confirms recorded execution provenance, not that the paper's conclusions have been scientifically verified.
- Panel agreement is not full independence (see Layer 2) — treat a unanimous panel as suggestive, not conclusive.
- Domain agents are strongest in the life sciences; physics, maths and CS get a weaker signal. Field-specific norms (e.g. preregistration is not expected in pure maths) are an area of ongoing work.
- Signals can become stale when a preprint, model, rubric, or literature corpus changes. Version-complete automatic re-screening is still being developed.
Source code & reproducibility
The deterministic lookups, marker rules, and documented pipeline behaviour can be inspected through the docs and Evidence pages. Current reports expose the provenance fields actually recorded. Older rows may lack manuscript hashes, model versions, module states, or literature queries; those absences are limitations, not silently reconstructed evidence.