Evidence wall
What we measure, and what we don't
Every credibility claim on preprints.ai gets its own page — with a denominator, a sample size, a source-code link, and a list of things it does not measure. No marketing claims, no cherry-picked screenshots.
When we don't yet have data, the page should say so plainly. External benchmarks are attributed and separated from production performance. This wall documents screening modules and their failure modes; it is not evidence that the overall system performs peer review.
01
Layer 1 audit modules
Layer 1 audit pipeline
18 deterministic modules run before any LLM sees the paper. Coverage rises with the daily backfill cron.
Hidden-prompt detection
Rendering-level scan for white-on-white text, sub-pixel fonts, and off-page coordinates that hide instructions from humans.
Paper-mill signals
Template, fingerprint, and metadata patterns associated with paper-mill output. Methodology and caveats; quantitative results pending labelled corpus.
Image forensics
Duplicate-region detection and band-shift heuristics inspired by ELIS. Runs only when full text is available.
Citation verification
Cross-checks the reference list against Semantic Scholar and OpenCitations. Flags broken DOIs, retracted citations, and self-citation rates.
Replication context
Links assessed papers and cited foundations to documented replication and reproduction attempts in the open FLoRA dataset.
Scholarly evidence graph
A versioned relational layer connecting preprints, published versions, citations, repeat attempts, outcomes, and screening runs.
02
Model screening panel
Calibration corpus & review-12b
9,279 training examples (eLife + preprints.ai + PREreview). Calibration model is in training; not yet wired into the production grade pipeline.
Honest methodology limits
What the pipeline cannot detect: fabricated data, misread figures, and agreement that is consistency, not truth.
03
External grounding
Semantic Scholar literature context
Related-work and missing-citation suggestions sourced from Semantic Scholar. Coverage limited by S2 rate limits; gaps marked explicitly on the report.
Portfolio pattern signals
Denominator-first 30-day trends for replication dependencies, problematic-paper concentration, reference integrity, screening provenance, and citation overlap.
04
Quality measurement
05
Case studies
Retracted papers we would have caught
Retrospective analysis of retracted preprints scored by the live pipeline. Page forthcoming once the labelled set is finalised.
Prompt-injection attacks caught in production
Real adversarial PDFs flagged by the hidden-prompt detector, with redactions. Page forthcoming once cloud-path coverage stabilises.