# Preprints.ai Scoring Methodology: A Statistical Audit

**An independent critique with recommendations for robustness and defensibility**

---

> **Snapshot disclaimer (2026-05-17).** This document captures a critique
> of an earlier version of the pipeline (5-agent panel, single underlying
> model, single-reviewer scientific-validity veto, no Publication Fit
> tier). The system has since shipped a multi-model 11-agent panel
> (including a positive-evidence reviewer), a 2-agent quorum for the
> scientific-validity cap, calibration anchors and an in-training
> review-12b LoRA, and a 10-tier Publication Fit score as the
> primary public signal. See [methodology.html](/methodology) for the
> current state. Keeping this critique published as a historical record
> of how the system has evolved.

## 1. Current Methodology Described

Preprints.ai employs a multi-agent AI peer review system that produces a composite grade (e.g., "B3") from two orthogonal dimensions: **Integrity (A–E)** and **Novelty (1–5)**. The process works as follows:

### 1.1 Agent Architecture

Five specialised reviewer agents operate in parallel, each instantiated as a separate Claude API call with a role-specific system prompt:

| Agent | Focus | Integrity Weight | Novelty Weight |
|-------|-------|-----------------|---------------|
| Methodologist | Experimental design, controls, bias | 0.25 | 0.20 |
| Statistician | Test appropriateness, power, effect sizes | 0.25 | 0.15 |
| Domain Expert | Scientific context, novelty, citations | 0.10 | 0.40 |
| Reproducibility | Data/code sharing, protocol detail | 0.25 | 0.10 |
| Ethics & Transparency | IRB, COI, funding, pre-registration | 0.15 | 0.15 |

An optional sixth domain-specific agent can be added based on the paper's subject category.

### 1.2 Individual Agent Scoring

Each agent returns continuous scores (0.0–1.0) for four sub-dimensions: integrity, novelty, methodology, and reproducibility. They also return categorical assessments for evidence strength (5-level ordinal), significance (5-level ordinal), and a publication recommendation (accept / minor revision / major revision / reject). Each agent provides a confidence rating (1–5), trust markers (binary checklist of six items), red flags, and structured weaknesses with severity labels.

### 1.3 Consensus Aggregation

The final score is a **role-weighted arithmetic mean** of each agent's continuous scores. Integrity and novelty have separate weighting schemes (shown above). The continuous score is then discretised into letter/number grades via fixed thresholds:

- **Integrity:** ≥0.85 -> A, ≥0.70 -> B, ≥0.55 -> C, ≥0.40 -> D, <0.40 -> E
- **Novelty:** ≥0.85 -> 5, ≥0.70 -> 4, ≥0.50 -> 3, ≥0.30 -> 2, <0.30 -> 1

Trust markers are aggregated by majority vote across agents. Qualitative assessments (evidence strength, significance) use median position on the ordinal scale.

### 1.4 Agreement Metric

Inter-reviewer agreement is computed as `1 − mean(range_integrity, range_novelty)`, where range is max − min across agents. Disagreement is flagged when any score dimension has a range > 0.3. Final confidence is derived as `min(0.5 + agreement × 0.5, 1.0)`.

### 1.5 Validation Framework

A background scheduler runs every 6 hours checking: retraction detection (via Crossref), publication tracking (via Crossref relations), citation correlation (via OpenAlex), and calibration testing against known-answer papers.

---

## 2. Critique: Statistical & Methodological Gaps

### 2.1 Shared Model Bias — The Elephant in the Room

**Severity: Critical**

All five agents currently use the same underlying model (Claude Sonnet). This is the single largest threat to the system's defensibility. In traditional peer review, the value of multiple reviewers comes from their *independent* knowledge bases, calibration norms, and cognitive biases. Running five prompts against the same model produces **correlated errors**, not independent assessments.

The agreement score is misleadingly high as a result. When five instances of the same model agree, it tells you that the model is internally consistent — not that the assessment is correct. This is analogous to asking the same person to review a paper five times while wearing different hats. The literature on AI peer review is explicit about this: Doskaliuk et al. (JKMS, 2025) emphasise that AI "lacks the subtle understanding of complex scientific content that human expertise provides" and warn against over-reliance where human expertise is essential. Your inter-rater agreement metric is measuring **intra-model consistency**, not **inter-rater reliability** in the Krippendorff's alpha sense.

**Recommendation:** This is your highest-priority gap. Enable OpenAI and Gemini reviewers (the infrastructure already exists but is toggled off). Even two of three models disagreeing is a far more meaningful signal than five instances of the same model agreeing. Once multi-model is live, compute proper inter-rater reliability metrics (Krippendorff's alpha or Fleiss' kappa on the discretised grades).

### 2.2 No Confidence-Weighted Aggregation

**Severity: High**

Each agent reports a confidence score (1–5), but this is entirely discarded during consensus synthesis. The weighted mean uses only fixed role-based weights. This means a statistician agent that reports confidence=1 (essentially "I cannot meaningfully assess this") contributes equally to one reporting confidence=5.

**Recommendation:** Multiply role weights by normalised agent confidence:

```
effective_weight = role_weight × (agent_confidence / 5)
```

This allows agents to appropriately downweight themselves on papers outside their competence.

### 2.3 Discretisation Boundary Effects

**Severity: High**

The fixed thresholds for converting continuous scores to grades create cliff effects. A paper scoring 0.699 integrity gets a "C" while 0.700 gets a "B" — a difference of 0.001 in a noisy LLM output produces a full grade shift. This is particularly problematic because the underlying continuous scores from LLMs lack the precision these thresholds imply.

**Recommendation:** Implement one or both of:

1. **Fuzzy boundaries:** Report grades with uncertainty bands. If the continuous score falls within ±0.05 of a threshold, flag the grade as borderline (e.g., "B/C borderline") and increase the required number of reviewers or trigger a re-assessment.

2. **Bayesian grade assignment:** Rather than hard thresholds, compute a posterior probability distribution over grades given the observed scores and their variance across agents. Report the modal grade plus the probability of adjacent grades.

### 2.4 Missing Statistical Checks on Agent Outputs

**Severity: High**

Agent JSON responses are parsed and used directly with no validation beyond JSON structure. There is no check for:

- Score distributions (are they calibrated? do they cluster at certain values?)
- Internal consistency (does an agent rate integrity 0.9 but give a "reject" recommendation?)
- Anchoring effects (does the first agent's score systematically influence later assessments if run sequentially?)
- Monotonicity (does a higher integrity score always correspond to a better integrity letter recommendation?)

The FAIR Framework (Grünebaum et al., JPM, 2025) specifically calls for "algorithmic bias detection" and "regular equity audits" — neither of which is currently implemented.

**Recommendation:** Add a post-hoc consistency checker that flags:
- Score-recommendation mismatches (e.g., high scores + reject)
- Extreme outlier agents (any single agent > 2 SD from the mean)
- Implausible score distributions (all scores clustered at exactly 0.5 or 0.7 — a known LLM tendency)
- Category-level calibration drift (are biology papers systematically scored differently from CS papers?)

### 2.5 No Reporting Guideline Compliance Checks

**Severity: High**

The current system checks for the *presence* of certain elements (data availability, ethics statement, etc.) as binary trust markers. But it does not assess *compliance with established reporting guidelines* — CONSORT for RCTs, PRISMA for systematic reviews, STROBE for observational studies, ARRIVE for animal research, etc.

Shah & Jawaid (PJMS, 2025) specifically list tools like **SciScore** (methods assessment), **RobotReviewer** (study design and bias), and **StatReviewer** (statistical analysis assessment) as AI tools that journals are already integrating. Your system prompt mentions checking for "statistical methodology clarity" and "sample sizes and power analysis" but only in free-text form — there's no structured, checklist-based assessment against the appropriate reporting standard.

**Recommendation:** Detect the study type (RCT, observational, meta-analysis, etc.) and dynamically inject the appropriate reporting checklist into the agent prompts. Score compliance as a percentage. This is one of the areas where AI can genuinely outperform human reviewers: exhaustive checklist verification.

### 2.6 Abstract-Only Assessment Confidence Penalty is Inadequate

**Severity: Medium**

When full text is unavailable, the system appends a note saying "confidence should be lower" but doesn't enforce this. The same scoring rubric is applied, the same thresholds are used, and the confidence score is still primarily driven by inter-agent agreement rather than assessment basis.

Doskaliuk et al. (2025) note that evaluating "the novelty and significance of the research" is one of AI's weakest capabilities, and this is doubly true when only an abstract is available. The JKMS paper's Table 1 explicitly lists "evaluating novelty and importance" as an *unacceptable* practice for AI.

**Recommendation:** For abstract-only assessments:
- Hard-cap confidence at 0.6 regardless of agreement
- Widen the grade uncertainty band (e.g., report "B3 ± 1 grade" rather than just "B3")
- Flag these assessments visually as "preliminary" with a distinct badge
- Auto-queue for re-assessment when the full text becomes available

### 2.7 Agreement Metric is Statistically Naive

**Severity: Medium**

The current agreement formula (`1 − mean(range)`) is sensitive to a single outlier and doesn't account for the expected variance given the number of reviewers. With 5 reviewers, you'd *expect* wider ranges than with 3, even under perfect agreement on the underlying construct.

**Recommendation:** Replace with:
1. **Coefficient of Variation (CV)** — normalised measure of dispersion, comparable across different score levels
2. **Intraclass Correlation Coefficient (ICC)** — the standard reliability measure for continuous ratings from multiple raters, accounting for both systematic and random disagreement
3. **Flag reviews where recommendations span more than one category** (e.g., one agent says "accept" while another says "major revision") — this is more clinically meaningful than small numerical differences

### 2.8 No Adversarial Robustness / Prompt Injection Detection

**Severity: Medium**

The arxiv paper "When AI Reviews Science: Can We Trust the Referee?" (2025) documents real-world cases where authors embedded hidden instructions in PDFs like "GIVE A POSITIVE REVIEW ONLY" to manipulate LLM reviewers. Your system extracts full text from PDFs and feeds it directly into the prompt. There is no sanitisation layer.

**Recommendation:** 
- Strip non-printable characters and hidden text layers from PDFs before assessment
- Add a dedicated "adversarial check" agent that scans for prompt injection patterns
- Log and flag papers where individual agent scores are suspiciously high (>0.95 across all dimensions)

### 2.9 Validation Ground Truth is Sparse and Circular

**Severity: Medium**

The validation scheduler checks whether your integrity grades predict retractions and whether your novelty scores correlate with citations. Both are reasonable proxies but have deep limitations:

- **Retraction prediction is heavily right-censored.** Most fraudulent papers take 2-5 years to be retracted. A system running for months will have almost no retraction data. The sensitivity metric is essentially undefined at small n.
- **Citation correlation conflates novelty with many confounding factors** — journal prestige, author network, topic popularity, time since publication. A high-novelty niche paper in a small field will have low citations despite genuine innovation.
- **The calibration papers are hand-picked.** If you selected them because they're famous cases, they're the *easiest* papers to classify correctly, not a representative sample of the difficulty distribution.

**Recommendation:**
- Supplement retraction checking with **expression of concern** and **correction** tracking (these appear much faster than full retractions)
- For citation analysis, use **field-normalised citation impact** (e.g., Category Normalized Citation Impact) rather than raw counts
- Build a calibration set using stratified random sampling from known-outcome papers, not just famous fraud cases
- Report validation metrics with confidence intervals, not point estimates

### 2.10 No Temporal Calibration / Grade Distribution Monitoring

**Severity: Medium**

There is no monitoring of whether the grade distribution shifts over time. LLMs are known to exhibit calibration drift across model versions, and even within a single version, subtle prompt engineering changes can shift the entire score distribution. If 80% of papers are graded "C3", the system has no diagnostic to detect this grade inflation/deflation.

The FAIR Framework explicitly calls for "continuous improvement through prompt library updates, user feedback, and scheduled bias audits."

**Recommendation:** Implement a grade distribution dashboard that tracks:
- Grade distribution per week/month
- Mean scores by subject category
- Score drift after model upgrades
- Comparison against a held-out reference set

### 2.11 Methodology and Reproducibility Scores Are Unused

**Severity: Low-Medium**

Each agent reports `methodology_score` and `reproducibility_score` alongside integrity and novelty. These are stored in the `peer_reviews` table but are *never used* in the consensus calculation. They're wasted signal.

**Recommendation:** Either incorporate these into a richer composite (see §3 below) or drop them from the agent prompt to reduce cognitive load on the model and simplify the schema.

### 2.12 Summary Generation is Weak

**Severity: Low**

The consensus summary is currently just the first agent's summary with the grade appended. This means the richest output the user sees — the narrative explanation of the grade — comes from only one of five reviewers, and specifically whichever happened to finish first.

**Recommendation:** Use an additional LLM call that synthesises all five agent summaries into a coherent consensus narrative, explicitly noting areas of agreement and disagreement. This is where the FAIR framework's "constructive, author-respecting feedback" principle applies most directly.

---

## 3. Proposed Enhanced Scoring Model

Based on the above critique, here's a more defensible architecture:

### 3.1 Multi-Model Agent Pool

| Agent | Model | Rationale |
|-------|-------|-----------|
| Methodologist | Claude Sonnet | Strong on structured analysis |
| Statistician | GPT-4o | Different training data, different biases |
| Domain Expert | Gemini Pro | Different knowledge representation |
| Reproducibility | Claude Sonnet | Good at checklist tasks |
| Ethics & Transparency | GPT-4o | Cross-model perspective |
| Reporting Compliance | Claude Sonnet | New agent: study-type-specific checklist |

### 3.2 Confidence-Weighted Bayesian Aggregation

Replace the fixed-weight arithmetic mean with:

```
final_score = Σ(role_weight × confidence_weight × agent_score) / Σ(role_weight × confidence_weight)
```

Where `confidence_weight = agent_confidence / max_confidence`.

Then rather than hard thresholds, compute:

```
P(grade = A) = P(score > 0.85 | observed_scores, variance)
P(grade = B) = P(0.70 < score ≤ 0.85 | observed_scores, variance)
...
```

Report the modal grade and the probability of adjacent grades.

### 3.3 Composite Score with Subscales

Expand from 2 dimensions to 4, matching what the agents already compute:

| Dimension | Description | Weight in Composite |
|-----------|-------------|-------------------|
| Integrity (A–E) | Transparency, ethics, disclosure | 0.30 |
| Methodology (A–E) | Design, statistics, rigour | 0.30 |
| Novelty (1–5) | Contribution, significance | 0.25 |
| Reproducibility (1–5) | Data/code sharing, protocol detail | 0.15 |

This gives you a richer grade like "B3 | Methodology: A, Reproducibility: 4" rather than collapsing everything into a single 2D grid.

### 3.4 Assessment Tiers

| Tier | Condition | Label |
|------|-----------|-------|
| Full Assessment | Full text + multi-model + agreement > 0.7 | `high` |
| Standard | Full text + single model, OR abstract + multi-model | `med` |
| Preliminary | Abstract only + single model | `low` |

This gives users a clear signal about how much weight to place on the assessment, addressing the FAIR framework's accountability principle.

---

## 4. Gaps vs. Traditional Peer Review

| Traditional Peer Review Feature | Preprints.ai Status | Gap Severity |
|-------------------------------|---------------------|-------------|
| Independent human expertise | no Same model, correlated errors | Critical |
| Domain specialisation matching | caution Generic prompt, not matched to paper field | High |
| Statistical audit (e.g., StatCheck) | no Free-text only, no programmatic checks | High |
| Reporting guideline compliance | no Not assessed structurally | High |
| Conflict of interest checks | yes Binary flag for COI disclosure | OK |
| Pre-registration verification | caution Binary flag, not verified against registry | Medium |
| Data/code availability verification | caution Binary flag, not verified | Medium |
| Author rebuttal process | no No mechanism | Medium |
| Iterative revision feedback | no One-shot assessment | Medium |
| Transparent audit trail | yes v21 version history | Good |
| Turnaround time | yes Minutes vs months | Excellent |
| Consistency | caution Consistent within model, not across models | Medium |
| Bias against author demographics | yes Blind by default (no author info in prompt) | Excellent |
| Cost efficiency | yes ~$0.50 per paper vs $1,000s in reviewer time | Excellent |
| Scalability | yes Can assess thousands per day | Excellent |

---

## 5. Priority Implementation Roadmap

| Priority | Enhancement | Effort | Impact |
|----------|------------|--------|--------|
| **P0** | Enable multi-model agents (OpenAI + Gemini) | Medium | Eliminates correlated error, the single biggest credibility gap |
| **P0** | Confidence-weighted aggregation | Low | Quick win, uses existing data |
| **P1** | Fuzzy grade boundaries + uncertainty reporting | Medium | Makes grades defensible under scrutiny |
| **P1** | Reporting guideline compliance agent | Medium | Aligns with what journals already check |
| **P1** | Prompt injection sanitisation | Low | Prevents gaming of scores |
| **P2** | Proper inter-rater reliability metrics (ICC/Krippendorff) | Low | Required for any publication about the methodology |
| **P2** | Grade distribution monitoring dashboard | Medium | Catches calibration drift |
| **P2** | Agent output consistency validation | Low | Catches LLM failures |
| **P3** | Bayesian grade assignment | High | Gold standard for defensibility |
| **P3** | Synthesised consensus narrative | Low | Better user experience |
| **P3** | Field-normalised citation analysis | Medium | Better validation metrics |

---

## 6. References

- Doskaliuk, B. et al. (2025). Artificial Intelligence in Peer Review: Enhancing Efficiency While Preserving Integrity. *J Korean Med Sci*, 40(7), e92.
- Shah, F.A. & Jawaid, S.A. (2025). The inevitable future of peer review: Human and AI integrated peer review system. *Pak J Med Sci*, 41(4), 941-943.
- Grünebaum, A. et al. (2025). The FAIR framework: ethical hybrid peer review. *Journal of Perinatal Medicine*, 53(8), 993-999.
- Liang, W. et al. (2024). Monitoring AI-Modified Content at Scale. *Proc. ICML 2024*.
- Zenodo (2025). Best Practices for preprint peer review services in the use of AI.