Claim-level evidence audit: check each claim

Oryx Centre for Strategic Futures is a two-person research initiative founded 2 October 2026. It is a Udyam-registered micro enterprise (UDYAM-UP-56-0189301) — NOT a registered organisation and NOT a registered non-profit. Every number below from Oryx’s own work is labelled as a pilot result, not a universal rate.

Most checks of a document give one verdict for the whole thing: pass, fail, or a score. A claim-level evidence audit does the opposite. It breaks a document into its individual claims, checks each claim against its own source, and records a separate verdict for each one. A report can then be mostly sound and still contain three claims that fail — and you can see exactly which three, and why.

That is the whole idea. The rest of this page explains what counts as a claim, how each one is checked, what the verdicts mean, and what an audit report contains. You need nothing from Oryx to follow the method.

Why check claim by claim

A document is not one assertion. It is dozens of them, and they do not share a fate. One claim may rest on a real, current source that says exactly what is claimed. The claim next to it may cite a paper that does not exist, or a real paper that says something else entirely. A single score for the document hides that difference — and the difference is the decision.

Claim-by-claim checking also makes the result fixable. “Your paper scored 72%” tells an author nothing to do. “Claims 4, 9, and 17 fail; claims 4 and 9 have broken links, claim 17 cites a real paper for a statement it does not make” tells them exactly what to repair. The same applies to a vendor proposal: a failed claim is a named gap to fix or withdraw, not a reason to argue about the whole vendor.

What counts as a claim

A claim is a checkable statement plus a source that should support it. Two parts, both needed:

• The statement. Something that can be true or false: a finding, a figure, a comparison, a description of what a system does. “Our model scores 92% accuracy on FinanceBench” is a claim. “Enterprise-grade accuracy you can trust” is not — there is nothing in it to check until it points at evidence.

• The source. The document, dataset, evaluation, or reference that is supposed to back the statement up. A claim without a source is unverified, not unimportant: it goes on the list with the verdict “cannot verify” and the reason “no source offered”.

Slogans get one question instead of a check: “What evidence supports this, in writing?” If the answer is more slogans, the statement stays out of the decision.

The four verdicts

Every claim in an audit gets exactly one verdict:

1. Supports. The source exists, resolves, and says what the claim says. The claim can be relied on.

2. Partially supports. The source exists and is relevant, but it supports only part of the claim — a narrower result, a different population, an older version. The claim needs narrowing or a better source.

3. Does not support. The source exists but says something else, or contradicts the claim. A real source saying something else is a failed check, not a pass.

4. Cannot verify. The source does not resolve, cannot be located, or was never offered. The claim is not checked — it is uncheckable as presented, and it stays that way until a source is supplied.

The first three verdicts are about the relationship between claim and source. The fourth is about the absence of a relationship at all. Keeping them separate matters: “does not support” and “cannot verify” fail for different reasons and are fixed differently.

The five steps, per claim

The audit runs the same five steps on every claim:

1. Extract the claim, word for word. Record the statement exactly as made, with where it appeared (page, section, slide, email). A claim you cannot quote back is a claim you cannot hold anyone to.

2. Identify the source it points to. Note the citation, link, DOI, document, or dataset attached to the claim. If no source is attached, mark the claim “cannot verify — no source offered” and move on; no further step can rescue it.

3. Check the source resolves. Follow the DOI or link yourself. Open the document. A dead link or a document that cannot be located is a failed citation, not a minor flaw. In Oryx’s published pilot audit of 66 AI-generated citations, 31 (47%) had broken links (Paper 2, study 1 — a pilot, not a universal rate). Resolvability fails often enough to check every time.

4. Check the source actually supports the claim. Read the relevant passage and mark the verdict plainly: supports, partially supports, or does not support. In the same pilot audit, 7 of the 66 citations (11%) mismatched the claim they were attached to, 11 (17%) could not be verified, and only 5 both resolved and fully supported the claim (Paper 2, study 1 — a pilot, not a universal rate). Existence and support are two different checks, and only the second tells you whether the claim is true.

5. Record the verdict, with the reason. One line per claim: the claim, the source, the verdict, and the one-sentence reason. The log is the audit. Without it, the work cannot be re-checked, and a reviewer cannot tell what was actually verified.

A worked example: four claims, four verdicts

These are illustrative examples of how the verdicts work, not a test of any real document.

#	Claim (as made)	Source	Resolves?	Supports?	Verdict
1	“The model scored 92% on FinanceBench, March 2026 evaluation”	Evaluation report PDF	Yes	Supports — the report shows 92% on the stated benchmark	Supports
2	“Cuts review time by 40% for mid-size firms”	A two-page case note, no client named, no dates	Yes	Partially supports — 40% appears, but for one unnamed firm over two weeks, not mid-size firms generally	Partially supports
3	“Trained only on licensed data”	No source offered; “trust our compliance team”	No source	Not checked	Cannot verify
4	“A 2024 study found X improves outcomes by 30%”	A real 2024 paper — which reports a 12% improvement, on a different outcome	Yes	Does not support — the source exists but says something else	Does not support

Claim 4 is the most instructive. The citation is real, the paper is real, the year is right — and the claim still fails, because the paper does not say what the claim says. Only reading the passage catches that. A check that stops at “the link works” would have passed it.

How a claim-level audit differs from other checks

• Peer review judges a paper’s contribution, methods, and argument. A claim-level audit judges only whether each claim’s stated evidence supports it. It is narrower, and it can be run before peer review, on a draft, or on a document that will never be peer-reviewed at all — a vendor proposal, a policy brief, an internal report.

• A plagiarism check asks whether the words are original. A claim-level audit asks whether the claims are supported. A fully original document can fail every claim; a heavily borrowed one can pass them all.

• A fact-check usually verifies a small number of high-profile claims. A claim-level audit is systematic: every material claim in the document gets a verdict, not just the ones that look suspicious.

• An AI detector guesses whether text was machine-written. A claim-level audit does not care who wrote the text — it checks whether the claims hold. AI-generated citations are a common reason to run one, because they look plausible while failing often, but the method applies to human-written documents too.

The human review gate

An audit is not finished when the checks are run. It is finished when a named human reviews the verdicts and signs off on them. That is the human review gate: a person, named in the report, who has read the verdicts, checked the reasoning on the failures, and accepts responsibility for the result. The gate exists because verdicts involve judgement — “partially supports” is a judgement call, and a judgement call needs a named judge.

Oryx’s working paper describes a six-layer framework in which the human gate is one layer among others; the audit itself is the claim-by-claim method described above. The gate does not re-do the checks. It reviews them, challenges the borderline verdicts, and signs the report. Without a named gate, an audit is a spreadsheet, not a decision.

What an audit report contains

A complete claim-level evidence audit report has four parts:

1. The claim list. Every material claim in the document, quoted word for word, with where it appeared.

2. The verdict log. One row per claim: the source, whether it resolved, the verdict, and the one-sentence reason. Failed claims carry the reason in full — a broken link, a mismatched passage, a missing source.

3. The summary counts. How many claims were checked, and how many fell under each verdict. Counts are reported as counts, not as a score: “5 of 66 claims fully supported” is informative; “72% trustworthy” is not a claim-level result.

4. The sign-off. The name of the human reviewer, the date, and what happens next for the failed claims — fix, re-source, withdraw, or re-check by a stated date.

Anything less is a partial audit, and should say so. A report that lists verdicts without reasons cannot be re-checked; a report without a named reviewer cannot be relied on.

What an audit cannot do

A claim-level audit checks whether the stated evidence supports the stated claim. It does not check:

• Whether the underlying science is right. A claim can be faithfully supported by its source and still be wrong, if the source itself is wrong. The audit verifies the chain from claim to source, not the source’s own correctness.

• Whether the document is complete. An audit checks the claims that are there. It does not find the claims the author left out.

• Whether a number is meaningful. A percentage with no method stated — what was measured, on what inputs, by whom — can pass the support check (the source really does report that number) while still telling you nothing useful. Method questions are a separate check, and worth running alongside.

Saying these limits plainly is part of the method. An audit that claims more than it checks is itself an unsupported claim.

When to run one

• Before submitting a paper that contains AI-assisted citations, or any citations you have not personally verified.

• Before signing a vendor contract, on the material claims in the proposal and demo — the claims the purchase decision rests on.

• Before publishing a report that will be quoted, cited, or used to justify a decision.

• After a draft changes, on the claims that changed. A re-check of changed claims only is a legitimate partial audit, as long as it says so.

The cost of running one is time: roughly a few minutes per claim for the resolve-and-support checks, plus the extraction pass. The cost of not running one is a failed claim discovered by a reviewer, a client, or a reader — after the document is already out.

Sources and further reading

• Oryx Working Paper v3, From Answers to Auditable Intelligence (Zenodo DOI 10.5281/zenodo.23718472) — the six-layer framework behind the claim, source, and human-gate checks above.

• Paper 2 v2, Trust, But Verify (Zenodo DOI 10.5281/zenodo.23731235) — the published pilot citation audit quoted in steps 3 and 4.

• Oryx’s guide, “Auditable AI research: what makes research checkable” (oryxai.in) — the five properties that make any report, including an audit report, checkable.

• Oryx’s guide, “How to check AI-generated citations in your own paper” (oryxai.in) — the five-step citation check this audit method applies claim by claim.

• Oryx’s guide, “How to check an AI vendor’s claims before you sign” (oryxai.in) — the six-step vendor-claim check, and the worked example of the same method on a proposal.

• Oryx’s “AI vendor claims checklist” (oryxai.in) — the seven-check version of this method, run per claim with a copyable log.

• Oryx’s guide, “Human review gate AI” (oryxai.in) — the named-human gate that signs off an audit.

---

Oryx AI offers claim-level evidence audits for research drafts and vendor claims — ₹7,500 per pilot report, payable only on acceptance, currently invite-only beta (contact: amartya@oryxai.in).

Oryx Centre for Strategic Futures is a two-person research initiative working on auditable AI.