AI vendor claims checklist: check every claim

Oryx Centre for Strategic Futures is a two-person research initiative founded 2 October 2026. It is a Udyam-registered micro enterprise (UDYAM-UP-56-0189301) — NOT a registered organisation and NOT a registered non-profit. Every number below from Oryx’s own work is labelled as a pilot result, not a universal rate.

A vendor proposal is not one decision. It is a bundle of claims, and each claim is a separate bet. One claim may be solid — a real evaluation, on the product you are buying, with a named person behind it — while the claim next to it has no source at all. A checklist that scores the vendor as a whole hides that difference. This one does not.

The checklist below runs on each claim, one at a time, and ends with a claim log: a short record of what was checked, what the verdict was, and who accepted the risk. You need nothing from Oryx to use it. Work through it with the vendor’s proposal, demo materials, and draft contract in front of you, and keep the completed log with the contract file.

What counts as a claim

A claim is a checkable statement plus a source that should support it. Slogans are not claims until they point at evidence:

• Claim: “Our model scores 92% accuracy on the FinanceBench benchmark, evaluation report dated March 2026.” Checkable: the benchmark, the score, and the report can all be opened and verified.

• Claim: “We reduce document review time by 40% for a mid-size law firm, measured in a six-week deployment at [named client].” Checkable: the deployment, the measurement, and the firm type can all be asked about.

• Slogan, not a claim: “Enterprise-grade accuracy you can trust.” No benchmark, no measurement, no source. Until the vendor ties it to evidence, treat it as marketing, not as a basis for signing.

Only claims go through the checklist. Slogans get one question instead: “What evidence supports this, in writing?” If the answer is more slogans, the statement stays out of the decision.

The checklist: seven checks per claim

Run every check on every material claim. A claim that fails one check is not a pass; mark the failure and note what would fix it.

1. The claim is written down, exactly as made.

• ☐ The claim is recorded word-for-word, with where it appeared (proposal page, demo slide, email, sales call). A claim you cannot quote back to the vendor is a claim you cannot hold them to.

2. A source is supplied, in writing.

• ☐ The vendor has named the source: a document, dataset, evaluation, or reference you can open. “Our internal testing” with nothing attached is not a source. If no source is offered, stop here — the claim is unverified, and no further check can rescue it.

3. The source resolves.

• ☐ The link, DOI, or document is real and retrievable. Follow it yourself; do not take the vendor’s word for it. A dead link or a document that cannot be located is a failed citation, not a minor flaw. In Oryx’s published pilot audit of 66 AI-generated citations, 31 (47%) had broken links (Paper 2, study 1 — a pilot, not a vendor test and not a universal rate). The lesson is that resolvability fails often enough to check every time.

4. The source actually supports the claim.

• ☐ Read the relevant passage and mark the verdict plainly: supports, partially supports, or does not support. A real source saying something else is a failed check, not a pass. In the same pilot audit, 7 of the 66 citations (11%) mismatched the claim they were attached to, and only 5 both resolved and fully supported the claim (Paper 2, study 1 — a pilot, not a universal rate). Existence and support are two different checks, and only the second tells you whether the claim is true.

5. The evidence matches the version being sold.

• ☐ The version and date of the evidence match the product version in the contract. Evidence for an older model, an earlier dataset, or a pilot configuration does not support claims about the version you are buying. If the vendor re-ran the evaluation for the current version, ask for that run, not the older one.

6. Any number in the claim has a method behind it.

• ☐ A percentage, a time saving, or a score states what was measured, on what inputs, and by whom. A figure with no method is a slogan with a number on it. If the method is proprietary, the vendor should still say what was measured and on how many cases — without that, you cannot tell whether the number means anything for your use.

7. A named human accepts the residual risk.

• ☐ A named person on the vendor’s side stands behind the claim, and a named person on your side records acceptance or rejection. “The team” and “the system” are not names. If the claim later proves false, this is the person the remedy is owed to — and the remedy itself should be written before signing, not negotiated after.

A worked example: three claims, three verdicts

These are illustrative examples of how the log works, not a test of any real vendor.

#	Claim (as made)	Source supplied	Resolves?	Supports?	Version match?	Verdict
1	“92% accuracy on FinanceBench, March 2026 evaluation”	Evaluation report PDF	Yes	Supports — the report shows 92% on the stated benchmark	Yes, current version	Pass
2	“Cuts review time by 40% for mid-size firms”	A two-page case note, no client named, no dates	Yes	Partially supports — 40% appears, but for one unnamed firm over two weeks, not mid-size firms generally	Unclear — version not stated	Conditional — needs a dated, named deployment on the version being sold
3	“Trained only on licensed data”	No source offered; “trust our compliance team”	No source	Not checked	Not checked	Reject — unverified claim, no source offered

The point of the table is the discipline, not the outcomes. Claim 2 is not a failure of honesty — it is a claim with evidence that is too thin for the decision it is being asked to support. Claim 3 is the commonest failure of all: a claim that matters, with nothing behind it.

Copy the claim log

Copy this table and fill one row per material claim. Keep it with the contract file.

#	Claim (word-for-word)	Where it appeared	Source	Resolves?	Supports?	Version match?	Method stated?	Named acceptor	Verdict
1
2
3

One line per claim, one verdict per line. If the log is longer than a page, that is fine — a proposal with many claims deserves a log with many rows. If a claim cannot fill a row, it has not been checked.

What to do with the result

Score each claim, then decide on the bundle:

• Pass: all seven checks met, evidence on file. The claim can be relied on.

• Conditional: named gaps, each with a fix and a re-check date. A conditional is not a pass, and it does not become one by waiting. Do not sign on a conditional claim without the fix in writing.

• Reject: no source offered, the source does not resolve, or the source does not support the claim. A rejected material claim is a reason to hold signing until it is fixed or withdrawn — the rest of the proposal does not compensate for it.

Record the verdicts in one page, attach it to the contract file, and set a re-check date after signing: models and sources change, so a claim that passed on the demo can fail silently on the live product unless re-checking is built in.

Common traps

• A percentage with no method. “94% accurate” means nothing until you know what was measured, on what inputs, and by whom.

• A demo-only feature. Something shown in the demo that is not in the purchased version is not part of what you are buying, whatever the slide said.

• “AI-powered” as a claim. It describes a technique, not an outcome. Ask what the system does, measured how, on whose data.

• Evidence from the wrong version. A strong evaluation of last year’s model is not evidence about this year’s.

• A source that supports a different claim. The most common mismatch: a real, respectable document cited for a statement it never makes. Reading the passage is the only cure.

• A claim you cannot quote back. If it was said on a call and never written down, get it in writing before it enters the decision.

How this checklist differs from due diligence

Due diligence is wider: it covers data handling, exit terms, subcontractors, and the contract itself. This checklist is narrower on purpose — it is the claims half only, designed to be run claim by claim in a single evaluation meeting. Use both: run this checklist on the claims, and Oryx’s published due-diligence checklist on everything else. The claim log from this page becomes the evidence section of the due-diligence record.

Sources and further reading

• Oryx Working Paper v3, From Answers to Auditable Intelligence (Zenodo DOI 10.5281/zenodo.23718472) — the six-layer framework behind the claim, source, and human-gate checks above.

• Paper 2 v2, Trust, But Verify (Zenodo DOI 10.5281/zenodo.23731235) — the published pilot citation audit quoted in checks 3 and 4.

• Oryx’s published guide, “How to check an AI vendor’s claims before you sign” (oryxai.in) — the six-step claim check this checklist applies claim by claim.

• Oryx’s published “AI vendor due diligence checklist” (oryxai.in) — the wider due-diligence checklist this page complements.

• Oryx’s guide, “Auditable AI research: what makes research checkable” (oryxai.in) — the five properties that make any report, including a vendor’s evidence pack, checkable.

---

Oryx AI offers claim-level evidence audits for research drafts and vendor claims — ₹7,500 per pilot report, payable only on acceptance, currently invite-only beta (contact: amartya@oryxai.in).

Oryx Centre for Strategic Futures is a two-person research initiative working on auditable AI.