A procurement questionnaire arrives and one line on it says: provide your SOC 2 Type II report. It is a reasonable request, and for an AI system it answers a narrower question than the person asking intends. The gap is not a flaw in SOC 2. It is a scope boundary that the report states plainly, in the section that is easiest to skip.
A clean SOC 2 Type II report and a system that fabricates citations are entirely compatible outcomes. Nothing in the attestation looks at whether an answer is true.
What a SOC 2 report actually is
A SOC 2 engagement is, in the AICPA's own definition, an examination engagement reporting on the fairness of the presentation of management's description of the service organization's system, the suitability of the design of the controls included in that description, and, in a type 2 engagement, the operating effectiveness of those controls. The CPA performing it is the practitioner, and in an examination the practitioner expresses an opinion.
Two consequences follow immediately, and both are routinely lost in the summary a buyer receives. The deliverable is a report containing an opinion, and in a Type II the auditor's tests of controls and their results are described in it. There is no certificate, so a vendor that says it is SOC 2 certified is describing something that does not exist. And the opinion attaches to a scope: a Type I opines on whether controls are suitably designed as of a specified date, a Type II adds an opinion on whether they operated effectively throughout a specified period, and neither speaks to anything outside the system description the vendor wrote.
What it covers, and who decides
The engagement is built on the Trust Services Criteria. One set of them, the common criteria, applies across all five categories, and on their own the common criteria are the complete criteria for the security category. Availability, Processing Integrity, Confidentiality and Privacy each add further criteria of their own, and the practitioner may report on any of the five categories individually or in combination, which means the service organization chooses which categories the engagement covers. That selection, together with the fact that management writes the description of the system under examination, is why two documents both titled SOC 2 Type II can cover substantially different ground.
- Security. Protection against unauthorized access, unauthorized disclosure, and damage to systems. Covered by the common criteria, which apply whichever categories are selected, so this is the part a buyer asking for the report usually has in mind and it is genuinely covered.
- Availability. Whether information and systems are available for operation and use to meet the entity's objectives. Relevant if you depend on an endpoint being up, and silent on what it returns when it is.
- Processing Integrity. Whether system processing is complete, valid, accurate, timely and authorized to meet the entity's objectives. Note the last clause: the objectives are the entity's own. It is the nearest criterion to output quality and it still does not ask whether generated text is supported by evidence.
- Confidentiality. Whether information designated as confidential is protected to meet the entity's objectives. Worth having in scope if you are sending documents.
- Privacy. Collection, use, retention, disclosure and disposal of personal information, to meet the entity's objectives.
What it cannot tell you about a model
No Trust Services Criterion contains a requirement that a generated answer be supported by the sources the system retrieved. The nearest, under processing integrity, asks whether the entity delivers output completely, accurately and timely in accordance with specifications. Read that clause carefully, because it is the whole answer: the specifications are the vendor's own commitments and system requirements. The framework does not ignore accuracy. It takes the definition of accuracy from the vendor.
That is not an oversight. The criteria were written for service organizations handling data, and they do that job. It means the failure mode that makes an AI system different from the software a SOC 2 report was designed for is not something the engagement is built to reach. A vendor can hold a clean Type II across every category and still ship a system that invents a citation, answers confidently from a document it misread, or follows an instruction that arrived inside a retrieved file.
- Fabrication. Whether the system produces claims its sources do not support. No criterion names it. This is measurable, and measurable without an auditor: you need a held-out set, a definition of supported, and a published rate.
- Grounding. Whether each answer can point at the passage behind it. An architectural property of the system, and invisible to an examination that never reads an answer.
- Indirect prompt injection. Whether instructions arriving inside retrieved content can change behavior. No criterion names it. The nearest, which concerns the introduction of unauthorized or malicious software, is about code reaching the system, not about instructions arriving inside a document the retriever was supposed to ingest and did.
- Training on your data. Whether your documents reach a training run, at the vendor or at a model provider behind them. Confidentiality, where it is in scope, concerns protection of information designated confidential. The contractual question of training use is answered in a data processing agreement, not in an opinion.
- Model change control. Change management is in scope for the vendor's own system. A provider swapping the model underneath them, and what regression testing catches it, is a question to ask directly.
What to ask alongside it
The useful move is not to stop asking for SOC 2. It is to stop expecting it to answer questions it does not cover, and to ask those separately. Every question below has a checkable answer, which means a vendor who cannot answer it has told you something.
- What is your measured rate of unsupported answers, and on what test set?
- A rate without a test set is a number without a method. Ask for both, and ask whether the set is held out from development. A vendor who has never measured this will say so in the shape of their answer.
- Is that measurement wired into your release process, or reported after the fact?
- A dashboard competes for attention with everything else on it. A gate that fails a build stops a release. The difference is total, and it is the difference between a measurement and a control.
- Can every answer point to the passage it came from?
- Ask to see it on a real query, not in a diagram. If a citation is presented but generated rather than retrieved, you have the appearance of grounding and none of the property.
- What happens when a retrieved document contains an instruction?
- This separates a team that has thought about retrieval from one that has only built it. A good answer describes privilege separation between retrieved content and instructions, and limits on what the output is permitted to do.
- Do my documents reach a training run, at you or at anyone behind you?
- Get it in the data processing agreement rather than in a sales answer, and ask it about the subprocessors too, because a model provider behind your vendor may be the party with the training pipeline.
- Who is accountable when the system is confidently wrong, and what is the remedy?
- A contract question. It tells you whether the vendor has priced the failure mode or only the feature.
Where we stand, plainly
MetaMinds does not hold a SOC 2 report, of any type. We are not a CPA firm, so we cannot perform a SOC 2 examination or issue the attestation at all; only a CPA firm can. What we sell adjacent to this is readiness and evidence preparation, which the AICPA itself treats as a different kind of engagement: readiness work is performed under the consulting standards, where the practitioner develops findings and makes recommendations and does not express an opinion on the subject matter.
We publish that gap rather than leaving it to be discovered in diligence, and the reasoning is not modesty. A buyer who finds a vendor candid about one gap has a reason to believe the other answers. A vendor who implies an attestation it does not hold has put a claim in writing that diligence will test, and in the United States a firm's failure to possess a reasonable basis for an objective advertising claim is an unfair and deceptive practice under Section 5 of the FTC Act.
What we can show instead is the thing SOC 2 does not reach. On CourtNetra, our own legal retrieval platform over a corpus of 18,863,754 Indian judgments, an automated grounding evaluation runs weekly in continuous integration and fails the build when the ungrounded rate exceeds 2%. That threshold is ours, chosen for our corpus and our risk tolerance, and it is not a recommendation for anyone else. The mechanism is the point: the measurement is wired to a gate rather than to a report.
Ask for the SOC 2 report. Then ask the questions it does not cover, because those are the ones about the system you are actually buying.
Common questions
Does a SOC 2 report cover whether an AI model hallucinates?
No. A SOC 2 report is an examination engagement in which a CPA firm expresses an opinion on controls over a service organization's system, built on the Trust Services Criteria: the common criteria, plus additional criteria for availability, processing integrity, confidentiality and privacy. No criterion contains a requirement that a generated answer be supported by the sources the system retrieved. The nearest, under processing integrity, asks whether output is delivered completely, accurately and timely in accordance with specifications, and those specifications are the vendor's own. So a vendor can hold a clean SOC 2 Type II report and still operate a system that fabricates citations.
Is there such a thing as being SOC 2 certified?
No. A SOC 2 engagement produces a report containing a CPA firm's opinion, not a certificate, so no organization is SOC 2 certified. A vendor using that phrasing is describing something that does not exist, which is worth noticing. The correct phrasing is that an organization has a SOC 2 Type I or Type II report, for a stated scope and, in a Type II, a stated period.
What is the difference between SOC 2 Type I and Type II?
A SOC 2 Type I report gives an opinion on whether controls are suitably designed as of a specified date. A Type II additionally gives an opinion on whether those controls operated effectively throughout a specified period, and it includes a detailed description of the auditor's tests of controls and their results, which a Type I does not. Both are examinations by a CPA firm and both are bounded by the description of the system that management wrote, so a Type II is stronger evidence than a Type I but is still only evidence about what was in scope.
Why are two SOC 2 Type II reports not directly comparable?
Because the service organization chooses the scope. In a SOC 2 engagement the practitioner may report on any of the five trust services categories, individually or in combination, so the organization being examined selects which categories are covered, and it also writes the description of the system under examination. One set of criteria, the common criteria, applies across all five categories and on its own is the complete set for security; availability, processing integrity, confidentiality and privacy each add further criteria. Two documents both titled SOC 2 Type II can therefore cover substantially different ground, which is why the scope section and the period matter more than the opinion when comparing vendors.
What should I ask an AI vendor instead of, or alongside, SOC 2?
Ask questions with checkable answers about output behavior, since no Trust Services Criterion covers whether a generated answer is supported by its sources: the measured rate of unsupported answers and the test set it was measured on; whether that measurement blocks a release or is only reported; whether every answer can point to the passage it came from, shown on a real query; what happens when a retrieved document contains an instruction; whether your documents reach a training run at the vendor or at any subprocessor behind them; and who is accountable when the system is confidently wrong.
Does MetaMinds hold a SOC 2 report?
No. MetaMinds holds no SOC 2 report of any type, and states that openly rather than leaving it to be found in diligence. MetaMinds is not a CPA firm, so it cannot perform a SOC 2 examination or issue the attestation; the adjacent work it does sell is readiness and evidence preparation, which the AICPA treats as a consulting engagement in which no opinion is expressed, and which is named here as the different service it is.
Written by Aniruddh Atrey, founder and CTO of MetaMinds. Published 7 October 2026 and verified 7 October 2026. Framework statements here describe the scope of a SOC 2 engagement as set out in AICPA TSP section 100, the 2017 Trust Services Criteria (March 2020 update), read at source on 7 October 2026 and recorded in the claims register. They are not claims about MetaMinds, which holds no SOC 2 report of any type and is not a CPA firm. The grounding figures are from CourtNetra, a production system MetaMinds operates over a corpus of 18,863,754 Indian judgments: a weekly automated grounding evaluation in continuous integration that fails the build above 2% ungrounded. Nothing here states what buyers require, demand or gate a purchase on, because that claim was withdrawn from the claims register on 14 September 2026 for having no primary source.