SOC 2 for AI vendors: what the report covers, and what it cannot tell you about a model
Asking an AI vendor for their SOC 2 report is a reasonable question that answers a different one. The report tells you how they run their infrastructure. It is silent on whether the system makes things up, which may well be the risk that brought you to the question.
What is in it
- A SOC 2 report is an examination engagement producing a CPA firm's opinion, not a certificate, so no organization is SOC 2 certified.
- The service organization selects which trust services categories the engagement covers and writes the description of the system examined, so two Type II reports are not comparable until you have read both scopes.
- No Trust Services Criterion names whether a generated answer is supported by its sources, and the nearest one measures accuracy against the vendor's own specifications, so a clean report and a fabricating model can coexist.
- The questions that do cover model behavior have checkable answers, and they are answerable today rather than at the end of an audit period.
- MetaMinds holds no SOC 2 report. That is stated here for the same reason it is stated on the site: a gap you volunteer is what makes the rest believable.
Topics
- SOC 2
- AI assurance
- Vendor assessment
- Procurement