docs

LLM API verification observations: why sample size matters (November 2026)

A November 4, 2026 snapshot of ModelVerify's retained public observations contains 7,970 diagnostic samples across 2,590 endpoint-hostname groups. Of those groups, 1,365 have only one diagnostic sample. Sample depth matters when reading a provider's displayed match rate.

← Back to docs

What was observed, and when?

This descriptive snapshot was generated on November 4, 2026 at 02:50:57 UTC from ModelVerify's public provider aggregates. It contains 7,970 retained diagnostic observations across 2,590 endpoint-hostname groups. Diagnostic observations follow the public aggregate's definition: retained samples with an identity verdict and a model label, rather than every API request.

These observations are self-selected. Hostname groups are not independently verified businesses or independent users; one operator can use multiple hostnames. There is no comparable earlier snapshot in this brief from which to establish a trend.

How many diagnostic samples were available per hostname?

The four groups below partition all 2,590 hostname groups by the number of retained diagnostic observations. Separately, 890 hostname groups had a diagnostic observation within the API's last-30-days window at collection time. Neither count measures market share or traffic.

  • No diagnostic observations: 6 hostname groups.
  • One diagnostic observation: 1,365 hostname groups.
  • Two to nine diagnostic observations: 1,097 hostname groups.
  • Ten or more diagnostic observations: 122 hostname groups.

Why is a 100% match rate not enough to compare providers?

One matched observation can produce a displayed 100% match rate, while offering much less evidence than multiple observations across relevant model labels and dates. Even ten observations are not a reliability threshold: requests may share an account, route or configuration, and repeated runs need not be independent.

Compare sample depth, model labels, timestamps and verdicts before considering an aggregate rate. A provider page's recent-model breakdown covers only its returned recent entries; it must not be read as the complete historical distribution.

What can this snapshot tell an API buyer?

It shows why evidence depth deserves attention when using this particular public dataset. It cannot estimate how often the wider market substitutes models, establish that a provider is fraudulent, or prove that future requests will use the same backend. We do not publish a market-wide authenticity percentage from these samples.

For your own endpoint, keep the requested model ID, selected reference, date and non-secret configuration together. Use behavioral evidence alongside representative workload evaluation. A connection error is not a mismatch, and missing or stale data is not a normal result.