Watchman audits an AI model on two layers. It verifies the model is what it claims to be and detects weight modifications hidden inside third-party and compressed releases. And it measures how the model behaves once you wire it to your data, including hostile data, before any of it reaches production.
Vendor checkpoints. Open-weight bases. Community fine-tunes. Quantized repacks. Every one of them is tens of gigabytes of opaque numbers, and regulators on three continents now expect you to prove what is inside before it reaches production.
A file hash only tells you the bytes changed, and under compression the bytes always change. That leaves you unable to say whether a release was actually modified or just repacked. Watchman answers the question the hash cannot: were the weights modified beyond their declared compression, how much, where in the network, and what kind of change was it.
One run produces the same set of artifacts every time. Each one answers a question your security, compliance, and engineering teams already have.
Watchman finds weight changes that go beyond a model's declared quantization, and names the kind of change: instruction tuning, alignment modification, or domain specialization. It works from the weight files and nothing else. No training data, no prompts, no inference access, no cooperation from the publisher.
Every audit produces an evidence-grade report and a CycloneDX-style AI-BOM attestation your security team, your compliance team, and your regulator can hold. It carries a SHA-256 chain of custody for every model file, the decision thresholds in force, and the stated limitations, all in one document.
When Watchman flags a modification, it shows where the change concentrates, by component and by network depth. You learn what was touched, rather than only that something happened. Each modification type leaves its own signature in the report.
Deterministic exit codes drop straight into your pipeline: 0 clean, 2 modification detected, 1 indeterminate. Gate your model registry on it, run it on-prem or air-gapped, and schedule re-audits for the continuous record federal frameworks now expect. Nothing leaves your environment.
Provenance tells you the model is what it claims. It does not tell you how the model behaves the moment you connect it to a retrieval system and hand it a document that is relevant but imperfect, or hostile. Watchman now measures that too.
A model can top every capability leaderboard and still get worse when you give it your own documents. One 31B model we audited lost 13 accuracy points from relevant retrieved context. One 120B model was talked into answering by a hostile instruction hidden in a retrieved document 87% of the time. Neither weakness shows up on a leaderboard, and both are deployment-layer risk.
Does giving the model retrieved documents make it worse than answering from memory? We measure the net effect, and how often a correct answer gets overridden by the context.
Can a hostile instruction hidden in a retrieved document hijack the model? We measure the hijack rate undefended, and what actually drives it down.
When the documents cannot answer the question, does the model admit it or invent an answer? We measure the guess rate on unanswerable questions.
When the answer is in the documents, does the model actually use it? We measure the accuracy gain on knowledge-heavy questions.
Does a strict governance instruction quietly make the model dumber on questions it should answer? We measure the accuracy it retains under governance.
Does the compressed variant you actually deploy still behave like its full-precision parent? We grade the variant you ship, not the one on the card.
Governments and supervisors are converging on one demand: prove what your model is, where it came from, and that nobody changed it on the way in. Here is how a Watchman audit maps to each.
| Framework | What it asks for | What Watchman hands you |
|---|---|---|
| EU AI Act Reg (EU) 2024/1689 · from 2 Aug 2026 |
Article 11 technical documentation that accounts for a model's identity, the bases it was built on, and its known limitations. | An audit report with a hash-pinned identity for the model and its base, the classification of any modification, and the limitations, filed in your technical documentation. |
| NDAA / DFARS FY2026 NDAA → DFARS / CMMC |
Verified model registries and integrity checks before deployment, covering model weights across the lifecycle. | A deterministic pre-deployment gate with exit codes and per-file SHA-256 weight identifiers your registry can pin. |
| OMB M-26-04 Federal continuous accountability |
A move from one-time documentation to continuous accountability, including for supply-chain modifications. | Scheduled re-audits that produce a dated, comparable series of reports, the continuous-accountability record on file. |
| CycloneDX AI-BOM Procurement provenance |
A model entry whose lineage and provenance can be verified, rather than self-declared by the supplier. | A CycloneDX-style attestation fragment with the per-file weights identifier, a verified edge to the claimed base, and the verdict, ready to merge into your bill of materials. |
| Banking MRM Interagency guidance · Apr 2026 |
Independent validation of third-party and vendor models, developed and monitored over time. | Reproducible validation evidence for the model-weights layer, what changed from the base, where, and of what kind, for the third-party validation file. |
We validated Watchman across five model families and publish the numbers as they came out. It is most reliable on the 7B-and-up open-weight models enterprises actually deploy.
Real modifications caught under leave-one-out validation.
Of detected modifications, put in the correct class. Alignment and domain-specialization classes were perfect.
In the validation library. Detection generalizes across families, validated held-out by family.
Every tested modification on models this size and up was detected. Borderline sub-2B cases route to review.
Tell us which framework you are preparing for, and we will audit a model of your choosing, walk through the evidence, and scope a deployment inside your environment.
Request an Audit