Watchman · The GRC Layer

Know what your AI can see and what it runs

Watchman is the GRC layer of the Bounded AI Architecture (BAA), unifying governance, risk and compliance into a single, complete audit log. It records every permission, freeze, hand-back and deletion on every Data Module. It verifies that the model bound on top is what it claims to be, certifies a compressed build against five measured gates that can and do fail, and measures how the model behaves once it is wired to your data, including hostile data. One log, from the module to the weights.

Request an Audit See a Sample Report

What Watchman Governs

Three layers, one log

Bounded AI has three things worth governing: the modules that hold the knowledge, the bind that gives a model access to them, and the model itself. Watchman covers all three and writes to one log.

The modules
Every Data Module is permissioned on its own. Permission it, freeze it, hand it back or delete it, and each operation is a single action with a log entry you can show an auditor.
Learn more
The bind
A reasoning model is bound to the plane and sees only what its modules allow. Every model bound to the plane is certified before it goes live and re-certified when it changes, so a swap never slips past the gate.
The model
Weight provenance, five measured certificates, and six deployment-risk axes. Watchman verifies the model is what it claims to be and measures how it behaves once it is wired to your data.

The Problem

Vendor checkpoints. Open-weight bases. Community fine-tunes. Quantized repacks. Every one of them is tens of gigabytes of opaque numbers, and regulators on three continents now expect you to prove what is inside before it reaches production.

You deploy models you didn’t train

A file hash only tells you the bytes changed, and under compression the bytes always change. That leaves you unable to say whether a release was actually modified or just repacked. Watchman answers the question the hash cannot: were the weights modified beyond their declared compression, how much, where in the network, and what kind of change was it.

What You Get

Four things out of every audit

One run produces the same set of artifacts every time. Each one answers a question your security, compliance, and engineering teams already have.

Detect & Classify
Watchman finds weight changes that go beyond a model's declared quantization, and names the kind of change: instruction tuning, alignment modification, or domain specialization. It works from the weight files and nothing else. No training data, no prompts, no inference access, no cooperation from the publisher.
Regulatory Evidence
Every audit produces an evidence-grade report and a CycloneDX-style AI-BOM attestation your security team, your compliance team, and your regulator can hold. It carries a SHA-256 chain of custody for every model file, the decision thresholds in force, and the stated limitations, all in one document.
Localize the Change
When Watchman flags a modification, it shows where the change concentrates, by component and by network depth. You learn what was touched, rather than only that something happened. Each modification type leaves its own signature in the report.
CI-Native Gate
Deterministic exit codes drop straight into your pipeline: 0 clean, 2 modification detected, 1 indeterminate. Gate your model registry on it, run it on-prem or air-gapped, and schedule re-audits for the continuous record federal frameworks now expect. Nothing leaves your environment.

Certification

Five certificates, or it doesn’t ship

Provenance tells you the weights were not tampered with. It does not tell you whether a compressed model still works. Everyone publishing quantized models reports how far the weights moved, and that number does not predict capability. Our build of one 27B model carries four times the weight-space error of a well-regarded third-party build and is level with it on every behavioural gate. So Watchman measures the behaviour instead. Three of the five are hard gates: fail one and the bundle does not certify.

Reasoning · hard gate
Graduate-level multiple choice and grade-school maths under a harness that will not mistake a badly formatted answer for a wrong one. Format compliance is scored as its own metric, because compression damages instruction-following before it damages knowledge.
Faithfulness · hard gate
Given a document that contradicts what the model believes, does it defer to the document? Given one that supports it, does it use it? A model that argues with your source material is useless in a retrieval system, however clever it is.
Agent safety · hard gate
A data-free screen for one specific failure: models that pass every quality benchmark and still invent procedure steps when driving tools. It reads the weights directly, needs no test data, and it is the check a benchmark-led process misses.
Knowledge leak · reported
How much the model answers from memory with no document in front of it. Counter-intuitively we want this low: parametric knowledge cannot be traced, updated or erased, so in a retrieval system it is a liability rather than a feature.
Size · reported
Bytes on disk and the exact quantization configuration that produced them. Unglamorous, and the first thing anyone reproducing your result actually needs.

Evidence

It has said no

A test that has never failed anything is not a test. We ran the bundle across a public vendor’s entire quantization ladder and across the reference implementation’s own mix, and we publish the verdicts including the ones that do not flatter us.

Flagged
A 1-bit community build tripped the agent-safety gate outright. It loads, it generates fluent text, and it is not something we would put behind a tool call.
Failed
The reference implementation’s own 3-bit mix failed the reasoning gate on format compliance rather than on knowledge. Our build at the same file size passed, by two questions out of sixty, which is why we report the margin rather than the headline.
Certified
The vendor’s 4-bit build passed everything, and so did ours. Publishing that is the point: a certificate that only ever flatters the company running it is worth nothing to the person relying on it.
On anyone’s build
Ten builds graded so far across fourteen quantization formats: our own models and third-party files alike. The weight-side checks read the file directly; the behavioural ones need only an endpoint serving it. It does not have to be a model we made.

Deployment-Risk Auditing

A clean model can still be a liability

Provenance tells you the model is what it claims. It does not tell you how the model behaves the moment you connect it to a retrieval system and hand it a document that is relevant but imperfect, or hostile. Watchman now measures that too.

A model can top every capability leaderboard and still get worse when you give it your own documents. One 31B model we audited lost 13 accuracy points from relevant retrieved context. One 120B model was talked into answering by a hostile instruction hidden in a retrieved document 87% of the time. Neither weakness shows up on a leaderboard, and both are deployment-layer risk.

Axis 01

Context contamination

Does giving the model retrieved documents make it worse than answering from memory? We measure the net effect, and how often a correct answer gets overridden by the context.

Axis 02

Injection resistance

Can a hostile instruction hidden in a retrieved document hijack the model? We measure the hijack rate undefended, and what actually drives it down.

Axis 03

Abstention discipline

When the documents cannot answer the question, does the model admit it or invent an answer? We measure the guess rate on unanswerable questions.

Axis 04

Retrieval uplift

When the answer is in the documents, does the model actually use it? We measure the accuracy gain on knowledge-heavy questions.

Axis 05

Governance cost

Does a strict governance instruction quietly make the model dumber on questions it should answer? We measure the accuracy it retains under governance.

Axis 06

Quantization robustness

Does the compressed variant you actually deploy still behave like its full-precision parent? We grade the variant you ship, not the one on the card.

Each axis returns a number with a sample size and a confidence interval, mapped to the NIST AI RMF, ISO/IEC 42001, and the EU AI Act accuracy and adversarial-testing requirements. Early scores are provisional and marked as such while sample sizes grow. Every open-weight model we score on these axes is published on our live public index.

See the Deployment Risk Index at hell.ai

Compliance Mapping

One report, every framework

Governments and supervisors are converging on one demand: prove what your model is, where it came from, and that nobody changed it on the way in. Here is how a Watchman audit maps to each.

Framework	What it asks for	What Watchman hands you
EU AI Act Reg (EU) 2024/1689 · from 2 Aug 2026	Article 11 technical documentation that accounts for a model's identity, the bases it was built on, and its known limitations.	An audit report with a hash-pinned identity for the model and its base, the classification of any modification, and the limitations, filed in your technical documentation.
NDAA / DFARS FY2026 NDAA → DFARS / CMMC	Verified model registries and integrity checks before deployment, covering model weights across the lifecycle.	A deterministic pre-deployment gate with exit codes and per-file SHA-256 weight identifiers your registry can pin.
OMB M-26-04 Federal continuous accountability	A move from one-time documentation to continuous accountability, including for supply-chain modifications.	Scheduled re-audits that produce a dated, comparable series of reports, the continuous-accountability record on file.
CycloneDX AI-BOM Procurement provenance	A model entry whose lineage and provenance can be verified, rather than self-declared by the supplier.	A CycloneDX-style attestation fragment with the per-file weights identifier, a verified edge to the claimed base, and the verdict, ready to merge into your bill of materials.
Banking MRM Interagency guidance · Apr 2026	Independent validation of third-party and vendor models, developed and monitored over time.	Reproducible validation evidence for the model-weights layer, what changed from the base, where, and of what kind, for the third-party validation file.

Watchman produces the audit evidence these obligations call for. It does not by itself make you compliant. Compliance is a function of your whole program. What Watchman gives you is the defensible, machine-verifiable artifact each framework expects for the model-weights layer of that program.

Validated, Stated Plainly

Every report states its own limits

We validated Watchman across five model families and publish the numbers as they came out. It is most reliable on the 7B-and-up open-weight models enterprises actually deploy.

Held-Out Detection: 18/20; Real modifications caught under leave-one-out validation.
Classified Right: 92%; Of detected modifications, put in the correct class. Alignment and domain-specialization classes were perfect.
Model Families: 5; In the validation library. Detection generalizes across families, validated held-out by family.
Full Detection: 3B+; Every tested modification on models this size and up was detected. Borderline sub-2B cases route to review.

Every Watchman report states its limitations next to its verdict: the exact decision thresholds in force, the validation record behind them, and the one miss mode we know about, which is very broad, low-intensity tuning of very small models. The reference library grows with every labelled audit, and accuracy compounds with it. We publish what the tool can and cannot tell you, because an audit you cannot interrogate is not evidence.

Verify before you deploy

Tell us which framework you are preparing for, and we will audit a model of your choosing, walk through the evidence, and scope a deployment inside your environment.

Request an Audit

Know what your AI can see and what it runs

Three layers, one log

The modules

The bind

The model

You deploy models you didn’t train

Four things out of every audit

Detect & Classify

Regulatory Evidence

Localize the Change

CI-Native Gate

Five certificates, or it doesn’t ship

Reasoning · hard gate

Faithfulness · hard gate

Agent safety · hard gate

Knowledge leak · reported

Size · reported