RFC 9421: How to Cryptographically Verify an AI Agent

Every bot detection system built before 2024 rests, ultimately, on a string that anyone can type. The user agent header is self-declared, unauthenticated, and trivially forgeable — you can claim to be Googlebot with a single line of curl. The entire industry's response has been reverse DNS lookups, published IP ranges, and behavioural heuristics: an elaborate scaffolding erected around the fact that HTTP has no native way to prove who is calling.

RFC 9421 changes that. It gives HTTP a standard mechanism for signing a request so the recipient can verify cryptographically that it came from the holder of a particular key. For AI agents — which are about to make a very large number of requests on behalf of people who are not watching — this is the difference between a claim and a proof.

The problem in one line

User-Agent: Mozilla/5.0 (compatible; ExampleBot/1.0) is a claim. Nothing anywhere checks it.

Everything the industry does to compensate is inference on top of that gap. IP allowlists require the operator to publish ranges and keep them current, and they break the moment traffic moves. Reverse DNS verification works but only for operators who run it properly and costs a lookup on the request path. Behavioural heuristics are probabilistic and adversarial by nature.

None of these verify the request. They verify something near the request and hope the association holds.

What a signature actually covers

The core insight of RFC 9421 is that you do not sign the raw bytes of an HTTP message. You cannot: proxies legitimately reorder headers, normalise whitespace, add hops, and re-encode. A byte-exact signature would break in transit constantly.

Instead, the signer chooses a list of components to cover, and both parties construct a canonical signature base from them in an agreed way. Sign that.

Components come in two kinds:

• Header fields — ordinary headers by name: content-type, content-digest, authorization.
• Derived components — properties of the message itself, distinguished by an @ prefix: @method, @authority, @path, @target-uri, @scheme, @query.

Derived components are the important addition. @method and @target-uri mean the signature covers what is being requested and where, not just the headers alongside it — so a captured signature cannot be replayed against a different endpoint.

The two headers

A signed request carries both:

Signature-Input: sig1=("@authority" "@method" "@path" "signature-agent");\
  created=1735689600;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";\
  alg="ed25519";expires=1735693200;nonce="b3n5jxr2"
Signature: sig1=:MEUCIQ...base64...==:

Read Signature-Input as a recipe. Inside the parentheses is the ordered list of covered components. After the semicolons are the signature parameters:

• created and expires — a validity window, in seconds since the epoch. Together they bound replay.
• keyid — an identifier for the key needed to verify. This is what turns a signature into something you can actually check.
• alg — the algorithm. Ed25519 is the common choice for agent traffic: small keys, small signatures, fast verification.
• nonce — a unique value, so a verifier tracking recently-seen nonces can reject an exact replay inside the validity window.

The label (sig1 here) lets one message carry several signatures — an agent's own signature plus, say, a gateway's countersignature.

The bootstrap problem, and Web Bot Auth

RFC 9421 verifies a signature against a key. It says nothing about how you obtain that key.

For a partner integration this is fine — exchange keys out of band, done. For the open web it is the whole difficulty. An agent you have never heard of arrives at your server. Its signature references keyid=.... You have no relationship with its operator and no prior knowledge of the key.

Web Bot Auth is the IETF work addressing exactly this. Its architecture adds two things on top of RFC 9421:

A Signature-Agent header. The agent states where its keys can be found — a URL under a domain it controls. That domain becomes the identity being verified, which is the useful property: you are no longer verifying "a bot claiming to be ExampleBot", you are verifying "the holder of a key published at example.com". Domain control is something you can reason about.

A key directory. A well-known, fetchable, cacheable document listing the agent operator's public keys in a standard format. Verifiers fetch it once, cache it, and verify many requests against it.

The result is a verification chain with no prior relationship required: signature → key → directory → domain. Every step is checkable, and the last one is anchored in something that already has real-world accountability.

Cloudflare has been the main driver here and has shipped support at scale, which matters more than the specification text — a verification standard is worth precisely as much as its deployment.

What verification does and does not give you

This distinction gets muddled constantly, and getting it wrong produces bad security decisions.

The last two rows are the ones that matter operationally. A verified agent is one whose identity you know. That is enormously more than you had before, and it is not permission. You can verify an agent perfectly and still decide to rate-limit it, serve it something different, or refuse it — and now those decisions can be about a known party rather than a guess.

Implementation notes

If you are building a verifier, these are the parts that bite.

Canonicalisation is the whole game. Verification fails when the two sides build different base strings — from a proxy rewriting a covered header, a differing view of @authority behind a load balancer, or a mishandled structured-field parse. Log the reconstructed base on failure during development. You will need it.

Clock skew is real. created and expires are absolute timestamps. Machines disagree about the time. Allow a tolerance window measured in seconds; a zero-tolerance verifier will reject legitimate traffic and the failures will look random.

Cache the key directory aggressively, but bound it. Fetching a directory per request converts every incoming request into an outbound one — a fine denial-of-service amplifier if someone points traffic at you with a novel Signature-Agent. Cache by domain, respect cache headers, apply a floor, and rate-limit directory fetches for domains you have not seen before.

Verification is cheap; do not make it free. Ed25519 verification is fast, but "fast" is not "free" and it sits on the request path. Only attempt verification when a signature is actually present. Keep the unsigned path completely clear of the crypto.

Fail closed on verification, open on availability. If a signature is present and invalid, that is a real signal and should be treated as such. If your key directory fetch times out, that is your problem, not the agent's — do not convert your own outage into a wave of rejections.

Why this matters for attribution specifically

The connection to traffic measurement is direct.

The whole reason AI referral attribution is hard is that every available signal is weak or forgeable. The referrer is missing. The user agent is a claim. Client characteristics are statistical. You end up combining several unreliable signals and reporting a confidence score, because that is genuinely the best available.

A cryptographic signature is categorically different. It is the one signal in the entire stack that is not a guess. When an agent signs its request and the signature verifies against a key published under a domain you can name, you are not inferring anything.

Today that covers agents rather than the humans they act for, so it addresses the crawler side of the problem more than the referral side — and keeping those two apart is its own discipline. But the trajectory is clear. As more activity becomes genuinely agentic — software fetching pages, comparing options and completing transactions on someone's behalf — the fraction of traffic that can carry a proof rather than a claim only goes up.

The measurement stack of 2030 is likely to look like: verified where possible, inferred where not, with the boundary between them clearly marked. RFC 9421 is what makes the first half of that sentence possible.