How Websites Detect AI Browser Agents in 2026, and Web Bot Auth

As of October 2026, websites detect AI browser agents in three different ways, one per kind of agent. Declared crawlers and user-triggered fetchers are caught by user-agent strings, published IP lists and, increasingly, cryptographic signatures. Agentic browsers that drive real Chromium are a harder case. They carry a normal Chrome fingerprint, so they're detected mostly by how they behave: a cursor that jumps instead of moving, text pasted instead of typed, scrolling in discrete jumps. Web Bot Auth is the opt-in fix. The agent signs its requests so a site can recognise it without guessing.

How websites detect AI browser agents: three kinds of agent, three methods

"AI bot" isn't one category. To detect AI browser agents you first have to work out which kind is in front of you, because each kind identifies itself differently and gets caught by a different signal.

1. Training and search crawlers. These are fetchers that collect pages in bulk with no user waiting on the result: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot. They declare a user-agent token, publish IP ranges and, by their vendors' own statements, honour robots.txt.
2. User-triggered fetchers. A person asks an assistant a question and it fetches a page right then: ChatGPT-User, Claude-User, Perplexity-User, Google-Agent. They still declare themselves, but several vendors say robots.txt may not apply, because a human started the request.
3. Agentic browsers. A real Chromium instance clicks, types and scrolls for the user, either on the user's own machine (Perplexity Comet, the Claude in Chrome extension, open-source browser-use) or in the vendor's cloud (ChatGPT agent, Manus). These look like browsers because they are browsers.

Cloudflare's taxonomy from July 1, 2026 splits the same traffic by behaviour: Search, Agent (acting "in real time, on a person's behalf"), and Training. Its Agent bucket explicitly includes both "chat fetch bots (e.g., ChatGPT-User) and browser-use agents (e.g., Gemini or Claude driving Chrome)". Group 2 and group 3 share a policy category, but detecting them takes completely different methods.

The declared agents: user-agent tokens and published IPs (verified October 2026)

For groups 1 and 2, detection is mostly a matter of checking documentation. Each major vendor publishes its tokens and an IP list you can check requests against. A UA match plus an IP match is reliable. A UA match alone is not, because anyone can send the string.

Declarations only work when vendors stick to them. In August 2025 Cloudflare reported that, after sites blocked Perplexity's declared bots, it saw undeclared requests with a generic Chrome-on-macOS user agent, coming from IPs outside Perplexity's published ranges and rotating ASNs. Cloudflare said it identified that traffic with "a combination of machine learning and network signals" and removed Perplexity from its verified-bot list. Perplexity disputed the account. Whoever was right, the lesson for site owners is that UA and IP rules catch honest traffic. They don't catch traffic that has decided not to identify itself.

Why agentic browsers get past UA and IP rules

An agentic browser running on a user's laptop sends that user's Chrome user agent, Chrome's TLS and HTTP/2 fingerprint, and the user's home IP. There's nothing in the request for a UA or IP rule to match. Cloud-hosted agents are only partly better: their IPs belong to cloud providers, but so do VPN users, corporate proxies and the automation tools you already allow.

The FP-Agent paper (Wang, Shafiq and Vekaria, arXiv 2605.01247, May 2026) measured this directly. It ran seven agents (OpenAI's Atlas browser and ChatGPT agent, Claude for Chrome, Perplexity Comet, Manus, Browser Use and Skyvern) through flight-booking, shopping and forum tasks next to human participants. In its Cloudflare case study, "only Manus is fully blocked by Cloudflare, whereas the other six browsing agents complete the tasks successfully." Browser fingerprints alone reached an F1 of 0.7969 at telling the classes apart. The authors found fingerprints "correlated with their underlying execution environment": local agents simply inherit the host machine's.

One correction for anyone working from 2025 material: OpenAI shut down the Atlas browser on August 9, 2026 and folded agent browsing into its desktop app. The paper's Atlas results still matter, because they describe a local-agent pattern that other products share.

Signals that still identify agentic browsers

For agentic browsers, behaviour gives them away far more than environment does. The FP-Agent classifier combined behavioural and browser features and reached an F1 of 1.0000 on the paper's dataset. That's a lab result on seven agents and three tasks, not a production detection rate, but the direction matches what vendors report.

Cursor geometry and input cadence

The cleanest signal in the paper: "browsing agents do not exhibit any mouse movements and instead directly teleport to the click location." Typing is just as distinctive. Atlas, ChatGPT agent and Comet pasted text, and ChatGPT agent's Ctrl+V gave "high hold latencies… with an average of 66.58ms". Browser Use, Skyvern and Manus did type keystrokes, with mean inter-key latencies of 5.31 ms, 9.52 ms and 1.39 ms, while humans showed "much higher inter-key… latencies, together with substantially greater variance". Scrolling showed up as "discrete jumps" instead of continuous wheel movement.

Cloudflare's Precursor (announced July 13, 2026 for Enterprise Bot Management, rolling out free until general availability later in the year) is a production version of the same idea. It injects client-side JavaScript that collects "pointer movement, keyboard activity, focus changes, and visibility" and accumulates them per session, so a bot can't reset its behavioural record by reloading the page or starting a new challenge. Its premise: "What remains difficult to replicate is consistent human behavior over time."

Automation and extension artefacts

Some agents leave traces in the page itself. CHEQ's February 2026 teardown of the Claude Chrome extension found that it drives the page through the chrome.debugger API, so its synthesised clicks count as trusted events. While active, though, it injects DOM elements with ids like claude-agent-stop-container and claude-agent-animation-styles, and the style element tends to stay after the agent stops. HUMAN Security's research on Atlas and Comet likewise reports DOM artefacts from Comet's internal extension, and notes that Comet "does not provide a per-request, verifiable identity signal".

Don't build a strategy on single artefacts. Castle documented how a widely used CDP-detection trick (a getter on Error.stack that fires when an automation client has enabled the Runtime domain) quietly stopped working after V8 changes in May 2025. A vendor can rename a DOM id in a single release. Treat artefacts as supporting evidence that a behavioural model weighs, never as the whole verdict.

Environment consistency and egress

Cloud-hosted agents give away more about where they run. The paper saw ChatGPT agent report "Linux x86_64 and MacIntel" platforms alternately. A captured request published by Simon Willison shows the same thing: a macOS Chrome user agent sent with Sec-Ch-Ua-Platform: "Linux". Manus showed a fixed "Linux x86_64" with "4GB of RAM and 6 CPU cores". A mismatch between the UA and client hints, plus a cloud ASN, isn't proof on its own, but together they're a strong prior. Our browser fingerprinting explainer covers how those consistency checks are built.

Web Bot Auth: signed agents instead of guessed ones

Web Bot Auth lets an agent prove who operates it. It signs each request with HTTP Message Signatures (RFC 9421) and publishes its public keys at a well-known URL, so a site can verify the operator cryptographically instead of inferring it from UA, IP or behaviour. The IETF work is still a draft. The webbotauth working group adopted its protocol document as draft-ietf-webbotauth-httpsig-protocol (revision -00 dated September 1, 2026, authors from Cloudflare and Google). Production deployments are already running ahead of the standard.

A real signed request, decoded

Here are the signature headers from the ChatGPT agent request Simon Willison captured in August 2025 (the nonce is truncated here):

Signature-Agent: "https://chatgpt.com"
Signature-Input: sig1=("@authority" "@method" "@path" "signature-agent");created=1754340838;keyid="otMqcjr17mGyruktGvJU8oojQTSMHlVm7uO-lrcqbdg";expires=1754344438;nonce="_8jbGwfL…";tag="web-bot-auth";alg="ed25519"
Signature: sig1=:…:

• Signature-Agent names who is vouching for the request. The verifier fetches https://chatgpt.com/.well-known/http-message-signatures-directory to get the keys.
• The covered components (@authority, @method, @path, signature-agent) bind the signature to this host, method and path. A signature captured on one site can't be replayed against another.
• expires − created = 3600: the signature is valid for one hour, and the nonce lets a verifier reject exact replays inside that window.
• tag="web-bot-auth" and alg="ed25519" match Cloudflare's requirements: Ed25519 keys, the web-bot-auth tag, and at least @authority and signature-agent covered. keyid is the key's JWK thumbprint.

The UA in that same request was ordinary desktop Chrome. Without verifying the signature, the request looks like any other browser.

Who actually signs today

We requested /.well-known/http-message-signatures-directory on the main agent vendors' domains on October 3, 2026. chatgpt.com served an Ed25519 key with "signature_agent":"https://chatgpt.com", under a different keyid from the 2025 capture, so the key has been rotated. agent.bot.goog served Ed25519 keys; Google's docs say it is "experimenting with the Web Bot Auth protocol, using the https://agent.bot.goog identity" for Google-Agent. browserbase.com served keys too (Browserbase was in Cloudflare's first signed-agents cohort in August 2025, alongside ChatGPT agent, Block's Goose and Anchor Browser). The same path returned 404 or an HTML page on claude.ai, claude.com, anthropic.com, perplexity.ai and browser-use.com. A missing file at that path doesn't prove a vendor never signs anything, but none of them publishes a directory there or documents signing on its bot pages.

On the verifier side, Cloudflare validates signatures at the edge. As of July 1, 2026 it has merged signed agents into Verified bots, with a new Direct vs Intermediary field that records whether one operator runs the bot or many end users drive it. DataDome's Agent Trust combines Web Bot Auth, official IP lists and reverse DNS with intent models. Both treat a signature as proof of identity, not of good intent.

Agent type → identification → detection → typical policy

What this means for site owners

Decide per category and per endpoint, not once for "AI". Something that's fine on a public article may not be fine on a login form.

• Verify, then decide. Check UA against IP lists for declared bots and signatures for signed agents. Unverified traffic claiming to be GPTBot is just traffic with a forged UA.
• Allow, block or charge. Cloudflare's controls work per category on all plans. Pay per crawl answers unpaid crawlers with HTTP 402, but it was still in closed beta at the time of writing.
• Protect actions, not pages. The real risk from local agentic browsers is what they do inside authenticated sessions. Put behavioural scoring and step-up checks on checkout, account changes and form submissions. Our Turnstile internals piece explains how challenge scoring fits in.
• Expect false positives in both directions. Accessibility tools also paste text and move focus without a mouse. "No mouse movement" alone shouldn't block a session.

What this means for engineers building agents

Identify honestly, sign if you can, and use proxies for geography and load distribution, not to look like a human. Detection is moving towards "verify the signed ones, score the rest on behaviour". Disguising an agent keeps getting more expensive, and staying honest keeps getting cheaper.

• Declare yourself. Use a descriptive UA token with a contact URL for server-side fetchers. If you operate an agent service, publish a key directory and sign requests. Cloudflare's docs and open-source web-bot-auth libraries cover the Ed25519 setup.
• Respect robots.txt and per-site agent rules for anything that isn't directly triggered by a user, and keep request rates reasonable. Both are conditions for Cloudflare verification.
• Use proxies for the right reasons. Residential exits give correct country-level results and keep a fleet of agents from piling onto one cloud IP. They don't make an agent invisible to behavioural models, and using them to get around a site that has blocked agents is the behaviour Cloudflare cited when it removed Perplexity from its verified list. Our practical guide to proxy setup for AI agents covers configuration. Best Proxies for AI Agents compares IP types for these workloads.
• Stop at challenges. When an agent hits a CAPTCHA or a 403 from a bot-management vendor, the site has made a decision. Hand that to a human instead of retrying.

What Web Bot Auth does not solve

A signature proves who operates the agent, not what the user asked it to do. As Castle's walkthrough of Operator signatures puts it, verification confirms origin but "doesn't prove user intent". A signed agent can still be pointed at credential stuffing, scalping or a prompt-injected purchase. That's why Cloudflare's Intermediary label and DataDome's trust scores still layer behaviour on top of identity. Signing also does nothing for the hardest case, local agentic browsers, which currently send nothing to verify at all. Until those products sign, or sites rely on cursor geometry and timing alone, detecting them stays a matter of probability, not proof.