Cloudflare AI Bot Defaults and Pay Per Crawl: What Changed in 2026

Cloudflare's AI bot defaults, as of October 2026, mostly hit crawlers that say who they are. A new domain whose owner says it runs ads now starts with AI training refused and AI agents blocked on ad pages, while search crawlers stay allowed. Price monitors, SEO audits and other "Data Collection" traffic fall outside those three presets and are judged by the same bot rules as before. The change that matters for a data team is not a new wall. It is a stronger reason to be a bot that identifies itself, with a payment lane (Pay Per Crawl, and since September 30 Pay Per Use) for content that used to be closed off entirely.

A lot of the coverage you will find is already out of date. Cloudflare announced one plan on July 1, 2026 and shipped a revised one on September 15. The timeline below follows what Cloudflare's own blog and docs say was actually shipped.

What Cloudflare's AI bot defaults are as of October 2026

Short version: Search, Agent and Training are now three separate switches on every Cloudflare plan, Free included, and each can be set to Allow, Block on pages with ads, or Block. Training also has a fourth option, Disallow AI Training. New domains get a recommended preset depending on whether the owner monetises with ads. Existing domains kept the effect of their old settings.

The July plan versus what shipped on September 15

The July announcement said multi-purpose crawlers "such as Googlebot, Applebot, and BingBot will be blocked" wherever a customer blocks Training. That would have taken sites out of search. The September 15 post replaced it. Under Disallow AI Training, Cloudflare writes a no-training preference into robots.txt. Accountable mixed-use crawlers (Applebot, Bingbot, Googlebot) stay allowed for search, and "every other training crawler is blocked, including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI." Only an explicit Block or Block on pages with ads now stops Googlebot, and with it search.

The presets Cloudflare now offers when someone onboards a new domain:

For existing domains, Cloudflare's migration notes say a domain that never turned on the old "Block AI bots" toggle maps to Allow on all three. Domains that had it on move to Disallow AI Training. TechCrunch's July 1 report said the new defaults would also reach "all existing free customers". Cloudflare's own September 15 migration tables don't show that, so check your target sites rather than assuming either.

Search, Agent, Training: how a request gets classified

Cloudflare classifies bots, not requests in isolation. A known bot gets one or more behaviour labels in BotBase, Cloudflare's bot directory. Each preset then blocks "Verified bots classified with that behavior, plus additional unverified bots that fall under these classifications," per the Block AI Bots docs.

• Search: crawling to build a search index or RAG database that answers questions later.
• Agent: user-directed, usually real-time: chat fetchers such as ChatGPT-User, or browser-use agents driving Chrome.
• Training: crawling to train or fine-tune a model.

The verified bots page lists eight more behaviours that are not among the three AI presets. They are Transact, Data Collection ("price scraping, competitive intelligence gathering, and third-party analytics"), Security Testing, SEO, Ads Verification, Social/Link Preview, Feed Fetching, and Monitoring & Operations. A crawler with several purposes is now tracked under all of them, and the most restrictive matching rule wins.

Identity comes from one of three places:

1. Verified bot or agent. The operator applied, and Cloudflare checks identity deterministically. That means a Web Bot Auth signature (Ed25519 keys published at /.well-known/http-message-signatures-directory, per IETF drafts), a published IP list with a stable user agent, or reverse DNS. Since July 1, 2026, signed agents count as Verified. They are labelled Direct (one operator) or Intermediary (a platform many end users drive).
2. Known but unverified. The traffic matches a pattern Cloudflare attributes to a crawler category, for example the heuristics it added after the Perplexity investigation.
3. Everything else. Generic bot detection decides: bot score, fingerprints, challenges. That layer existed before any of this. Our Turnstile internals post covers how the challenge side scores a session.

Being Verified no longer means being let in. Cloudflare redefined it on July 1, 2026: a Verified bot is "allowable" within whatever category the site owner permits. Honest identification gets you classified correctly. Getting in is still the site owner's call.

What "blocked by default" means for an actual request

"Blocked" here means up to three separate mechanisms. Which one you hit decides what your crawler sees.

1. A preference in robots.txt

Bot Preference Sync (the successor to managed robots.txt) prepends rules generated from the zone's settings to any existing robots.txt, so your own Disallow lines survive. The managed file in Cloudflare's docs example includes a Content Signals line and per-crawler disallows:

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The preamble says the signals are "express reservations of rights" under Article 4 of the EU DSM Copyright Directive (2019/790). That has legal weight for text-and-data-mining in the EU, even though robots.txt itself is voluntary.

2. An edge block

If the zone blocks your category, the request is refused at Cloudflare's edge before it reaches the origin. Pay Per Crawl documents this case as crawler-error: CrawlerForbidden with HTTP 403. With Block on pages with ads, the same crawler can get a 200 on one URL and a block on another URL of the same domain. Cloudflare detects ad-serving pages automatically and says the list "is too large and changes too frequently to enumerate in robots.txt." From outside you can't predict which pages are covered.

3. A price: HTTP 402 and Pay Per Crawl

On zones using Pay Per Crawl (still a closed beta as of October 2026), a verified crawler without payment intent gets a 402 that carries the price:

HTTP/2 402
crawler-price: USD 0.01

According to Cloudflare's crawl-pages docs, the crawler retries with exactly one of crawler-exact-price (must match crawler-price) or crawler-max-price. The payment header must be in the Web Bot Auth signature-input components. A paid 200 response carries crawler-charged: USD 0.01. Failures come back with a crawler-error header. Codes include MissingCrawlerPrice (402), InvalidCrawlerMaxPrice (402), ConflictingPriceHeaders (400) and StrongAuthRequired (400). Only verified, signing crawlers can pay. An anonymous scraper has no payment lane at all. A WAF or Bot Management block also overrides the "charge" action.

If you run a crawler, the useful habit is to treat 402 as data, not as a failure to retry around. A minimal pattern with requests:

import requests

def fetch(url, session: requests.Session):
    r = session.get(url, timeout=30)
    if r.status_code == 402 and "crawler-price" in r.headers:
        # Priced content: record it for a licensing/budget decision.
        return {"url": url, "status": 402, "price": r.headers["crawler-price"]}
    if r.status_code == 403 and r.headers.get("crawler-error") == "CrawlerForbidden":
        # The owner refused this crawler. Drop the URL; don't rotate around it.
        return {"url": url, "status": 403, "refused": True}
    r.raise_for_status()
    return {"url": url, "status": r.status_code, "body": r.text}

Pay Per Use, launched in beta on September 30, works differently. It doesn't happen at request time. A buyer defines a "use" (an excerpt returned in AI search, say), publishers accept or decline the offer, and the buyer self-reports each use to a Cloudflare API, which bills monthly. The first named partners, per TechCrunch, were Ceramic.ai and You.com.

Who is affected, and who isn't

The new defaults change the most for crawlers that identify themselves as AI, and the least for traffic that was never in an AI category. Evasion doesn't count as an exemption.

A self-identified training crawler (GPTBot, ClaudeBot, CCBot, Bytespider and so on) is the clearest case. On a new ad-supported domain it is refused by default. On many other zones it is blocked or priced. Its options are honest ones: honour the refusal, pay where Pay Per Crawl or Pay Per Use is offered, or license directly.

An AI agent product fetching pages for a user is the new target. Agents are blocked on ad pages by default on new ad-supported domains. There is no "Disallow" variant for agents yet, because no robots.txt-style standard for agents exists. Cloudflare says it will revisit this as proposals such as ai-prefs mature. Signing requests (as a signed agent) at least gets you classified as an agent and not as an anonymous bot. That matters on every zone that allows agents.

A scraper with a normal browser user agent isn't covered by "AI preset" logic as long as Cloudflare doesn't attribute it to an AI category. That doesn't make it safe. It still faces bot scoring, JavaScript challenges and rate limits. If it is in fact feeding a training set or an answer engine, it is doing what got Perplexity de-listed in 2025. Cloudflare explicitly blocks "additional unverified bots" that fall under each category. We won't help anyone disguise an AI crawler as a human visitor. Beyond the ethics, it is a losing game once the network starts matching your behaviour.

An in-house price monitor belongs to Data Collection, which isn't one of the three presets. The September changes alone don't change its access. What decides it is whatever the site already runs: Bot Fight Mode, Bot Management rules, rate limits. Our white-hat guide to Cloudflare blocks walks through those error codes.

Decision table for data teams

Start from what you are collecting and why. That decides your category, and the category decides your options.

Two notes on the price-monitoring row. First, "geo-appropriate egress" means seeing the price a local shopper sees. It doesn't mean hiding what you are. With ProxyHat that is a username flag, e.g. http://USERNAME-country-DE:PASSWORD@gate.proxyhat.com:8080. Our price-monitoring guide covers the pipeline around it. Second, request volume is still the fastest way to get flagged. Our rate-limit explainer covers backoff that sites tolerate.

Why ad-supported publishers asked for this

Ads only pay when a person sees the page. In Cloudflare's words, "Training replaces that visit with an answer; agents fetch the page with nobody there to see the ads." That is why the ad-page presets are stricter. Cloudflare's September 15 post puts numbers on the split: under 1% of its sites block Search bots, while 17% enable some mechanism to block training. Publishers want to stay findable but not supply content to models for free, and before September they couldn't block Googlebot's training use without losing Google search. Disallow AI Training and the Accountable commitments are Cloudflare's answer to that bind. Pay Per Use is the attempt to turn "no" into "yes, for a price."

What this does not change

• Sites not on Cloudflare. Akamai, DataDome, HUMAN and in-house stacks have their own rules.
• Generic bot defence. Bot scores, Turnstile, rate limiting and WAF rules work as before. A Data Collection crawler can still be blocked by them.
• Bing training opt-out via robots.txt. Microsoft is building no-training support in robots.txt with a target of early 2027. Until then, Disallow AI Training doesn't convey the preference to Bing that way.
• Predictability. Ad detection is undocumented, and owners can change presets at any time. Read each target's robots.txt and response headers; don't rely on blog summaries (this one included).
• Your legal position. Category labels don't decide copyright, contract or personal-data questions. Content Signals' Article 4 reservation applies in the EU whatever your bot score is.

A short checklist before your next crawl

1. Write down the purpose of each crawler. If one bot does search and training, split it. Cloudflare now tracks every purpose, and the most restrictive rule wins.
2. Use an honest, stable user agent with a contact URL. It costs nothing and is a precondition for verification.
3. Read robots.txt including Content-signal. Parse ai-train, search, ai-input and use, not just Disallow.
4. Handle 402 and crawler-error explicitly. Log prices; send refusals to a do-not-fetch list.
5. Consider Web Bot Auth if you run an agent or crawler at scale. It is required for Pay Per Crawl and is the route to Verified status.
6. Prefer licensed sources (APIs, data partnerships, Pay Per Use) for content you need regularly from publishers who refuse free crawling.

The 2026 changes push one way: crawlers that say who they are and what they do have more legitimate options than they did a year ago, and disguised crawlers have fewer. If your team is choosing infrastructure for agent or LLM data work under those rules, our guide to proxies for AI agents covers where proxies help and where they don't.