Articles Article 010

AI answers broke search’s deal with the web. Nothing has replaced it

Where Google shows an AI answer, sites get about 40% fewer clicks while ad clicks hold. Who loses, what still works, and what pays.

Published
Nov 18, 2026
Length
15 min read · 3270 words
Diagram titled 'AI answers keep the crawl and drop the click'. Two rows of boxes joined by arrows. Before: Googlebot crawls, ranked link, user clicks, site earns. Now: same crawl, AI answer on top, user stops there, then a broken green arrow to a greyed-out box, site earns less, labelled '73% no-click with an AI answer, 54% with plain links'. A grey note reads 'Clicks on Google’s ads: unchanged in both'.
Contents
  1. 1AI answers keep the crawl and drop the click
  2. 2Where an AI answer appears, about 40% of clicks disappear
  3. 3The loss falls on sites whose product is the answer
  4. 4The clicks are going; whether that’s a loss is the argument
  5. 5Optimizing for AI citations is mostly SEO
  6. 6No payment scheme yet replaces referral revenue for independent sites
  7. 7Plan for search as a scarce channel
In short
  1. A randomized experiment on US desktop users (January–February 2026) found that an AI Overview cut clicks to outside websites by about 40% on the searches where it appeared. Ad clicks did not change, and users reported no better search experience. Google said in 2025 that total clicks were stable but published no figures.
  2. The loss falls on sites whose product is the answer: guides, explainers, generic news, Wikipedia. Sites with loyal readers hold up better. Optimizing for AI citations is mostly ordinary search engine optimization (SEO), and the one large study of llms.txt found no correlation with citations.
  3. Independent publishers still lack a proven, broadly available replacement for referral revenue. Cloudflare’s payment betas, Really Simple Licensing (RSL) and Google’s contribution pilot have published no payout figures as of November 2026, and licensing deals have gone mainly to large publishers that block AI crawlers first. Treat search as a scarce channel: build direct readers, keep SEO basics, and decide on purpose whether to use Google’s Search Console opt-out from its AI answers, available worldwide since 31 August 2026.

Google now puts an AI-written answer above the links on a growing share of searches, and in May 2026 it told advertisers that “Google Search is AI Search, through and through.” The cost to websites is no longer a guess: a randomized experiment found that where an AI Overview appears, clicks to outside sites fall by about 40%, while clicks on ads don’t move. The old deal, in which sites let Google crawl them in exchange for visitors, is breaking, and independent publishers still have no proven, broadly available replacement for the referral revenue it brought.

AI answers keep the crawl and drop the click

For two decades, a site let Googlebot crawl its pages, Google ranked them, people clicked, and the site earned money from the visit through ads, affiliate links or subscriptions. Cloudflare says the old search crawlers scanned a page “a couple of times, or less, for each visitor sent”. Crawling was the price; the visit was the payment.

Now the same crawl also feeds an answer: one Googlebot serves both the classic results and the AI features. When Google decides an answer adds something, the model runs a query fan-out, “issuing multiple related searches across subtopics and data sources”, and writes a summary with source links on top. Follow-up questions then flow into AI Mode, Google’s chat-style search.

Then most people stop. In the experiment below, 73% of searches with an AI Overview ended without a click to any website, against 54% when the overview was hidden, and the links inside the overview drew about 8% of all clicks.

The diagram sets the two flows side by side. The green arrow is the step AI answers cut; the ad clicks in the middle don’t change.

Search used to pay for crawling with clicks
One crawl, two outcomes
Source: Agarwal and Sen’s randomized experiment, US desktop, January–February 2026 (no-click shares, ad clicks); Google Search Central (one crawl, query fan-out). The dashed loop is the predicted feedback, not a measured one.

Where an AI answer appears, about 40% of clicks disappear

The experiment

Saharsh Agarwal (Indian School of Business) and Ananya Sen (Carnegie Mellon) ran a randomized controlled trial (RCT) of AI Overviews on real searches, which they call the first causal evidence of the effect. Using a Chrome extension, they randomly split 1,065 US desktop users into a normal-Google group, a group whose overviews were hidden in real time, and an AI Mode group, each observed for two weeks between 7 January and 10 February 2026. On searches that would show an overview, users who saw it made 0.37 clicks to outside sites per search, against 0.62 for those who didn’t: 39.8% fewer1. Clicks on ads and the number of searches did not change.

Testing the benefits Google claims, the authors found that clicks were no different in bounce rate or time on site, and that users reported no measurable gain in satisfaction, perceived quality or ease of finding information. The effect was concentrated where the overview took the top slot and in informational queries, 71% of all searches; other query types were too few to judge.

It is a working paper, revised in June 2026, not peer reviewed, and desktop only. A second preregistered experiment, by Wang and colleagues with 1,100 participants (August 2026), reports the same direction: removing AI Overviews and AI Mode raises clicks to publishers, and AI Mode alone “erodes user experience and trust”.

With an AI answer on the page, fewer people click through
Without vs with
Clicks to outside sites per search (RCT)
Without AI answer0.62
With AI answer0.37
Searches ending with no click to an outside site (RCT)
Without AI answer54%
With AI answer73%
Results-page visits with a click on a regular result (Pew, observational)
Without AI answer15%
With AI answer8%
Results-page visits that ended the browsing session (Pew, observational)
Without AI answer16%
With AI answer26%
Source: RCT: Agarwal and Sen, 1,065 US desktop Chrome users, January–February 2026, comparing the normal-Google and hidden-overview groups, on searches where an AI Overview would appear; working paper. Pew Research Center: browsing data from 900 US adults, March 2025; not controlled for query type. Percentage bars share a 0–100% scale.

Observational data points the same way

Pew Research Center’s browsing data from 900 US adults in March 2025 shows the same pattern (see the chart); only 1% of visits included a click on a link inside the AI summary. Pew’s comparison isn’t controlled: summaries appear more often on longer, question-style searches.

Chartbeat’s 2026 report on its news-heavy client network puts Google Search referrals down 40.2% from July 2025 to July 2026, after a 21.9% fall the year before2. Chartbeat, which sells analytics, doesn’t define what that yearly table counts. Google Search’s share of all pageviews, a different measure, fell from about 9% to 5% over two years. Reddit told shareholders its search traffic was “down meaningfully” in the third quarter of 2026, and its CEO said AI Overviews “are not traffic drivers”.

How much a whole site loses depends on how often answers appear for its queries, and that share is rising. Overviews appeared on about 41% of the experiment’s searches and on 18% of the searches Pew tracked in March 2025; DataForSEO’s keyword panel, as reported by RevScope, went from 5.93% of tracked US keywords in September 2024 to 61.43% in August 2026, on a list that skews commercial. The bases differ; this is not one trend line. The experiment’s 40% is measured only on searches that show an overview, so it is not a site-wide loss. As overviews spread, the authors note, the total loss “could become substantially larger, even if the per-query effect remains constant.”

Google’s counter-claim

Google’s head of Search, Liz Reid, wrote in August 2025 that total organic clicks were “relatively stable year-over-year”, with “slightly more quality clicks”, meaning visits where users don’t quickly click back3. She called third-party reports “often based on flawed methodologies”. Google sees traffic nobody else does, so the claim deserves a hearing. But it comes with no figures from an interested party, and the RCT found no difference in click quality.

The loss falls on sites whose product is the answer

The experiment’s effect sat in informational queries, and Chartbeat says “generic, easily-summarized content is the most exposed”, while original reporting is harder to replace. Size matters too: in Chartbeat data reported by Axios, search referrals to small publishers, those with 1,000 to 10,000 daily pageviews, fell 60% over two years, against 47% for mid-sized publishers and 22% for those above 100,000.

All About Berlin is the case people cite. Its owner has run the Berlin guide since 2017 and lived from it since 2020. In May 2026 he blamed AI Overviews for a 70 to 75% traffic loss, and in November he wrote on Hacker News: “My traffic is down over 80%. The information I create is really easy to steal, but it’s as expensive as ever to create.”

Wikipedia faces a version of the same problem. Its foundation reported human pageviews down roughly 8% year on year in October 2025 and warns that fewer visits may mean fewer volunteers to improve the content and fewer donors to fund it.

Some parties don’t lose, at least so far, starting with Google’s ads. Loyal readers held up across Chartbeat’s network: those who visit on at least 8 of the last 16 days generated 14.98 billion monthly pageviews in 2024 and 15.02 billion in 2026. Chatbot referrals are a separate, tiny channel: 0.28% of traffic across 74,752 sites in March 2026, by the count of Ahrefs, an SEO tool vendor. On Ahrefs’ own site, though, visitors from AI chatbots and AI search were 0.5% of the total and 12.1% of sign-ups.

Not every collapse is AI. HouseFresh, an independent air-purifier review site, lost 91% of its search traffic in Google’s March 2024 core update, two months before AI Overviews launched in the US.

The clicks are going; whether that’s a loss is the argument

Almost nobody disputes the loss itself, including people who welcome AI answers. The disagreement is about what is being lost.

The best case for AI answers is that the pages they replace had become bad. Recipes buried under life stories, cookie banners, overlays and affiliate review farms made the answer box the better product for simple questions. Barry Adams, a news SEO consultant, likes the overviews himself: “they tend to be pretty good.” Many go further and argue Google broke search first, by profiting from SEO spam and pushing results below ads long before AI arrived.

Two things weaken that case. In the RCT, users shown AI Overviews reported no gain in satisfaction: on desktop, at least, they didn’t experience the cleaner web the argument promises. And the case judges only today’s interface. The experiment measures an immediate loss of outbound clicks; what that loss does to what people publish is still open.

The feedback loop

The shared worry is a loop: AI answers depend on publishing that they make unprofitable. If guides, reviews and Wikipedia edits stop, the answers built on them get worse. Agarwal and Sen end their summary on the same point: innovation in information access should not “systematically undermine the incentives to produce the information on which those innovations depend.”

The best objection is that much of the web was never made for traffic. Hobby communities live on YouTube, Discord and Reddit, and Wikipedia’s most active editors may care about their subject, not about casual readers. Nobody has data on this yet; editor counts and new-page publication rates would settle it.

Optimizing for AI citations is mostly SEO

Generative engine optimization (GEO), the practice of shaping pages to be cited by AI answers, sells itself as a new discipline. The evidence says otherwise. Google’s documentation says there are “no additional technical requirements” to appear in AI Overviews and AI Mode, and that “specific optimization isn’t required” beyond SEO fundamentals.

llms.txt is a proposed file that summarizes a site for language models. SE Ranking checked about 300,000 domains: 10.13% had the file, and the study found “no correlation between AI citations and LLMs.txt”; removing the variable even improved its prediction model4. SE Ranking itself still calls the file an optional, low-effort experiment.

What does correlate is old-fashioned reputation. Ahrefs found branded web mentions the strongest correlate of a brand appearing in AI answers, with a Spearman correlation of 0.664, and “best X” lists make up 43.8% of the page types AI systems draw on. Probably because of fan-out, a page can be cited without ranking for the query itself: Ahrefs found 76% of AI Overview citations came from top-10 pages in July 2025 and about 38% in March 2026, though the two studies’ methods differ.

All of this is correlational, and most of it comes from vendors that sell GEO tools. Adams puts it bluntly: people who call GEO a separate discipline “have never had to optimize search beyond classic search.”

No payment scheme yet replaces referral revenue for independent sites

Cloudflare’s pay per crawl revives HTTP status 402. A verified AI crawler that requests a priced page gets a payment-required response, and can retry with a header agreeing to pay; Cloudflare is the merchant of record.

HTTP/1.1 402 Payment Required
crawler-price: USD XX.XX

A year after launch, pay per crawl was still a closed beta. On 30 September 2026 Cloudflare opened two more. In Pay Per Use, an AI company offers a price per use of a page, such as returning an excerpt, and reports each use itself; Cloudflare bills it and pays the publishers who accept, monthly. The Monetization Gateway, a US-only closed beta, charges AI agents per request through x402, an open payment protocol built on HTTP 402, and settles in USDC, a dollar-pegged stablecoin; Cloudflare aims it at APIs and data rather than articles. Neither has published a payout figure.

Really Simple Licensing (RSL) 1.0, published in December 2025, lets a site declare machine-readable licence terms through robots.txt, HTTP headers or RSS. At launch, its payment partner Supertab said bots “aren’t actually being billed at this point”; no billing figures have been published since.

Google’s “AI contribution pilot”, offered to selected publishers, does pay: each sees a monthly payout figure in Search Console, with no explanation. According to Digiday, it has appealed mostly to small and mid-sized publishers, where most independent sites sit. One executive called it “quite black box”; one source said early returns were “peanuts” relative to ad revenue.

Ways to get paid for AI use, and whether they pay
As of November 2026
Model, and who runs itStatusMoney flowing?
Pay per crawl (HTTP 402). CloudflarePrivate beta from 1 July 2025; still a closed beta in August 2026No payout figures published
Pay Per Use: an AI company pays per defined use, such as returning an excerpt, and self-reports each use. CloudflareBeta from 30 September 2026No payout figures published
x402 per-request payments (Monetization Gateway). Cloudflare, with Coinbase’s x402 facilitatorClosed beta from 30 September 2026, US only, aimed at APIs, tools and dataNo payout figures published
Block, then license. Large publishers on Cloudflare: People Inc, FT, The Atlantic, Ziff Davis, Condé Nast, AP“Real momentum” (People Inc CEO); “not only for large publishers” (Cloudflare)Yes, for large publishers (self-reported)
RSL 1.0 licence terms. RSL Collective, backed by Cloudflare, Akamai, AP, Stack Overflow and SupertabSpec published 10 December 2025Not at launch: bots “aren’t actually being billed” (Supertab, December 2025); no figures since
AI contribution pilot. GoogleOffered to selected publishers, appealing mostly to small and mid-sized ones, at least dozens approached, reported September 2026Yes, undisclosed monthly sums; “peanuts”, per one source
Search generative AI control (opt out of AI features, keep ranking). Google, prompted by a UK CMA requirementWorldwide since 31 August 2026; set per Search Console property (domain, subdomain or URL path); covers AI Overviews, AI Mode and Discover’s AI features, not training; CMA page-level control due three months after its 3 December 2026 deadlineNot a payment
Source: Status as reported by Cloudflare, Press Gazette, The Register, Digiday, 9to5Google, Google and the CMA. “No payout figures published” means none disclosed, not that none were made. RSL: Really Simple Licensing. CMA: the UK’s Competition and Markets Authority.

What works so far is blocking. Cloudflare says customers block AI crawlers to create “reliable scarcity” and then negotiate licences, and People Inc’s CEO reports “real momentum”. The named deals are with large publishers; Cloudflare says the route is “not only for large publishers”, a vendor’s claim. So far, money reaches two groups: large publishers, through deals, and pilot members, in undisclosed sums.

Blocking doesn’t keep you out of Google’s AI answers. People Inc’s CEO had said the company blocks “almost all AI crawlers other than OpenAI, where we have a deal, and Google where we can’t block them because they use one crawler for search and AI”. As of November 2026, Google still hasn’t split its crawler. Instead, prompted by a requirement from the UK’s Competition and Markets Authority (CMA), it offers a Search Console setting that, since 31 August 2026, any site worldwide can use to leave AI Overviews, AI Mode and the generative AI features in Discover5. Google says the choice isn’t a ranking signal elsewhere in Search.

The setting works per Search Console property, so a domain, subdomain or URL path such as example.com/blog/ can opt out on its own; there is no per-page switch. It doesn’t cover training, which still runs through Google-Extended. The CMA’s deadlines are separate: Google must comply with its requirement by 3 December 2026 and has three more months to roll out a page-level control.

Adams doubts the value of opting out: the overview stays, and “it’s just going to cite your competitor.” Google’s help page concedes that content from other sites “may appear similar to yours”. Agarwal and Sen add that opting out restores no traffic, and that licensing may concentrate money with publishers big enough to negotiate.

Plan for search as a scarce channel

Adams calls “Google Zero” a myth. In his experience, Google Search and Discover together still provide “at least 30% to 35%” of a successful news publisher’s traffic. What changed is that an “endlessly renewable channel has now become a scarce channel”: being fourth or fifth in a niche used to pay, and now you need to be a “top three publisher in that space”. For an independent site, that suggests a short list:

  • Write what a summary can’t replace. Original reporting, first-hand testing and local knowledge. Chartbeat says original reporting is harder for AI to replace.
  • Build channels you control. In Chartbeat’s network, direct traffic gained share, from 13.5% to 15.9% of pageviews between July 2024 and July 2026, and dark social (links shared in messages) from 7.1% to 11.3%. Newsletters and RSS are the independent-site version.
  • Keep SEO basics; don’t count on shortcuts. The llms.txt study found no correlation with citations, so don’t expect the file to bring any. Get mentioned and linked where AI looks: Reddit, YouTube, “best X” lists, the press.
  • Measure revenue per channel, not rank. Search Console folds AI-feature clicks into its Web totals, and its separate AI reports show impressions only.
  • Set your crawler policy on purpose. Block other AI training crawlers if what you publish is scarce. Cloudflare’s “Disallow AI Training” setting (September 2026) does this while Googlebot, Bingbot and Applebot keep crawling for search. For Google, a Google-Extended rule in robots.txt opts you out of Gemini training without affecting ranking (Google’s crawler documentation). To leave AI Overviews, AI Mode and Discover’s AI features, use the Search Console setting, for the whole site or a path. Snippet controls such as nosnippet also limit them, but trim your regular snippets too.
Search loses share of pageviews; direct and dark social gain
Share of all pageviews
Source: Chartbeat, 2026 Publisher Playbook (Q3 2026 report); client network skewed toward news publishers, site count not disclosed. Shares are of all pageviews, including internal clicks between a site’s own pages, and Google Search excludes Discover. Direct is 15.9% in the report’s chart and 15.4% in its text; the chart value is shown. Dark social: links shared in texts, DMs and private channels. The vertical scale is broken between 17% and 37%.

What to watch: whether Google meets the CMA’s 3 December deadline and ships the page-level control in the three months after; whether Search Console’s AI reports start showing clicks, which the CMA requires; the first published payouts from Cloudflare’s new betas, RSL or Google’s pilot; and a mobile experiment, where no causal data exists. In 2027 Google plans to honour a shared robots.txt training opt-out from the Internet Engineering Task Force (IETF), which covers training, not AI Overviews. If none of that moves by mid-2027, assume the bargain stays repriced and plan for readers who come to you directly.

Notes

  1. 1Saharsh Agarwal and Ananya Sen, “The Impact of Google AI Overviews on Publisher Traffic and User Experience: Evidence from a Field Experiment”, working paper, SSRN, revised 17 June 2026. Not peer reviewed. ssrn.com
  2. 2Chartbeat, “2026 Publisher Playbook: 9 Strategies to Boost Acquisition, Engagement, and Loyalty”, Q3 2026 report. Chartbeat sells analytics. chartbeat.com
  3. 3Liz Reid, “AI in Search is driving more queries and higher quality clicks”, Google blog, 6 August 2025. Google is an interested party and gives no figures. blog.google
  4. 4Yulia Deda, “LLMs.txt: Why Brands Rely On It and Why It Doesn’t Work”, SE Ranking blog, 15 October 2026. Vendor study, correlational. seranking.com
  5. 5Google, “Search generative AI control”, Search Console Help, with a note dated 31 August 2026 on the worldwide rollout. support.google.com
Get new articles by email

One email when something new is published. Unsubscribe any time.

Keep reading

All articles

We burn the tokens so you don't have to.

Interesting technical work, plus deep investigations with every claim sourced, so you get the answer without the rabbit hole.