On 2 November 2026 the New York Times sued Brave, the company behind the Brave browser and Brave Search. The complaint says Brave copied more than 30 billion web pages, never identified its crawler to site owners, and sold access to that content to AI companies. If your site is in that index, no line you add to robots.txt takes it out while leaving Google’s access intact.
What the Times complaint claims
The filing says Brave copied the pages “without identifying itself or providing website owners with any mechanism to block its crawler”, then sold that content on. It also alleges copyright infringement and a route around the newspaper’s paywall.
Five AI companies are named in the complaint as customers: Anthropic, Perplexity, Mistral, xAI and Cohere. None of them is being sued. Brave had not answered the filing when Search Engine Journal reported on it on 6 November 2026, so these are claims and not findings.
The Daily News, the Chicago Tribune and the Orlando Sentinel filed a separate case the same day. Four news organisations, one week, and one crawler at the centre of it.
How Brave sells access to its search index
Brave runs its own web index, built by its own crawler, and sells queries against it through a product called the Brave Search API. Brave’s About page says the API is used by “some of the biggest names in AI, including Cohere, Mistral, Perplexity, and You.com”. Its AI page puts the index at “over 30 billion pages, with millions of new pages indexed each day”.
The part that matters for a publisher is the product called LLM Context. The complaint quotes Brave describing it as handing over “extracted page chunks and source metadata in a shape and length that satisfies the needs of an LLM without need for further scraping”. A page arrives at the AI company already cut into answer-sized pieces, which is the same query fan-out idea run one step further, with the reading done before the question is asked.
Nothing in that sale sends a reader back to your site. Ordinary search results do that by design, because the click is the product. An answer built from your chunk is a different transaction. Being read and being credited are two separate events, and the gap between them is where AI search and getting cited work happens.
Why robots.txt cannot block Brave’s crawler
robots.txt only works because crawlers announce who they are. OpenAI publishes its rules under names such as GPTBot, Anthropic under ClaudeBot, so a site can let one in and keep the other out. Cloudflare ran a pilot on that exact question for ChatGPT Search crawlers in October 2026.
Brave’s crawler has no name of its own. Brave’s help page, which I read on 6 November 2026, says the crawler “does not advertise a differentiated user agent because we must avoid discrimination from websites that allow only Google to crawl them”. The same page gives the rule it follows instead: “if a domain or page is not crawlable by Googlebot, then Brave Search’s bot will not crawl it either”.
The page is direct about the alternative. robots.txt is not used to prevent a page from being indexed, because that file governs crawling. Delisting needs a noindex directive, and Brave has to re-fetch the page before the directive takes effect.
Which crawlers you can name in robots.txt
Four crawlers that matter to a publisher, and what switching each one off costs. The names in the third column are the ones you can type into a file and have obeyed.
| Crawler | Who runs it | Named in robots.txt | What blocking it costs |
|---|---|---|---|
| Googlebot | Yes | Every visit from Google Search | |
| Brave Search crawler | Brave | No separate name | Cannot be blocked without blocking Googlebot too |
| GPTBot | OpenAI | Yes | Use of your pages in OpenAI’s training crawl |
| ClaudeBot | Anthropic | Yes | Claude’s access to your pages |
Brave’s row is the one the case turns on. The other three can be refused tonight and Brave’s cannot, so a robots.txt that looks careful can leave the largest unnamed crawler untouched.
The 30 second check for your own pages
Go to search.brave.com and type site: followed by your domain, with no space after the colon. Every page that comes back sits in the index Brave sells, and getting that count is faster than reading the complaint.
Read the results as a stored copy rather than a live view of the site. Search Engine Journal’s report notes that Brave served a page description its author had replaced earlier the same day.
What a publisher can actually do
Three options, and each one has a cost attached:
- Leave it alone. Treat Brave as part of the open web and keep working on the results that send clicks. Cheapest option, and it accepts the sale.
- Pull specific pages with noindex. It works, and Google drops those pages too, because the directive Google obeys is the one Brave follows.
- Refuse the named AI crawlers. GPTBot and ClaudeBot can be turned away in robots.txt today. Publishing an llms.txt file is a separate lever and does not change any of this.
None of the three reaches backwards. A noindex changes what happens next, and a copy already handed to an AI company is not recalled by a directive or a file. The counts in AI referral traffic in GA4 will not show you the difference either, because a sale of page chunks never appears as a visit.
The News Corp case sets the timeline
Brave has been through this argument before. News Corp sent a cease-and-desist over crawling, and Brave sued first, arguing that indexing the web is what “all search engine operators must do to exist”. In July 2026 News Corp sued back, saying Brave hides its crawlers so publishers cannot detect or block them. Semafor reported that exchange, and the Times complaint lands four months later.
A ruling on the Times’s pages will not automatically cover anyone else’s site. It will settle the question the case was built on, which is whether an index gathered by an unnamed crawler can be sold on in pieces, and every site in that index is affected by the answer.
The site: query on Brave is free and takes half a minute. Run it before deciding that a lawsuit between a newspaper and a browser company has nothing to do with a small blog.
