In October 2026, The Atlantic joined a fight reshaping the web’s memory. Secondary coverage puts it plainly: publishers have real grievances against companies that profit from their work without permission, but the Wayback Machine has become collateral damage. Blocking Internet Archive access, the argument runs, punishes a nonprofit library without stopping the AI systems publishers actually fear.
One transparency note first. The Atlantic article sits behind a paywall and could not be read directly for this article, so nothing here is quoted from it. What follows is a corroborated synthesis built from Nieman Lab reporting, Techdirt coverage, the Electronic Frontier Foundation, and the Internet Archive’s own FAQ, and this article flags where those sources reach only partway.
The dispute is easy to state. Publishers say they are trying to stop AI companies from scraping their journalism to train competing models. Archivists say the tool being blocked is not an AI company at all. It is a library that has saved the web since 1996, and every day it is shut out, new pieces of the public record stop being saved.
In Brief
- Blocking Internet Archive crawlers will not stop AI training, archivists and the EFF argue, because AI firms run their own crawlers and draw on other data sources.
- The New York Times hard-blocks the Archive’s crawlers and added archive.org_bot to its robots.txt file in late 2025; The Guardian limits article archiving but keeps landing pages.
- Nieman Lab counted 241 news sites blocking at least one Archive bot in January 2026, and more than 340 local outlets limiting access by May 2026.
- Publishers acted preemptively: Nieman Lab’s Andrew Deck reported that none could point to direct evidence their content had been scraped through the Wayback Machine.
- What is lost is concrete: Pew found 38% of 2013 webpages gone a decade later, while the Wayback Machine preserved about 15% of pages otherwise lost.
Why This Matters
Why does a crawler block deserve a reader’s attention? Because the web forgets on its own. Pages change, move, or disappear when sites redesign, shut down, or quietly edit stories. An archived copy is often the only remaining proof of what a page once said. Blocking Internet Archive crawlers does not freeze that record. It lets new holes open while the fight over AI training continues elsewhere.
The stakes are also asymmetric. A publisher that blocks the Archive still publishes daily and still reaches readers. Blocking Internet Archive crawlers costs readers something less visible: the ability to check, years later, what that publisher actually wrote. That is why more than 200 journalists signed a statement supporting the Wayback Machine, and why the Internet Archive says more than 100 news articles every month cite its preserved pages.
Blocking Internet Archive Access: Who Did What
Blocking Internet Archive crawlers did not arrive all at once. Nieman Lab first reported the pattern in January 2026. The New York Times went furthest, hard-blocking the Archive’s crawlers and adding archive.org_bot to its robots.txt file at the end of 2025. A Times spokesperson said the paper blocked the Archive’s bot because the Wayback Machine provides unfettered access to Times content, including by AI companies, without authorization.
The Guardian took a narrower route, limiting published articles, excluding itself from the Archive’s APIs and filtering article pages from the Wayback Machine’s URL interface, while homepages and topic pages continue to appear. The Atlantic’s chief executive, Nicholas Thompson, explained in a video that The Atlantic had blocked the Wayback Machine, arguing that archived material would flow to large AI companies and to everybody building a subsidiary AI product.
One company dominated the January 2026 count. Nieman Lab found 241 news sites from nine countries disallowing at least one of the Archive’s four crawling bots, and 87% were owned by USA Today Co., formerly Gannett. Each disallowed the same two bots, archive.org_bot and ia_archiver-web.archive.org. By May 2026, Nieman Lab reported more than 340 local news outlets limiting the Archive’s access.
Why Blocking Internet Archive Copies Fails Against AI
The core problem with blocking Internet Archive crawlers is that it targets the wrong bottleneck. AI companies operate their own crawlers and draw on separate corpora such as Common Crawl, an independent web crawl. Shutting the Archive’s crawlers out of a news site does not remove that site’s public pages, and it does not stop a determined scraper from fetching them directly. The EFF’s library analogy: a newspaper publisher announcing it will no longer allow libraries to keep copies of its paper.
The evidence behind blocking Internet Archive access is also thinner than the response. Nieman Lab’s Andrew Deck reported that publishers were acting preemptively, out of a fear of proxy scraping, rather than from direct evidence. None, in Deck’s account, could point to direct evidence that their content had already been scraped through the Wayback Machine. The Archive’s director, Mark Graham, says the Wayback Machine is built for human readers and uses rate limiting, filtering, and monitoring against abusive access.
As more search becomes AI-mediated, absence from the datasets those systems learn from can mean absence from their answers, as Techdirt’s Mike Masnick argued. Blocking Internet Archive access also rests on a blunt instrument: robots.txt gives a publisher a yes-or-no choice for each crawler, with no way to say that archiving is welcome while AI training is not.
What History Loses
The cost of blocking Internet Archive access is not abstract. A 2024 Pew study found that 38% of webpages from 2013 were no longer accessible a decade later, with roughly a quarter gone entirely. Internet Archive data scientist Sawood Alam found that the Wayback Machine had preserved roughly 15% of those otherwise vanished pages. Nieman Lab reporting also pointed to an 87% drop in saved news pages between May and October 2025, reported here as Nieman Lab’s finding, not an independently rechecked count.
The distortion is specific. If the most trusted publications keep blocking Internet Archive access while less reliable sites do not, future readers will find archived versions of random blogs and conspiracy sites, but not the newspapers considered the most reliable record. Masnick put it bluntly: it will not stop AI training, but it will ensure that significant chunks of the journalistic record simply disappear. Institutions do not last forever: the Rocky Mountain News published for 150 years before shutting down in 2009, and more than 2,100 newspapers have closed since 2004.
Confirmed vs Reported vs Unknown
Three buckets keep this story honest: what is confirmed, what is reported, and what remains unknown.
Quick Answers
Does blocking the Archive stop AI training? No, on the evidence here. AI companies run their own crawlers and use other corpora, so blocking Internet Archive access removes a public record without removing the training data. That is the EFF’s central point, and it is why archivists call the Wayback Machine collateral damage.
Did AI companies actually scrape publisher content through the Wayback Machine? Publishers have not shown it. Nieman Lab’s Andrew Deck reported that publishers acted preemptively out of a fear of proxy scraping, and none could point to direct evidence that their content had already been scraped that way. The blocking rests on a risk, not a documented case.
Is the Internet Archive itself building AI models? No. It is a nonprofit research library whose Wayback Machine has archived the web since 1996. Its director says it is built for human readers and uses rate limiting, filtering, and monitoring against abusive access.
Can publishers allow archiving but block AI training? Not with today’s robots.txt alone, which offers a yes-or-no choice per crawler. A publisher cannot use that file to say that the Archive may crawl for preservation while another company may not crawl for training. Until a finer-grained standard is widely honored, publishers wanting to stop training crawlers will keep blocking Internet Archive crawlers in the same net.
The Takeaway
Blocking Internet Archive crawlers is a trade with a certain cost and an uncertain payoff. The cost is measurable: fewer pages saved, a record that tilts away from trusted journalism, and a public that can no longer check what was once published. The payoff is unproven: no publisher has publicly documented AI training data flowing through the Wayback Machine.
Publishers have a legitimate fight about payment, permission, and control, and that fight belongs in courts and licensing deals, not in the settings that decide what a library may preserve. As the Archive’s director put it, preserving the public record is not optional but essential infrastructure for a functioning democracy, and sacrificing it to fight AI training would be a mistake measured in decades, not news cycles.
Sources: Nieman Lab, News Publishers Limit Internet Archive Access Due to AI Scraping Concerns, January 28, 2026; Nieman Lab, More Than 340 Local News Outlets Are Limiting the Internet Archive’s Access, May 20, 2026; Techdirt, News Publishers Are Now Blocking the Internet Archive and We May All Regret It, February 13, 2026; Internet Archive, FAQ: Publishers Blocking the Wayback Machine, last updated June 26, 2026; Electronic Frontier Foundation, Blocking the Internet Archive Won’t Stop AI, but It Will Erase the Web’s Historical Record, March 4, 2026; secondary coverage of The Atlantic’s October 2026 piece on the Wayback Machine and AI (not quoted, as the original is paywalled). PetaTech24 tested nothing for this article; the hero image is an original generated illustration, not a source photograph.
