Last updated: August 9, 2026 · By Vishal Swami, Founder & Lead AI Reviewer, AISagely
Gentoo's bug tracker went dark this week. A Gentoo infrastructure developer, Michał Górny, took Bugzilla offline after AI training bots overwhelmed it, and the story spread fast under the headline "Gentoo bugzilla closed due AI bot scraper overload" once it hit Hacker News on August 9, 2026.
Short answer: Gentoo bugzilla closed due to AI bot scraper overload on August 9, 2026, when developer Michał Górny took Bugzilla offline after LLM training bots — rotating through thousands of IPs and ignoring robots.txt — hammered its database with repeated searches. It's the latest flare-up in a pattern that's been hitting SourceHut, Fedora, and other open-source infrastructure since early 2025.
I run this site behind Cloudflare and already deal with a version of this problem at a much smaller scale, so when Gentoo's story broke I went digging into what actually happened and how bad the wider issue has gotten for projects that don't have a commercial CDN's bot-management budget.
What actually happened
Gentoo Bugzilla didn't fail overnight. Górny had already written about the slow-motion version of this problem back in April, in a post titled “The pinnacle of enshittification, or Large Language Models”. His description of the pattern is blunt: "LLM scrapers are ignoring these rules, and firing Bugzilla search after search, report after report. The server is churning like crazy, the database is churning like crazy, and real users who actually need to find a bug report or file one, are suffering because of that." He called the effect "like a constant DDoS attack at independent infrastructure" and warned that "the Internet is losing its independent websites."
That April post followed an earlier, quieter fix. Gentoo had already restricted anonymous Bugzilla search back in October 2025 — Gentoo developer sam_ explained on the Gentoo Forums that the change wasn't optional: "Searching Bugzilla is intense on database queries. We had to introduce that restriction because of bots hammering it." That fix required a login for list-based searches but still let anyone view or file individual bugs without an account.
By August 9, 2026, that wasn't enough. Górny took Bugzilla down entirely. The resulting Hacker News thread filled up with sysadmins comparing notes. The worst traffic reportedly comes from residential proxy networks and rotating IP ranges, not a single identifiable crawler. That's exactly why simple IP blocking hasn't worked for Gentoo, or for anyone else fighting this.
Gentoo isn't an isolated case, either. Drew DeVault, founder of the Git-hosting service SourceHut, told TechCrunch in March 2025 that he was spending "from 20-100% of my time in any given week mitigating hyper-aggressive LLM crawlers at scale." He also described "dozens of brief outages per week." That same reporting covered Fedora's infrastructure team and the Linux news site LWN, which ran into similar floods. That's what pushed a wave of open-source projects toward proof-of-work tools like Anubis, a free, MIT-licensed reverse proxy. Anubis makes a browser solve a small compute puzzle before it can load a page. It picked up roughly 2,000 GitHub stars, 20 contributors, and 39 forks within days of its March 19, 2025 release. That's how many maintainers were already looking for a fix like it.
What you'll need to check your own exposure
You don't need Gentoo's traffic volume to have this problem — a slow bug tracker, wiki, or search page on any self-hosted project can get flattened by the same crawlers. Pull up your server's access logs (or your host's traffic dashboard) for the last week and look for repeated hits on search or list-style URLs rather than individual pages; that pattern is the tell. Check whether your site currently has a robots.txt file at all, and if it does, whether it's ever been updated since AI crawlers started ignoring the standard Disallow rules anyway. If you're behind Cloudflare, Fastly, or a similar CDN, know your login credentials for its dashboard, since most of the fixes below live there rather than in your application code.
Step-by-step: protecting a self-hosted project from the same overload
1. Confirm you're actually being scraped, not just getting normal traffic
Filter your access logs for requests with no referrer, a generic or spoofed browser user-agent, and a search or filter URL pattern. A spike of thousands of near-identical requests from a spread of unrelated IPs, all hitting your most database-intensive page, is the signature Gentoo, SourceHut, and Fedora all reported.
2. Add a Content Signals line to robots.txt as a baseline
This won't stop a bot that's already ignoring Disallow, but it costs nothing and it's the documented, unambiguous way to state your preference:
“ User-agent: * Content-Signal: search=yes,ai-train=no Disallow: /search Disallow: /buglist.cgi “
3. Put a real blocker in front of expensive endpoints
A robots.txt line is a request, not an enforcement mechanism. That's exactly what Gentoo's April post is complaining about. If you're on Cloudflare, its AI Crawl Control lets you block bots by category — Search, Agent, or Training — for free on every plan, including the free tier. I cover the exact toggle path in my Cloudflare AI traffic options walkthrough. If you're self-hosting outside a CDN, Anubis (github.com/TecharoHQ/anubis) sits in front of your app as a reverse proxy. It makes a browser solve a proof-of-work challenge before it reaches your server. That's the same fix Gentoo's peers in the open-source hosting world adopted through 2025.
4. Rate-limit list and search endpoints specifically
Even with a bot-blocking layer, cap request rates on the URLs that hit your database hardest — search forms, bug lists, tag pages. A basic Nginx zone limits repeated hits without blocking normal browsing:
“` limit_req_zone $binary_remote_addr zone=search_limit:10m rate=5r/m;
location /search { limit_req zone=search_limit burst=3 nodelay; } “`
5. Consider a login wall for the most expensive queries only
Gentoo's October 2025 fix — requiring login for list-based Bugzilla searches while leaving individual bug pages open — is a reasonable middle ground if a full proof-of-work layer feels heavy for a small project. It trades some convenience for real load reduction, and it's the option that bought Gentoo about ten extra months before the problem outgrew it.
Common mistakes to avoid
In my testing on this site's own Cloudflare setup, the mistake I see most often is treating IP blocking as a real fix. Per the Hacker News discussion of Gentoo's outage, the traffic was spread across large residential proxy ranges. A single static IP block is trivial to rotate around. Second, don't assume robots.txt alone protects you. Górny's own post describes bots that already ignore it — a Content Signals line is a courtesy flag, not a wall. Third, don't wait for a full outage before doing anything. Gentoo's timeline went from a search restriction in October 2025, to a warning post in April 2026, to a complete shutdown in August 2026, and each step bought less time than the last. Fourth, don't block every non-human visitor indiscriminately. Legitimate search crawlers and the AI "Search" category (the kind that sends you referral traffic) behave differently from training scrapers, and blunt blocking can cost you real visibility for no benefit.
Mitigation options compared
| Option | Cost | How it works | Best for |
|---|---|---|---|
| Login wall (Gentoo's Oct. 2025 fix) | Free | Requires an account for list/search-style queries | Small teams that can accept less-convenient public access |
| Anubis (proof-of-work proxy) | Free, open source (MIT) | Makes the browser solve a compute puzzle before serving a page | Self-hosted forges, wikis, bug trackers |
| Cloudflare AI Crawl Control | Free on every plan | Classifies bots as Search, Agent, or Training and blocks by category | Sites already behind Cloudflare |
| Nginx rate limiting | Free (built in) | Caps request rate per IP on expensive endpoints | A first layer on top of whatever else you use |
| robots.txt Content Signals | Free | States a search=yes,ai-train=no preference |
A baseline everyone should set, though it isn't enforcement |
Tools that make this easier
If you're already behind Cloudflare, start with the toggles in my Cloudflare AI traffic options walkthrough rather than reaching for a separate proxy. It covers the same Search/Agent/Training split Anubis and similar tools try to approximate. If you're curious how much of this problem is really about training data collection, my breakdown of scanning Hugging Face datasets for secrets shows what gets swept up when scrapers harvest at this scale. Open-source projects thinking about infrastructure risk more broadly should also look at Chiaro’s open-source SOC 2 methodology, which covers documenting this kind of availability risk. To track whether AI crawlers send your site traffic worth allowing, AI Search Console explains how to read that split inside Google's own reporting. And before you pay for a bot-management or monitoring tool, my guides to reading AI tool ratings without getting fooled and the free AI tools worth using are good starting points — a free option often already covers this.
My take
Gentoo's outage isn't a one-off story about one project's bad week. It's what happens when a volunteer-run service with no dedicated sysadmin budget runs into the same scraper volume that's already forced SourceHut, Fedora, and LWN into active defense. The fixes that work are boring and layered, not clever: a robots.txt signal as a baseline, a real blocking or proof-of-work layer in front of expensive pages, and rate limits as a backstop. None of that is exotic. None of it requires paying for enterprise bot management. What it does require is doing it before the outage, not after — Gentoo's own timeline shows each partial fix bought less time than the last.
Frequently Asked Questions
Is Gentoo Bugzilla back online now?
Gentoo restricted anonymous search in October 2025 and then took Bugzilla fully offline on August 9, 2026, according to Górny's own announcement post, referenced in the Hacker News discussion of the outage. Check bugs.gentoo.org directly for current status, since a volunteer-run project's timeline for restoring access isn't fixed in advance.
Why can't Gentoo just block the AI bots by IP address?
Because the traffic isn't coming from a small, fixed set of addresses. Commenters on the Hacker News thread and Górny's own April 2026 post both describe scraper traffic spread across large, rotating IP ranges, including residential proxy networks, which makes static IP blocklists ineffective within days.
What is Anubis, and is it free?
Anubis is a free, MIT-licensed reverse proxy created by Xe Iaso that makes a visitor's browser solve a small proof-of-work puzzle before it can reach a site, which blocks most non-browser scrapers without a login wall. It's free to self-host, and TechCrunch reported it picked up roughly 2,000 GitHub stars within days of its March 19, 2025 release.
Does blocking AI bots hurt my search rankings?
Not if you separate categories the way Cloudflare's AI Crawl Control does. Blocking training-only crawlers doesn't affect classic search indexing; blocking a bot classified under both Search and Training will, since the more restrictive rule applies to any multi-purpose crawler.
Is this only a problem for big open-source projects?
No — the tools that make Gentoo, SourceHut, and Fedora targets (a public search page, a database-backed lookup, no dedicated bot-management budget) exist on plenty of small self-hosted sites. The fixes in this guide — a robots.txt signal, a free proxy like Anubis, and basic rate limiting — scale down to a single-maintainer project just as well as they scale up.