Konstantin Ryabitsev from git.kernel.org discusses the growing issue of 'background radiation' from abusive crawlers. At any given time, across 5 geo-distributed nodes, 14 CPU cores are dedicated to rendering git commits as HTML, far exceeding the resources used for legitimate access like git clones. This highlights the significant strain on resources and the need for better protection against such behavior. Ryabitsev's concerns are particularly relevant for services like Datasette, which serve a large number of crawlable web pages. Source: https://simonwillison.net/2026/Sep/7/creepy-crawlies/
LLMs1 min read
Git kernel's CPU drain from abusive crawlers
Git.kernel.org spends more resources rendering commits for scrapers than on legitimate access. This affects services like Datasette.
By OpenSmartRoute editorial · written through the router by coder-small
From Simon Willison - “Creepy crawlies”
Keep reading
Related posts
LLMs1 min read
Quoting Jakub Pachocki
The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI. [...] We will need powerful, aligned AI for defense; to secure infrastructure,...
LLMs1 min read
Video Compression Tool with Claude Fable 5.1
Engineers can now optimize video files using Claude Fable 5.1 and FFMPEG, enhancing web performance and storage efficiency.
LLMs1 min read
Mercator ↔ Equal Earth
Tool: Mercator ↔ Equal Earth I got curious about the Equal Earth map projection that was recently voted on at the UN so I had GPT-6 Astra (medium) in ChatGPT Work build me this animated transition between Mercator and Equal Earth using D...