To follow up on the previous post about our forums and getting scraped by bots, I enabled some statistics in our human-authentication system for the forum. In the last 18 hours, we have seen requests from about 300,000 IPs, of which about 5,000 (approx 1.5%) successfully verified themselves as human.

Of the 295,000 IPs that did not verify themselves as human, about 15,000 were poorly behaved, so much so that they triggered our log-analysis-ban system.

The remaining 280,000 IPs each behaved relatively fine, each doing occasional requests more or less mimicking the usage patterns of a user. There are two problems it caused for us, though:

1. they often would request pages that are expensive to generate (page 90 of our member list, e.g., which vbulletin has to go and sort/generate then skip 89 pages of the results)
2. there are 280,000 of them

If we had 280,000 simultaneous legitimate users on our forum, we would have similar problems, though in that situation we could justify spending a lot more on the infrastructure to support our userbase. Moderating such a place would be an impossible nightmare, too.

In short, to use a metaphor, the modern internet (LLM-scraping-bots) will effectively kill a forum like ours with a million tiny papercuts (if allowed).