Comment by 💀 requiem
Re: "My intermittent capsule outages are being caused by what..."
That’s the way to do it! Also can you publish which crawler it is - what IP it is from? Maybe the creator will see it here…
2024-05-13 · 1 year ago
2 Later Comments ↓
🚀 jsreed5 [OP] · 2024-05-13 at 14:24:
Good point! The crawler's IP address is 104.207.150.107.
💀 requiem · 2024-05-13 at 16:29:
Reverse DNS resolves to celery.eu.org; over HTTP it says 'unplanned maintenance', copy-pasting the IP into the browser redirects you to a rickroll. TBH I would just keep the domain in your blacklist for now.
Original Post
My intermittent capsule outages are being caused by what appears to be a very aggressive crawler. The capsule's robots.txt file tells bots not to index my CGI scripts, but this crawler is ignoring the file and sending multiple requests per second against my scripts, which overloads the server and causes it to crash. I've temporarily solved the problem by blocking the crawler entirely; I'll look for a more permanent solution.
💬 3 comments · 3 likes · 2024-05-13 · 1 year ago
Source