Wow didn't realize it has comes to this. I guess the AI bots are ignoring the ro... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		zenmac 4 months ago \| parent \| context \| favorite \| on: Schleswig-Holstein completes migration to open sou... Wow didn't realize it has comes to this. I guess the AI bots are ignoring the robots.txt now days huh, so just removing the index seems enough to stop the AI scrappers?

cyberax 4 months ago [–]

Yep. I have my own small gitweb server, and I was getting literally thousands requests per second from bots who try to crawl through all the links.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact