I don't think it's ignoring robots.txt, it's probably just cached at another thr... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		zorked on July 15, 2011 \| parent \| context \| favorite \| on: How to make fun of Google Bot (PicoLisp Wiki) I don't think it's ignoring robots.txt, it's probably just cached at another thread/machine/datacenter. It will pick up the new robots.txt eventually. This behavior is well-known.

afhof on July 15, 2011 [–]

But whats the point of it downloading the robots.txt file if it isn't going to honor it?

robtoo on July 15, 2011 | [–]

Presumably, one cluster (or whatever google calls them) is honoring it, while a different cluster isn't.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact