Is there any way to do this once and for all for a link? As you're crawling the database you look up the link and do this once and for all for it. So that the next time a user asks a query if it's over there you can directly return that, if it isn't then you can use more resources to crunch it for them.

Is this possible? Also, is there any way I can help you to make this? I don't expect any monetary benefit. I just want to learn.