Fantastic work by Xe here -- not the first but this seems like the most traction I've seen on a PoW anti-scraper project (with an MIT license to boot!).
PoW anti-scraper tools are a good first step, but why don't we just jump straight to the endgame? We're drawing closer to a point where information's value is actually fully realized -- people will stop sharing knowledge for free. It doesn't have to be that way, but it does in a world where people are pressed for economic means, knowledge becomes an obvious thing to convert to capital and attempt to extract rent on.
The simple way this happens is just a login wall -- for every website. It doesn't have to be a paid login wall of course (at first), but it's a super simple way to both legally and practically protect from scrapers.
I think high quality knowledge, source code (which is basically executable knowledge), being open in general is a miracle/luxury of functioning, economically balanced societies where people feel driven (for many possible reasons) to give back, or have time to think of more than surviving.
Don't get me wrong -- the doomer angle is almost always wrong -- every year humanity is almost always better off than we were the previous year on many important metrics, but it's getting harder to see a world where we cartwheel through another technological transformation that this time could possibly impact large percentages of the working population.
> people will stop sharing knowledge for free. It doesn't have to be that way
Yeah. People over-estimate the flashy threats from AI, but to me the more significant threat is killing the open exchange of knowledge and more generally the open, trusting society by flooding it with agents which are happy to press "defect" on the prisoner's dilemma.
> being open in general is a miracle/luxury of functioning, economically balanced societies where people feel driven (for many possible reasons) to give back, or have time to think of more than surviving
"High trust society". Something that took the West a very long time to construct through social practices, was hugely beneficial for economic growth, but is vulnerable to defectors. Think of it like a rainforest: a resource which can be burned down to increase quarterly profit.
> Yeah. People over-estimate the flashy threats from AI, but to me the more significant threat is killing the open exchange of knowledge and more generally the open, trusting society by flooding it with agents which are happy to press "defect" on the prisoner's dilemma.
I don't think societies are open/trusting by default -- it takes work and a lot of anti-intuitive thinking, sustained over long periods of time.
> "High trust society". Something that took the West a very long time to construct through social practices, was hugely beneficial for economic growth, but is vulnerable to defectors. Think of it like a rainforest: a resource which can be burned down to increase quarterly profit.
I think the trust is downstream of the safety (and importantly "economic safety", if we can call it that). Everyone trusts more when they're not feeling threatened. People "defect" from cultures that don't work for them -- people leave the culture they like and go to another one usually because of some manifestation of danger.
Maybe the dream of knowledge being free and open was always doomed to fail; if knowledge has value, and people are encouraged to spend more of their time and energy to create it rather than other kinds of work, they will have to be compensated in order to do it increasingly well.
It's kinda sad though, if you grew up in a world where you could actually discover stuff organically and through search.
I don't think it's inevitable doom, but a realignment of incentives will probably be needed. Perhaps in the form of payment. In several EU countries it's illegal to have any internet connection without linking it to your ID card or passport in some central database, so that could also be thing - people are generally reluctant to get arrested.
I think we only have two choices here:
1) every webpage requires Facebook login, and then Facebook offers free hosting for the content.
2) every webpage requires some other method of login, but not locked into a single system.
I read the GP comment as suggesting we push on the second option there rather than passively waiting for the first option.
You were right on the second one! Facebook wasn't even a thought in my mind per say (they're not unique in that every social network wants to build a walled garden).
My focus was more on the areas outside the large walled gardens -- they might be come a bunch of smaller... fenced backyards, to put it nicely.
IP addresses are not anonymous. Have you tried to make your IP address anonymous, e.g., with Tor or one of those NordVPN-like companies? (not picking on Nord, though they deserve to be picked on - they're just the most advertised.)
You'll find CAPTCHAs almost everywhere, outright 403s or dropped connections in a lot of places. Even Google won't serve you sometimes.
The reason you're not seeing that situation right now is that your IP address is identifiable.
I see captchas all the time on my home internet connection without a VPN these days. That era seems to be ending, probably because AI scraping is now using residential IP blocks.
There's been talk on NANOG about whole residential ISPs getting marked as VPNs now. Turns out selling excessive security to businesses is easy, I guess. Like CrowdStrike.
anonymity and the open web are different things, and neither of them were promised/guaranteed to anyone on the internet.
For people that value anonymity, they'll create their own spaces. People that value openness will continue to be open.
What we're about to find out is what happens when the tide goes out and people show you what they really believe/want -- anything other than that is a form of social control, whether via browbeating or other means.
> anonymity and the open web are different things, and neither of them were promised/guaranteed to anyone on the internet. > > For people that value anonymity, they'll create their own spaces. People that value openness will continue to be open
Hardly anything of what's the internet today was promised, but who are you to decide what the internet has to become now, and that people with different ideas need to confine themselves in their own ghettos?
Everyone values privacy, it's just out of social pressure if most give up so much of it.
> What we're about to find out is what happens when the tide goes out and people show you what they really believe/want -- anything other than that is a form of social control, whether via browbeating or other means
Thanks! I'm gonna try and bootstrap this into a company. My product goal for the immediate future is unbranded Anubis (already implemented) with a longer term goal of being a Canadian-run Cloudflare competitor.
I mean, LLM scrapers set fire to the commons, and when you do that, now you have a flaming hole in the ground where the commons used to be. It's not the fault of website operaters who have to act in self-defense lest their site get DDoSed out of existence.
PoW anti-scraper tools are a good first step, but why don't we just jump straight to the endgame? We're drawing closer to a point where information's value is actually fully realized -- people will stop sharing knowledge for free. It doesn't have to be that way, but it does in a world where people are pressed for economic means, knowledge becomes an obvious thing to convert to capital and attempt to extract rent on.
The simple way this happens is just a login wall -- for every website. It doesn't have to be a paid login wall of course (at first), but it's a super simple way to both legally and practically protect from scrapers.
I think high quality knowledge, source code (which is basically executable knowledge), being open in general is a miracle/luxury of functioning, economically balanced societies where people feel driven (for many possible reasons) to give back, or have time to think of more than surviving.
Don't get me wrong -- the doomer angle is almost always wrong -- every year humanity is almost always better off than we were the previous year on many important metrics, but it's getting harder to see a world where we cartwheel through another technological transformation that this time could possibly impact large percentages of the working population.