The risk is that it behaves like a reinforcement learning algorithm which essentially rewards itself by making you more predictable, I'd argue that's what curated social networks do today.
If you're unpredictable you're a problem. Thus, it makes sense to slowly push you to a pole so you conform to a group's preferences and are easier to predict.
A hole in my own argument is that today's networks are incentivized to do increase engagement where a neutral agent is in most ways not.
So perhaps the problem isn't just the need for agents but for a proper business model where the reward isn't eyeball time as it is today.
But you are predictable, even if you think you are unpredictable, you are just a bit more adventurous. Algorithm can capture that as well. It will be easier for algorithm that works on your behalf.
This makes me think of a talk with an AI-optimistic Microsoft sales guy I had a few years ago. His argument was essentially the same:"Look, it's no problem to have an AI curate everything for you because the algorithm will just know what you want, even if your habits are unusual!"
Of course this hasn't happened yet and I doubt it ever will. Maybe I'm just insane, but most of the recommendations from services I have fed data for hundreds of hours (YouTube) are actually repulsive.
Interesting because I think I have some rather random assortment of hobbies that generally tend to have no overlap and I get pretty good recommendations all the time.
If you're unpredictable you're a problem. Thus, it makes sense to slowly push you to a pole so you conform to a group's preferences and are easier to predict.
A hole in my own argument is that today's networks are incentivized to do increase engagement where a neutral agent is in most ways not.
So perhaps the problem isn't just the need for agents but for a proper business model where the reward isn't eyeball time as it is today.