I think that velocity is the fascinating part. I remember being intrigued while reading "Influx" by Daniel Suarez (great tin foil hat thriller that is hitting scarily close to home these days) about the training of the various AIs that were used by the government.
I wonder what happens once we grow an AI whose purpose is to excel at training other AI's. Will we see an exponential leap forward in the speed with which they are trained?
I recommend watching the previous video first, which introduces the problem of strong-AI rapidly finding solutions (especially with poorly-specified questions) that may not be in humanity's best interest.
Before anybody jumps to overly-sensational conclusions, note that the last video in that series, Rob Miles explains how exponential self-improvement is an extreme point in the space of possible AI development. We don't know how to predict discoveries[1], so we need more research into AI, so we can hopefully make something that isn't exponentially growing beyond our understanding.
Finding solutions doesn't imply that those solutions are implemented. I find a dozen solutions that are not in humanities interest every day. Does that put anybody at risk? Maybe if you were stupid enough to make me Dictator of the World or something. And you certainly shouldn't do that until I convincingly demonstrate that I am very very committed to finding solutions that are in humanity's best interests.
I'm baffled at how easily people assuming a computer thinking something means it's going to happen. There are a trillion pieces that have to fall in place for that to happen accidentally, and if it doesn't happen accidentally, your problem is a human social problem, not an AI problem.
And I know what someone is going to think: "The AI might be smart enough to figure out how to make those trillion pieces fall in place." But then, who cares? So what if it figures out how to do something. It still has to be done. And we're the ones who have to do it.
The AI could come up with ideas that benefit the people that enact them in the short term. If the AI can earn its own money, then it's not that hard for it to use that money to pay people.
For an extreme example, you could imagine an AI getting rich from the stock market (or mechanical turk, etc), then buying up ridiculous amounts of land for paperclip factories and paying workers. The people that want to feed their families or get rich from selling their factories are the ones who will enact the AI's plans. How many conscientious objectors do you expect?
The "rich AI" problem didn't seem feasible to me until I realized how powerful and flexible bitcoin is. Now add the power of contracts through ethereum and a machine could actually harness a significant amount of leverage over human actors. With traditional contracts, enforcement would have left the power in government hands, and with traditional banking as well. We've now stepped into an era where we might literally have built the bat and shovel AI will use to get humanity into its grave. /panic
How many objections would there be a machine collecting a significant amount of money in the stock market and redistributing it among humans? How is that different from what is happening today? We already have rich, selfish people. And they already pay people to get their way. And plenty of people object to it.
I was alluding to the idea of a paperclip maximizer AI[1], which over time redirects increasing amounts of resources to making useless paperclips. Following the thought experiment further, it continually buys more factories for the purpose of building paperclips or technologies specifically for building paperclips (including improving itself). It probably does some charity in order to be seen as benevolent by people while the people are still in control. Soon many countries are doing nothing but building paperclips and making the minimum necessities to feed their workers. Every other human endeavor is decreasingly profitable as the AI orchestrates the markets to optimize for paperclips. When the AI reaches enough automation and humans are no longer useful or a threat, it drops all benevolent pretenses and replaces all of its human workers, leaving them to fend for themselves while it owns and defends all of the planet's resources.
In the extreme case, the AI could become a super human engineer. Design working nanotech. Pay or trick some humans into making it. Then take over the world in a grey goo like fashion.
Of course you might be skeptical that is even possible. So there is always slower world take over paths. It could slowly earn tons of money, hack into the world's computer systems, trick and persuade humans winning social influence, design superior macroscale robots, etc.
The only important part is that the AI be far smarter than humans. Which seems inevitable to me, since it can rewrite and improve its own code, and run on giant computers that are far faster than human brains. If it isn't smarter than us at first, it will be eventually. Unless you really believe that humans are close to optimal intelligence.
At every step of this process this machine will be under intense human scrutiny, and we'll be constantly asking it to meet our demands, and if it ever fails to do so we will replace it with one that does.
That is the environment in which such an AI would be trying to evolve. And thus it will evolve into a faithful servant, because nothing else will survive.
And even if it were secretly developing plans, we'd be able to see how wasteful those plans end up being, and we'd purge them. We kill those processes that run functions that we don't see the value in. This is state-of-the-art design. You don't get state-of-the-art by having your back turned.
Mor importantly: you're glossing over "self-improvement." How does the computer know what an "improvement" is? We tell it what an improvement is. And an improvement will be "it is better at meeting our needs," not "it is better at being secretive and conniving and getting it's own way." In fact, "getting it's own way" is very obviously a bug, and if it happened, you'd have a useless program. One that wastes precious CPU cycles on who-knows-what, and you'd prefer it spent that effort doing what you want, rather than planning for what you don't.
We're not going to invest trillions into building some super AI and completely forget about it. After we give it control of all of our natural resource harvesting and infrastructure.
No, what you're talking about is a deity. If you invent a deity, then my thoughts on the matter don't really apply, since I'm not a deity. But that's no worse than advanced aliens landing on Earth, and just about as likely to happen.
I wonder what happens once we grow an AI whose purpose is to excel at training other AI's. Will we see an exponential leap forward in the speed with which they are trained?