The big question with all this stuff to me is whether we've just figured out a couple of new tricks (primarily around neural nets processing 2D data and word sequences for translation) and are now going to hit a new plateau in machine learning- or whether "this time it's different" and we're going to similar improvements year after year for the next decade or more...
If the only major impact of current deep learning methods on culture was to squeeze out an additional 1-5% of performance on every task set in front of it, the fact that it has made large-scale speech recognition and image recognition good enough for public use, it would be enough to call it substantial progress.
But I think one of the major successful things of the deep learning renaissance has been the ability to embed different 'worlds' into the same vector space (French and English, text and music, images and text, etc.). By co-locating these different worlds, we gain the ability to perform more accurate search, to create images from text like in the "Generative Adversarial Text to Image Synthesis" paper, and a wide variety of other multi-modal tasks. We can multi-embed almost anything, even things that are nonsensical. You want to make a music-to-font translation system? Or a sneaker-to-handwriting generator? Gather the training data, and the world is now your oyster. The impact of deep learning as differentiable task pipelines has only begun to scratch the surface of what is possible.
It may be different for other reasons but the main difference I note today is the number of opensource AI/ML platforms that are trivial to install, use, play, experiment at pretty much near the peak computing capacity of the hardware we use today. Exploring the vast search space of reality has never been easier and faster than today.
Probably true for the "internet scale" applications. But my impression is that there's still lots of medium-sized opportunities to build interesting things around the edges, in fields outside of the mobile/PC ecosystem.
You could say the same about smartphones. Mobile computing was a couple of new tricks and features that only really became powerful when people had computers in their pockets (like GPS), and for the most part the real discoveries have already been made, but it's all a step in the process and there's still a ton of applications present-day ML can do.
We haven't created true AI yet, but that's ok. We can make things better than before as we work on an even more advanced future.
Beyond that, as an outsider looking in I may be off base, but the rise of machine learning seems mostly fueled by the rise of GPGPU. So I would say it's not different, in that we experienced something similar with regular CPUs until about 2007. But it may be different depending upon how much, and how long, we can continue to cheaply speed up GPUs.
It's the rise of cheap computation and the availability of enormous data sets.
It's a bit of exaggeration to say so, so don't take it literally but in many ways the NN work is where it was when Papert and Minsky wrote the "Perceptrons" book in 1969. But as Stalin is supposed to have said, "quantity has a quality all its own" (thus the immense quantity of cycles and data people have at their fingertips causes a discontinuous change in functionality). Don't over-read this; I don't mean to denigrate a lot of cool work done in the past few years. But conceptually you are correct on the computation side.
This is fair, and one should certainly acknowledge the progress we have made as a functionality all its own, but there seems to be something else at work in the deep learning paradigm. I think (ha) we are dipping our toes in the water of intelligence replication, and that deep learning is a very real crystallization of hundreds of years (using Decartes as my x-axis) of scientifically accurate data amalgamation on what it is we are doing when we maintain a thought, so while the Perceptron is definitely the architecture for passing from the realm of information to the realm of data, what fascinates me is the proximity to actual human reasoning that is occurring presently within the 'mind' of, say, Watson, which I have to say is an incredibly providential name given IBM's founder, Sherlock's best bud, and James Watt's influence on electricity.
It's almost as if intelligent design isn't all that far fetched....lol
If that is the case, if the limiting factor was hardware, then we should see continuing growth for a good stretch going forward. There are quite a few companies looking to build specialized chips for machine learning, that could outcompete GPUs the way GPUs outcompete CPUs in this field.
>> are now going to hit a new plateau in machine learning- or whether "this time it's different"
I think it is actually both, yes we are going to hit a new plateau and yes this time it's different.
It is different not because we have found something profoundly new, but because we are able to quickly, easily and successfully experiment with huge (deep), new neural network architectures and learning methodologies.
This has become possible because a combination of factors that have come together towards the end of the 2000’s: e.g. much more computation power (GPGPUs), much more data available online, "simple" insigths such as progressive training of deep nets by stacking (auto encoder) layers, "Hey! Stochastic Gradient Descent works quite well actually!", Drop-out to improve generalisation capabilities, etc..
The great open source libraries such as TensorFlow, Theano and others make it even easier to do experiments. A framework like Keras even abstracts TensorFlow & Theano so you don’t have to worry what is used as deep learning framework.
So we shifted to a much higher gear when it comes to machine learning research, and this will be like this for a while. Computing capabilities keep expanding: GPGPUs become ever faster for Deep Learning, but also Intel has the Xeon Phi Knights Landing with 72 cores and upcoming variants with Deep Learning specific instructions (Knights Mill).
On the other hand we will definitely hit a plateau:
1) To make truly intelligent systems, we need to encode a lot of knowledge; knowledge that is common to us, but not at all to machines.
Bootstrapping a general AI with human-like intelligence, will prove very difficult. I think such AIs need to develop just like children acquire knowledge and cognitively develop. The type of problems we encounter to achieve this are of a whole other type, for one, just imagine how much time this will take before we get this right!
Imagine an AI that learns for a few years but fails to improve, can we reuse what it has learned in a new and better version of the AI? Will we capture all of its experiences to relearn a new version from scratch?
2) Apparently the human brain has a 100 billion neurons and trillions of connections, AlexNet (2012) has 650,000 neurons and 60 million parameters. We have grown the networks considerably since then, but compared to the complexity of the brain we have a (very, very) long way to go.
3) FLOPS/Watt : this is going to play an ever growing role in the success of AI. Our brain is incredibly efficient when it comes to energy use. We shouldn’t need a power plant for every robot we deliver to customers, right?
The former, if all things remain the same in terms of computing power and availability of data. I think betting against improvement in either of those areas is a bad bet so I'm optimistic we'll continue to see really interesting things in the near future
if all things remain the same in terms of computing power and availability of data
Right, but that's exactly the thing that is changing. All the promises of "Big data" can now actually be realized. The trick is that for the data sources that were set up long ago, is cleaning them and making sure they are structured correctly.
