Self-supervised learning is the future of AI

Despite the huge contributions of deep learning to the field of artificial intelligence, there’s something very wrong with it: It requires huge amounts of data. This is one thing that both the pioneers and critics of deep learning agree on. In fact, deep learning didn’t emerge as the leading AI technique until a few years ago because of the limited availability of useful data and the shortage of computing power to process that data.

Reducing the data-dependency of deep learning is currently among the top priorities of AI researchers.

In his keynote speech at the AAAI conference, computer scientist Yann LeCun discussed the limits of current deep learning techniques and presented the blueprint for “self-supervised learning,” his roadmap to solve deep learning’s data problem. LeCun is one of the godfathers of deep learning and the inventor of convolutional neural networks (CNN), one of the key elements that have spurred a revolution in artificial intelligence in the past decade.

Self-supervised learning is one of several plans to create data-efficient artificial intelligence systems. At this point, it’s really hard to predict which technique will succeed in creating the next AI revolution (or if we’ll end up adopting a totally different strategy). But here’s what we know about LeCun’s masterplan.

A clarification on the limits of deep learning

First, LeCun clarified that what is often referred to as the limitations of deep learning is, in fact, a limit of supervised learning. Supervised learning is the category of machine learning algorithms that require annotated training data. For instance, if you want to create an image classification model, you must train it on a vast number of images that have been labeled with their proper class.

“[Deep learning] is not supervised learning. It’s not just neural networks. It’s basically the idea of building a system by assembling parameterized modules into a computation graph,” LeCun said in his AAAI speech. “You don’t directly program the system. You define the architecture and you adjust those parameters. There can be billions.”

Deep learning can be applied to different learning paradigms, LeCun added, including supervised learning, reinforcement learning, as well as unsupervised or self-supervised learning.

But the confusion surrounding deep learning and supervised learning is not without reason. For the moment, the majority of deep learning algorithms that have found their way into practical applications are based on supervised learning models, which says a lot about the current shortcomings of AI systems. Image classifiers, facial recognition systems, speech recognition systems, and many of the other AI applications we use every day have been trained on millions of labeled examples.

Reinforcement learning and unsupervised learning, the other categories of learning algorithms, have so far found very limited applications.

Where does deep learning stand today?

random vectors

Supervised deep learning has given us plenty of very useful applications, especially in fields such as computer vision and some areas of natural language processing. Deep learning is playing an increasingly important role in sensitive applications, such as cancer detection. It is also proving to be extremely useful in areas where the scale of the problem is beyond being addressed with human efforts, such as—with some caveats—reviewing the huge amount of content being posted on social media every day.

“If you take deep learning from Facebook, Instagram, YouTube, etc., those companies crumble,” LeCun says. “They are completely built around it.”

But as mentioned, supervised learning is only applicable where there’s enough quality data and the data can capture the entirety of possible scenarios. As soon as trained deep learning models face novel examples that differ from their training examples, they start to behave in unpredictable ways. In some cases, showing an object from a slightly different angle might be enough to confound a neural network into mistaking it with something else.

ImageNet images vs ObjectNet images