Earlier this week, AIM released a story, ‘The Missing Link of Self-Supervised Learning’. Today, MetaAI announced the second version of data2vec, which addresses some of the challenges faced by self-supervised learning. 

data2vec 2.0 is a self-supervised algorithm that works across three different modalities—speech, vision, and text. The new and improved version can train self-supervised learning models up to 16x times faster for computer vision while maintaining similar accuracy to the most popular existing algorithm. 

This high-performance, single-purpose algorithm solves several limitations associated with self-supervised learning (SSL) models which are trained on a single input modality—such as images or text—and require a lot of computational power. 

SSL limitations 

The current scaling laws by DeepMind, state that for a given compute budget, optimal performance is achieved by equally scaling the model size (parameters) and the number of training tokens (dataset size). In this regard, while self-supervised learning achieved high-quality output without needing supervised labelled data, the amount of training data required for a single input function (text or image) was enormous. 

This proved to be a huge challenge when using self-supervised models for scaling LLMs since the issue was not only about having that amount of data but also requiring good-quality data to ensure high-quality output. But, according to some researchers, we are approaching a shortage of good-quality data, especially considering that most of the text on the internet is duplicated. 

Along similar lines—in reference to the scale at which GPT-generated content is growing on the internet—Google’s François Chollet also said that the performance of generative models (which are based on LLMs) will start to degrade as it will start training on its own output. According to him, “The dataset sizes for text models have already peaked, simply because the [Signal to Noise] S2N ratio will start declining.” 

As the researchers of the Data2vec 2.0 paper also say, “While the resulting models [of self-supervised learning] are excellent few-shot learners, the preceding self-supervised learning stage is far from efficient: for some modalities, models with hundreds of billions of parameters are trained, which often pushes the boundaries of what is computationally feasible”.

When Analytics India Magazine spoke to MetaAI’s Chief and the Guru of self-supervised learning, Yann LeCun—regarding data scarcity being the major bottleneck to the scaling of large models—emphasised that the issue is not data scarcity as much as it is data optimisation. He said, “We have more data than we can use, but we don’t know how to use it.”

LeCun referred to the example of how humans learn while explaining his case. He said that an infant is exposed to quite a small amount of language before learning how to converse with clarity. This is especially true in comparison to machines that are trained upon billions of words or fragments of text to perform well. Similarly, humans are far more efficient than current AI when it comes to different kinds of information—be it text, speech or other modalities. 

LeCun’s faith in self-supervised models was also reflected in his recent tweet, where he mentioned that the impact of self-supervised models had been much larger than he had previously predicted. 

But how exactly does MetaAI’s data2vec 2.0 solve the issue of data optimisation and ensure that the Signal to Noise (S2N) ratio doesn’t decline? 

Here’s how 

The second iteration of data2vec improves compute efficiency of self-supervised learning with a contextualised representation of data—or the layers of a neural network—instead of discrete representations of pixels of an image, the words of a text passage, or the sounds of speech. Target contextualisation captures information about the entire sample. So, for example, when it comes to text, the targets can represent different meanings based on the context in which the word appears. 

While contextualised representation with a separate teacher model should require additional training and computation, the researchers show that they brought significant efficiency improvements resulting in the model eliciting richer learning tasks and faster learning. 

The efficiency improvements to the original algorithm were made possible in several ways: 

  • Firstly, the target representations built for a particular training example are reused for masked versions, i.e., where different parts of the training example are hidden. Each version is then fed into the student model, which predicts the same contextualised target representation for different masked versions. The researchers say that this will effectively write off the computation required to create target representations.
  • Secondly, the student encoder network is not run for the blanked-out portions of the training examples (in their case, around 80% of an image), significantly reducing the required compute time. 
  • Finally, the researchers employ a more efficient decoder model that uses a multilayer convolutional network rather than Transformer networks.

In this way, the data2vec 2.0 algorithm uses efficient data encoding, a fast convolutional decoder and reusing target representations for multiple versions of each sample (as seen in the image above). Thus, for different input modalities, the algorithm trains separate models for each using the Transformer architecture with different feature encoders.  

Findings of the paper

The results showed that the data2vec 2.0 algorithm offers a much superior speed–accuracy trade-off to the previous versions of the model across all three modalities. Further, it also showed that training with considerably fewer number of epochs and batch size is possible because multi-masking extracts more learning signals from each training sample, thereby upholding the Signal to Noise (S2N) ratio of the training data. 

The post Meta’s Answer to the Missing Link of Self-Supervised Learning appeared first on Analytics India Magazine.