Intel co-founder Gordon Moore, in 1965, said that the number of components per integrated circuit would double every year; he predicted that such rate of growth would continue for at least another decade. He extrapolated that computing power would increase dramatically at a relative decrease in cost. Also called the Moore’s Law, this became the golden rule for the electronics industry. That said, Moore’s Law has been debunked multiple times. 

In a recent paper titled “Compute Trends Across Three Eras of Machine Learning”, the authors have studied the compute, data, and algorithmic advances that eventually guide the progress of machine learning. They demonstrated that before 2010, the training compute grew according to Moore’s law, doubling every 20 months. However, with the advent of deep learning, the scaling of training compute has been doubling every six months.

By late 2015, firms started developing large scale machine learning models with 10 to 100-fold larger requirements in training compute. Based on these advancements, the authors of this research categorise the history of compute in machine learning into three eras – the Pre Deep Learning Era, the Deep Learning Era and the Large-Scale Era. 

Trends in the three eras of machine learning

The progress in the machine learning domain has a significant impact on the overall advancement of industry, policy and society; compute forms a major component in understanding this progress. There is a relationship between AI capabilities and the way machine learning models scale, as also demonstrated by various researchers.

In this study, the author curated a dataset of 123 milestone ML systems and annotated them with the compute that it took to train them. They then categorised the trends into three eras (pre-deep learning, deep learning, and large-scale) and calculated the estimates of the doubling times in each of them. The authors finally checked the results against a series of appendices.

The researchers found that there was a major difference in the trends before and after the advent of deep learning. Before deep learning, the amount of compute required to train a machine learning system doubled every 17-29 months. The trend in the pre-deep learning era was found to be roughly according to Moore’s Law. 

The starting of the deep learning era is not clearly marked as there is no noticeable discontinuity in the transition between the two eras. 

Data further showed that the new trend of large scale models emerged in 2015-16, especially with the introduction of AlphaGo. Models like these were trained by large corporations with high training budgets. The trend of increasing compute is slower, which doubled every 9 to 10 months. The authors also observed that the trend of regular-scale models was not perturbed.

“In particular, we identify an 18-month doubling time between 1952 and 2010, a 6-month doubling time between 2010 and 2022, and a new trend of large-scale models between late 2015 and 2022, which started 2 to 3 orders of magnitude over the previous trend and displayed a 10-month doubling time,” the authors write.

Read the full paper here.

Why is this study important

Modern AI is computationally intensive. As mentioned earlier, it was the bigger, richer firms with higher training budgets who had the actual advantage in modern AI research. A 2020 study named ‘The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research’ demonstrated the increased presence of big firms and elite universities in AI research.

The authors also showed evidence that the unanticipated rise of deep learning has resulted in compute divide between large firms and non-elite universities. This compute divide raises questions about the fairness of AI technology on the whole. The results also suggest that the lack of fair access to specialised equipment like compute could lead to ‘de-democratisation’ of knowledge production.