The efficiency of ML models and the algorithms they run on has historically been attributed to the presence of more data or big data. Google’s Research Director Peter Norvig has gone on to claim that “We don’t have better algorithms. We just have more data.” Established data scientists have consistently advocated the supremacy and sovereignty of data in the efficient running of ML models. However, new research has debunked such “misinterpretations”. According to recent findings, ML models have improved algorithmically and run more efficiently without any increase in compute.
Cases of Algorithmic Efficiency
OpenAI, in a report in 2020, has highlighted certain interesting cases where ML models have run efficiently without any increase in computing. Within translation, the Transformer22 surpassed seq2seq23 performance on English to French translation on WMT’14 with 61x less training compute three years later. According to OpenAI, AlphaZero24 took 8x less compute to get to AlphaGoZero25 level performance one year later. OpenAI Five Rerun required 5x less training compute to surpass OpenAI Five26 (which beat the world champions, OG) 3 months later.
Moreover, according to John Wentsworth, a blogger, rewriting code in C and using Big O speedups can improve the efficiency of an algorithm and run a code speedily. In this regard, Wentsworth also highlighted how Facebook sped up its algorithm by 250k. They did so by figuring out how close people were to various friends by looking at their mutual friend graph and using Big O speedups.
In a recent study by the University of Waterloo, it was found that it is possible for the algorithm to learn to differentiate between different objects even when there is not enough data for certain classes. This process has been termed “less-than-one-shot learning”. By using carefully designed soft-label examples, ML developers have successfully taught the algorithm with fewer data.
According to OpenAI, “It can be helpful to think of compute in 2012 not being equal to compute in 2019 in a similar way that dollars need to be inflation-adjusted over time. A fixed amount of compute could accomplish more in 2019 than in 2012.” Now that we have piqued your interest in increasing ML algorithm efficiency without increasing compute, let’s take a look at the various steps and methods one can adopt to improve algorithms.
Calculate math accurately
Using raw Python to process large datasets can be a long and tedious process. But, according to Maksym Balatsko, Machine Learning Developer, using built-in math functions from NumPy helps optimise algorithmic functions and allows the code to run as fast as possible.
Featured Engineering
This method helps extract novel information from existing data. Feature engineering heavily relies on hypotheses generation. A good hypothesis results in good features. For this, one must adopt feature transformation and feature creation. The main goal of featured engineering is to prepare the proper input dataset compatible with the machine learning algorithm requirements and improve the performance of machine learning models.
Big O Speedups
Big O notation is used as a measurement unit that helps programmers evaluate or estimate the efficiency of an algorithm. For example, a big O notation is used to evaluate how quickly the runtime grows relative to the input data processed by that algorithm.
MapReduce
MapReduce functions as a programming model that transforms the input into output. This step is repeated through every step. In the sequential step, the reduce function aggregates all output of the map function.
In conclusion, it is pertinent to remember that better quality data and algorithms are more important and efficient than large amounts of data and an increase in compute.
