Machine learning is now becoming quite prominent in IT. This has not just raised concerns on its cost, efficiency and ethics, majorly the computation costs. These include not just monetary, but the harm to the environment. Carbon footprint and greenhouse gas emissions are not just buzz words anymore when it comes to ML training.

There have been many research papers since 2018 showing environmental concerns over the computational requirements for storing vast amounts of data for ML training. But, going against the tide is Google’s paper – “The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink.” The paper focuses on operational carbon emissions, which includes the energy cost of operating ML hardware. The paper puts into consideration the data centre overheads from the training of natural language processing (NLP) models. 

Yann LeCun, Meta’s VP and Chief AI Scientist, also supported the report on Twitter and called the claims of ‘AI is an environmental disaster in the making- ‘total BS.’

He also supported the report in the comments that claimed that Google could not be trusted. He said, “I would totally trust David Patterson, who is the author: prof at Berkeley, distinguished engineer at Google, and Turing Award Laureate.”

Lets’s look at the paper.

Google’s Views

The paper claims that even though ML workloads have grown rapidly in the last decade, and the number of computations per training run has also increased by orders of magnitude, technology improvements have compensated for this increased load. To prove this, the paper compares GPT-3 and GLaM, where they consider the parameters, accelerator years of computation, energy consumption, and gross CO2e.

The team also performed calculations on data for one week of April in 2019, 2020, and 2021. They released that each time the ML portion was 10% to 15% of Google’s total energy consumption for that week, despite ML representing 70%-80% of the floating point operations (FLOPS) at Google. The paper claims that while ML usage increased during these three years, algorithmic and hardware improvements kept that growth to a rate comparable with overall energy growth at Google. 

When it comes to data centre energy consumption, the paper states that global data centre energy consumption increased by only 6% from 2010 to 2018. This happened when the data centre computing capacity increased by 550%. This is because of the shift from conventional data centres to cloud data centres, as it allows the same workloads to be served with less hardware. 

Most cloud companies also compensate partially for their carbon emissions. Google (since 2017) and Facebook (since 2020) have purchased renewable energy annually to match 100% of their usage. Microsoft, too, has similar goals for 2025. 

Conflicting research

But can this be believed? According to a study called “Energy and Policy Considerations for Deep Learning in NLP,” by the University of Massachusetts Amherst, the energy demands of models are high, and it is a point of concern. 

The team estimated that it is a must to cut carbon emissions by half over the next decade to deter escalating rates of natural disasters.

The study noticed that AI language-processing system can generate 1,400 pounds of emission and can even reach up to 78,000 pounds, depending on the scale of the AI experiment.

Solutions

Whether Google is right or wrong will be seen in the coming years. It is essential to focus on reducing the carbon footprint. A team of researchers from Stanford, Facebook AI Research, and McGill University realised the issue in 2020 and came up with an easy-to-use tool that can measure how much electricity an ML project will use and what that means in carbon emissions. They believe that the AI developers and companies must know how much their ML experiments are spewing and how the volumes could be reduced.

Google’s David Patterson, a distinguished engineer, Google Research, Brain Team, has also suggested best practices to reduce energy and carbon footprints. He calls it the 4Ms. It includes Model – which requires selecting efficient ML model architectures. It can advance ML quality and reduce computation by 3x–10x. The second is Machine – which requires using processors and systems optimised for ML training rather than general-purpose processors. The third is Mechanisation – Computing in the cloud rather than on-premise reduces energy usage and, therefore, emissions by 1.4x–2x. The last is Map Optimisation – The cloud lets customers pick the location with the cleanest energy, further reducing the gross carbon footprint by 5x–10x. 

Patterson believes that these four practices together can reduce energy by 100x and emissions by 1000x.