LLMs Ride the Overconfidence Wave

Developers trying to fine-tune LLMs often encounter a plethora of issues. An experiment by Jonathan Whitaker and Jeremy Howard from fast.ai highlighted a rather unscrutinized problem with LLM models — overconfidence, which shouldn’t be confused with the widely discussed LLM hallucination.

Overconfidence is when the model insists on certain information provided in the dataset even if it is incorrect for the said question, which is possibly caused by the infamous two terms — underfitting and overfitting. 

To start with, overfitting is when a model becomes overly intricate and tailors itself too closely to the training data. And underfitting, as the name suggests, is exactly the opposite, when the model does not have enough training data to make predictions. This balance is often referred to as the bias-variance tradeoff.

To tackle these problems, developers apply several techniques, some work, and some bring up other problems. When it comes to the case of fast.ai researchers, they tried to train the model on a single example, which, to their surprise, gave out very different results than they expected.

Enter Overconfident LLMs 

When the model is given new unseen data, it can display unwarranted confidence in its predictions, despite being wrong. This is contrary to the conventional belief that neural networks typically require a multitude of examples due to the bumpy nature of loss surfaces during training.  

Imagine a language model that has been fine-tuned on a comprehensive medical dataset to diagnose diseases based on patient descriptions. When provided with cases featuring evident symptoms and clear diagnostic criteria, the model confidently assigns high probabilities to specific diseases. For instance, if a patient describes classic symptoms of the flu, the model might assign a near 1.0 probability to influenza as the diagnosis. 

However, when confronted with complex medical cases with ambiguous symptoms or multiple potential diseases, the model might distribute probabilities more evenly among different diagnostic options, indicating its uncertainty about the correct diagnosis.

Similarly, when training neural network classifiers, which are typically exposed to extensive datasets repeatedly, Howard and Whitaker noticed that even a single example of an input-output pair had a remarkable impact on these models. It was discovered that during training, the models exhibited overconfidence. As their confidence increased, they assigned close to 1.0 probability to their predictions, even if those predictions were incorrect.

This overconfidence, particularly in the early stages of training, raised concerns about how neural networks handle new information and adapt to it.

They found out that the model can learn to make accurate predictions after seeing a single example, the model essentially memorised the training data (a single example) and demonstrated a robust generalisation, making it less likely to overfit. The intention was to get the machine to learn efficiently and make reliable predictions regulating its confidence scores.

Is overfitting the cause of overconfident models?

While overfitting, the phenomenon where a model becomes too specific to the training data, is a well-known challenge in machine learning, the real problem here appears to be overconfidence. These predictions led to an unexpected result: the validation loss, which measures the model's performance on unseen data, got worse, even as the model's training loss improved.

As expected, the experiment brought up several discussions on a HackerNews thread. When the model learns the training data too well, it performs poorly on new data. The researchers of the model explained that they are not pointing out any problem, but just pointing out an opportunity if it is possible to train models with a single example. 

Interestingly, the two terms are closely related, overconfidence can be a symptom of overfitting. When a model is overfit, it learns the statistical noise in the training data, as well as the underlying patterns. This can lead to the model being overly confident in its predictions, even when those predictions are not accurate.

However, overconfidence is not always caused by overfitting. A model can also be overconfident if it is not trained on enough data, or if the data is not representative of the real world.

Lucas Beyer, researcher at GoogleAI, clarifies that these findings are specific to fine-tuning pre-trained models and don't necessarily change how models are initially pre-trained. He also pointed out that the findings are more applicable to fine-tuning scenarios and might not be as relevant to training models entirely from scratch. 

While there are other questions and critiques of this experiment, one oversight not missed by anyone is the lack of the base model or any detail of the model trained on for this experiment. It is not even clear if they used the same dataset again and again to fine-tune the model, which resulted in overfitting, and thus, overconfidence. 

---

LLMs (Large Learning Models) and Gen AI have been in the news ever since ChatGpt was introduced to the public. Generative AI is powered by very large machine learning models that are pre-trained on vast amounts of data. A lot of research is being done on these models to better understand the behavior and refine them for broader usage.  
Researchers recently noticed an unusual training pattern in fine tuning LLMs which led them to infer that the specific model was rapidly learning to recognize examples even by just seeing them once. Though this behavior is very good, it goes against the idea that slowly trained models over lengthy periods of time with varied data sets produce better results. The “overconfidence” highlighted in the paper could be due to overfitting of models. Ideally, one would want to select a model at the sweet spot between underfitting and overfitting. This is the goal, but is very difficult to do in practice. There are several techniques that could be used to mitigate the challenges that may also arise with such fast-learning models. The researchers’ “memorization hypothesis” is based on observations of behavior while fine tuning pre-trained models using specific data sets consisting of Kaggle science exam questions. This needs to be further studied with other data sets and tested to confirm their findings.  

One thing that is almost certain for any ML model is the model decay. The model will sooner or
later provide erroneous or erratic results with deteriorating value and predictability. Complex
systems that depend on multiple models are impacted more by model decay. The model with
the shortest half-life impacts the value and predictability of a multi-model system. The model
decay can be due to many different reasons. The more common reason is the data drift that
can happen when the data changes because of unforeseen reasons. Usually, the data drifts are
slow and can be verified and corrected in the model by keeping close watch on the model
quality. Often times, collecting additional data and minor recalibrations may correct for the
data drift. However, correcting for concept drift or hypothesis drift, the other reason for model
failure or decay, is extremely difficult. In any case, after some time, it makes more sense to
rebuild a new model. Rebuilding a new ML model requires a lot of data scientist efforts and
costs both time and money.
LLM models are on the other hand more extensive and expansive than traditional ML models
and require much more resources to build. The model decays for LLM models are not very well
understood yet. However, the thumbs rule is that the more complex a model is the less stable
the model can be and more susceptible to model decay it will be. It is expected that LLM
models because of their inherent complexity would decay quicker. The above article identifies
that some of the LLM models are already performing worse than how they started with. It is
difficult to speculate whether the performance degradation is due to model decay. However, it
might still be circumspect to set aside additional resources for model maintenance as people
are building more and more complex models.

---