Building machine learning models for research can be challenging, especially if you are developing them from scratch. Several factors come into play before you successfully deploy ML models for production. Some of these aspects include data collection, training data in a limited amount of time, non-representative training data, poor quality of data, irrelevant/unwanted features, overfitting the training data, underfitting the training data and offline learning, development of the model, etc.
Last year, Gartner reported that only 53 per cent of projects made it from prototype to production — and that’s at organisations with some level of AI/ML experience. On the other hand, that number is likely to be much higher, with some failure-rate estimates touching nearly 90 per cent. In the McKinsey report, out of 160 companies, 88 per cent did not progress beyond the experimental stage.
Chris Chapo, vice president, advanced analytics at Amperity, said one of the biggest reasons is that people believe by throwing money at a problem or putting technology in, success comes out the other end. “That just doesn’t happen,” said Chapo.
So, what is the secret sauce here? Is there an easy way out or a step-by-step guide to successfully deploy machine learning models?
Michael A. Lones, in a research paper titled ‘How to avoid machine learning pitfalls: a guide for academic researchers,’ explained the common mistakes that occur when using machine learning techniques and what can be done to avoid them. Lones is an associate professor in the School of Mathematical and Computer Science, Heriot-Watt University, Edinburgh.
Data is everything
Before diving into the sophisticated techniques, you probably should perform some fundamental data clearing methods on every machine learning project. Unfortunately, these are overlooked by even seasoned machine learning professionals and researchers yet are so critical that if skipped, models may break or report overly optimistic performance results/outcomes.
Lones says that it is normal to want to rush into training and evaluating models, but it is important to take the time to think about the goals of a project, to understand the data, its limitations, previous works, etc. He says, if you don’t do these things, you might end up with results that are hard to publish or models that are not suitable for their intended purpose.
So, before you start building your models, here are some things that are important to consider.
- Take time to understand your data
- Look at all your data
- Make sure you have enough data
- Talk to domain experts
- Survey the literature
- Think about how you want your model to be deployed
Building reliable models
Building models is one of the most important parts of machine learning. With modern ML frameworks, it is easy to throw all techniques at your data and see what works. This, sometimes, leads to a disorganised mess of experiments that is hard to justify and record. Lones believes that approaching model building in an organised manner is the way forward. You make sure to utilise data correctly and put adequate consideration into the choice of models. He suggests the following steps to ensure you build a reliable model:
- Do not allow test data to leak into the training process
- Try out a range of different models
- Use inappropriate models
- Optimise your model’s hyperparameters
- Be cautious where you optimise hyperparameters and select
Evaluating the robustness of models
Lones says one needs to have valid results that you can draw reliable conclusions from. Without this, it becomes difficult to assess the outcomes. He says one should think carefully about how they are going to use data in their experiments, measure the true performance of their models, and how they are going to report this performance in a meaningful way.
Here are the tips to consider while evaluating your machine learning model
- Use an appropriate test set
- Use a validation set
- Evaluate a model multiple times
- Save some data to assess your final model instance
- Do not use accuracy with imbalanced datasets
Comparing models fairly
Comparing models is one of the most common practices in AI/ML research work, but it is also challenging to get it right. Lones says that if a researcher carries out a comparison unfairly and publishes it, other researchers may subsequently be led astray. Therefore, he suggested that researchers evaluate different models within the same context, explore multiple perspectives, and utilise statistical tests appropriately.
Here are the things to consider while comparing your machine learning models:
- Do not assume a bigger number means a better model
- Use statistical tests when comparing models
- Do correct for multiple comparisons
- Don’t always believe results from community benchmarks
- Consider combinations of models
Reporting results
Lones says that the aim of academic research should not be self-aggrandising but rather an opportunity to contribute to knowledge. Therefore, one needs to provide a complete picture of their work, showcasing what worked and did not. “Machine learning is often about trade-offs, and you should try to reflect this with a nuanced and considered approach to reporting results and conclusions,” says Lones.
Here are some of the critical things to consider before reporting the results of your model:
- Be transparent
- Do report performance in multiple ways
- Do not generalise beyond the data
- Be careful when reporting statistical significance
- Do look at your models
Conclusion
Lones believes that the documents do not tell you everything you need to know, the lessons sometimes have no firm conclusions, and some of the things discussed here are debatable.
“This, I am afraid, is the nature of research. The theory of how to do ML almost always lags behind the practice; academics will always disagree about the best ways of doing things, and what we think is correct today may not be correct tomorrow,” says Lones. He said that one has to approach machine learning the same way they would approach any other research area — with an open mind, willingness to stay updated with the latest developments, and the humility to accept that you do not know everything.
