Machine learning is an exciting area of research and development. ML tools are important in many industries and science fields. ML research is also very tricky and has several challenges. If not addressed suitably, these challenges can lead the project in the wrong direction.
Micheal A Lones, Associate Professor in the School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh, recently produced a paper on the common mistakes that occur when using machine learning techniques and outlining a few techniques to dodge them. This guide is especially useful for students. This paper covers five stages of the machine learning process: what to do before building the model, reliably building the model, model evaluation, fairly comparing models, and reporting the results.
Before starting to build the model
First, the data has to be collected from a reliable source and through reliable technology. Second, one should also make sure that they have enough data, which is a prerequisite to training a model that generalises.
Once the good quality and sufficient data have been collected, the researcher should avoid making untestable assumptions. This means that the developer should avoid looking at any test data too closely in the initial exploratory analysis stage as it may limit the generality of the model.
Talking to domain experts should be considered an important part of the preparation. They can help one understand which problem to solve, the most appropriate feature set and ML model to use, and help in publishing to the most appropriate audience. Apart from this, it is important to do thorough literature surveys to understand what has and hasn’t been done previously.
At last, if the eventual goal of the project is to produce an ML model that would be used in the real-world, then it is worth thinking about how it is going to be deployed.
Building models reliably
With modern ML frameworks, it is easy to test different approaches to building models and see what works. However, this can lead to disorganisation. It is important to approach model building in an organised manner.
Generally speaking, there is no single best ML model. No ML approach is better than any other when considering a range of possible problems. The job of the researcher is to find the ML model that works best for a given problem.
Researchers should also make sure not to use inappropriate models. When the barrier to implementation is lowered, modern ML libraries make it easy to apply inappropriate models to the data.
Further, it is better to use a hyperparameter optimisation strategy, which may include random and grid search, or use tools that intelligently search for optimal configurations.
Lastly, one should avoid leakage of test data into the training process. In case of such leakage, the data no longer provides a reliable measure of generality. This is often the reason why published ML models fail to generalise to real-world data.
Robustly evaluate model
Measuring the true performance of an ML model is a way to go about if one wishes to contribute to the progress in the field genuinely. In order to do so, a researcher would need to have valid results to draw reliable conclusions from.
Firstly, a researcher must always use a test to measure the generality of the ML model. One must ensure that the data in the test set is appropriate and should not overlap with the training set, apart from being representative of the wider population.
While it is not usual to train multiple models in succession and use previously gained knowledge to guide the configuration of the next, it is important not to use the test set within this process. Researchers could instead use a separate validation set to measure performance.
To get a reliable estimate of the model instance’s generality, researchers may use another test set. If there is enough data, it is wise to keep some aside and use it once to provide an unbiased estimate of the final selected model instance.
Fair comparison of models
Model comparison is the basis of academic research but it isn't easy to get right. An incorrect and unfair comparison may lead to confusion and misleading reports. Researchers must evaluate different models within the same context, explore multiple perspectives, and correctly use statistical tests.
Following steps could be followed to avoid unfair comparisons:
- Suspend the belief that bigger numbers imply better models
- Use statistical tests when comparing models
- Exercise caution when considering results from community benchmarks
- Consider combinations of model
Reporting the results
Author Lones writes that the aim of academic research should be seen as an opportunity to contribute to knowledge rather than used for self-aggrandisement. To effectively contribute to knowledge, the researchers must provide a complete picture of their work that covers both what worked and what did not. Since it is rare that one model is better than another in all aspects, a researcher must try to reflect this with a nuanced approach to reporting results and conclusions.
Read the full paper here.
Your reaction
Discussion
What's your take on this story?
No comments yet — be the first to share your view.