5 min
A key balancing act in machine learning is choosing an appropriate level of model complexity: if the model is too complex, it will fit the data used to construct the model very well but generalise poorly to unseen data (overfitting); if the complexity is too low the model won’t capture all the information in the data (underfitting).
In deep learning or machine learning scenarios, model performance depends heavily on the hyperparameter values selected. The goal of hyperparameter exploration is to search across various hyperparameter configurations to find a configuration that results in the best performance. Typically, the hyperparameter exploration process is painstakingly manual, given that the search space is vast and evaluation of each configuration can be expensive.
Source: Sigopt
The above figure illustrates a plot of hyperparameter space in three dimensions. X and Y direction give the hyperparameters, the Z-direction gives the score of the model under consideration.
There are mainly three methods to perform high dimensional non-convex optimisation. They are as follows:
Grid search a very common and often advocated approach where you lay down a grid over the space of possible hyperparameters, and evaluate at each point on the grid; the hyperparameters from the grid which had the best objective value is then used in production.
Using Python and Azure to perform tuning:
from azureml.train.hyperdrive import GridParameterSampling
param_sampling = GridParameterSampling(
{"num_hidden_layers": choice(1, 2, 3),
"batch_size": choice(16, 32)
})
Random search is performed by evaluating n uniformly random points in the hyperparameter space and select the one producing the best performance. But this method has its own disadvantages like high variance. So, a better, more intelligent alternative would be Bayesian optimisation.
from azureml.train.hyperdrive import RandomParameterSampling
param_sampling = RandomParameterSampling( {
"learning_rate": normal(10, 3),
"keep_probability": uniform(0.05, 0.1),
"batch_size": choice(16, 32, 64, 128)
})
Bayesian optimisation builds a surrogate for the objective and quantifies the uncertainty in that surrogate using a Bayesian machine learning technique, Gaussian process regression, and then uses an acquisition function defined from this surrogate to decide where to sample.
from azureml.train.hyperdrive import BayesianParameterSampling
param_sampling = BayesianParameterSampling( {
"learning_rate": uniform(0.05, 0.1),
"batch_size": choice(16, 32, 64, 128)
})
Azure Machine Learning service supports random sampling, grid sampling, and Bayesian sampling. Know more about it here
For architectures like LSTM, the learning rate and the size of the network are its prime hyperparameters.
In reinforcement learning algorithms, to measure the sensitivity of choice of hyperparameters, a larger number of data points because the performance is adequately captured with a lesser number of points due to high variance.
The efficiency of a machine learning model pivots on two factors: Speed and scores. Few applications require real-time outputs where speed is required and others require accuracy. There is always a tradeoff between these two and to get insights on how to proceed with a specific problem, hyperparameters offer decent assistance.
Read more about Grid and Random search here
Read more about Bayesian optimisation here
The Process Of Learning Parameters
It is a 4 step process:- Input data is taken
- A generalised function is obtained
- Predicting values
- Parameter learning
How Can Hyperparameters Address Issues With Model Design
Hyperparameters help answer questions like:- The depth of the decision tree
- How many trees are required in random forest
- How many layers should a neural network have
- The learning rate for gradient descent method
Setting The Right Tune
Tuning in simple words can be thought of as “searching”. What is being searched are the hyperparameter values in the hyperparameter space.
Source: Sigopt
The above figure illustrates a plot of hyperparameter space in three dimensions. X and Y direction give the hyperparameters, the Z-direction gives the score of the model under consideration.
There are mainly three methods to perform high dimensional non-convex optimisation. They are as follows:
Grid search a very common and often advocated approach where you lay down a grid over the space of possible hyperparameters, and evaluate at each point on the grid; the hyperparameters from the grid which had the best objective value is then used in production.
Using Python and Azure to perform tuning:
from azureml.train.hyperdrive import GridParameterSampling
param_sampling = GridParameterSampling(
{"num_hidden_layers": choice(1, 2, 3),
"batch_size": choice(16, 32)
})
Random search is performed by evaluating n uniformly random points in the hyperparameter space and select the one producing the best performance. But this method has its own disadvantages like high variance. So, a better, more intelligent alternative would be Bayesian optimisation.
from azureml.train.hyperdrive import RandomParameterSampling
param_sampling = RandomParameterSampling( {
"learning_rate": normal(10, 3),
"keep_probability": uniform(0.05, 0.1),
"batch_size": choice(16, 32, 64, 128)
})
Bayesian optimisation builds a surrogate for the objective and quantifies the uncertainty in that surrogate using a Bayesian machine learning technique, Gaussian process regression, and then uses an acquisition function defined from this surrogate to decide where to sample.
from azureml.train.hyperdrive import BayesianParameterSampling
param_sampling = BayesianParameterSampling( {
"learning_rate": uniform(0.05, 0.1),
"batch_size": choice(16, 32, 64, 128)
})
Azure Machine Learning service supports random sampling, grid sampling, and Bayesian sampling. Know more about it here
For architectures like LSTM, the learning rate and the size of the network are its prime hyperparameters.
In reinforcement learning algorithms, to measure the sensitivity of choice of hyperparameters, a larger number of data points because the performance is adequately captured with a lesser number of points due to high variance.
The efficiency of a machine learning model pivots on two factors: Speed and scores. Few applications require real-time outputs where speed is required and others require accuracy. There is always a tradeoff between these two and to get insights on how to proceed with a specific problem, hyperparameters offer decent assistance.
Read more about Grid and Random search here
Read more about Bayesian optimisation here- From Humanoid Robots to Telecom Trials, 12 Major Announcements at NVIDIA GTC Berlin 2026
- Anyone Can Make a Robot Move, Few Can Make It Stop
- After ISRO’s PSLV Failure, TakeMe2Space Looks to SpaceX for Reliable Launch
- Why AI Agents are the Next Frontier in Quantum Security
- Physical AI Squeezes Labour Market, And Low-Skilled Immigrant Jobs May Be First to Fall
- Rocket Math Isn’t Mathing: Why India’s Space Launches are Expensive