Introduced a few years ago by Tianqi Chen and his team of researchers at the University of Washington, eXtreme Gradient Boosting or XGBoost is a popular and efficient gradient boosting method. XGBoost is an optimised distributed gradient boosting library, which is highly efficient, flexible and portable.
The method is used for supervised learning problems and has been widely applied by data scientists to get optimised results for various machine learning challenges. It implements ML algorithms under the Gradient Boosting framework and helps in solving data science problems in a fast and accurate manner.
Here are the top ten interview questions on XGBoost that Data Scientists must know.
1| Is XGBoost faster than random forest?
Solution: XGBoost is usually used to train gradient-boosted decision trees (GBDT) and other gradient boosted models. Random forests also use the same model representation and inference as gradient-boosted decision trees, but it is a different training algorithm. XGBoost can be used to train a standalone random forest. Also, random forest can be used as a base model for gradient boosting techniques.
Further, random forest is an improvement over bagging that helps in reducing the variance. Random forest builds trees in parallel, while in boosting, trees are built sequentially. Meaning, each of the trees is grown using information from previously grown trees, unlike bagging, where multiple copies of original training data are created and fit separate decision tree on each. This is the reason why XGBoost generally performs better than random forest.
Know more here.
2| What are the advantages and disadvantages of XGBoost?
Advantages:
- XGB consists of a number of hyper-parameters that can be tuned — a primary advantage over gradient boosting machines.
- XGBoost has an in-built capability to handle missing values.
- It provides various intuitive features, such as parallelisation, distributed computing, cache optimisation, and more.
Disadvantages:
- Like any other boosting method, XGB is sensitive to outliers.
- Unlike LightGBM, in XGB, one has to manually create dummy variable/ label encoding for categorical features before feeding them into the models.
Know more here.
3| How XGBoost Works?
Solution: When using gradient boosting for regression, where the weak learners are considered to be regression trees, each of the regression trees maps an input data point to one of its leaves that includes a continuous score. XGB minimises a regularised objective function that merges a convex loss function, which is based on the variation between the target outputs and the predicted outputs. The training then proceeds iteratively, adding new trees with the capability to predict the residuals as well as errors of prior trees that are then coupled with the previous trees to make the final prediction.
Click here to learn the step by step process of how XGB works.
4| What does the weight of XGB leaf nodes mean? How to calculate it?
Solution: The “leaf weight” can be said as the model’s predicted output associated with each leaf (exit) node. Here is an instance of how to calculate the weights of the leaf nodes in XGB-
Consider a test data point, where age=10 and gender=female.To get the prediction for the data point, the tree is traversed from the top to bottom, performing a series of tests. At each of the intermediate nodes, a feature is needed to compare against a threshold.
Now, depending on the result of the comparison, one must proceed to either the left or right child node of the tree. In case of (10, female), the test “age < 15” is to be performed first and then proceed to the left branch, because “age < 15” is true. Then, the second test “gender = male?” is performed, which evaluates to false, so we proceed to the right branch. We end up at the Leaf 2, whose output (leaf weight) is 0.1.
Click here to know more in detail.
5| What are the data pre-processing steps for XGB?
Solution: The data pre-processing steps for XGB include the following-
- Load the data
- Explore the data and remove the unneeded attributes
- Transform textual values to numeric
- Find and replace the missing values if needed
- Encoding the categorical data
- Break the dataset into training set as well as test set
- Perform feature scaling or data normalisation
Know more here.
6| How does XGB calculate features?
Solution: XGB automatically provides the estimations of feature importance from a trained predictive model. After a boosting tree is constructed, it retrieves feature importance scores for each attribute. The feature importance contributes a score which indicates how much valuable each feature was in the construction of the boosted decision trees within the model.
Also, in terms of accuracy, XGB models show better performance for the training phase and comparable performance for the testing phase when compared to SVM models. Besides accuracy, XGB has higher computation speed than SVM.
Know more here.
7| Why does XGBoost perform better than SVM?
Solution: In case of missing values, XGB is internally designed to handle missing values. The missing values are interpreted in such a way that if there endures any trend in the missing values, it is captured by the model. Users are required to supply a different value than other observations and pass that as a parameter.
XGBoost tries different things as it encounters a missing value on each node and learns which path to take for missing values in future. On the other hand, Support Vector Machine (SVM) does not perform well with the missing data and it is always a better option to impute the missing values before running SVM.
Know more here.
8| Differences between XGBoost and LightGBM.
Solution: XGBoost and LightGBM are the packages belonging to the family of gradient boosting decision trees (GBDTs).
- Traditionally, XGBoost is slower than lightGBM but it achieves faster training through the Histogram binning process.
- LightGBM is a newer tool as compared to XGBoost. Hence, it has fewer users and thus a narrow user base than XGBoost and contains less documentation.
Know more here.
9| How does XGB handle missing values?
Solution: XGBoost supports missing values by default. In tree algorithms, branch directions for missing values are learned during training. It is important to note that the gblinear booster treats missing values as zeros. During the training time XGB decides whether the missing values should fall into the right node or left node. This decision is taken to minimise the loss. If there are no missing values during the training time, the tree made a default decision to send any new missings to the right node.
Know more here.
10| What is the difference between AdaBoost and XGBoost?
Solution: XGBoost is flexible compared to AdaBoost as XGB is a generic algorithm to find approximate solutions to the additive modeling problem, while AdaBoost can be seen as a special case with a particular loss function.
- Unlike XGB, AdaBoost can be implemented without the reference to gradients by reweighting the training samples based on classifications from previous learners
- now more here.
