Introduction to XGBoost

Table of contents

1. What is Gradient Boosting?
2. What is XGBoost?
3. XGBoost in R.
4. Pros and Cons?
5. Applications

What is Gradient Boosting?

Before diving deep into XGBoost, let us first understand Gradient Boosting.
Boosting is just taking random samples of data from our dataset and learning a weak learner (a predictor with not so great accuracy) for it. There is one catch though, we give more weights to those data points which were misclassified by the previous learners.
I know that this is making a little sense to you. Let us clear it with the help of an example and a visualization.

Example:

Suppose that you are given a binary classification problem. Also let us assume that we are using Decision tree stumps (small decision trees) as our weak learner. There are weights assigned to each data point (all equal initially) and the error is the sum of weights of misclassified examples.

Step:3

Current Hypothesis: \(H\) = \(h_1\) + \(h_2\) Now, we will learn another predictor \(h_3\) on the data points and try to minimize the \(residual\) = \(y\)-(\(h_1\) + \(h_2\)).

We will perform these steps upto say \(m\) times, each time giving boosted importance to misclasified points and trying reducing the residual.
Our final hypothesis will be a weighted sum of these weak learners. \(H\) = \(w_1\)\(h_1\) + \(w_2\)\(h_2\) + … + \(w_m\)\(h_m\)

Visualization

Below, there are three weak predictors and they are combining to become a strong one. Also notice that each subsequent predictor tries to classify correctly what the last predictor misclassified.

Ques: How do I set m?

Well, \(m\) is a hyperparameter and is tuned using cross-validation.

What kind of predictors can ‘h’ be?

There are mathematical restrictions on it (diffrentiablity requirements). Genreally, decision tree stumps are used.

I highly recommend you to see Viagra vs Cialis, Which Is Better? for a Mathematical-cum-intuitive explanation and this for visualization of the process.

---

What is XGBoost?

XGBoost stands for Extreme Gradient Boosting. It is a supervised learning algorithm. It is a library for developing fast and high performance gradient boosting tree models. Parallel computation behind the scenes is what makes it this fast. It has been very popular in recent years due to its versatiltiy, scalability and efficiency.
Like any other ML algorithm, this too has its own pros and cons. But fortunately pros easily outweigh cons given we have an astute understanding of the algorithm and an intuition for proper parameter tuning. This has been proven by its huge popularity on Kaggle.

How to use XGBoost?

There are library implementations of XGBoost in all major data analysis languages. We just have to train the model and tune its parameters. Below, is the series of steps to follow:

1. Load your dataset.
2. Prepare your data to contain only numeric features (yes, XGBoost works only with numeric features).
3. Split the data into train and test sets.
4. Train the model and tune the parameters.
5. Deploy your model on test data.

A burning question here is “Which are the major hyperparameters in case of XGBoost?”

Major hyperparameters in XGBoost

XGBoost in R

# Install xgboost from CRAN using 'install.packages("xgboost")'

#Use XGBoost library for this post.
require(xgboost)

## Loading required package: xgboost

## Warning: package 'xgboost' was built under R version 3.3.2

#For plotting
require(Ckmeans.1d.dp)

## Loading required package: Ckmeans.1d.dp

## Warning: package 'Ckmeans.1d.dp' was built under R version 3.3.2

require(DiagrammeR)

## Loading required package: DiagrammeR

## Warning: package 'DiagrammeR' was built under R version 3.3.2

#We will use agaricus dataset in the package XGBoost.
#Agaricus comes with separate train and test dataset
#But usually, you need to split the data into training and test sets yourself.

data(agaricus.train, package='xgboost')
data(agaricus.test, package='xgboost')

train <- agaricus.train
test <- agaricus.test

#setting hyperparametrs
param <- list("objective" = "binary:logistic", 
              "eval_metric" = "logloss",
              "eta" = 1,           #Can play around with this
              "max.depth" = 2      #Dataset is not that complex
              )

#xgb.cv is used for cross-validation and finding best value for hyperparameter.
#Here, we will find out the best value for nrounds.
best_cv <- xgb.cv(param = param,
                  data = as.matrix(train$data),
                  label = train$label,
                  nfold = 10, 
                  nrounds = 100,
                  verbose = FALSE)

#Use nrounds where error curve has minimum (50 in this case)
plot(log(best_cv$evaluation_log$test_logloss_mean), type = 'l')

#Use nrounds = 50

final_model <- xgboost(data = as.matrix(train$data), label = train$label, 
                 params = param, nrounds = 50, verbose = FALSE)

#Predict with this model
pred <- predict(final_model, test$data)

#Print importance matrix (importance of variables in classification).

#get the feature names
feature_name <- dimnames(train$data)[[2]]
importance_matrix <- xgb.importance(feature_name , model = final_model)
xgb.plot.importance(importance_matrix[1:10,])

# You can also plot the decision tree 

# xgb.plot.tree(feature_names = feature_name, model = final_model, n_first_tree = 1) 

Pros and Cons?

Applications

• Winning solution for many Kaggle competitions.
• Heavily used in industries due to its scalability.

Additional resources:

I highly recommend you to go through the links below for an in-depth understanding of the Maths behind this algorithm.

1. XGBoost talk by inventor
2. Artificial Intelligence(MIT)