Most data science practitioners are starving for higher accuracy from their models and neural networks are regularly giving us state-of-the-art performances. In this article, we are going to see a method applying that we can boost the accuracy level of our modelling procedure. This method can be considered a way to join multiple machine learning models. We call this method a stacking ensemble. The major points to be discussed in the article are listed below.

Table of contents 

  1. What is stacking 
  2. Implementing stacking ensemble deep learning
    1. Data preparation 
    2. Model preparation 
    3. Stacking models 
    4. Evaluating results  

Let’s start with understanding stacking.

What is stacking?

We can consider stacking as a process of ensembling multiple machine learning models together. There can be various methods for ensemble models such as bagging, boosting, and stacking is one of them. When talking about the bagging method, it works by applying multiple models with high variance and these high variances are averaged to decrease variance. And boosting work by building multiple incremental models so that bias can be decreased. 

Stacking is very different from these ensemble methods because this method focuses on exploring the space of different models applied to the same problem. Basically, the idea behind this method is to handle a machine learning problem using different types of models that are capable of learning to an extent, not the whole space of the problem. Using these models we can make intermediate predictions and then add a new model that can learn using the intermediate predictions. 

The final model that is learned using the intermediate predictions can be considered as stacked on top of the intermediate models.  Using such methods we can improve our performances and always we can get a model which is better than the intermediate model. The below picture can explain the stacking method.

Image source

The above picture represents that a final classifier is stacked on top of three intermediate classifiers. In this article, we are going to see how we can do stack ensembling with deep learning models. Let’s start the implementation. 

Are you looking for a complete repository of Python libraries used in data science, check out here,

Implementation

In this section, we will look at how we can stack an MLP classifier on top of some neural networks. This whole section includes the usage of packages from TensorFlow, Keras, and sci-kit learn libraries. Let’s start with preparing data for the process.  

Data preparation

In this article, we will try to stack classification models. For this, we are going to make toy data using the sklearn provided make_classification  function. Let’s see how we can do that.

from sklearn.datasets import make_classification
X, y = make_classification()
print(X.shape, y.shape)

Output:

Let’s split the dataset into training and test sets.

from sklearn.model_selection import train_test_split
X_train, X_val, y_train, y_val = train_test_split(X, y,
                                                  test_size=0.2,
                                                  random_state=0)

Let’s check what our toy data looks like.

print(X_train)

Output:

print(y_train)

Output:

Model preparation

Now in this example of stacking, we are going to use neural networks as our intermediate models so we require importing some of the packages from Keras.

from tensorflow import keras
from keras import layers
from keras.constraints import maxnorm
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation,  Flatten, Input

In this article, we are going to use the stacking facility provided by scikit learn so we are required to import a wrapper provided in Keras for SciKit-Learn so that it can be compatible with scikit learn facilities. Lets import KerasClassifier.

from keras.wrappers.scikit_learn import KerasClassifier

Defining a model function

def create_model ():
    # create model
    model = Sequential()
    model.add(Dense(20, input_dim=20, activation='relu'))
    model.add(Dropout(0.2))
    model.add(Flatten())
    optimizer= keras.optimizers.RMSprop(lr=0.001)
    model.add(Dense(units = 1, activation = 'sigmoid'))  # Compile model
    model.compile(loss='binary_crossentropy',
                  optimizer=optimizer, metrics=[keras.metrics.AUC(), 'accuracy'])
    return model

In the above, we have defined a simple function for creating models in which we have added 2 dense layers and a dropout and flatten function. Let’s make some classifiers using the above function.

Defining models 

NN_clf1=KerasClassifier(build_fn=create_model, epochs=5, batch_size= 32)
NN_clf1._estimator_type = "classifier"
NN_clf2=KerasClassifier(build_fn=create_model, epochs=10, batch_size= 32)
NN_clf2._estimator_type = "classifier"
NN_clf3=KerasClassifier(build_fn=create_model, epochs=15, batch_size= 32)
NN_clf3._estimator_type = "classifier"
NN_clf4=KerasClassifier(build_fn=create_model, epochs=20, batch_size= 32)
NN_clf4._estimator_type = "classifier"
NN_clf5=KerasClassifier(build_fn=create_model, epochs=25, batch_size= 32)
NN_clf5._estimator_type = "classifier"

Here we have defined 5 models where we have altered the number of epochs we can also use some other methods to perform alteration.

Stacking models

Let’s start by combining the above-defined models.

intermediate = [('NN1', NN_clf1), ('NN2', NN_clf2), ('NN3', NN_clf3), ('NN4', NN_clf4),
             ('NN5', NN_clf5)]
intermediate

Output:

Here we can see that we have combined all the above-defined neural networks. Now we are ready to stack these models. In this article, we are going to use MLPClassifier as the final model that will learn from the learning of the intermediate neural networks.   

from sklearn.ensemble import StackingClassifier
from sklearn.neural_network import MLPClassifier
clf = StackingClassifier(estimators=intermediate, final_estimator=MLPClassifier())
clf.fit(X_train, y_train)

Output:

Here we can see the final details of the model where it is representing estimators or intermediate models and final models.

Evaluating the final model

Let’s check for accuracy.

 print("Stacking model score: %.3f" % clf.score(X_test, y_test))

Output

Here is our final accuracy. Here we can see an example of stacking deep learning models. Let’s check the probability score.

clf.predict_proba(X_test)

Output:

Here we can see the probability of a sample being part of class 0 or 1. This article represents the idea that if applying neural networks in a stacking manner, we can also use base models like random forest and logistic regression models as the intermediate or final models.

Final words

In this article, we have discussed stacking which is a process of ensembling multiple machine learning models together in a stacked manner and we also see how we can stack models from sklearn on top of the neural networks.     

References