A hands-on guide to principal component regression in Python
Principal Components Regression (PCR) is a technique for analyzing multiple regression data that suffer from multicollinearity. PCR is derived from Principal Component Analysis (PCA). So, it is PCA applied to a regression algorithm that has multicollinear features. Principal components regression reduces errors in regression estimates by adding a degree of bias and by doing so, it will be possible to provide more reliable estimates. In this article, we will put focus on implementing PCR on a regression problem. Following are the topics covered.
Table of contents
- About the architecture of PCA
- About multicollinearity in regression
- About Principal Components Regression (PCR)
- Implementing PCR in Python
Let’s start with the architect PCA from which PCR is derived.
About the architecture of PCA
Principal Component Analysis (PCA) is the analysis of principal features of the data. The analysis is done by reducing the dimensionality of the feature space. In other words, it is a tool to reduce the features from the data to get only the required features or principal components for the learner. PCA has three major components which help to reduce dimensionality:
- The covariance matrix is the measure of how much the variables are associated with each other.
- The eigenvectors are the directors in which the data is dispersed.
- The eigenvalues are the relative importance of the directions.
About multicollinearity in regression
From the name, it is clear that the collinearity between the independent variables in a regression problem is stated as multicollinearity in regression. The reasons behind curing a multicollinear regression problem are:
- Understanding of the significance of features for the regression learner.
- Instability in estimating the coefficient
- Overfitting of the learner
Multicollinearity in regression could be vandalized with the help of Principal Component Regression (PCR). Let’s understand how does PCR control multicollinearity.
About Principal Components Regression (PCR)
The Principal Component Regression (PCR) algorithm is an approach for reducing the multicollinearity of a dataset. Although multi-variate linear regression can fit well on the test set, there is normally a high-variance problem with it. For this reason, PCR adds a small bias to the model, such that it aims to maintain a high level of accuracy, but reduce the variance substantially. This is achieved by applying the PCA to the features before the training. Let’s try to learn PCR by implementing it on data.
Implementing PCR in Python
The objective of this regression model is to predict the salary of the player based on different features.
Importing packages
import numpy as np import pandas as pd
Reading the data set
salary=pd.read_csv("Hitters.csv")
df=salary.copy()
df.dropna(inplace=True)
df.shape,salary.shape
((263, 20), (322, 20))
This data set is taken from the Kaggle repository, to use this dataset the links are given in references. Copying the dataset to another data frame for further pre-processing so that the original data frame remains unchanged.
Then drop all the missing values and at last check the number of rows and columns of the original and copied data frames. There are two categorical columns and the rest are continuous features, needed to encode the categorical features for further utilization.
Encoding categorical features:
df = pd.get_dummies(df,columns=['League', 'Division', 'NewLeague']) df.head()
Encoded the data using the pandas get_dummies function. The original features which are used for creating the dummies are replaced with a total of six new columns as shown in the above image. Since the regression problem is to predict the salary of the players so the dependent variable is “Salary”.
y=df['Salary'] X=df.drop(['Salary'],axis=1)
There are a total of 23 columns in this data frame out of which only a few are important for the analysis and prediction. There is a requirement for feature elimination which would be performed by PCA.
Applying PCA
from sklearn.decomposition import PCA pca=PCA() X_red = pca.fit_transform(scale(X))
PCA is been imported from the sklearn library and stored in a variable for easier applications. The PCA is fitted with the independent variables for dimensionality reduction. The percentage of variance in the dependent variable is explained by adding each principal component to the model.
np.cumsum(np.round(pca.explained_variance_ratio_, decimals = 4)*100)[0:5]
The above output is explained as:
- By using the first principal component, we can explain 32.62% of the variation in the dependent variable.
- By using the second principal component, we can explain 53.02% of the variation in the dependent variable.
- Similarly, by using others we can explain 68.82%, 78.07%,85.39%, 89.31%
The conclusion of this is that we need to use a total of five principal components in the regression learner. Now, the principal components are decided it’s for split the dataset into a 70:30 ratio for train and test for training and testing the learner.
X_train, X_test, y_train, y_test = train_test_split(X,
y,
test_size=0.30,
random_state=42)
Let’s build the final model using five principal components in the linear regression model.
X_red_train = pca.fit_transform(scale(X_train)) X_red_test = pca.transform(scale(X_test))[:,0:5] lm = LinearRegression() pcr = lm.fit(X_red_train[:,0:5], y_train) y_pred = pcr.predict(X_red_test)
The final model is built and trained on the train data set and also predicted the salary using the test dataset. Let’s check how good the model has performed.
np.round(np.sqrt(mean_squared_error(y_test, y_pred)),2)
397.52
The root means square error (RMSE) is low approximately 398 this can be improved by tunning the model further I would leave that to you.
Nutshell
Applying Principal Component Analysis before the regression can reduce multicollinearity and helps in better prediction with fewer features in lesser time. PCR also reduces the chances of overfitting learners.
