Bi-Variate Analysis
In my previous post, we have covered Uni-Variate Analysis as an initial stage of data-exploration process. In this post, we will cover the bi-variate analysis. Objective of the bi-variate analysis is to understand the relationship between each pair of variables using statistics and visualizations. We need to analyze relationship between:
- the target variable and each predictor variable
- two predictor variables
Why Analyze Relationship between Target Variable and each Predictor Variable
It is important to analyze the relationship between the predictor and the target variables to understand the trend for the following reasons:
- The bi-variate analysis and our model should communicate the same story. This will help in understand and analysing the accuracy of our models, and to make sure, that our model has not over-fit the training data.
- If the data has too many predictor variables, we should include only those predictor variables in our regression models which show some trend with the target variable. Our aim with the regression models is to understand the story each significant variable is communicating, and its behaviour with other predictor variables and the target variable. A variable that has no pattern with the target variable, may not have a direct relation with the target variable (while a transformation of this variable might have a direct relation).
- If we understand the correlations and trends between the predictor and the target variables, we can arrive at better and faster transformations of predictor variables, to get more accurate models faster.
Eg. Following curve indicates logarithmic relation between target and predictor variables. A curve of below-mentioned shape indicates that there is a logarithmic relation between x and y. In order to transform the above curve into linear, we need to take an exponential of 10 of the predictor variable. Hence, a simple scatter plot can give us the best estimate of variables – transformations required to arrive at the appropriate model.
Why Analyze Relationship between 2 Predictor Variables
- It is important to understand the correlations between each pair of predictor variables. Correlated variables lead to multi-collinearity. Essentially, two correlated variables are transmitting the same information, and hence are redundant. Multi-collinearity leads to inflated error term and wider confidence interval (reflecting greater uncertainty in the estimate).
- When there are too many variables in a data-set, we use techniques like PCA for dimensionality reduction. Dimensionality reduction techniques work upon reducing the correlated variables, to reduce the extraneous information and so that we run our modelling algorithms on the variables that explain the maximum variance.
Method to do Bi-Variate Analysis
We have understood why bi-variate analysis is an essential step to data exploration. Now we will discuss the techniques to bi-variate analysis.
For illustration, we will use Hitters data-set from library ISLR. This is Major League Baseball Data from the 1986 and 1987 seasons. It is a data frame with 322 observations of major league players on 20 variables. The target variable is Salary while the rest of the 19 are dependent variables. Through this data-set, we will demonstrate bi-variate analysis between:
- two continuous variables,
- one continuous and one categorical variable,
- two categorical variables
Following is the summary of all the variables in Hitters data-set:
data(Hitters)
hitters=Hitters
summary(hitters)## AtBat Hits HmRun Runs
## Min. : 16.0 Min. : 1 Min. : 0.00 Min. : 0.00
## 1st Qu.:255.2 1st Qu.: 64 1st Qu.: 4.00 1st Qu.: 30.25
## Median :379.5 Median : 96 Median : 8.00 Median : 48.00
## Mean :380.9 Mean :101 Mean :10.77 Mean : 50.91
## 3rd Qu.:512.0 3rd Qu.:137 3rd Qu.:16.00 3rd Qu.: 69.00
## Max. :687.0 Max. :238 Max. :40.00 Max. :130.00
##
## RBI Walks Years CAtBat
## Min. : 0.00 Min. : 0.00 Min. : 1.000 Min. : 19.0
## 1st Qu.: 28.00 1st Qu.: 22.00 1st Qu.: 4.000 1st Qu.: 816.8
## Median : 44.00 Median : 35.00 Median : 6.000 Median : 1928.0
## Mean : 48.03 Mean : 38.74 Mean : 7.444 Mean : 2648.7
## 3rd Qu.: 64.75 3rd Qu.: 53.00 3rd Qu.:11.000 3rd Qu.: 3924.2
## Max. :121.00 Max. :105.00 Max. :24.000 Max. :14053.0
##
## CHits CHmRun CRuns CRBI
## Min. : 4.0 Min. : 0.00 Min. : 1.0 Min. : 0.00
## 1st Qu.: 209.0 1st Qu.: 14.00 1st Qu.: 100.2 1st Qu.: 88.75
## Median : 508.0 Median : 37.50 Median : 247.0 Median : 220.50
## Mean : 717.6 Mean : 69.49 Mean : 358.8 Mean : 330.12
## 3rd Qu.:1059.2 3rd Qu.: 90.00 3rd Qu.: 526.2 3rd Qu.: 426.25
## Max. :4256.0 Max. :548.00 Max. :2165.0 Max. :1659.00
##
## CWalks League Division PutOuts Assists
## Min. : 0.00 A:175 E:157 Min. : 0.0 Min. : 0.0
## 1st Qu.: 67.25 N:147 W:165 1st Qu.: 109.2 1st Qu.: 7.0
## Median : 170.50 Median : 212.0 Median : 39.5
## Mean : 260.24 Mean : 288.9 Mean :106.9
## 3rd Qu.: 339.25 3rd Qu.: 325.0 3rd Qu.:166.0
## Max. :1566.00 Max. :1378.0 Max. :492.0
##
## Errors Salary NewLeague
## Min. : 0.00 Min. : 67.5 A:176
## 1st Qu.: 3.00 1st Qu.: 190.0 N:146
## Median : 6.00 Median : 425.0
## Mean : 8.04 Mean : 535.9
## 3rd Qu.:11.00 3rd Qu.: 750.0
## Max. :32.00 Max. :2460.0
## NA's :59Before we conduct the bi-variate analysis, we’ll seperate continuous and factor variables and perform basic cleaning through following code:
#deleting rows with missing values in target variable
hitters=hitters[-which(is.na(hitters$Salary)),]
#seperating continuous and factor variables
hitters_factor=data.frame(1:nrow(hitters))
hitters_cont=data.frame(1:nrow(hitters))
i=1
while(i<=length(names(hitters)))
{
if(class(hitters[,i])=="factor")
hitters_factor=cbind(hitters_factor,hitters[i])
else
hitters_cont = cbind(hitters_cont,hitters[i])
i=i+1
}
hitters_cont = hitters_cont[,-1]
hitters_factor=hitters_factor[,-1]Techniques to do Bi-Variate Analysis.
Please note, that the way to do the bi-variate analysis is same irrespective of predictor or target variable.
Bi-Variate Analysis between Two Continuous Variables
To do the bi-variate analysis between two continuous variables, we have to look at scatter plot between the two variables. The pattern of the scatter plot indicates the relationship between the two variables. As long as there is a pattern between two variables, there can be a transformation applied to the predictor / target variable to achieve a linear relationship for modelling purpose. If no pattern is observed, this implies no relationship possible between the two variables. The strength of the linear relationship between two continuous variables can be quantified using Pearson Correlation.A correlation coefficient of -1 indicates high negative correlation, 0 indicates no correlation, 1 indicates high positive correlation.
Correlation is simply the normalized co-variance with the standard deviation of both the factors. This is done to ensure we get a number between +1 and -1. Co-variance is very difficult to compare as it depends on the units of the two variables. So, we prefer to use correlation for the same.
Please note: * If two variables are linearly related, it means they should have high Pearson Correlation Coefficient. * If two variables are correlated does not indicate they are linearly related. This is because correlation is deeply impacted by outliers. Eg. the correlation for both the below graphs is same, but the linear relation is not there in the second graph:
- If two variables are related, it does not mean they are necessarily correlated. Eg. in the below graph of y=x^2-1, x and y are related but not correlated (with correlation coefficient of 0).
x=c(-1, -.75, -.5, -.25, 0, .25, .5, .75, 1)
y=x^2-1
plot(x,y, col="dark green","l")
cor(x,y)## [1] 0For our hitters illustration, following are the correlations and the scatter-plots:
cor(hitters_cont[1:4]) ## AtBat Hits HmRun Runs
## AtBat 1.0000000 0.9639691 0.5551022 0.8998291
## Hits 0.9639691 1.0000000 0.5306274 0.9106301
## HmRun 0.5551022 0.5306274 1.0000000 0.6310759
## Runs 0.8998291 0.9106301 0.6310759 1.0000000This gives the correlation-coefficients between the continuous variables in hitters data-set. Since it is difficult to analyse so many values, we prefer to attain a quick visualization of correlation through scatter plots. Following command gives the scatter plots for the first 4 continuous variables in hitters data-set:
pairs(hitters_cont[1:4], col="brown") 
Observations:
- Linear pattern can be observed between AtBat and Hits, and can be confirmed from the correlation value = 0.96
- Linear pattern can be observed between Hits and Runs, and can be confirmed from the correlation value = 0.91
- Linear pattern can be observed between AtBat and Runs, and can be confirmed from the correlation value = .899
To get a scatter plot and correlation between two continuous variables:
plot(hitters_cont[,1], hitters_cont[,2], col="yellow", xlab="AtBat", ylab="Hits")
The graph shows a strong positive linear correlation.
Pearson Correlation Coefficient between AtBat and Hits
cor(hitters_cont[,1], hitters_cont[,2])## [1] 0.9639691Correlation value of .96 verifies our claim of strong positive correlation. We can obtain better visualizations of the correlations through library corrgram and corrplot:
library(corrgram)
corrgram(hitters) 
Strong blue means strong +ve correlation. Strong red means strong negative correlation. Dark color means strong correlation. Weak color means weak correlation.
To find correlation between each continuous variable
library(corrplot)
continuous_correlation=cor(hitters_cont)
corrplot(continuous_correlation, method= "circle", type = "full", is.corr=T, diag=T) 
This gives a good visual representation of the correlation and relationship between variables, especially when the number of variables is high. Dark blue and large circle represents high +ve correlation. Dark red and large circle represents high -ve correlation. Weak colors and smaller circles represent weak correlation.
Bi-Variate Analysis between Two Categorical Variables
- 2-way Frequency Table: We can make a 2-way frequency table to understand the relationship between two categorical variables.
2-Way Frequency Table
head(hitters_factor)## League Division NewLeague
## -Alan Ashby N W N
## -Alvin Davis A W A
## -Andre Dawson N E N
## -Andres Galarraga N E N
## -Alfredo Griffin A W A
## -Al Newman N E Atab=table(League=hitters_factor$League, Division=hitters_factor$Division)#gives the frequency count
tab ## Division
## League E W
## A 68 71
## N 61 63This gives the frequency count of League vs Division
- Chi-Square Tests of Association: To understand if there is an association / relation between 2 categorical variables.
Chi-Square Test for Hitters Data-set
chisq.test(tab)##
## Pearson's Chi-squared test with Yates' continuity correction
##
## data: tab
## X-squared = 0, df = 1, p-value = 1The chisquare probability value = 1, we retain the null hypothesis that there is not sufficient association between the two factor variables.
- Visualization of two categorical variables can be obtained through Heat Map and Fluctuation Charts as illustrated in the example:
Heat Map
library(ggplot2)
runningcounts.df <- as.data.frame(tab)
ggplot(runningcounts.df, aes(League,Division)) +
geom_tile(aes(fill = Freq), colour = "black") +
scale_fill_gradient(low = "white", high = "red")+theme_classic() 
Dark Color represents higher frequency and lighter colors represent lower frequencies.
Fluctuation Plot
ggplot(runningcounts.df, aes(League, Division)) +
geom_point(aes(size = Freq, color = Freq, stat = "identity", position = "identity"), shape = 15) + scale_size_continuous(range = c(3,15)) +
scale_color_gradient(low = "white", high = "black")+theme_bw()
Dark Color represents higher frequency and lighter colors represent lower frequencies.
Bi-Variate Analysis between a Continuous Variable and a Categorical Variable
- Aggregations can be obtained using functions xtabs, aggregate or using dplyr library. Eg. In the Hitters Data-set, we will use Factor Variable: “Division”" and Continuous Var: “Salary”.
Aggregation of Salary Division-Wise
xtabs(hitters$Salary ~ hitters$Division) #gives the division-wise sum of salaries## hitters$Division
## E W
## 80531.01 60417.50aggregate(hitters$Salary, by=list(hitters$Division), mean,na.rm=T) #gives the division-wise mean of salaries## Group.1 x
## 1 E 624.2714
## 2 W 450.8769hitters%>%group_by(Division)%>%summarise(Sum_Salary=sum(Salary, na.rm=T), Mean_Salary=mean(Salary,na.rm=T), Min_Salary=min(Salary, na.rm=T),Max_Salary = max(Salary, na.rm=T))## # A tibble: 2 × 5
## Division Sum_Salary Mean_Salary Min_Salary Max_Salary
## <fctr> <dbl> <dbl> <dbl> <dbl>
## 1 E 80531.01 624.2714 67.5 2460
## 2 W 60417.50 450.8769 68.0 1900- T-Test: 2-Sample test (paired or unpaired) can be used to understand if there is a relationship between a continuous and a categorical variable. 2-sample T test can be used for categorical variables with only two levels. For more than two levels, we will use Anova.
Eg. Conduct T-test on Hitters data-set to check if Division (a predictor factor variable) has an impact on Salary (continuous target variable). H0: Mean of salary for Divisions E and W is same, so there is not much impact for divisions on salary HA: Mean of salay is different; so there is a deep impact of divisions on salary
df=data.frame(hitters$Division, hitters$Salary)
head(df)## hitters.Division hitters.Salary
## 1 W 475.0
## 2 W 480.0
## 3 E 500.0
## 4 E 91.5
## 5 W 750.0
## 6 E 70.0library(dplyr)
df%>%filter(hitters.Division =="W")%>%data.frame()->sal_distribution_W
df%>%filter(hitters.Division=="E")%>%data.frame()->sal_distribution_E
x=sal_distribution_E$hitters.Salary
y=sal_distribution_W$hitters.Salary
t.test(x, y, alternative="two.sided", mu=0) ##
## Welch Two Sample t-test
##
## data: x and y
## t = 3.145, df = 218.46, p-value = 0.001892
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## 64.73206 282.05692
## sample estimates:
## mean of x mean of y
## 624.2714 450.87692-sided indicates two tailed test. P-value of .19% indicates that we can reject the null hypothesis and there is a significant difference in mean salary based on division. Hence, division does make an impact on the salary.
- Anova: Analysis of variance (ANOVA) is a collection of statistical models used to analyze the differences among group means and their associated procedures (such as “variation” among and between groups).
Anova
anova=aov(Salary~Division, data = hitters)
summary(anova)## Df Sum Sq Mean Sq F value Pr(>F)
## Division 1 1976102 1976102 10.04 0.00171 **
## Residuals 261 51343011 196717
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Since probability value is < 1%, so, the difference between the average salary for different divisions is significant.
- Visualization can be obtained through box-plots, bar-plots and density plots.
Box Plots to establish relationship between Division variable and Salary Variable in Hitters data-set
x=plot(hitters$Division, hitters$Salary, col="red", xlab="Division", ylab="Salary")