Table of Contents
- 1 Problem Statement
- 2 Analysis Approach & Conclusions
- 3 Importing data
- 4 Data Quality Checks
- 5 Exploratory Data Analysis
- 5.1 Univariate Analysis / Outlier Treatment
- 5.2 Bivariate Analysis
- 5.2.1 Pairplot
- 5.2.2 Child Mortality vs Life Expectancy
- 5.2.3 Child Mortality vs Total Fertility
- 5.2.4 GDP per capita vs Health Spending
- 5.2.5 Imports vs GDP per capita
- 5.2.6 Exports vs GDP per capita
- 5.2.7 GDP per capita vs Trade Deficit
- 5.2.8 Net Income per person vs Inflation
- 5.2.9 GDP per capita vs Inflation
- 5.3 Correlation Analysis
- 6 Hopkin's Statistic
- 7 Standardizing Values
- 8 K-Means Clustering
- 9 Hierarchical Clustering
- 10 Mixed K-Means Clustering
- 11 Cluster Profiling
- 12 Conclusion
HELP - Countries to Aid¶
Jayanth Boddu
Clustering
Problem Statement¶
An NGO wants to use their funding strategically so that they could aid five countries in dire need of help. The analyst's objective is to use clustering algorithms to group countries based on socio-economic and health factors to judge the overall development of countries. Further,the final deliverable is - suggesting 5 countries that need the aid the most.
Analysis Approach & Conclusions¶
- The objective of the analysis is to recommend 05 countries in dire need of aid of help.
- The achieve this, the following features of 167 countries have been analyzed.
- GDP per capita
- Child mortality per 1000 live births
- Net Income per person
- Fertility rates at the current previaling age/fertility rate
- Inflation index
- Life Expectancy
- Imports per capita
- Exports per capita
- Total Health Spending per capita
- The above features are further grouped into health, economic and policy indicators
- Health Indicators
- Child mortality per 1000 births
- Fertility rates at the current previaling age/fertility index
- Life Expectancy
- Economic Indicators
- Net Income per person
- Imports per capita
- Exports per capita
- Inflation Index
- Policy Indicators
- Total Health Spending per capita
- Health Indicators
- From the above, countries deduced to be in dire need of aid are countries with bad economic indicators and bad health indicators having low health spending
- Further on the feature level, the following are general indicators of need of aid.
| Feature | High / Low |
|---|---|
| Child Mortality | High |
| Life Expectancy | Low |
| Fertility | High |
| Health Spending | Low |
| GDP per capita | Low |
| Inflation Index | High |
| Income per Person | Low |
| Imports | High |
| Exports | Low |
- Data has been check for quality.No Data Quality Issues have been found - No missing values, No duplicates , No incorrect data types
A new feature 'Trade Deficit' has been derived.
- Trade Deficit = Imports - Exports
- Trade Deficit or Trade balance is a good indicator of economic health of countries with low GDP per capita
- Lower or negative is better than higher and positive.
Univariate analysis revealed the following information
| Feature | Highest | Lowest |
|---|---|---|
| Child Mortality | Haiti,Sierra Leone | Iceland |
| Life Expectancy | Japan , Singapore | Haiti, Lesotho |
| Total fertility | Chad, Niger | Singapore, South Korea |
| Health Spending | Switzerland, US | Madagascar, Eritrea |
| GDP per capita | Luxemborg , Norway | Burundi , Liberia |
| Inflation Index | Nigeria, Venezuela | Seycelles, Ireland |
| Net Income per person | Qatar, Luxemborg | Liberia, Congo Dem. Rep. |
| Imports per capita | Luxemborg, Singapore | Myanmar, Burundi |
| Exports per capita | Luxemborg, Singapore | Myanmar, Burundi |
| Trade Deficit per capita | Bahamas, Greece | Luxemborg, Qatar |
- Since there's less data per cluster, soft range of 1st percentile - 99th perentile has been used to classify and cap outliers. Only those outliers which do not skew the areas of interest have been capped.
- Outlier Treatment Summary
| Feature | Upper Outliers | Lower Outliers |
|---|---|---|
| Child Mortality | Not Changed | Capped |
| Life Expectancy | Capped | Not Changed |
| Fertility | Not Changed | Capped |
| Health Spending | Capped | Not Capped |
| GDP per capita | Capped | Not Changed |
| Inflation Index | Not Changed | Capped |
| Income per Person | Capped | Not Changed |
| Imports | Capped | Not Changed |
| Exports | Capped | Not Changed |
- Bivariate Analysis revealed the following insights
- There is a negative relationship between Child Mortality and Life expectancy. As child mortality increases, Life Expectancy decreases. Haiti and Sierra Leone have the highest child mortality and lowest life expectancy.
- There is positive relationship between child mortality and total fertility. Ex : Chad and Mali
- Health Spending has a positive relationship with GDP per capita , but the situation is dire in case of low GDP countries like Eritrea & Madagascar.
- There's very high correlation between child mortality and life expectancy , child mortality and total fertility.
Columns : 'child_mort', 'exports', 'health', 'imports', 'income','inflation', 'life_expec', 'total_fer', 'gdpp','trade_deficit' were used for clustering.
Hopkin's Statistic was calculated which showed a very high mean clustering tendency of 96% with a standard deviation of 4%
- The dataset was standardized with mean = 0 , standard deviation = 1 before clustering.
- Optimum no of clusters for k-means was found to be 4 , both from Elbow curve and Silhoeutte Analysis curve.
- Clustering with k-means was performed (Cluster centers initialized with k-means++)
- Hierarchical Clustering with Single, Complete and Average linkages and Euclidean , correlation based distances were explored.
- Mixed Clustering - K-Means clustering initialised with centroids of Hierarchical clusters , 6 no of clusters - was carried out.
- Finally, Hierarchical Clustering with Complete linkage and correlation based distance measure was chosen to arrive at the results. Choice was made based on interpretability of results.
- Characteristics of Clusters obtained :
| Cluster | GDP | Income | Child Mortality |
|---|---|---|---|
| 0 | Very low | Very Low | Very High |
| 1 | low | low | low |
| 2 | Moderate | Moderate | Moderate |
| 3 | High | High | Low |
| 4 | Very High | Very High | Low |
| 5 | Very Low | Very Low | Low |
- From the characteristics , we see that the Cluster 0 is our area of interest.
- According to the UN goals of 2030, the top priority is health and then poverty.
- Hence countries in cluster 0 were ranked based on Child Mortality followed by income and GDP per capita.
- By that criteria, the following are the five countries HELP should consider extending their aid.
- Haiti
- Sierra Leone
- Chad
- Central African Republic
- Mali
In [85]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('whitegrid')
from sklearn.cluster import KMeans
from scipy.cluster.hierarchy import linkage
from scipy.cluster.hierarchy import dendrogram
from scipy.cluster.hierarchy import cut_tree
import warnings
warnings.filterwarnings('ignore')
!pip install tabulate
!jt -f roboto -fs 12 -cellw 100%
from tabulate import tabulate
from bokeh.models import ColumnDataSource, HoverTool
from bokeh.plotting import figure,show,output_notebook,reset_output
from bokeh.transform import factor_cmap
from bokeh.layouts import row
output_notebook()
In [86]:
# to table print a dataframe
def tab(ser) :
print(tabulate(pd.DataFrame(ser), headers='keys', tablefmt='psql'))
Importing data¶
In [87]:
countries = pd.read_csv('./Country-data.csv')
tab(countries.head())
- The dataset contains
exports,imports,healthas a proportion of GDP per capita - Let's convert them to actual values
In [88]:
# conversion to actual values
countries['exports'] = 0.01 * countries['exports'] * countries['gdpp']
countries['imports'] = 0.01 * countries['imports'] * countries['gdpp']
countries['health'] = 0.01 * countries['health'] * countries['gdpp']
Data Quality Checks¶
In [89]:
tab(countries.info())
- 167 countries
- No apparent missing values
- No incorrect data types
In [90]:
# taking a closer look to see if any country is duplicated
countries[countries.duplicated(subset=['country'])].index.values
Out[90]:
- No duplicate countries
In [91]:
# exports, health, imports are a percentage of GDPP . Checking if there are any anomalies
condition1 = countries['health'] < countries['gdpp']
condition2 = countries['imports'] < countries['gdpp']
condition3 = countries['exports'] < countries['gdpp']
# countries which don't satisfy the above conditions
index = countries[~(condition1 & condition2 & condition3)].index.values
countries.loc[index]
Out[91]:
- Some countries have greater imports than GDP per capita
In [92]:
# child mortality rate is calculated for 1000 live births. Let's check if its above 1000
countries[countries['child_mort'] > 1000].index.values
Out[92]:
- Sanity checks completed.
- No Data Quality issues found.
Exploratory Data Analysis¶
In [93]:
# summary statistics
tab(countries.describe())
- The Maximum values of all the attributes are much higher than their 75% values.
In [94]:
# let's look at quantiles are to take a closer look at outliers
tab(countries.quantile(np.linspace(0.75,1,25)).reset_index())
exports,child_mort,imports,income,inflation,life_expec,total_ferandgdpphave outliers.- These outliers shall be treated in Univariate analysis
Univariate Analysis / Outlier Treatment¶
In [95]:
# function to perform outlier analysis
def outlier_analysis(column) :
'''
This function prints a violin plot and box plot of the column provided.
It also prints the five major quantiles, lower oultier threshold value, upper outlier threshold value, tables of countries which are outliers
Output : lower outlier threshold condition, upper outlier threshold condition
Input : column name
Side effects : Violin plot, box plot, outlier tables
'''
plt.figure(figsize=[12,6])
plt.subplot(121)
plt.title('Violin Plot of '+column)
sns.violinplot(countries[column])
plt.subplot(122)
plt.title('Box Plot of '+column)
sns.boxplot(countries[column])
print('Quantiles\n')
print(tab(countries[column].quantile([.1,0.25,.50,0.75,0.99])))
lower_outlier_threshold = countries[column].quantile(0.01)
upper_outlier_threshold = countries[column].quantile(0.99)
print('\n\nLOWER OUTLIER THRESHOLD [1st PERCENTILE] FOR' ,column,': ',lower_outlier_threshold)
l_condition = countries[column] < lower_outlier_threshold
l_outliers = countries[l_condition][['country',column]].sort_values(by=column)
if l_outliers.shape[0] :
print('\n\nLower Outliers : ')
tab(l_outliers)
else :
print('No lower outliers found in ' + column)
print('\n\nUPPER OUTLIER THRESHOLD [99th PERCENTILE] FOR' ,column,': ',upper_outlier_threshold)
u_condition = countries[column] > upper_outlier_threshold
u_outliers = countries[u_condition][['country',column]].sort_values(by=column)
if u_outliers.shape[0] :
print('\n\nUpper Outliers : ')
tab(u_outliers)
print('\n\n')
return l_condition, u_condition
Child Mortality¶
In [96]:
# Countries with oultiers in child mortality
column = 'child_mort'
l_condition, u_condition = outlier_analysis(column)
- Notice that half of the countries have a child mortality below 20.
- Further,countries with child mortality less than 1st percentile might not need aid at all. We shall cap their child mortalities to 1st percentile values.
- Countries with extremely high child mortality rate - upper outliers (> 99th percentile) are the perfect candidates for aid. Let's keep them as they are for further analysis.
In [97]:
# Removing countries with lower outliers in `child_mort`
countries.loc[l_condition, column] = countries[column].quantile(0.01)
Life Expectancy¶
In [98]:
# LIFE EXPECTANCY
column = 'life_expec'
l_condition, u_condition = outlier_analysis(column)
- Some of the lowest life expectancy is seen in Haiti and Lesoto.
- About 50% of the countries have a life expectany of 73 or below and the other 50% have above 73.
- Countries like Singapore and Japan have the highest life expective, better than 99% of the countries.
- Countries with very low life expectancy are possible candidates for aid.
- Hence, let's keep the lower outliers.
- One the other hand, countries like Singapore and Japan might not need aid but these values would skew our analysis. Let's cap these outliers.
In [99]:
# Capping upper outliers in life expectancy
countries.loc[u_condition, column] = countries[column].quantile(0.99)
Fertility¶
In [100]:
column = 'total_fer'
l_condition, u_condition = outlier_analysis(column)
- About 50% of the countries have a fertility of 2.41 or less.
- Lower fertility is seen in developed nations like Singapore and South Korea, where the fertility is less than 99% countries.
- Countries with higher total fertility might need the aid more since this mean more health risk.
- Fertility in Chad and Niger is higher than 99 percent of the countries.
- Since these countries might need the aid, let's leave these values for further analysis.
- Countries with fertility rates less than 1 perecent of the population look like they are developed nations. Let's cap these outlier so that they don't skew our analysis.
- Further, from the violin plot, notice that fertility has two peaks - 2 and 5. This indicates that fertility rate could be used to effectively segregate countries. More analysis is needed here.
In [101]:
# capping lower outliers in fertility
countries.loc[l_condition,column] = countries[column].quantile(.1)
Health Spending¶
In [102]:
# health spending per capita
column = 'health'
l_condition, u_condition = outlier_analysis(column)
- The countries with more health spending are less in need of aid.
- Like Switzerland and United States, whose health spending is higher than 99% of the countries. But values like these would skew the entire analysis for aid needing countries. Hence, we'd cap these values with the 99th percentile value.
- The countries with less health spending might have variety of reasons like - optimum general health , or bad economic conditions. Let's keep these values for further analysis.
In [103]:
# removing upper outliers in health spending
countries.loc[u_condition,column] = countries[column].quantile(0.99)
GDP per capita¶
In [104]:
# GDP per capita
column = 'gdpp'
l_condition, u_condition = outlier_analysis(column)
- GDP per capita is a very good indicator of a country's prosperity
- Notice that there are three peaks in the violin plot of GDP. These might indicate the three clusters - under developed, developing and developed nations.
- Under developed nations are the most in need of aid. Countries like Burundi and Libera have GDPs less than 99% of the countries. Even though they are outliers, let's keep them for further analysis.
- But, we could cap GDPs which are greater than GDPs of 99% of the countries (Luxembourg & Norway)
In [105]:
# capping upper outliers in gdpp
countries.loc[u_condition, column] = countries[column].quantile(.99)
Inflation Index¶
In [106]:
column = 'inflation'
l_condition, u_condition = outlier_analysis(column)
- Very high inflation indicates a bad economic state.
- The violinplot again indicates three peaks - indicating three possible clusters of inflation - each with progressively less countries.
- Countries with very high inflation are might need aid. Since this is our area of interest, let's keep the upper outliers in inflation as they are for further analysis.
- Let's cap lower outliers since these look like good economies with no need of aid.
In [107]:
# capping lower outliers in inflation
countries.loc[l_condition, column] = countries[column].quantile(0.01)
Net Income per person¶
In [108]:
# income
column = 'income'
l_condition, u_condition = outlier_analysis(column)
- High net income per person is an indicator of general prosperity of a country. Such countries do not need aid.
- Hence, we shall cap the upper outliers i,e net incomes greater than 99% of that of the other countries.
- Lower outliers, countries with net incomes less than 1% of that of the other countries is our area of interest. Lets leave these values as they are for further analysis.
Imports per capita¶
In [109]:
column = 'imports'
l_condition, u_condition = outlier_analysis(column)
Exports per capita¶
In [110]:
column = 'exports'
l_condition, u_condition = outlier_analysis(column)
Trade Deficit per capita¶
In [111]:
# trade deficit
countries['trade_deficit'] = countries['imports'] - countries['exports']
column = 'trade_deficit'
l_condition, u_condition = outlier_analysis(column)
- Trade deficit indicates the balance between exports and imports. Negative trade deficit is favourable indicator of economic health. It means that the country has higher exports compared to imports.
- When the trade deficit is negative, it means that the country is import predominant.
- From the above plot, Greece and Bahamas have a trade deficit higher than 99% of the countries. This could mean a bad economic state. Since this is our key area of interest, we could leave these outliers as they are.
- Countries with negative trade deficit , i.e export predominant countries could be capped to 1st percentile values.
In [112]:
# capping lower outliers of trade_deficit
countries.loc[l_condition,column] = countries[column].quantile(0.01)
In [113]:
# Pair Plots of all variables
sns.pairplot(countries[['child_mort', 'exports', 'health', 'imports', 'income',
'inflation', 'life_expec', 'total_fer', 'gdpp']]);
In [114]:
## bivariate analysis boken function
def bivariate_analysis(x_var,y_var,dataframe=countries) :
# Bivariate Plots with tooltips : country,x,y
dataframe = dataframe.copy()
source =ColumnDataSource(dataframe)
# pallete = ["rgba(38, 70, 83, 1)", 'rgba(42, 157, 143, 1)', "rgba(233, 196, 106, 1)", "rgba(244, 162, 97, 1)", "rgba(231, 111, 81, 1)", '#009cc7']
pallete = ["#9b59b6", "#3498db", "#95a5a6", "#e74c3c", "#34495e","#38895e",'rgba(42, 157, 143, 1)']
# tooltips
tooltips1 = [
("Country", "@country"),
(x_var,'@'+x_var),
(y_var,'@'+y_var)
]
p = figure(plot_width=420, plot_height=400,title='Scatter Plot : '+ x_var+' vs '+y_var, tooltips=tooltips1)
p.scatter(x=x_var,y=y_var, fill_alpha=0.6, size=8, source=source, fill_color = "#3498db")
p.xaxis.axis_label = x_var
p.yaxis.axis_label = y_var
reset_output()
output_notebook()
show(p)
Child Mortality vs Life Expectancy¶
In [115]:
# CHILD MORTALITY vs LIFE EXPECTANCY
x='child_mort'
y = 'life_expec'
bivariate_analysis(x,y)
- looks like
child_mortandlife_expecare almost linearly related. - Life expectancy decreases as child mortality increases.
- One of these features is enough for analysis.
In [116]:
# Countries with high child mortality and low life expectancy
life_expect_cond = countries['life_expec'] < 60
child_mort_cond = countries['child_mort'] > 100
print('Countries with low life expectancy and low Child Mortality Rate')
tab(countries[life_expect_cond & child_mort_cond][['country','child_mort','life_expec']].sort_values(by=['child_mort','life_expec'], ascending=[False,True])[:10])
Child Mortality vs Total Fertility¶
In [117]:
# CHILD MORTALITY vs TOTAL FERTILITY
x='child_mort'
y = 'total_fer'
bivariate_analysis(x,y)
In [118]:
# countries with high fertility and child mortality rate
total_fer_cond = countries['total_fer'] > 6
child_mort_cond = countries['child_mort'] > 100
print('Countries with High Total Fertility and Child Mortality Rate')
tab(countries[total_fer_cond & child_mort_cond][['country','total_fer','child_mort']].sort_values(by=['child_mort','total_fer'], ascending=[False,False]))
In [119]:
# Fertility in countries with extremely high mortality rate
child_mort_cond = countries['child_mort'] > 150
print('Countries with High Total Fertility and Child Mortality Rate')
tab(countries[child_mort_cond][['country','total_fer','child_mort']].sort_values(by=['child_mort','total_fer'], ascending=[False,False]))
GDP per capita vs Health Spending¶
In [120]:
# GDP per capita vs Health Spending
y='health'
x= 'gdpp'
bivariate_analysis(x,y)
In [121]:
# Countries with Low GDPP and health spending might need aid
health_cond = countries['health'] < 200
gdpp_cond = countries['gdpp'] < 2500
print('Countries with Low GDP and low Health Spending')
tab(countries[health_cond & gdpp_cond][['country','health','gdpp']].sort_values(by=['health','gdpp'], ascending=[True,True])[:10])
Imports vs GDP per capita¶
In [122]:
# IMPORTS vs GDPP
y='imports'
x= 'gdpp'
bivariate_analysis(x,y)
Exports vs GDP per capita¶
In [123]:
# Exports vs GDPP
y='exports'
x= 'gdpp'
bivariate_analysis(x,y)
GDP per capita vs Trade Deficit¶
In [124]:
y='trade_deficit'
x= 'gdpp'
bivariate_analysis(x,y)
- Trade deficit is a country's net imports minus exports.
- It indicates the dependence of a country on imports
- Countries with low GDP per capita and positive trade deficit are likely to require aid
- From the above plot, one can see that there are many such countries
In [125]:
# Countries with positive high trade deficit and low GDP per capita
tr_deficit_cond = countries['trade_deficit'] > 1000
gdpp_cond = countries['gdpp'] < 10000
print('Countries with high Trade Deficit and Low GDP')
tab(countries[tr_deficit_cond & gdpp_cond][['country','gdpp','trade_deficit']].sort_values(by=['trade_deficit','gdpp'], ascending=[True,False])[:10])
Net Income per person vs Inflation¶
In [126]:
# Income vs Inflation
x = 'inflation'
y = 'income'
bivariate_analysis(x,y)
In [127]:
# Low Income - High inflation countries might require aid
inflation_cond = countries['inflation'] > 20
income_condition = countries['income'] < 10000
print('Countries with high inflation and low income')
tab(countries[inflation_cond & income_condition][['country','inflation','income']].sort_values(by=['inflation','income'], ascending=[False, True]))
- Countries with High inflation and low income are possible candidates for aid requirement.
GDP per capita vs Inflation¶
In [128]:
x = 'gdpp'
y = 'inflation'
bivariate_analysis(x,y)
- Countries with low GDP percapita and high Inflation are in dire need of support.
- For example, Nigeria has an inflation > 100 while its GDP is 2330.
Correlation Analysis¶
In [129]:
plt.figure(figsize=[12,12])
sns.heatmap(countries.corr(),annot=True,cmap='YlGnBu', center=0)
Out[129]:
Top Correlations
- Negative Correlation between
life_expecandchild_mort - Positive Correlation between
total_ferandchild_mort
Although clustering analysis is not affected by multicollinearity, this plot shows us the possible linear relationships between different features to help with results obtained from cluster analysis.
Hopkin's Statistic¶
In [130]:
# hopkins test function
from sklearn.neighbors import NearestNeighbors
from random import sample
from numpy.random import uniform
from math import isnan
def hopkins(X):
d = X.shape[1]
#d = len(vars) # columns
n = len(X) # rows
m = int(0.1 * n)
nbrs = NearestNeighbors(n_neighbors=1).fit(X.values)
rand_X = sample(range(0, n, 1), m)
ujd = []
wjd = []
for j in range(0, m):
u_dist, _ = nbrs.kneighbors(uniform(np.amin(X,axis=0),np.amax(X,axis=0),d).reshape(1, -1), 2, return_distance=True)
ujd.append(u_dist[0][1])
w_dist, _ = nbrs.kneighbors(X.iloc[rand_X[j]].values.reshape(1, -1), 2, return_distance=True)
wjd.append(w_dist[0][1])
H = sum(ujd) / (sum(ujd) + sum(wjd))
if isnan(H):
print(ujd, wjd)
H = 0
return H
In [131]:
## Data used for Clustering
columns_for_clustering = ['child_mort', 'exports', 'health', 'imports', 'income',
'inflation', 'life_expec', 'total_fer', 'gdpp', 'trade_deficit']
clustering_data = countries[columns_for_clustering].copy()
In [132]:
# Hopkin's test
n = 10
hopkins_statistic = []
for i in range(n) :
hopkins_statistic.append(hopkins(clustering_data))
print('Min Hopkin\'s Statistic in ',n,'iterations :', min(hopkins_statistic))
print('Max Hopkin\'s Statistic in ',n,'iterations :', max(hopkins_statistic))
print('Mean Hopkin\'s Statistic in ',n,'iterations :', np.mean(hopkins_statistic))
print('Std deviation of Hopkin\'s Statistic in ',n,'iterations :', np.std(hopkins_statistic))
Since hopkin's statistic is greater than 80% , the data shows good clustering tendency
Standardizing Values¶
In [133]:
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
clustering_data[columns_for_clustering] = scaler.fit_transform(clustering_data[columns_for_clustering])
tab(clustering_data.describe())
K-Means Clustering¶
In [134]:
# Plotting Elbow curve of Sum of Squared distances of points in each cluster from the centroid of the nearest cluster.
ssd = []
range_n_clusters = np.arange(2,9)
for num_clusters in range_n_clusters :
kmeans = KMeans(n_clusters=num_clusters)
kmeans.fit(clustering_data)
ssd.append(kmeans.inertia_)
plt.plot(range_n_clusters,ssd)
plt.title('Elbow Curve');
plt.xlabel('No of clusters');
plt.ylabel('SSD');
- From the above Elbow curve, one can clearly see that there is a high gradient descent from k=2 to k=4 and then the curve tapers (Change in slope is not as SIGNIFICANT as earlier)
- Hence, k=4 is optimum no of clusters, statistically.
Sihoutte Analysis¶
In [135]:
from sklearn.metrics import silhouette_score
no_of_clusters = np.arange(2,10)
score = []
for n_cluster in no_of_clusters :
kmeans = KMeans(n_clusters=n_cluster, init='k-means++')
kmeans = kmeans.fit(clustering_data)
labels = kmeans.labels_
score.append(silhouette_score(clustering_data,labels))
plt.title('Silhouette Analysis Plot')
plt.xlabel('No of Clusters')
plt.ylabel('Silhouette Score')
plt.plot(no_of_clusters, score);
print(score)
- Higher the silhouette score the better
- However , from the above plot, we see silhouette score is the highest for k = 2, sharply falls at 3 and there's a local maximum at k=4
- k = 4 seems to be the optimum no of clusters
Final k - Means Clustering¶
In [136]:
# k - means clustering algo with k = 4
n_cluster = 4
kmeans = KMeans(n_clusters=n_cluster, init='k-means++', random_state = 100)
kmeans = kmeans.fit(clustering_data)
labels = kmeans.labels_
countries['k_means_cluster_id'] = labels
In [137]:
# Countries in each Cluster - k means
for cluster_no in range(n_cluster) :
condition = countries['k_means_cluster_id'] == cluster_no
print( 'CLUSTER #',cluster_no,'\n',countries.loc[condition,'country'].values,'\n\n')
In [138]:
# Agglomerative Single Linkage
mergings = linkage(clustering_data,method='single',metric='euclidean')
plt.figure(figsize=[16,10])
plt.title('Single Linkage - Hierarchical Clustering')
dendrogram(mergings);
HAC : Complete Linkage, Euclidean Measure¶
In [139]:
# Complete Linkage
mergings = linkage(clustering_data,method='complete',metric='euclidean')
plt.figure(figsize=[16,10])
plt.title('Complete Linkage - Hierarchical Clustering')
dendrogram(mergings);
- Hierarchical clustering with complete linkage has a more discriminative dendrogram
In [140]:
# Using Complete Linkage, cutting the tree for 6 clusters
n_clusters = 6
cluster_labels = cut_tree(mergings, n_clusters=n_clusters)
countries['hac_complete_cluster_id'] = cluster_labels
In [141]:
# Countries in each Cluster - Hierarchical - Complete Linkage
for cluster_no in range(n_clusters) :
condition = countries['hac_complete_cluster_id'] == cluster_no
print( 'CLUSTER #',cluster_no,'\n',countries.loc[condition,'country'].values,'\n\n')
HAC : Average Linkage, Euclidean Measure¶
In [142]:
# Average Linkage
mergings = linkage(clustering_data,method='average',metric='euclidean')
plt.figure(figsize=[16,10])
plt.title('Average Linkage - Hierarchical Clustering')
dendrogram(mergings);
In [143]:
# Using Average Linkage, cutting the tree for 6 clusters
n_clusters = 6
cluster_labels = cut_tree(mergings, n_clusters=n_clusters)
countries['hac_average_cluster_id'] = cluster_labels
In [144]:
# Countries in each Cluster - Hierarchical - average Linkage
for cluster_no in range(n_clusters) :
condition = countries['hac_average_cluster_id'] == cluster_no
print( 'CLUSTER #',cluster_no,'\n',countries.loc[condition,'country'].values,'\n\n')
HAC : Complete Linkage , Correlation Measure¶
In [145]:
## HAC Clustering : Dissimilarity Measure : Correlation
hac_correlation_mergings = linkage(clustering_data,method='complete', metric='correlation')
plt.figure(figsize=[12,12])
plt.title('Hierarchical Clustering : Complete Linkage, Correlation Measure')
dendrogram(hac_correlation_mergings);
In [146]:
n_clusters = 6
labels = cut_tree(hac_correlation_mergings, n_clusters=n_clusters)
countries['hac_correlation_cluster_id'] = labels
In [147]:
# HAC clustering : Correlation measure : complete distance : Countries in each cluster
for cluster_no in range(n_clusters) :
condition = countries['hac_correlation_cluster_id'] == cluster_no
print( 'CLUSTER #',cluster_no,'\n',countries.loc[condition,'country'].values,'\n\n')
Mixed K-Means Clustering¶
In [148]:
# Performing k-means using results of Hierarchical clustering
# 1. No of clusters of Hierarchical Clustering
# 2. Centroids obtainded from Hierarchical Clustering as the initialization points.
clustering_data['k_means_cluster_id'] = countries['k_means_cluster_id']
clustering_data['hac_correlation_cluster_id'] = countries['hac_correlation_cluster_id']
columns = columns_for_clustering.copy()
columns.extend(['hac_correlation_cluster_id'])
centroids = clustering_data[columns].groupby(['hac_correlation_cluster_id']).mean()
n_clusters = 6
mixed_kmeans = KMeans(n_clusters=6 , init = centroids.values, random_state=100)
results = mixed_kmeans.fit(clustering_data[columns_for_clustering])
In [149]:
countries['mixed_cluster_id'] = results.labels_
In [150]:
# Mixed clustering : Euclidean measure : k-means : Countries in each cluster
for cluster_no in range(n_clusters) :
condition = countries['mixed_cluster_id'] == cluster_no
print( 'CLUSTER #',cluster_no,'\n',countries.loc[condition,'country'].values,'\n\n')
In [151]:
# silhouette scores of all the methods
print('Mixed Clustering',silhouette_score(clustering_data[columns_for_clustering],countries['mixed_cluster_id']))
print('K-means Clustering',silhouette_score(clustering_data[columns_for_clustering],countries['k_means_cluster_id']))
print('Hierarchical Correlation Clustering',silhouette_score(clustering_data[columns_for_clustering],countries['hac_correlation_cluster_id']))
Cluster Profiling¶
In [152]:
# Clustering Profling - Plots using Bokeh
cluster_id_column = 'hierarchical-c-link-cluster-id'
title="Hierarchical Clustering"
def cluster_analysis_plot(cluster_id_column,title,x_var='income',y_var='child_mort',z_var='gdpp',dataframe=countries) :
# Plots
# works upto 6 clusters
dataframe = dataframe.copy()
dataframe[cluster_id_column] = dataframe[cluster_id_column].astype('str')
source =ColumnDataSource(dataframe)
cluster_ids = sorted(dataframe[cluster_id_column].unique())
# pallete = ["rgba(38, 70, 83, 1)", 'rgba(42, 157, 143, 1)', "rgba(233, 196, 106, 1)", "rgba(244, 162, 97, 1)", "rgba(231, 111, 81, 1)", '#009cc7']
pallete = ["#9b59b6", "#3498db", "#95a5a6", "#e74c3c", "#34495e","#38895e",'rgba(42, 157, 143, 1)']
mapper = factor_cmap(cluster_id_column,palette=pallete[:len(cluster_ids)], factors = cluster_ids)
# plot 1
tooltips1 = [
("Country", "@country"),
(z_var,'@'+z_var),
(x_var,'@'+x_var),
('Cluster', '@'+cluster_id_column)
]
p = figure(plot_width=420, plot_height=400,title=title+ ' : '+ z_var+' vs '+x_var, tooltips=tooltips1, toolbar_location=None)
for num,index in enumerate(cluster_ids) :
condition = dataframe[cluster_id_column] == index
source = dataframe[condition]
p.scatter(x=z_var,y=x_var, fill_alpha=0.6, size=8, source=source, fill_color = pallete[num] , muted_alpha=0.1, legend_label=index)
p.xaxis.axis_label = z_var
p.yaxis.axis_label = x_var
p.legend.click_policy="mute"
# ----------------------------
#Plot 2
tooltips2 = [
("Country", "@country"),
(z_var,'@'+z_var),
(y_var,'@'+y_var),
('Cluster', '@'+cluster_id_column)
]
q = figure(plot_width=420, plot_height=400,title=title+ ' : '+ z_var+' vs '+y_var, tooltips=tooltips2, toolbar_location=None)
for num,index in enumerate(cluster_ids) :
condition = dataframe[cluster_id_column] == index
source = dataframe[condition]
q.scatter(x=z_var,y=y_var, fill_alpha=0.6, size=8, source=source, fill_color = pallete[num] , muted_alpha=0.1, legend_label=index)
q.xaxis.axis_label = z_var
q.yaxis.axis_label = y_var
q.legend.click_policy="mute"
# ----------------------------
#Plot 3
tooltips3 = [
("Country", "@country"),
(x_var,'@'+x_var),
(y_var,'@'+y_var),
('Cluster', '@'+cluster_id_column)
]
r = figure(plot_width=420, plot_height=400,title=title+ ' : '+ x_var+' vs '+y_var, tooltips=tooltips3, toolbar_location=None)
for num,index in enumerate(cluster_ids) :
condition = dataframe[cluster_id_column] == index
source = dataframe[condition]
r.scatter(x=x_var,y=y_var, fill_alpha=0.6, size=8, source=source, fill_color = pallete[num] , legend_label=index, muted_alpha=0.1 )
r.xaxis.axis_label = x_var
r.yaxis.axis_label = y_var
r.legend.click_policy="mute"
show(row(p,q,r))
K-Means¶
In [153]:
# Cluster Profiling for k-means with 4 clusters
# hover for country names and x, y values , cluster no
# Click on legend to selectively view clusters
cluster_analysis_plot('k_means_cluster_id','k-means clusters')
In [154]:
# Comparing k-means Clusters using mean values of features
clustering_data[['child_mort','income','gdpp','k_means_cluster_id']].groupby('k_means_cluster_id').mean().plot(kind='barh')
plt.title('Comparison of Cluster Means for K-means results');
In [155]:
plot_columns = ['child_mort','income','gdpp']
for idx,column in enumerate(plot_columns) :
plt.suptitle('Comparison of Clusters Characteristics for K-means');
plt.subplot('13'+str(idx+1))
sns.boxplot(y=column, x='k_means_cluster_id',data=clustering_data)
In [156]:
plt.figure(figsize=[8,8])
pd.plotting.parallel_coordinates(clustering_data, 'k_means_cluster_id', color=["#9b59b6", "#3498db", "#95a5a6", "#e74c3c", "#34495e"]);
plt.title('Parallel Coordinate Plot for K-means')
plt.xticks(rotation=45);
From the above plots, we could characterize features of each cluster as within the following levels.
- Levels : Low, Moderate, High , very high
Characteristics of each cluster
| Cluster | GDP | Income | Child Mortality |
|---|---|---|---|
| 0 | High | High to Very High | Low |
| 1 | Low | Low | High to Very High |
| 2 | Low to Moderate | Low to Moderate | Low to Moderate |
| 3 | Very High | Very High | Low |
From the characteristics , we see that the Cluster 1 is our area of interest. Lets look at the countries in Cluster 1.
In [157]:
# Countries in cluster with area of interest
condition = countries['k_means_cluster_id'] == 1
countries.loc[condition,['country','child_mort','income','gdpp']].sort_values(by=['child_mort','income','gdpp'], ascending=[False,True,True])[:5]
Out[157]:
Hierarchical Clustering - Complete Linkage, Correlation based distance¶
In [158]:
# HAC : Complete linkage, Correlation based distance : cluster analysis plot
# hover for country names and x, y values , cluster no
# Click on legend to selectively view clusters
cluster_analysis_plot('hac_correlation_cluster_id','HAC CORRELATION CLUSTERS')
In [159]:
# Comparing Hierarchical Clusters using mean values of features
clustering_data['hac_correlation_cluster_id'] = countries['hac_correlation_cluster_id']
clustering_data[['child_mort','income','gdpp','hac_correlation_cluster_id']].groupby('hac_correlation_cluster_id').mean().plot(kind='barh');
plt.title('Comparison of Cluster Means for Hierarchical Clustering');
In [160]:
# box plots
plot_columns = ['child_mort','income','gdpp']
for idx,column in enumerate(plot_columns) :
plt.suptitle('Comparison of Clusters Characteristics for Hierarchical Clustering');
plt.subplot('13'+str(idx+1))
sns.boxplot(y=column, x='hac_correlation_cluster_id',data=clustering_data)
In [161]:
# Parallel Coordinates plot for Hierarchical clustering with correlation measure and complet linkage
plt.figure(figsize=[8,8])
pd.plotting.parallel_coordinates(clustering_data, 'hac_correlation_cluster_id', color=["#9b59b6", "#3498db", "#95a5a6", "#e74c3c", "#34495e","#38895e",'rgba(42, 157, 143, 1)']);
plt.title('Parallel Coordinate Plot for HAC with Correlation Measure and Complete Linkage')
plt.xticks(rotation=45);
From the above plots, we could characterize features of each cluster as within the following levels.
- Levels : Very low , low, Moderate, High , very high
Characteristics of each cluster
| Cluster | GDP | Income | Child Mortality |
|---|---|---|---|
| 0 | Very low | Very Low | Very High |
| 1 | low | low | low |
| 2 | Moderate | Moderate | Moderate |
| 3 | High | High | Low |
| 4 | Very High | Very High | Low |
| 5 | Very Low | Very Low | Low |
From the characteristics , we see that the Cluster 0 is our area of interest. Lets look at the countries in Cluster 0.
In [162]:
# Countries in cluster with area of interest
condition = countries['hac_correlation_cluster_id'] == 0
countries.loc[condition,['country','child_mort','income','gdpp']].sort_values(by=['child_mort','income','gdpp'], ascending=[False,True,True])[:5]
Out[162]:
Mixed Clustering : K-means initialized with Hierarchical Cluster Centroids¶
In [163]:
cluster_analysis_plot('mixed_cluster_id','MIXED K-MEANS CLUSTERS')
In [164]:
# Comparing MIXED k-means Clusters using mean values of features
clustering_data['mixed_cluster_id'] = countries['mixed_cluster_id']
clustering_data[['child_mort','income','gdpp','mixed_cluster_id']].groupby('mixed_cluster_id').mean().plot(kind='barh')
plt.title('Comparison of cluster means for Mixed Clustering');
In [165]:
# box plots
plot_columns = ['child_mort','income','gdpp']
for idx,column in enumerate(plot_columns) :
plt.suptitle('Comparison of Clusters Characteristics for Mixed Clustering');
plt.subplot('13'+str(idx+1))
sns.boxplot(y=column, x='mixed_cluster_id',data=clustering_data)
In [166]:
# parallel coordinate plot
plt.figure(figsize=[8,8])
pd.plotting.parallel_coordinates(clustering_data, 'mixed_cluster_id', color=["#9b59b6", "#3498db", "#95a5a6", "#e74c3c", "#34495e","#38895e",'rgba(42, 157, 143, 1)']);
plt.title('Parallel Coordinate Plot for Mixed Clustering');
plt.xticks(rotation=45);
- From the above plots, Cluster 0 has the very low income and gdpp and very high child mortality. Hence, Cluster 0 is the area of interest.
In [167]:
# Countries in cluster with area of interest
condition = countries['mixed_cluster_id'] == 0
countries.loc[condition,['country','child_mort','income','gdpp']].sort_values(by=['child_mort','income','gdpp'], ascending=[False,True,True])[:5]
Out[167]:
- Hence , the top five countries to extend aid are
- Haiti
- Sierra Leone
- Chad
- Central African Republic
- Mali
Conclusion¶
Columns : 'child_mort', 'exports', 'health', 'imports', 'income','inflation', 'life_expec', 'total_fer', 'gdpp','trade_deficit' were used for clustering.
Hopkin's Statistic was calculated which showed a very high mean clustering tendency of 96% with a standard deviation of 4%
- The dataset was standardized with mean = 0 , standard deviation = 1 before clustering.
- Optimum no of clusters for k-means was found to be 4 , both from Elbow curve and Silhoeutte Analysis curve.
- Clustering with k-means was performed (Cluster centers initialized with k-means++)
- Hierarchical Clustering with Single, Complete and Average linkages and Euclidean , correlation based distances were explored.
- Mixed k-means clustering - K-means initialized with centroids of hierarchical clustering - was also explored.
Due to more interpretable results, Hierarchical Clustering with Complete linkage and correlation based distance measure was chosen to arrive at the results.
Characteristics of Clusters obtained :
| Cluster | GDP | Income | Child Mortality |
|---|---|---|---|
| 0 | Very low | Very Low | Very High |
| 1 | low | low | low |
| 2 | Moderate | Moderate | Moderate |
| 3 | High | High | Low |
| 4 | Very High | Very High | Low |
| 5 | Very Low | Very Low | Low |
- From the characteristics , we see that the Cluster 0 is our area of interest.
- According to the UN goals of 2030, the top priority is health and then poverty.
- Hence countries in cluster 0 were ranked based on Child Mortality followed by income and GDP per capita.
- By that criteria, the following are the five countries HELP should consider extending their aid.
- Haiti
- Sierra Leone
- Chad
- Central African Republic
- Mali