Table of Contents
- 1 Problem Statement
- 2 Business Goals
- 3 Analysis Approach & Conclusions
- 4 Importing Data
- 5 Data Cleaning
- 6 Retained Data
- 7 Data Imbalance
- 8 Univariate Analysis
- 9 Bivariate Analysis
- 10 Data Preparation
- 11 Modelling
- 12 Final Features
- 13 Predictions
- 13.1 Predictions on Train set
- 13.1.1 Actual Conversions vs Conversion Predictions
- 13.1.2 Predictions with cut off = 0.5
- 13.1.3 Confusion Matrix
- 13.1.4 Accuracy of the Model
- 13.1.5 Metrics beyond simple accuracy
- 13.1.6 Plotting ROC Curve.
- 13.1.7 Finding Optimal Cutoff Point
- 13.1.8 Precision and Recall
- 13.1.9 Precision and Recall Tradeoff
- 13.2 Predictions on Test set
- 13.1 Predictions on Train set
- 14 Lead Scoring
- 15 Score Sheet for X Education
- 16 KS Statistic
- 17 Gain Chart
- 18 Lift Chart
- 19 Conclusion
Lead Scoring for X Education¶
Problem Statement¶
An education company named X Education sells online courses to industry professionals. On any given day, many professionals who are interested in the courses land on their website and browse for courses.
The company markets its courses on several websites and search engines like Google. Once these people land on the website, they might browse the courses or fill up a form for the course or watch some videos. When these people fill up a form providing their email address or phone number, they are classified to be a lead. Moreover, the company also gets leads through past referrals. Once these leads are acquired, employees from the sales team start making calls, writing emails, etc. Through this process, some of the leads get converted while most do not. The typical lead conversion rate at X education is around 30%.
Now, although X Education gets a lot of leads, its lead conversion rate is very poor. For example, if, say, they acquire 100 leads in a day, only about 30 of them are converted. To make this process more efficient, the company wishes to identify the most potential leads, also known as ‘Hot Leads’. If they successfully identify this set of leads, the lead conversion rate should go up as the sales team will now be focusing more on communicating with the potential leads rather than making calls to everyone. A typical lead conversion process can be represented using the following funnel:
X Education has appointed us to help them select the most promising leads, i.e. the leads that are most likely to convert into paying customers. The company requires us to build a model wherein we need to assign a lead score to each of the leads such that the customers with higher lead score have a higher conversion chance and the customers with lower lead score have a lower conversion chance. The CEO, in particular, has given a ballpark of the target lead conversion rate to be around 80%.
Business Goals¶
- X-Education wants to improve their lead conversion.
- Rather than randomly pursuing leads, the company wants to create a pool of Hot Leads the sales team could focus on.
- They have tasked us to score their leads betwen 0-100 based on the probability of conversion. 100 being the most likely to convert and 0 being unlikely to convert.
Analysis Approach & Conclusions¶
Lead scoring is a class probability estimation problem, a form of classification problem. The target variable in the data set has two classes : 0 - Un converted and 1 - Converted. The objective is to model the probability(p) that each lead belongs to the class - Converted. Since there are just two classes - it follows that the probability of belonging to class - Un-Converted is (1-p). The relationship between probability of conversion of each lead and its characteristics is modelled using Logistic Regression. And the leads are scored on a scale of 0-100, 100 being most probable conversion candidate.
The final solution has been provided in two parts.
1. Scoring the leads provided by the company in the order of probability of conversion (0-100)
2. Insights into the relationship between characteristics of a lead and the log-odds probability of conversion that could help the company score leads in the future.
A logistic regression model is created using lead features. To arrive at the list of features which significantly affect conversion probability, a mixed feature elimination approach is followed. 25 most important features are obtained through Recursive Feature Elimination and then reduced to 15 via p-value / VIF approach. The dataset is randomly divided into train and test set. (70 - 30 split).
The final relationship between log Odds of Conversion Probability and lead features is
logOdds(Conversion Probability) = -0.6469 - 1.5426 Do Not Email -1.2699 Unknown Occupation -0.9057 No Specialization -0.8704 Hospitality Management - 0.6584 Outside India + 1.7923 SMS Sent + 1.1749 Other Last Activity + 2.3769 Working Professional - 0.8614 Olark Chat Conversation + 5.3886 Welingak Website + 3.0246 Reference + 1.1876 Olark Chat -1.0250 Landing Page Submission + 1.1253 Total Time Spent on Website + 0.6106 * Email Opened
where Total Time Spent on Website is standardized to $\mu=0,\sigma=1$
Interpreting Top 6 features affecting Conversion Probability :
- A lead from
Welingak Websitehas 5.4 times higher log odds of conversion than those fromGoogle. - Leads through
Referencehave 3 times higher log odds of conversion than those fromGoogle. - Leads from
Working Professionalhave 2.38 times higher log odds of conversion than those fromBusinessman. - Leads with
SMS Senthave 1.8 times higher log odds of conversion than those with no SMS sent. - Leads with
Do Not Emailhave 1.5 times lesser log odds of conversion compared to leads who would like email updates. - Leads with
Unknown Occupationhave 1.27 times lesser log odds of conversion compared to those fromBusinessman.
Lead Scores :
- Score sheet can be generated by running coding in the cell named
Score Sheet for X Educationcell in the analysis notebook.
At an optimum cut-off probability of 0.36, model performance is as follows.
Model Performance on Training Set :
- Accuracy : 81.7%
- Sensitivity / Recall: 80.393 %
- Specificity : 81.772 %
- Precision / Positive Predictive Power : 72.924 %
- False Positive Rate : 18.228 %
- AUC Score : 0.81
Model Performance for Test Set :
- Accuracy : 79.593 %
- Sensitivity / Recall : 77.605 %
- Specificity : 80.81%
- Precision / Positive Predictive Power : 71.224 %
- False Positive Rate : 19.19 %
- AUC Score : 0.79
KS statistic :
- Max KS Statistic is 59.76 for 5th decile
- This model discriminates between Converted and Non-converted leads well since KS Statistic in 4th decile (58.11) is greater than 40%. Hence, this is a reasonably good model.
Gain :
- Instead of pursuing leads randomly, pursuing the top 40% leads scored by the model would let the sales team reach 80% of leads likely to convert.
Lift :
- The model outperforms a random model by alteast 2 times in identifying the top 40% potentially convertible leads.
- As opposed to 10% conversions from 10% leads pursued randomly, pursuing the top 10% leads scored by this model would lead to 24% conversions.
Note :
- Incorrect data types have been corrected
- Columns with high missing values have been dropped.
- Columns which do not explain variability in the model have been dropped.
- Columns with sales teams notes like
Tagswhere the classes are not mutually exclusive have been dropped. - Features with low missing values have been imputed with the most frequent values.
- Categories in a feature with less than 1% contribution have been grouped together to reduce the number of levels.
- Inconsistencies in Categories have been corrected.
- 97.5 % of the leads provided by the company have been used for analysis.
- Class imbalance = 0.6
- Indicator variables have been created for all categorical variables with the first category as the reference.
- Continuous variables have been standardized $\mu : 0 , \sigma = 1$ before modelling.
In [151]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('whitegrid')
import warnings
warnings.filterwarnings('ignore')
!pip install tabulate
from tabulate import tabulate
import sidetable
!jt -t grade3 -f roboto -fs 12 -cellw 100%
In [152]:
# to table print a dataframe
def tab(ser) :
print(tabulate(pd.DataFrame(ser), headers='keys', tablefmt="psql"))
Importing Data¶
In [153]:
# importing the dataset
leads = pd.read_csv('./leads.csv')
# Inspecting few column heads at a time
for i in range(0,leads.shape[1], 5) :
if i+4 <= leads.shape[1] :
print('Columns : ',i,' to ',i+4)
else :
print('Columns : ',i,' to last')
tab(leads.iloc[:,i : i+5].head())
print('\n')
In [154]:
# dataset information
leads.info()
- This data set has a total of 9240 records ,each with 36 features.
- Each record represents the characteristics of a lead and whether the lead was converted.
Convertedcolumn indicates whether the particular lead was converted to a client. This is our target variable.
Data Cleaning¶
Incorrect data types¶
In [155]:
# 'Converted' is a binary categorical variable but the info shows it is `int64`. Converting to `category` data type
leads['Converted'] = leads['Converted'].astype('category')
Duplicates¶
In [156]:
# Checking for any duplicate leads / prospects
duplicate_prospect_ids = leads['Prospect ID'][leads['Prospect ID'].duplicated()].sum()
duplicate_lead_no = leads['Lead Number'][leads['Lead Number'].duplicated()].sum()
print('No of Duplicate Prospect IDs : ', duplicate_prospect_ids)
print('No of Duplicate Lead Nos : ', duplicate_lead_no)
- There are no duplicate prospect IDs or Lead Numbers
- Since, these are dimensions (i.e identification columns) not required for analysis, they could be popped for re-indentification at a later step.
Separating ID columns¶
In [157]:
# Popping Prospect ID and Lead Number columns for later use
prospect_ids = leads.pop('Prospect ID')
lead_no = leads.pop('Lead Number')
Missing Values¶
In [158]:
# Null values in each Column
nulls = pd.DataFrame(100*leads.isnull().sum()/leads.shape[0])
nulls.columns = ['Null Percentage']
# Sorting null percentages in descending order and highlighting null % > 45
nulls[nulls['Null Percentage'] !=0].sort_values(by ='Null Percentage', ascending=False).style.applymap(lambda x : 'color : red' if x > 45 else '')
Out[158]:
- More than 45% of the leads have missing values in
Lead Quality,Asymmetrique Profile Score,Asymmetrique Activity Score,Asymmetrique Profile Index,Asymmetrique Activity Index - Further, the data in these columns is filled by the sales team and the values depend heavily on the team's judgement. These columns are not good candidates for modelling since the values are subjective.
- Hence these columns could be dropped.
In [159]:
# Dropping columns with null percentage > 45
high_null_col = nulls[nulls['Null Percentage'] >=45].index
leads.drop(columns=high_null_col, inplace=True)
In [160]:
# Rows Missing Target Variable
print('Number of rows with missing Target Variable : ',leads['Converted'].isnull().sum())
- No rows with missing target value
In [161]:
# Rows missing more than 50% of values
highNullRowsCondition = leads.isnull().sum(axis=1)/leads.shape[1] > 0.5
leads[highNullRowsCondition].index
Out[161]:
- No rows missing more than 50% of values.
Disguised missing Values¶
In [162]:
# Categorical columns
condition = leads.dtypes == 'object'
categoricalColumns = leads.dtypes[condition].index.values
categoricalColumns
Out[162]:
In [163]:
# value counts of each label in a categorical feature
def cat_value_counts(column_name) :
'''
prints unique values and value counts of each label in categorical column
'''
print(tabulate(pd.DataFrame(leads.stb.freq([column_name])), headers='keys', tablefmt='psql'))
print(pd.DataFrame(leads[column_name]).stb.missing(),'\n\n\n')
In [164]:
# Looking at value counts of each label in categorical variables
for col in sorted(categoricalColumns) :
print(col)
cat_value_counts(col)
- The following columns have a label
Selectwhich is a disguised missing value. Selectis the default option in online forms and this value might mean that the lead hasn't selected any option.- We shall replace them with
np.nan.SpecializationLead ProfileCityHow did you hear about X Education
In [165]:
# Replacing Select with NaN value
leads.replace({'Select' : np.nan},inplace=True)
In [166]:
# Looking at Missing Values again
# Null values in each Column
nulls = pd.DataFrame(100*leads.isnull().sum()/leads.shape[0])
nulls.columns = ['Null Percentage']
# Sorting null percentages in descending order and highlighting null % > 50
nulls[nulls['Null Percentage'] !=0].sort_values(by ='Null Percentage', ascending=False).style.applymap(lambda x : 'color : red' if x > 50 else '')
Out[166]:
Lead Profile&How did you hear about X Educationhave very high percentage of nulls. Let's drop these columns
In [167]:
leads.drop(columns=['Lead Profile','How did you hear about X Education'],inplace=True)
Imputation¶
In [168]:
# Sorting null percentages in ascending order and highlighting null % < 16
def lowNulls() :
nulls = pd.DataFrame(100*leads.isnull().sum()/leads.shape[0])
nulls.columns = ['Null Percentage']
return nulls[nulls['Null Percentage'] !=0].sort_values(by ='Null Percentage', ascending=True).style.applymap(lambda x : 'color : green' if x < 16 else '')
lowNulls()
Out[168]:
- 'Lead Source','Last Activity','TotalVisits','Page Views Per Visit' have less than 2% missing values. These rows could be dropped.
- We could impute columns with higher missing values on a case by case basis.
- Missing values are imputed by the metric most representative of the feature's distribution.
- For categorical features, missing values could be imputed by the most frequently occuring label i.e MODE value, since this is the most representative metric of a categorical feature.
- For continuous features, if there are outliers, the most representative metric of the feature's distribution is MEDIAN, else it is MEAN. Continuous feature imputations are thus dependent on presence of outliers.
- About 26% of data in Country Column is missing.
In [169]:
# Country Imputation :
leads.stb.freq(['Country'])
Out[169]:
- Since 95% of leads come from India, it is probable that missing values are from India.
In [170]:
# Imputing missing values in Country feature with "India"
leads['Country'].fillna('India', inplace=True)
Specializationfeature has 36% of missing values.- Since there's no one label that's driving leads, replacement would mislead the analysis.
- Hence, we could impute missing values with a new label 'No Specialization'
In [171]:
# Imputing Null Values by filling it using "No Specialization".
leads['Specialization'].fillna("No Specialization",inplace=True)
print('Missing values in Specialization feature ', leads['Specialization'].isnull().sum())
In [172]:
leads['Specialization'].value_counts()
Out[172]:
- About 39% of values in
Citycolumn are missing
In [173]:
# Imputation of missing cities
leads.stb.freq(['City'])
Out[173]:
In [174]:
# Missing Cities vs Country
condition_india = leads['Country'] == 'India'
print('Total Missing City values :', leads['City'].isnull().sum())
print('Missing City values in leads from India : ',leads.loc[condition_india,'City'].isnull().sum())
- Looks like 3609 out 3669 leads with Missing City label are from India.
- As can be seen from the value counts of
Cityfeature, 60% of the leads come from Mumbai. - Since, we could impute missing
Cityvalue for leads from India withMumbai
In [175]:
# Replacing Null Cities in India with Mumbai
condition = (leads['City'].isnull()) & condition_india
leads.loc[condition,'City'] = 'Mumbai'
- 29% of values in
What is your current occupationcolumn are missing. - Let's look at the distribution of levels in this column
In [176]:
tab(leads.stb.freq(['What is your current occupation']))
tab(leads['What is your current occupation'].reset_index().stb.missing())
- Since, the business problem clearly says the company targets working professionals, this is an extremely important variable.
- So to keep the analysis unbiased, we could impute missing values with a new level for now.
- Let's replace missing values in
What is your current occupationwith 'Unknown Occupation'
In [177]:
leads['What is your current occupation'].fillna('Unknown Occupation',inplace=True)
In [178]:
# Missing Values in `What matters most to you in choosing a course`
ftr = 'What matters most to you in choosing a course'
tab(leads.stb.freq([ftr]))
tab(leads[ftr].reset_index().stb.missing())
- Almost all leads ( >99%) show interest in the company's offerings for
Better Career Prospects, excluding leads who havent filled this feature. - Since this might be a very important feature from the analysis perspective, to keep the analysis unbiased, instead of imputation with an existing label, let's impute with a new label for now.
- Let's replace missing values in
What matters most to you in choosing a coursewith 'Unknown Target' for now.
In [179]:
# Filling Missing Values with `Unknown`
leads[ftr].fillna('Unknown Target',inplace=True)
Incorrect Labels¶
In [180]:
# Looking at labels in each Categorical Variable to check for incorrect labels.
categoricalFeatures = leads.dtypes[leads.dtypes == 'object'].index.values
print('Categorical Features : ', categoricalFeatures,'\n\n')
for feature in categoricalFeatures :
print('Levels in ',feature,' are ' , leads[feature].unique(),'\n\n')
- We can clearly see that Google is appearing twice in 'Lead Source'- (Google,google)
In [181]:
# Replacing 'google' with 'Google
leads['Lead Source']=leads['Lead Source'].str.replace("google","Google")
In [182]:
# Missing Values and Value Counts for all categorical Variables
tab(leads.stb.missing())
print('Value Counts of each Feature : \n')
for feature in sorted(categoricalFeatures) :
tab(leads.stb.freq([feature]))
- Let's look for columns have more than 99% leads have the same level
- Such variables do not explain any variability.
- Hence, these columns are unnecessary to the analysis . They could be dropped
In [183]:
# Dropping columns having only one label - since these donot explain any variability in the dataset
invariableCol = ['Digital Advertisement','Do Not Call','Get updates on DM Content','Magazine','Newspaper','Newspaper Article','Receive More Updates About Our Courses','Search',
'Update me on Supply Chain Content','Through Recommendations',
'I agree to pay the amount through cheque',"What matters most to you in choosing a course",'X Education Forums']
leads.drop(columns=invariableCol, inplace=True)
In [184]:
# Tags feature
tab(leads.stb.freq(['Tags']))
Tagscolumn shows remarks by the sales team. This a subjective variable based on judgement of the team and cannot be used for analysis since the lables might change or might not always be available.- Also, it has a lot of levels that don't seem like mutually exclusive classes
- Let's drop this feature for this analysis.
In [185]:
# dropping Tags feature
leads.drop(columns=['Tags'], inplace=True)
Last Notable Activity&Last Activityseem to have similar levels- Lets look at the possibility of dropping one of them
In [186]:
# Last Notable Activity vs Last Activity
leads_copy = leads.copy()
leads_copy['Converted'] = leads_copy['Converted'].astype('int')
tab(leads_copy.stb.freq(['Last Notable Activity'],value='Converted'))
tab(leads_copy.stb.freq(['Last Activity'],value='Converted'))
Last Activityhas more levels compared toLast Notable ActivityLast Notable Activityseems like a column derived by the sales team usingLast Activity.- Since this insight might not be available for a new lead, let'd drop
Last Notable Activity.
In [187]:
leads.drop(columns = ['Last Notable Activity'], inplace=True)
In [188]:
# Looking at Missing Values again
leads.stb.missing()
Out[188]:
- From the above, the number of missing values is less than 2%. These are deemed missing completely at random. And these rows could be dropped without affecting the analysis
In [189]:
leads.dropna(inplace=True)
leads.stb.missing()
Out[189]:
Grouping Labels with less leads¶
Country¶
In [190]:
# Country distribution
leads.stb.freq(['Country'])
Out[190]:
- We see that leads from India make 97% of all leads. And others collectively make up 3% and the contribution of each of these countries is <= 1%.
- To reduce the levels, let us group the minority labels into a new label called 'Outside India'
In [191]:
# Grouping Countries with very low lead count into 'Outside India'
leadsByCountry = leads['Country'].value_counts(normalize=True)
lowLeadCountries = leadsByCountry[leadsByCountry <= 0.01].index
leads['Country'].replace(lowLeadCountries,'Outside India',inplace=True)
leads.stb.freq(['Country'])
Out[191]:
Lead Origin¶
In [192]:
feature = 'Lead Origin'
leads.stb.freq([feature])
Out[192]:
- We see that lead origins like
Lead Add Form,Lead Importare less than 1% of all origins. - Let's group them into a level called 'Other Lead Origins'
In [193]:
# Grouping lead origins
leadOriginsToGroup = ["Lead Add Form","Lead Import"]
leads[feature] = leads[feature].replace(leadOriginsToGroup, ['Other Lead Origins']*2)
leads.stb.freq([feature])
Out[193]:
Lead Source¶
In [194]:
feature = 'Lead Source'
leads.stb.freq([feature])
Out[194]:
- Lead sources from #7 to #18 contribute to less than 1% of all sources.
- Let's group these into a new label called 'Other Lead Sources'
In [195]:
# Grouping lead Sources
labelCounts = leads[feature].value_counts(normalize=True)
# labels with less than 1% contribution
labelsToGroup = labelCounts[labelCounts < 0.01].index.values
leads[feature] = leads[feature].replace(labelsToGroup, ['Other '+feature+'s']*len(labelsToGroup))
leads.stb.freq([feature])
Out[195]:
Last Activity¶
In [196]:
feature = 'Last Activity'
leads.stb.freq([feature])
Out[196]:
- Leads from #9 to #16 contribute to less than 1% of all last activity labels.Moreover, each of these labels contributes to less than 1% of leads.
- Let's group these into a new label called 'Other Last Activity'
In [197]:
# Grouping Last Activity
labelCounts = leads[feature].value_counts(normalize=True)
# labels with less than 2% contribution
labelsToGroup = labelCounts[labelCounts < 0.01].index.values
leads[feature] = leads[feature].replace(labelsToGroup, ['Other '+feature]*len(labelsToGroup))
leads.stb.freq([feature])
Out[197]:
Specialization¶
In [198]:
feature = 'Specialization'
leads.stb.freq([feature])
Out[198]:
- Lead from from #16 to #18 contribute to less than 2% of all Specialization categories. Moreover, each of these categories contributes to less than 1% of leads.
- Let's group these into a new label called 'Other Specializations'
In [199]:
# Grouping Last Activity
labelCounts = leads[feature].value_counts(normalize=True)
# labels with less than 2% contribution
labelsToGroup = labelCounts[labelCounts <=0.012121].index.values
leads[feature] = leads[feature].replace(labelsToGroup, ['Other '+feature]*len(labelsToGroup))
leads.stb.freq([feature])
Out[199]:
- Cleaning tasks have been completed. No more missing values exist
Retained Data¶
In [200]:
# Columns retained
print('Retained Columns\n\n', leads.columns.values)
In [201]:
# Retained rows
print('Retained rows : ',leads.shape[0])
print("Ratio of retained rows", 100*leads.shape[0]/9240)
Data Imbalance¶
In [202]:
leads.stb.freq(['Converted'])
Out[202]:
In [203]:
converted_cond = leads['Converted'] == 1
imbalance = leads[converted_cond].shape[0]/leads[~converted_cond].shape[0]
print('Class Imbalance : Converted /Un-converted =', np.round(imbalance,3))
- From the above, you can see that this data set contains 37% of converted leads and 62% of un-converted leads.
- Ratio of classes = 0.6
- The dataset is skewed towards 'unconverted leads'
Univariate Analysis¶
In [204]:
def categoricalUAn(column,figsize=[8,8]) :
''' Function for categorical univariate analysis '''
print('Types of ' + column)
tab(leads.stb.freq([column]))
converted = leads[leads['Converted'] == 1]
unconverted = leads[leads['Converted'] == 0]
print(column + ' for Converted Leads')
tab(converted.stb.freq([column]))
print(column + ' for Un-Converted Leads')
tab(unconverted.stb.freq([column]))
print(column + ' vs Conversion Rate')
tab((converted[column].value_counts()) / (converted[column].value_counts() + unconverted[column].value_counts()))
# bar plot
plt.figure(figsize=figsize)
ax = sns.countplot(y=column,hue='Converted',data=leads)
title = column + ' vs Lead Conversion'
ax.set(title= title)
Lead Origin¶
In [205]:
column = 'Lead Origin'
categoricalUAn(column,figsize=[8,8])
- Leads from
Landing Page Submissionfollowed byAPImake up 93% of all leads. - But it is interesting that 8.3% of leads coming from other sources have the highest conversion rate of 87.5%
Lead Source¶
In [206]:
column = 'Lead Source'
categoricalUAn(column,figsize=[8,8])
- Most leads that get converted come from
Google(31%), followed byDirect Traffic(28%) andOlark Chat(19%) - And leads through
Referencehave a very high conversion rate (91%)
Do not Email¶
In [207]:
feature = 'Do Not Email'
categoricalUAn(feature,figsize=[8,8])
- 92% of leads prefer to be sent Emails about the company.
Do not Email = No - And these are the most converted customers (40%)
Last Activity¶
In [208]:
# 'Last Activity'
feature = 'Last Activity'
categoricalUAn(feature,figsize=[8,8])
- Most leads open emails sent to them (38%) and that's their last activity.
- Among those leads who's last activity is opening emails, 37% are converted.
- Only 4% of last activity indicators show
Converted to Lead - Last activiy as 'SMS Sent' has highest conversion rate (62%).
- Last activiy as 'Email Bounced' has lowest conversion rate (7.9%).
Country¶
In [209]:
feature = 'Country'
categoricalUAn(feature,figsize=[8,8])
- Most leads come from India (97%)
- Out of these 38% are converted.
Specialization¶
In [210]:
feature = 'Specialization'
categoricalUAn(feature)
- Specialization of 36% of leads is missing.
- We have mapped those missing values with 'No Specialization'.There might be two reason for this,
- Lead might be a fresher.
- Lead missed to fill it.
- Among all the specializations, ' Banking, Investment And Insurance' has the highest conversion rate(48.9%).
What is your current occupation¶
In [211]:
feature = 'What is your current occupation'
categoricalUAn(feature,figsize=[8,8])
- Although the conversion rate for Working Professional is the highest ! 91.6%, they only make 7.4% of all leads. 60% leads are Unemployed customers followed by 29% with unknown nature of employment
- Among all the converted leads, Unemployed and Working Professionals top the list.
- Conversion for Housewife segment is 100%
City¶
In [212]:
feature = 'City'
categoricalUAn(feature)
- Most Leads come from 'Mumbai' and they have a decent conversion rate of 36.4%.
- Leads from Thane and outskirts make up 8.2% with a conversion rate of 44%
A free copy of Mastering The Interview.¶
In [213]:
feature = 'A free copy of Mastering The Interview'
categoricalUAn(feature)
- 68% of the leads said "No" for a free copy of 'Mastering The Interview'.
- Conversion rate of leads who said "No" is high (39.8%).
In [214]:
def num_univariate_analysis(column_name,scale='linear') :
converted = leads[leads['Converted'] == 1]
unconverted = leads[leads['Converted'] == 0]
plt.figure(figsize=(8,6))
ax = sns.boxplot(x=column_name, y='Converted', data = leads)
title = 'Boxplot of ' + column_name+' vs Conversion'
ax.set(title=title)
if scale == 'log' :
ax.set_xscale('log')
ax.set(ylabel=column_name + '(Log Scale)')
print("Spread for range of "+column_name+" that were Converted")
tab(converted[column_name].describe())
print("Spread for range of "+column_name+" that were not converted")
tab(unconverted[column_name].describe())
TotalVisits¶
In [215]:
column_name = 'TotalVisits'
num_univariate_analysis(column_name,scale='log')
- Looks like
Total Visitshave a lot of outliers among bothConvertedandUn-convertedleads. - Let's take a look at the quantiles between 90 and 100.
In [216]:
# Looking at Quantiles
tab(leads[column_name].quantile(np.linspace(.90,1,20)))
- From the above, it is clear that outliers exist and these might skew the analyses.
- For now, let's cap the outliers about 99th percentile to 99th percentile value.
soft rangecapping.
In [217]:
# Capping outliers to 99th perentile value
cap = leads[column_name].quantile(.99)
condition = leads[column_name] > cap
leads.loc[condition, column_name] = cap
Total TIme Spent on Website.¶
In [218]:
column = 'Total Time Spent on Website'
num_univariate_analysis(column)
- 'Total Time Spend on Website' has many outliers.
- Let's look quantiles to confirm this.
In [219]:
tab(leads[column].quantile(np.linspace(0.75,1,25)))
In [220]:
leads[column].quantile(np.linspace(0.75,1,50)).plot()
Out[220]:
In [221]:
# Capping `Total Time Spent on Website` values to 99th percentile
cap = leads[column].quantile(.99)
condition = leads[column] > cap
leads.loc[condition, column] = cap
Page Views Per Visit.¶
In [222]:
column = 'Page Views Per Visit'
num_univariate_analysis(column)
- 'Page Views Per Visit' has many outliers.
- Let's look quantiles to confirm this.
In [223]:
leads[column].quantile(np.linspace(0.75,1,30)).plot()
Out[223]:
- There is a sudden jump between 99th percentile and maximum value.
- let's cap the values to 99th percentile to avoid skewing the analysis
In [224]:
# Capping `Page Views Per Visit` values to 99th percentile
cap = leads[column].quantile(.99)
condition = leads[column] > cap
leads.loc[condition, column] = cap
Bivariate Analysis¶
In [225]:
leads.columns.values
Out[225]:
In [226]:
continuous_vars = ['TotalVisits', 'Page Views Per Visit', 'Total Time Spent on Website']
TotalVisits vs A free copy of Mastering The Interview¶
In [227]:
plt.figure(figsize=[8,8])
sns.barplot(x=continuous_vars[0], y = 'A free copy of Mastering The Interview', data=leads, hue='Converted')
Out[227]:
- One can see that the proportion of leads with high Total Visits to the website also like a free copy of Mastering The Interview.
- Incidentally, these are leads with higher conversion rate.
- More convertable leads are being attracted by the website through providing 'A free copy of Mastering The Interview'.
Lead Source vs Country¶
In [228]:
# sns.barplot(x='Lead Source', y = 'Country', hue='Converted', data=leads)
leads.groupby(['Country','Lead Source'])['Converted'].value_counts(normalize=True)\
.unstack()\
.plot(
layout=(2,2),
figsize=(14,12), kind='barh', stacked=True);
- Most leads from India through Reference Sources have very high conversion rate.
- Leads from outside of India from other Lead sources do not convert at all.
Occupation vs City¶
In [229]:
x = "What is your current occupation"
y = 'City'
leads.groupby([x,y])['Converted'].value_counts(normalize=True)\
.unstack()\
.plot(
layout=(2,2),
figsize=(14,12), kind='barh', stacked=True);
- Working Professionals in Other cities of Maharashtra have higher conversion rates compared to those from Mumbai , Thane and other cities.
- BusinessMen from Mumbai and Thane & Outskirts are poor leads in comparison to Tier 2 and Other cities .
Last Activity vs Country¶
In [230]:
x = "Country"
y = 'Last Activity'
leads.groupby([x,y])['Converted'].value_counts(normalize=True)\
.unstack()\
.plot(
layout=(2,2),
figsize=(14,12), kind='barh', stacked=True);
- SMS and Emails are more favourable for conversion over Website Visits outside of India.
- Leads from outside of India who click email links have higher conversion rate compared those from India.
Data Preparation¶
Mapping Binary Variables to 0 / 1¶
In [231]:
binary_var = ['Do Not Email', 'A free copy of Mastering The Interview']
leads[binary_var] = leads[binary_var].replace({'Yes' : 1, 'No' : 0})
Creating Indicator Variables¶
In [232]:
categoricalCol = ['Lead Origin', 'Lead Source','Last Activity', 'Country', 'Specialization',
'What is your current occupation', 'City']
print('Levels in Each Cateogrical Variable\n')
for col in sorted(categoricalCol) :
print(col, leads[col].unique(), '\n')
# Creating dummy variables
leadOriginDummies = pd.get_dummies(leads['Lead Origin'], drop_first=True)
leadSourceDummies = pd.get_dummies(leads['Lead Source'], drop_first=True)
lastActivityDummies = pd.get_dummies(leads['Last Activity'], drop_first=True)
countryDummies = pd.get_dummies(leads['Country'] ,drop_first=True)
specDummies = pd.get_dummies(leads['Specialization'],drop_first=True)
occupationDummies = pd.get_dummies(leads[ 'What is your current occupation'],drop_first=True)
cityDummies = pd.get_dummies(leads[ 'City'],drop_first=True)
# adding dummy variables to leads dataframe
leads = pd.concat([leads, leadOriginDummies,leadSourceDummies,lastActivityDummies, countryDummies, specDummies, occupationDummies, cityDummies], axis=1)
# dropping categorical columns
leads.drop(columns = categoricalCol, inplace=True)
print('Final Columns')
leads.columns
Out[232]:
Correlation¶
In [233]:
# Top Correlations
def correlation(dataframe) :
cor0=dataframe.corr()
type(cor0)
cor0.where(np.triu(np.ones(cor0.shape),k=1).astype(np.bool))
cor0=cor0.unstack().reset_index()
cor0.columns=['VAR1','VAR2','CORR']
cor0.dropna(subset=['CORR'], inplace=True)
cor0.CORR=round(cor0['CORR'],2)
cor0.CORR=cor0.CORR.abs()
cor0.sort_values(by=['CORR'],ascending=False)
cor0=cor0[~(cor0['VAR1']==cor0['VAR2'])]
return pd.DataFrame(cor0.sort_values(by=['CORR'],ascending=False))
In [234]:
#Correlations for Converted Leads
convertedCondition= leads['Converted']==1
print('Correlations for Converted Leads')
correlation(leads[convertedCondition])[1:30:2].style.background_gradient(cmap='GnBu').hide_index()
Out[234]:
- Conversions of leads from other lead origins and the ones through reference have similar conversion behaviour.
In [235]:
#Correlations for un-Converted Leads
unconvertedCondition=leads['Converted']==0
print('Correlations for Non-Converted Leads')
correlation(leads[unconvertedCondition])[1:30:2].style.background_gradient(cmap='GnBu').hide_index()
Out[235]:
- From the above,
Unknown OccupationandUnemployedare highly correlated for non-converted leads. - This might mean that unemployed leads and leads with unknown occupation have the same conversion behaviour.
Train-Test Split¶
In [236]:
from sklearn.model_selection import train_test_split
y = leads.pop('Converted')
X = leads
In [237]:
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.3,random_state=100)
Standardizing Continuous Variables¶
In [238]:
continuous_vars
Out[238]:
In [239]:
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
# fitting and transforming train set
X_train[continuous_vars] = scaler.fit_transform(X_train[continuous_vars])
# Transforming test set for later use
X_test[continuous_vars] = scaler.transform(X_test[continuous_vars])
Modelling¶
Recurvise Feature Elimination¶
In [240]:
print('No of features : ', len(X_train.columns))
- Currently, the dataset has 49 features.
- We shall follow a mixed feature elimination approach.
- We could use Recursive Feature Elimination for coarse elimination to 25 columns
- This is followed by manual elimination of features with high p-value / VIF.
In [241]:
# RFE
from sklearn.linear_model import LogisticRegression
from sklearn.feature_selection import RFE
minFeatures = 25
model = LogisticRegression()
rfe = RFE(model, n_features_to_select=minFeatures)
rfe = rfe.fit(X_train, y_train)
In [242]:
# Columns selected by RFE :
RFE_features = pd.DataFrame( {'feature' : X_train.columns, 'rank' : rfe.ranking_, 'support' : rfe.support_})
condition = RFE_features['support'] == True
rfe_features = RFE_features[condition].sort_values(by='rank',ascending=True )
print('Features selected by RFE\n')
rfe_features
Out[242]:
In [243]:
rfeFeatures = rfe_features['feature'].values
In [244]:
### Multicollinearity
from statsmodels.stats.outliers_influence import variance_inflation_factor
def vif(X) :
df = sm.add_constant(X)
vif = [variance_inflation_factor(df.values,i) for i in range(df.shape[1])]
vif_frame = pd.DataFrame({'vif' : vif[0:]},index = df.columns).reset_index()
tab(vif_frame.sort_values(by='vif',ascending=False))
In [245]:
# Model 1
import statsmodels.api as sm
features = rfe_features['feature'].values
X_train = X_train[features]
logm1 = sm.GLM(y_train, sm.add_constant(X_train), family=sm.families.Binomial())
print("VIF for X_train")
vif(X_train)
logm1.fit().summary()
Out[245]:
Unemployedhas the highest VIF. let's drop this feature.
Model 2¶
In [246]:
# Model 2 : Removing `Unemployed`
column_to_remove = 'Unemployed'
features = X_train.columns[X_train.columns !=column_to_remove]
X_train = X_train[features]
logm1 = sm.GLM(y_train, sm.add_constant(X_train), family=sm.families.Binomial())
print("VIF for X_train")
vif(X_train)
logm1.fit().summary()
Out[246]:
Other Lead Originshas a very high VIF.- Let's drop this variable
Model 3¶
In [247]:
# Model 3 : Removing `Other Lead Origins`
column_to_remove = 'Other Lead Origins'
features = X_train.columns[X_train.columns !=column_to_remove]
X_train = X_train[features]
logm1 = sm.GLM(y_train, sm.add_constant(X_train), family=sm.families.Binomial())
print("VIF for X_train")
vif(X_train)
logm1.fit().summary()
Out[247]:
Housewifehas a high p-value and hence the coefficient is insignificant. let's drop the same.
Model 4¶
In [248]:
# Model 4 : Removing `Housewife`
column_to_remove = 'Housewife'
features = X_train.columns[X_train.columns !=column_to_remove]
X_train = X_train[features]
logm1 = sm.GLM(y_train, sm.add_constant(X_train), family=sm.families.Binomial())
print("VIF for X_train")
vif(X_train)
logm1.fit().summary()
Out[248]:
Studenthas a p-value higher than 0.05 and the highest among all p-values. Let's drop this feature.
Model 5¶
In [249]:
# Model 5 : Removing `Student`
column_to_remove = 'Student'
features = X_train.columns[X_train.columns !=column_to_remove]
X_train = X_train[features]
logm1 = sm.GLM(y_train, sm.add_constant(X_train), family=sm.families.Binomial())
print("VIF for X_train")
vif(X_train)
logm1.fit().summary()
Out[249]:
Tier II Citieshas a p-value higher than confidence level and the highest among all the p - values.- Let's remove this feature
Model 6¶
In [250]:
# Model 6 : Removing `Tier II Cities`
column_to_remove = 'Tier II Cities'
features = X_train.columns[X_train.columns !=column_to_remove]
X_train = X_train[features]
logm1 = sm.GLM(y_train, sm.add_constant(X_train), family=sm.families.Binomial())
print("VIF for X_train")
vif(X_train)
logm1.fit().summary()
Out[250]:
Page Views Per Visithas a high p-value. Let's eliminate this.
Model 7¶
In [251]:
# Model 7 : Removing `Page Views Per Visit`
column_to_remove = 'Page Views Per Visit'
features = X_train.columns[X_train.columns !=column_to_remove]
X_train = X_train[features]
logm1 = sm.GLM(y_train, sm.add_constant(X_train), family=sm.families.Binomial())
print("VIF for X_train")
vif(X_train)
logm1.fit().summary()
Out[251]:
Media and Advertisinghas a high p-value. let's drop this feature
Model 8¶
In [252]:
# Model 8 : Removing `Media and Advertising`
column_to_remove = 'Media and Advertising'
features = X_train.columns[X_train.columns !=column_to_remove]
X_train = X_train[features]
logm1 = sm.GLM(y_train, sm.add_constant(X_train), family=sm.families.Binomial())
print("VIF for X_train")
vif(X_train)
logm1.fit().summary()
Out[252]:
- This model has a feature
Email Link Clickedwith high p-value of 0.059. Let's drop this feature.
Model 9¶
In [253]:
# Model 9 : Removing `Email Link Clicked`
column_to_remove = 'Email Link Clicked'
features = X_train.columns[X_train.columns !=column_to_remove]
X_train = X_train[features]
logm1 = sm.GLM(y_train, sm.add_constant(X_train), family=sm.families.Binomial())
print("VIF for X_train")
vif(X_train)
logm1.fit().summary()
Out[253]:
- All coefficients are significant / low p-value
- For further elimination , let's use the magnitude of coefficient as the weight/importance of the variable. Higher values are more important than lower values.
- By this reasoning,
TotalVisitshas the least coefficient. Let'd drop this.
Model 10¶
In [254]:
# Model 10 : Removing `TotalVisits`
column_to_remove = 'TotalVisits'
features = X_train.columns[X_train.columns !=column_to_remove]
X_train = X_train[features]
logm1 = sm.GLM(y_train, sm.add_constant(X_train), family=sm.families.Binomial())
logm1 = logm1.fit()
print("VIF for X_train")
vif(X_train)
logm1.summary()
Out[254]:
Other Lead Sourceshas high p-value. Let's drop this variable.
Model 11 - Final Model¶
In [255]:
# Model 11 : Removing `Other Lead Sources`
column_to_remove = 'Other Lead Sources'
features = X_train.columns[X_train.columns !=column_to_remove]
X_train = X_train[features]
logm_final = sm.GLM(y_train, sm.add_constant(X_train), family=sm.families.Binomial())
logm_final = logm_final.fit()
print("VIF for X_train")
vif(X_train)
logm_final.summary()
Out[255]:
- From the above, the features that remain are statistically significant and donot show any multi collinearity.
- Hence, we could use Model 11 is our final model.
Final Features¶
In [256]:
finalFeatures = X_train.columns.values
print('The Final Feature for Modelling are :', finalFeatures)
Predictions¶
Predictions on Train set¶
In [257]:
X_train_sm = sm.add_constant(X_train)
y_train_pred = logm_final.predict(X_train_sm)
Actual Conversions vs Conversion Predictions¶
In [258]:
# Creating a data frame with converted vs converted probabilities
y_train_pred_final = pd.DataFrame({'Converted':y_train.values, 'Converted_Prob':y_train_pred})
y_train_pred_final['CustID'] = y_train.index
y_train_pred_final.head(10)
Out[258]:
Predictions with cut off = 0.5¶
In [259]:
#Creating new column 'predicted' with 1 if Converted_Prob > 0.5 else 0
y_train_pred_final['predicted'] = y_train_pred_final.Converted_Prob.map(lambda x: 1 if x > 0.5 else 0)
# Let's see the head
y_train_pred_final.head(10)
Out[259]:
Confusion Matrix¶
In [260]:
from sklearn import metrics
# Confusion matrix
confusion = metrics.confusion_matrix(y_train_pred_final.Converted, y_train_pred_final.predicted )
print(confusion)
Confusion Matrix for Train Set
| $\frac{Predicted}{Actual}$ | Not Converted | Converted |
|---|---|---|
| Not Converted | 3462 | 455 |
| Converted | 699 | 1693 |
Accuracy of the Model¶
In [261]:
# Let's check the overall accuracy.
accuracy = metrics.accuracy_score(y_train_pred_final.Converted, y_train_pred_final.predicted)
print('Accuracy on Train set : ', round(100*accuracy,3),'%')
Metrics beyond simple accuracy¶
In [262]:
TP = confusion[1,1] # true positive
TN = confusion[0,0] # true negatives
FP = confusion[0,1] # false positives
FN = confusion[1,0] # false negatives
sensitivity = TP/(FN + TP)
specificity = TN/(FP + TN)
falsePositiveRate = FP/(FP + TN)
positivePredictivePower = TP/(TP +FP )
negativePredictivePower = TN/(TN + FN)
print('sensitivity / Recall: ', round(100*sensitivity,3),'%')
print('specificity : ', round(100*specificity,3),'%')
print('False Positive Rate : ', round(100*falsePositiveRate,3),'%')
print('Precision / Positive Predictive Power : ', round(100*positivePredictivePower,3),'%')
print('Negative Predictive Power : ', round(100*negativePredictivePower,3),'%')
Plotting ROC Curve.¶
In [263]:
def draw_roc( actual, probs ):
fpr, tpr, thresholds = metrics.roc_curve( actual, probs,
drop_intermediate = False )
auc_score = metrics.roc_auc_score( actual, probs )
plt.figure(figsize=(5, 5))
plt.plot( fpr, tpr, label='ROC curve (area = %0.2f)' % auc_score )
plt.plot([0, 1], [0, 1], 'k--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate or [1 - True Negative Rate]')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.show()
return None
In [264]:
draw_roc(y_train_pred_final.Converted, y_train_pred_final.Converted_Prob)
Finding Optimal Cutoff Point¶
- Optimal cutoff probability is that prob where we get balanced sensitivity and specificity
In [265]:
# Let's create columns with different probability cutoffs
numbers = [float(x)/10 for x in range(10)]
for i in numbers:
y_train_pred_final[i]= y_train_pred_final.Converted_Prob.map(lambda x: 1 if x > i else 0)
y_train_pred_final.head()
Out[265]:
In [266]:
# Now let's calculate accuracy sensitivity and specificity for various probability cutoffs.
cutoff_df = pd.DataFrame( columns = ['prob','accuracy','sensi','speci'])
# TP = confusion[1,1] # true positive
# TN = confusion[0,0] # true negatives
# FP = confusion[0,1] # false positives
# FN = confusion[1,0] # false negatives
num = [0.0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9]
for i in num:
cm1 = metrics.confusion_matrix(y_train_pred_final.Converted, y_train_pred_final[i] )
total1=sum(sum(cm1))
accuracy = (cm1[0,0]+cm1[1,1])/total1
speci = cm1[0,0]/(cm1[0,0]+cm1[0,1])
sensi = cm1[1,1]/(cm1[1,0]+cm1[1,1])
cutoff_df.loc[i] =[ i ,accuracy,sensi,speci]
print(cutoff_df)
In [267]:
# Let's plot accuracy sensitivity and specificity for various cutoff probabilities.
fig,ax = plt.subplots()
fig.set_figwidth(30)
fig.set_figheight(10)
plots=['accuracy','sensi','speci']
ax.set_xticks(np.linspace(0,1,50))
ax.set_title('Finding Optimal Cutoff')
sns.lineplot(x='prob',y=plots[0] , data=cutoff_df,ax=ax)
sns.lineplot(x='prob',y=plots[1] , data=cutoff_df,ax=ax)
sns.lineplot(x='prob',y=plots[2] , data=cutoff_df,ax=ax)
ax.set_xlabel('Probabilites')
ax.set_ylabel('Accuracy,Sensitivity,Specificity')
ax.legend(["Accuracy",'Sensitivity','Specificity'])
# cutoff_df.plot.line(, figure=[10,10])
plt.show()
- From the curve above, 0.36 is the optimum cutoff probability.
In [268]:
y_train_pred_final['final_predicted'] = y_train_pred_final.Converted_Prob.map( lambda x: 1 if x > 0.36 else 0)
y_train_pred_final.head()
Out[268]:
In [269]:
# Let's check the overall accuracy.
accu = metrics.accuracy_score(y_train_pred_final.Converted, y_train_pred_final.final_predicted)
print('Accuracy on Train set at Optimum Cut Off : ', round(100*accu,3),'%')
In [270]:
confusion2 = metrics.confusion_matrix(y_train_pred_final.Converted, y_train_pred_final.final_predicted )
confusion2
Out[270]:
In [271]:
TP = confusion2[1,1] # true positive
TN = confusion2[0,0] # true negatives
FP = confusion2[0,1] # false positives
FN = confusion2[1,0] # false negatives
sensitivity = TP/(FN + TP)
specificity = TN/(FP + TN)
falsePositiveRate = FP/(FP + TN)
positivePredictivePower = TP/(TP +FP )
negativePredictivePower = TN/(TN + FN)
print('sensitivity / Recall: ', round(100*sensitivity,3),'%')
print('specificity : ', round(100*specificity,3),'%')
print('False Positive Rate : ', round(100*falsePositiveRate,3),'%')
print('Precision / Positive Predictive Power : ', round(100*positivePredictivePower,3),'%')
print('Negative Predictive Power : ', round(100*negativePredictivePower,3),'%')
In [272]:
## ROC curve for cut off probability of 0.36
draw_roc(y_train_pred_final.Converted, y_train_pred_final.final_predicted)
Precision and Recall¶
In [273]:
#Looking at the confusion matrix again
confusion = metrics.confusion_matrix(y_train_pred_final.Converted, y_train_pred_final.predicted )
confusion
Out[273]:
- Precision :TP / TP + FP
- Recall :TP / TP + FN
In [274]:
print('Precision :', confusion[1,1]/(confusion[0,1]+confusion[1,1]))
print('Recall :', confusion[1,1]/(confusion[1,0]+confusion[1,1]))
In [301]:
#Doing the same using the sklearn.
from sklearn.metrics import precision_score, recall_score
print('Precision : ', precision_score(y_train_pred_final.Converted, y_train_pred_final.predicted))
print('Recall :', recall_score(y_train_pred_final.Converted, y_train_pred_final.predicted))
Precision and Recall Tradeoff¶
In [276]:
from sklearn.metrics import precision_recall_curve
p, r, thresholds = precision_recall_curve(y_train_pred_final.Converted, y_train_pred_final.Converted_Prob)
plt.plot(thresholds, p[:-1], "g-")
plt.plot(thresholds, r[:-1], "r-")
plt.show()
- The cut off point from precision-recall curve is ~0.4.
- Note that we have used the cut off obtained from 'Sensitivity-Specificity' trade off to predict conversions in this analysis.
Predictions on Test set¶
In [277]:
X_test_sm = sm.add_constant(X_test[finalFeatures])
y_test_pred = logm_final.predict(X_test_sm)
Actual Conversions vs Conversion Probability¶
In [278]:
# predicted conversions vs actual conversions and customer ID
y_test_predictions = pd.DataFrame({'Converted' :y_test, 'Conversion Probability' : y_test_pred, 'CustID' : y_test.index})
y_test_predictions.head()
Out[278]:
In [279]:
# predictions with optimal cut off = 0.35
cutoff=0.36
y_test_predictions['Predicted'] = y_test_predictions[
'Conversion Probability'
].map(lambda x : 1 if x > cutoff else 0 )
Confusion Matrix¶
In [280]:
confusion = metrics.confusion_matrix(y_test_predictions['Converted'], y_test_predictions['Predicted'])
confusion
Out[280]:
In [281]:
TP = confusion[1,1] # true positive
TN = confusion[0,0] # true negatives
FP = confusion[0,1] # false positives
FN = confusion[1,0] # false negatives
Accuracy¶
In [282]:
print('Accuracy on Test set : ', round(100*(TP + TN)/(TP + TN + FP + FN),3),'%')
Metrics Beyond Simple Accuracy¶
In [283]:
sensitivity = TP/(FN + TP)
specificity = TN/(FP + TN)
falsePositiveRate = FP/(FP + TN)
positivePredictivePower = TP/(TP +FP )
negativePredictivePower = TN/(TN + FN)
print('sensitivity / Recall: ', round(100*sensitivity,3),'%')
print('specificity : ', round(100*specificity,3),'%')
print('False Positive Rate : ', round(100*falsePositiveRate,3),'%')
print('Precision / Positive Predictive Power : ', round(100*positivePredictivePower,3),'%')
print('Negative Predictive Power : ', round(100*negativePredictivePower,3),'%')
In [284]:
## ROC curve for cut off probability of 0.364
draw_roc(y_test_predictions['Converted'],y_test_predictions['Predicted'])
- Note the AUC is 0.79 on the test test
Lead Scoring¶
In [285]:
# merging final predictions with leads dataset
conversionProb = pd.concat([y_test_predictions['Conversion Probability'],y_train_pred_final['Converted_Prob']],axis=0)
conversionProb = pd.DataFrame({'Conversion Probability' : conversionProb}, index=conversionProb.index)
leads = pd.concat([leads,conversionProb],axis=1)
leads['Prospect ID'] = prospect_ids
leads['Lead No'] = lead_no
leads['Converted'] = y
In [286]:
# Verifying prediction accuracy
leads['Predicted'] = leads['Conversion Probability'].map(lambda x : 1 if x > 0.36 else 0)
In [287]:
confusion = metrics.confusion_matrix(leads['Converted'], leads['Predicted'])
In [288]:
TP = confusion[1,1] # true positive
TN = confusion[0,0] # true negatives
FP = confusion[0,1] # false positives
FN = confusion[1,0] # false negatives
acc = metrics.accuracy_score(leads['Converted'], leads['Predicted'])
print('Accuracy : ', round(100*acc,3),'%')
sensitivity = TP/(FN + TP)
specificity = TN/(FP + TN)
falsePositiveRate = FP/(FP + TN)
falseNegativeRate = FN/(FP + TP)
positivePredictivePower = TP/(TP +FP )
negativePredictivePower = TN/(TN + FN)
print('sensitivity : ', round(100*sensitivity,3),'%')
print('specificity : ', round(100*specificity,3),'%')
print('False Positive Rate : ', round(100*falsePositiveRate,3),'%')
print('False Negative Rate : ', round(100*falseNegativeRate,3),'%')
print('Positive Predictive Power / Precision : ', round(100*positivePredictivePower,3),'%')
print('Negative Predictive Power : ', round(100*negativePredictivePower,3),'%')
In [289]:
## ROC curve
draw_roc(leads['Converted'], leads['Predicted'])
In [290]:
# Lead Scores
leads['Lead Score'] = leads['Conversion Probability']*100
leads[['Prospect ID','Lead No','Lead Score']].sort_values(by='Lead Score', ascending=False)[:10]
Out[290]:
Score Sheet for X Education¶
In [291]:
# Run the following to generate a sheet containing lead information provided by the company and corresponding scores
leads.to_csv('lead_scores.csv')
KS Statistic¶
In [292]:
# Gain Chart
y_test_predictions = y_test_predictions.sort_values(by='Conversion Probability', ascending=False)
y_test_predictions['decile'] = pd.qcut(y_test_predictions['Conversion Probability'],10,labels=range(10,0,-1))
y_test_predictions['Converted'] = y_test_predictions['Converted'].astype('int')
y_test_predictions['Un Converted'] = 1 - y_test_predictions['Converted']
y_test_predictions.head()
Out[292]:
In [293]:
df1 = pd.pivot_table(data=y_test_predictions,index=['decile'],values=['Converted','Un Converted','Conversion Probability'],
aggfunc={'Converted':[np.sum],
'Un Converted':[np.sum],
'Conversion Probability' : [np.min,np.max]})
df1 = df1.reset_index()
df1.columns = ['Decile','Max Prob', 'Min Prob','Converted Count','Un Converted Count']
df1 = df1.sort_values(by='Decile', ascending=False)
df1['Total Leads'] = df1['Converted Count'] + df1['Un Converted Count']
df1['Conversion Rate'] = df1['Converted Count'] / df1['Un Converted Count']
converted_sum = df1['Converted Count'].sum()
unconverted_sum = df1['Un Converted Count'].sum()
df1['Converted %'] = df1['Converted Count'] / converted_sum
df1['Un Converted %'] = df1['Un Converted Count'] / unconverted_sum
df1.head()
Out[293]:
In [294]:
df1['ks_stats'] = np.round(((df1['Converted Count'] / df1['Converted Count'].sum()).cumsum() -(df1['Un Converted Count'] / df1['Un Converted Count'].sum()).cumsum()), 4) * 100
df1
Out[294]:
- Max KS Statistic is 59.76 for 5th decile
- This model discriminates between Converted and Non-converted leads well since KS Statistic in 4th decile (58.11) is greater than 40%. Hence, this is a reasonable good model.
Gain Chart¶
In [295]:
df1['Cum Conversion %'] = np.round(((df1['Converted Count'] / df1['Converted Count'].sum()).cumsum()), 4) * 100
df1
Out[295]:
In [296]:
df1['Base %'] = np.arange(10,110,10)
df1 = df1.set_index('Decile')
In [297]:
df1
Out[297]:
In [298]:
### Gain chart
plot_columns =['Base %','Cum Conversion %']
plt.plot(df1[plot_columns]);
plt.xticks(df1.index);
plt.title('Gain chart');
plt.xlabel('Decile')
plt.ylabel('Cummulative Conversion %')
plt.legend(('Our Model','Random Model'));
- Instead of pursuing leads randomly, pursuing the top 40% leads scored by the model would let the sales team reach 80% of leads likely to convert.
Lift Chart¶
In [299]:
df1['Lift'] = df1['Cum Conversion %'] / df1['Base %']
df1['Baseline'] = 1
df1
Out[299]:
In [300]:
# Lift chart
plot_columns =['Lift', 'Baseline']
plt.plot(df1[plot_columns]);
plt.xticks(df1.index);
plt.title('Lift chart');
plt.xlabel('Decile')
plt.ylabel('Lift')
plt.legend(('Our Model','Random Model'));
- The model outperforms a random model by alteast 2 times in identifying the top 40% potentially convertible leads.
- As opposed to 10% conversions from 10% leads pursued randomly, pursuing the top 10% leads scored by this model would lead to 24% conversions.
Conclusion¶
A logistic regression model is created using lead features. To arrive at the list of features which significantly affect conversion probability, a mixed feature elimination approach is followed. 25 most important features are obtained through Recursive Feature Elimination and then reduced to 15 via p-value / VIF approach. The dataset is randomly divided into train and test set. (70 - 30 split).
The final relationship between log Odds of Conversion Probability and lead features is
logOdds(Conversion Probability) = -0.6469 - 1.5426 Do Not Email -1.2699 Unknown Occupation -0.9057 No Specialization -0.8704 Hospitality Management - 0.6584 Outside India + 1.7923 SMS Sent + 1.1749 Other Last Activity + 2.3769 Working Professional - 0.8614 Olark Chat Conversation + 5.3886 Welingak Website + 3.0246 Reference + 1.1876 Olark Chat -1.0250 Landing Page Submission + 1.1253 Total Time Spent on Website + 0.6106 * Email Opened
where Total Time Spent on Website is standardized to $\mu=0,\sigma=1$
Interpreting Top 6 features affecting Conversion Probability :
- A lead from
Welingak Websitehas 5.4 times higher log odds of conversion than those fromGoogle. - Leads through
Referencehave 3 times higher log odds of conversion than those fromGoogle. - Leads from
Working Professionalhave 2.38 times higher log odds of conversion than those fromBusinessman. - Leads with
SMS Senthave 1.8 times higher log odds of conversion than those with no SMS sent. - Leads with
Do Not Emailhave 1.5 times lesser log odds of conversion compared to leads who would like email updates. - Leads with
Unknown Occupationhave 1.27 times lesser log odds of conversion compared to those fromBusinessman.
Lead Scores :
- Score sheet can be generated by running this cell.
At an optimum cut-off probability of 0.36, model performance is as follows.
Model Performance on Training Set :
- Accuracy : 81.7%
- Sensitivity / Recall: 80.393 %
- Specificity : 81.772 %
- Precision / Positive Predictive Power : 72.924 %
- False Positive Rate : 18.228 %
- AUC Score : 0.81
Model Performance for Test Set :
- Accuracy : 79.593 %
- Sensitivity / Recall : 77.605 %
- Specificity : 80.81%
- Precision / Positive Predictive Power : 71.224 %
- False Positive Rate : 19.19 %
- AUC Score : 0.79
KS statistic :
- Max KS Statistic is 59.76 for 5th decile
- This model discriminates between Converted and Non-converted leads well since KS Statistic in 4th decile (58.11) is greater than 40%. Hence, this is a reasonably good model.
Gain :
- Inside of pursuing leads randomly, pursuing the top 40% leads scored by the model would let the sales team reach 80% of leads likely to convert.
Lift :
- The model outperforms a random model by alteast 2 times in identifying the top 40% potentially convertible leads.
- As opposed to 10% conversions from 10% leads pursued randomly, pursuing the top 10% leads scored by this model would lead to 24% conversions.
Note :
- Incorrect data types have been corrected
- Columns with high missing values have been dropped.
- Columns which do not explain variability in the model have been dropped.
- Columns with sales teams notes like
Tagswhere the classes are not mutually exclusive have been dropped. - Features with low missing values have been imputed with the most frequent values.
- Categories in a feature with less than 1% contribution have been grouped together to reduce the number of levels.
- Inconsistencies in Categories have been corrected.
- 97.5 % of the leads provided by the company have been used for analysis.
- Class imbalance = 0.6
- Indicator variables have been created for all categorical variables with the first category as the reference.
- Continuous variables have been standardized $\mu : 0 , \sigma = 1$ before modelling.