Algorithms Used

Based on the applicability of algorithms in the given table, we use the following algorithms in our model and compare their outputs:
a) k-Nearest-Neighbour – This algorithm works on the principle of proximity of instances and their relative distance or proximity makes them more likely to belong to the same class.
b) Linear SVM (Support Vector Machines) – This algorithm finds a line that separates the two classes in such a way that the nearest points in each class from the line are equidistant from the line separating them.
c) Radial Basis Function – We would use the RBF function as well as the instances may not be possible to be classified using a linear function, so we might need to convert it from Cartesian(x1,x2) to Vector(α,Θ) form.
d) XG Boost – It is considered a scalable and accurate implementation of gradient boosting machines and it has proven to push the limits of computing power for boosted trees algorithms as it was built and developed for the sole purpose of model performance and computational speed.
e) Decision Tree – This is ideal for classification as it identifies the attttribute that best classifies data, uses that attribute as the root , and repeats the process for each branch.
f) Random Forest – It consists of a large number of individual decision trees that operate as an ensemble. Each individual tree in the random forest spits out a class prediction and the class with the most votes becomes our model’s prediction.
g) MLP (Multi Layer Perceptron) – MLP is a deep learning method that is characterised by multiple layer of input nodes connected in a graphical manner with output layers.
h) AdaBoost (Adaptive Boosting) – It is an adaptive algorithm that tweaks the weak learners in favour of thise instances that were misclassfied by the previous classifiers.
i) Gaussian Naive Bayes – Assuming that the input attributes are independent of each other, this algorithm might be beset suited for such problems, also because of the fact that it needs less training data.
j) Logistic Regression – Since the dependent variable is dichotomous (binary), this must be tried as a basic ML algorithm.

Algorithms Comparison

We would now compare the 10 algorithms mentioned in the summary, by using the following method:
1. Installed all required packages and called the libraries.
2. We created a list of classifiers for all the 10 models.
3. We fitted all models using for loop on the ‘Train’ dataset and employed 10-fold Cross validation.
4. We printed the mean accuracy results for all the classifiers and plotted all results on a boxplot to check their relative performance.

import warnings
warnings.filterwarnings('ignore')
import xgboost as xgb
from xgboost import XGBClassifier
from sklearn import model_selection
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_moons, make_circles, make_classification
from sklearn.neural_network import MLPClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
from sklearn.linear_model import LogisticRegression

names = ["NN", "LSVM", "RBF SVM","XGB",
         "DTree", "RF", "MLP", "AdB",
         "NB", "LR"]

classifiers = [
    KNeighborsClassifier(3),                                              #Nearest Neighbors
    SVC(kernel="linear", C=0.025),                                        #Linear SVM
    SVC(gamma=2, C=1),                                                    #RBF SVM
    XGBClassifier(),                                                      #XGBoost
    DecisionTreeClassifier(max_depth=5),                                  #Decision Tree
    RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1), #Random Forest
    MLPClassifier(alpha=1, max_iter=1000),                                #MultiLayerPerceptron
    AdaBoostClassifier(),                                                 #AdaBoost
    GaussianNB(),                                                         #Naive Bayes
    LogisticRegression()]                                                 #Logistic Regression


dataset = (df_train_normalized, y)
id_=df_train.iloc[:,0].values # column of customer id

datasets = [dataset]
results = []
scoring = 'accuracy'

figure = plt.figure(figsize=(27, 20))
i = 1

# iterate over datasets
for ds_cnt, ds in enumerate(datasets):
    
    # preprocess dataset, split into training and test part
    X, y = ds
    X = StandardScaler().fit_transform(X)
    

    # iterate over classifiers
    for name, clf in zip(names, classifiers):
        kfold = model_selection.KFold(n_splits=10, random_state=42)
        cv_results = model_selection.cross_val_score(clf, X, y, cv=kfold, scoring=scoring)
        results.append(cv_results)
        print(name, cv_results.mean())
        
# boxplot algorithm comparison
fig = plt.figure()
fig.suptitle('Algorithm Comparison')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(names)
plt.show()

NN 0.791044776119403
LSVM 0.8093816631130064
RBF SVM 0.8119402985074627
XGB 0.8158848614072495
DTree 0.8123667377398721
RF 0.8149253731343282
MLP 0.8139658848614072
AdB 0.8109808102345415
NB 0.8053304904051174
LR 0.8146055437100213

# Variable importance plot 
from xgboost import plot_importance
plot_importance(xgb1)

#Additional sklearn functions which would be used for tuning steps
from sklearn import metrics   
from sklearn.model_selection import cross_val_score, GridSearchCV

import matplotlib.pylab as plt
%matplotlib inline
from matplotlib.pylab import rcParams
rcParams['figure.figsize'] = 12, 4

train = df_train
target = 'Category'
IDcol = 'Id'

#We start by defining a function for the XGBoost analysis
def modelfit(alg, dtrain, predictors,target, useTrainCV=True, cv_folds=5, early_stopping_rounds=50):
    
    if useTrainCV:
        xgb_param = alg.get_xgb_params()
        xgtrain = xgb.DMatrix(dtrain[predictors].values, label=dtrain[target].values)
        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], nfold=cv_folds,
            metrics='auc', early_stopping_rounds=early_stopping_rounds)
        alg.set_params(n_estimators=cvresult.shape[0])
    
    #Fit the algorithm on the data
    alg.fit(dtrain[predictors], dtrain[target],eval_metric='auc')
        
    #Predict training set:
    dtrain_predictions = alg.predict(dtrain[predictors])
    dtrain_predprob = alg.predict_proba(dtrain[predictors])[:,1]
        
    #Print model report:
    print("Accuracy : %.4g" % metrics.accuracy_score(dtrain[target].values, dtrain_predictions))
    print("AUC Score (Train): %f" % metrics.roc_auc_score(dtrain[target], dtrain_predprob))

#Step 1: Fix learning rate and number of estimators
#Choose all predictors except target

import xgboost as xgb
from xgboost.sklearn import XGBClassifier
predictors = [x for x in df_train.columns if x not in [target,IDcol]]
xgb1 = XGBClassifier(
 learning_rate =0.01,
 n_estimators=3000,
 max_depth=5,
 min_child_weight=1,
 gamma=0,
 subsample=0.8,
 colsample_bytree=0.8,
 objective= 'binary:logistic',
 nthread=4,
 scale_pos_weight=1,
 seed=27)
modelfit(xgb1, df_train, predictors, target)

Accuracy : 0.8532
AUC Score (Train): 0.852211

# Variable importance plot 
from xgboost import plot_importance
plot_importance(xgb1)

param_test1 = {
 'max_depth':range(3,10,1),
 'min_child_weight':range(1,10,1)
}
gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.01, n_estimators=3000, max_depth=5,
 min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,
 objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), 
 param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch1.fit(df_train[predictors],df_train[target])
gsearch1.cv_results_, gsearch1.best_params_, gsearch1.best_score_
#values for max_depth = 3 and for min_child_weight= 5

param_test3 = {
 'gamma':[i/10.0 for i in range(0,5)]
}
gsearch3 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.01, n_estimators=3000, max_depth=3,
 min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,
 objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), 
 param_grid = param_test3, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch3.fit(df_train[predictors],df_train[target])
gsearch3.cv_results_, gsearch3.best_params_, gsearch3.best_score_
# value for gamma comes to be : 0.2

param_test4 = {
 'subsample':[i/10.0 for i in range(6,10)],
 'colsample_bytree':[i/10.0 for i in range(6,10)]
}
gsearch4 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.01, n_estimators=3000, max_depth=3,
 min_child_weight=1, gamma=0.2, subsample=0.8, colsample_bytree=0.8,
 objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), 
 param_grid = param_test4, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch4.fit(df_train[predictors],df_train[target])
gsearch4.cv_results_, gsearch4.best_params_, gsearch4.best_score_
#subsample: 0.7, colsample: 0.9

param_test5 = {
 'subsample':[i/100.0 for i in range(60,80,5)],
 'colsample_bytree':[i/100.0 for i in range(60,80,5)]
}
gsearch5 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.01, n_estimators=3000, max_depth=3,
 min_child_weight=1, gamma=0.2, subsample=0.7, colsample_bytree=0.7,
 objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), 
 param_grid = param_test5, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch5.fit(df_train[predictors],df_train[target])
gsearch5.cv_results_, gsearch5.best_params_, gsearch5.best_score_
#optimum for subsample : 0.65, colsample = 0.85

param_test6 = {
 'reg_alpha':[1e-5, 1e-2, 0.1, 1, 100]
}
gsearch6 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.01, n_estimators=3000, max_depth=3,
 min_child_weight=1, gamma=0.2, subsample=0.65, colsample_bytree=0.75,
 objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), 
 param_grid = param_test6, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch6.fit(df_train[predictors],df_train[target])
gsearch6.cv_results_, gsearch6.best_params_, gsearch6.best_score_
#reg_alpha comes to be 1

#We have tuned the parameters from steps 2 to 5 and incorporated the results in the next section 
#because executing these steps repeatedly will hamper the model speed drastically 
#Final Check: Reducing LEarning rate and increase trees
xgbtest = XGBClassifier(
 learning_rate =0.01,
 n_estimators=3000,
 max_depth=5,
 min_child_weight=3,
 gamma=0.2,
 subsample=0.65,
 colsample_bytree=0.85,
 reg_alpha=1,
 objective= 'binary:logistic',
 nthread=4,
 scale_pos_weight=1,
 seed=27)
modelfit(xgbtest, df_train, predictors, target)

Accuracy : 0.8547
AUC Score (Train): 0.856438

Inferences from Parameter Tuning
When we used ‘Category’ variable in Scaling, then the model accuracy was 1.00 without parameter tuning. However, since that is a flawed logic, we removed ‘Category’ from the Standard Scaling and checked the accuracy, it was coming to 0.8159. Now, we employed the parameter tuning methods that improved this accuracy to 0.8564.

#We prepare test data for fitting into the model obtained by removing the column Id
df_test_final = df_test.drop(['Id'],axis=1)
#We convert test data into numpy arrays
id_person = df_test.iloc[:,0].values # column of customer id
X_test_final = df_test.iloc[:,1:].values # column of the inputs
X_test_normalized=scaler.fit_transform(X_test_final)

# Predict the target on the test dataset
predict_test = xgbtest.predict(df_test_final)
print('\nTarget on test data',predict_test) 
import pandas as pd
Prediction = pd.DataFrame()
Prediction.insert(0, 'Id', id_person.astype(int))
Prediction.insert(1, 'Category', predict_test.astype(int))
Prediction.to_csv("XGboost_Kaggle.csv", index=False)
Prediction

Target on test data [0 1 0 ... 0 0 0]