Advent of 2022, Day 19 – Statistical analysis and ML comparison of prediction models
This article is originally published at https://tomaztsql.wordpress.com
In the series of Azure Machine Learning posts:
- Dec 01: What is Azure Machine Learning?
- Dec 02: Creating Azure Machine Learning Workspace
- Dec 03: Understanding Azure Machine Learning Studio
- Dec 04: Getting data to Azure Machine Learning workspace
- Dec 05: Creating compute and cluster instances in Azure Machine Learning
- Dec 06: Environments in Azure Machine Learning
- Dec 07: Introduction to Azure CLI and Python SDK
- Dec 08: Python SDK namespaces for workspace, experiments and models
- Dec 09: Python SDK namespaces for environment, and pipelines
- Dec 10: Connecting to client using Python SDK namespaces
- Dec 11: Creating Pipelines with Python SDK
- Dec 12: Creating jobs
- Dec 13: Automated ML
- Dec 14: Registering the models
- Dec 15: Getting to know MLflow
- Dec 16: MLflow in action with xgboost
- Dec 17: Building responsible AI dashboard with Python SDK
- Dec 18: Statistical analysis, plotting graphs and feature engineering
Yesterday we have shown, that statistical analysis and all bolts and whistles can be done super simple in Azure machine learning. Today we will continue with feature engineering and modelling.
So, what is feature engineering? Is a general process and can involve both feature construction: adding new features from the existing data, and feature selection: choosing only the most important features for improving model performance, reducing data dimensionality, doing log-transformation, removing outliers, to do scaling (normalisation, standardisation), imputations, general transformation (and others, as polynomial), variable creation, variable extraction and so on.
One-hot encoding is another part of feature engineering, that we have seen in the previous post. and now, let’s create polynomial features of 2nd degree and train and transform the features.
poly_features = df[['duration','account_bal_neg_bal','account_bal_no_acc']] poly_target=df['target'] from sklearn.preprocessing import PolynomialFeatures poly_transformer = PolynomialFeatures(degree = 2) poly_transformer.fit(poly_features) poly_features = poly_transformer.transform(poly_features) poly_transformer.get_feature_names(input_features = ['duration','account_bal_neg_bal','account_bal_no_acc']) # delete duplicates for i in list(poly_features.columns): for j in list(df.columns): if (i==j): poly_features.drop(labels=i, axis=1, inplace=True) poly_features.drop(labels='1', axis=1, inplace=True) list(poly_features)
We have deleted the duplicated features and now we can calculate the correlations and make sure to remove any homoscedasticity and preferably also multicollinearity.
Once finished, we want to split the new dataset to train and test.
from sklearn.model_selection import train_test_split x, y = df.drop('target', axis=1), df['target'] x.shape, y.shape x_train, x_test, y_train, y_test= train_test_split(x,y, test_size=.2, random_state=42) x_train.shape, x_test.shape
And do normalisation and fitting.
from sklearn.preprocessing import MinMaxScaler scaler = MinMaxScaler(feature_range = (0, 1)) scaler.fit(x_train) scaler.fit(x_test) x_train= scaler.transform(x_train) x_test= scaler.transform(x_test)
We can now start comparing the models. We will be using Recall and AUC as metrics for comparison between the models.
import numpy as np y.value_counts(normalize=True) from sklearn.ensemble import RandomForestClassifier from sklearn.linear_model import LogisticRegression from sklearn.tree import DecisionTreeClassifier from sklearn.neighbors import KNeighborsClassifier from sklearn.ensemble import RandomForestClassifier from sklearn.discriminant_analysis import LinearDiscriminantAnalysis from sklearn.naive_bayes import GaussianNB from sklearn.svm import SVC from xgboost import XGBClassifier from sklearn.metrics import roc_auc_score, recall_score, classification_report from sklearn.model_selection import StratifiedKFold, cross_val_score, cross_validate # model preps models =  models.append(('DT', DecisionTreeClassifier(random_state=42))) models.append(('LR', LogisticRegression(random_state=42))) models.append(('RF', RandomForestClassifier(random_state=42))) models.append(('NB', GaussianNB())) models.append(('XGB', XGBClassifier(random_state=42))) models.append(('KNN', KNeighborsClassifier())) models.append(('SVM', SVC(gamma='auto',random_state=42))) # evaluate each model in turn results_recall =  results_roc_auc=  names =  recall= tp/ (tp+fn). Best value=1, worst value=0 scoring = ['recall', 'roc_auc'] for name, model in models: # split dataset into k folds. use one fold for validation and remaining k-1 folds for training skf= StratifiedKFold(n_splits=10, shuffle=True, random_state=42) cv_results = cross_validate(model, x_train, y_train, cv=skf, scoring=scoring) results_recall.append(cv_results['test_recall']) results_roc_auc.append(cv_results['test_roc_auc']) names.append(name) msg = "%s- recall:%f roc_auc:%f" % (name, cv_results['test_recall'].mean(),cv_results['test_roc_auc'].mean()) print(msg)
And finally, we can do a comparison of each model performance, since, we stored everything in a data.frame.
Compete set of code, documents, notebooks, and all of the materials will be available at the Github repository: https://github.com/tomaztk/Azure-Machine-Learning
Happy Advent of 2022!
Please visit source website for post related comments.