09_Errors_and_Inference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Errors and Inference\n",
    "-----\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "In this notebook, we go over these main concepts:\n",
    "- Perform sensitivity analysis with machine learning models,\n",
    "- Use imputation to fill in missing values.\n",
    "\n",
    "We'll build on the machine learning models of the previous notebooks to exemplify these concepts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import sqlite3\n",
    "import sklearn\n",
    "from sklearn.linear_model import LogisticRegression, LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Database Connection\n",
    "DB = 'ncdoc.db'\n",
    "conn = sqlite3.connect(DB)\n",
    "cur = conn.cursor()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation\n",
    "\n",
    "Recap: Our Machine Learning Problem\n",
    ">Of all prisoners released, we would like to predict who is likely to reenter jail within *5* years of the day we make our prediction. For instance, say it is Jan 1, 2009 and we want to identify which \n",
    ">prisoners are likely to re-enter jail between now and end of 2013. We can run our predictive model and identify who is most likely at risk. The is an example of a *binary classification* problem. \n",
    "\n",
    "We need to munge our data into **labels** (1_Machine_Learning_Labels.ipynb) and **features** (2_Machine_Learning_Features.ipynb) before we can train and evaluate **machine learning models** (3_Machine_Learning_Models.ipynb).\n",
    "\n",
    "This notebook assumes that you have already worked through the `1_Machine_Learning_Labels` and `2_Machine_Learning_Features` notebooks. If that is not the case, the following functions allow you to bring in the functions developed in those notebooks from `.py` scripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We are bringing in the create_labels and create_features functions covered in previous notebooks\n",
    "# Note that these are brought in from scripts rather than from external packages\n",
    "from create_labels import create_labels\n",
    "from create_features import create_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# These functions make sure that the tables have been created in the database.\n",
    "create_labels('2008-12-31', '2009-01-01', '2013-12-31', conn)\n",
    "create_features('2008-12-31', '2009-01-01', '2013-12-31', conn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a training set that takes people at the beginning of 2009 and defines the outcome based on data from 2009-2013. The features for each person are based on data up to the end of 2008."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sql_string = \"drop table if exists train_matrix;\"\n",
    "cur.execute(sql_string)\n",
    "\n",
    "sql_string = \"create table train_matrix as \"\n",
    "sql_string += \"select l.inmate_doc_number, l.recidivism, f.num_admits, f.length_longest_sentence, f.age_first_admit, f.age \"\n",
    "sql_string += \"from recidivism_labels_2009_2013 l \"\n",
    "sql_string += \"left join features_2000_2008 f on f.inmate_doc_number = l.inmate_doc_number \"\n",
    "sql_string += \";\"\n",
    "cur.execute(sql_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then load the training data into `df_training`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sql_string = \"SELECT *\"\n",
    "sql_string += \"FROM train_matrix \"\n",
    "sql_string += \";\"\n",
    "\n",
    "df_training = pd.read_sql(sql_string, con = conn)\n",
    "df_training.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in the previous notebooks, we'll drop any rows that have age < 14 and > 99.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "drop = (df_training['age_first_admit'] < 14) | (df_training['age_first_admit'] > 99)\n",
    "df_training = df_training[-drop]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing Data\n",
    "\n",
    "Sometimes, we have variables with missing (or unknown) data. Instead of dropping those values, there are methods to fill those in, in order to be able to use the data. \n",
    "\n",
    "Keep in mind that these imputed values will be **approximations**, and must be treated as such. If you choose to impute missing values in your project or future work, you must acknowledge your process and clearly state which variables you imputed values for. \n",
    "\n",
    "First, let's see how many missing values we have for each of our features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(df_training) - df_training['num_admits'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(df_training) - df_training['length_longest_sentence'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(df_training) - df_training['age_first_admit'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(df_training) - df_training['age'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will try to impute the unknown data using the following methods:\n",
    "- Mean imputation;\n",
    "- Regression imputation.\n",
    "\n",
    "We start by creating indicator variables that take the value 1 if we observe a missing value for a given feature (and 0 otherwise). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in df_training[[\"num_admits\", \"length_longest_sentence\", \"age_first_admit\", \"age\"]].columns:\n",
    "    df_training[col+\"_missing\"] = df_training[col].isnull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method 1. Mean Imputation\n",
    "\n",
    "One of the simplest ways of imputing values is by taking the mean and filling it in. It's possible to do this by using the overall mean, as well as by certain subgroups. \n",
    "\n",
    "First, we find the mean for each feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_admits = df_training.num_admits.mean() \n",
    "mean_longest_sentence = df_training.length_longest_sentence.mean() \n",
    "mean_age_first = df_training.age_first_admit.mean() \n",
    "mean_age = df_training.age.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a copy of our training data for imputation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_training_m = df_training.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can fill the missing values with the means we computed above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_training_m[\"num_admits\"].fillna(mean_admits, inplace = True)\n",
    "df_training_m[\"length_longest_sentence\"].fillna(mean_longest_sentence, inplace = True)\n",
    "df_training_m[\"age_first_admit\"].fillna(mean_age_first, inplace = True)\n",
    "df_training_m[\"age\"].fillna(mean_age, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_training_m.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also do this by subgroup. So, for example, we can compute the mean age for each value of number of admits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_age_by_admits = df_training[[\"num_admits\", \"age\"]].groupby('num_admits').mean()\n",
    "mean_age_by_admits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run model\n",
    "\n",
    "Here we select our (mean-imputed) features, the outcome variable and create X- and y-training objects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sel_features = ['num_admits', 'length_longest_sentence', 'age_first_admit', 'age' , \n",
    "                'num_admits_missing', 'length_longest_sentence_missing', 'age_first_admit_missing', 'age_missing']\n",
    "sel_label = 'recidivism'\n",
    "X_train = df_training_m[sel_features].values\n",
    "y_train = df_training_m[sel_label].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now build a prediction model that learns the relationship between our predictors (X_train) and recidivism (y_train), e.g., by using logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LogisticRegression(penalty = 'none')\n",
    "model.fit( X_train, y_train )\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the regression coefficients that were learned based on the imputed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.coef_[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.intercept_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method 2. Regression Imputation\n",
    "\n",
    "We can also use regression to try to get more accurate values. We build a regression equation from the observations for which we know the age, then use the equation to essentially predict the missing values. This is, in effect, an extension of the mean imputation by subgroup. Here, we will use the number of admits as well as the length of the longest sentence in order to impute age."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove missing rows first\n",
    "df_training_nona = df_training.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model creation process for a linear regression is very similar to that of the ML models when using `scikit-learn`. We create the model object, then give it the data, then use the model object to generate our predictions. The model object essentially contains all of the instructions on how to fit the model, and when we give it the data, it fits the model to that data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create model object\n",
    "ols = LinearRegression()\n",
    "\n",
    "# Predictors and Outcome\n",
    "predictors = df_training_nona[[\"num_admits\", \"length_longest_sentence\"]]\n",
    "outcome = df_training_nona.age\n",
    "\n",
    "# Fit the model\n",
    "ols.fit(X = predictors, y = outcome)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've fit our model, we can find the predicted values for age."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "missing_x = df_training.loc[df_training.age.isna() & df_training.num_admits.notna() & df_training.length_longest_sentence.notna(),['num_admits','length_longest_sentence']]\n",
    "missing_id = df_training.loc[df_training.age.isna() & df_training.num_admits.notna() & df_training.length_longest_sentence.notna(),'INMATE_DOC_NUMBER']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "missing_ages = pd.DataFrame({'INMATE_DOC_NUMBER':missing_id, 'age':ols.predict(missing_x)})\n",
    "missing_ages.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a copy of our training data for imputation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_training_r = df_training_m.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can fill the missing values of age with the predicted age values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_training_r.loc[df_training_r['INMATE_DOC_NUMBER'].isin(missing_id), 'age'] = missing_ages.age"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_training_r.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Advanced: Using machine learning to impute values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To impute values, we can also use the machine learning algorithm called the `K-nearest Neighbors`. The principle behind it is quite simple: the missing values can be imputed by values of \"closest neighbors\" - as approximated by other, known, features. \n",
    "\n",
    "The algorithm calculates the distance between the input values (the missing values) and helps to identify the nearest possible value based on other features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Re-run model\n",
    "\n",
    "As before, we select our (now mean- and regression-imputed) features, the outcome variable and create X- and y-training objects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sel_features = ['num_admits', 'length_longest_sentence', 'age_first_admit', 'age' , \n",
    "                'num_admits_missing', 'length_longest_sentence_missing', 'age_first_admit_missing', 'age_missing']\n",
    "sel_label = 'recidivism'\n",
    "X_train = df_training_r[sel_features].values\n",
    "y_train = df_training_r[sel_label].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, fit the logistic regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LogisticRegression(penalty = 'none')\n",
    "model.fit( X_train, y_train )\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the regression coefficients that were learned based on the newly imputed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.coef_[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.intercept_"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}