DS0103EN-3-3-1-From-Understanding-to-Preparation-py-v2.0.ipynb

{
 "cells": [
  {
   "cell_type": "markadown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<a href=\"https://cognitiveclass.ai\"><img src = \"https://ibm.box.com/shared/static/9gegpsmnsoo25ikkbl4qzlvlyjbgxs5x.png\" width = 400> </a>\n",
    "\n",
    "<h1 align=center><font size = 9>From Understanding to Preparation</font></h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Introduction\n",
    "\n",
    "In this lab, we will continue learning about the Data Science methodology, and focus on the **Data Understanding** and the **Data Preparation** stages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Table of Contents\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "1. [Recap](#0)<br>\n",
    "2. [Data Understanding](#2)<br>\n",
    "3. [Data Preparation](#4)<br>\n",
    "</div>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Recap <a id=\"0\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "In Lab **From Requirements to Collection**, we learned that the data we need to answer the question developed in the business understanding stage, namely *can we automate the process of determining the cuisine of a given recipe?*, is readily available. A researcher named Yong-Yeol Ahn scraped tens of thousands of food recipes (cuisines and ingredients) from three different websites, namely:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<img src = \"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DS0103EN/labs/images/lab3_fig1_allrecipes.png\" width=500>\n",
    "\n",
    "www.allrecipes.com\n", #insertingtherecipes
    "\n",
    "<img src = \"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DS0103EN/labs/images/lab3_fig2_epicurious.png\" width=500>\n",
    "\n",
    "www.epicurious.com\n",
    "\n",
    "<img src = \"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DS0103EN/labs/images/lab3_fig3_menupan.png\" width=500>\n",
    "\n",
    "www.menupan.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "For more information on Yong-Yeol Ahn and his research, you can read his paper on [Flavor Network and the Principles of Food Pairing](http://yongyeol.com/papers/ahn-flavornet-2011.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We also collected the data and placed it on an IBM server for your convenience.\n",
    "\n",
    "------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Data Understanding <a id=\"2\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DS0103EN/labs/images/lab3_fig4_flowchart_data_understanding.png\" width=500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<strong> Important note:</strong> Please note that you are not expected to know how to program in Python. The following code is meant to illustrate the stages of data understanding and data preparation, so it is totally fine if you do not understand the individual lines of code. We have a full course on programming in Python, <a href=\"http://cocl.us/PY0101EN_DS0103EN_LAB3_PYTHON_Coursera\"><strong>Python for Data Science</strong></a>, which is also offered on Coursera. So make sure to complete the Python course if you are interested in learning how to program in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Using this notebook:\n",
    "\n",
    "To run any of the following cells of code, you can type **Shift + Enter** to excute the code in a cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Get the version of Python installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# check Python version\n",
    "!python -V"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Download the library and dependencies that we will need to run this lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd # import library to read data into dataframe\n",
    "pd.set_option('display.max_columns', None)\n",
    "import numpy as np # import numpy library\n",
    "import re # import library for regular expression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Download the data from the IBM server and read it into a *pandas* dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "recipes = pd.read_csv(\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DS0103EN/labs/data/recipes.csv\")\n",
    "\n",
    "print(\"Data read into dataframe!\") # takes about 30 seconds"
    #takes adding some values of output
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Show the first few rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "recipes.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Get the dimensions of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "recipes.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "So our dataset consists of 57,691 recipes. Each row represents a recipe, and for each recipe, the corresponding cuisine is documented as well as whether 384 ingredients exist in the recipe or not, beginning with almond and ending with zucchini."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We know that a basic sushi recipe includes the ingredients:\n",
    "* rice\n",
    "* soy sauce\n",
    "* wasabi\n",
    "* some fish/vegetables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's check that these ingredients exist in our dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "ingredients = list(recipes.columns.values)\n",
    "\n",
    "print([match.group(0) for ingredient in ingredients for match in [(re.compile(\".*(rice).*\")).search(ingredient)] if match])\n",
    "print([match.group(0) for ingredient in ingredients for match in [(re.compile(\".*(wasabi).*\")).search(ingredient)] if match])\n",
    "print([match.group(0) for ingredient in ingredients for match in [(re.compile(\".*(soy).*\")).search(ingredient)] if match])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Yes, they do!\n",
    "\n",
    "* rice exists as rice.\n",
    "* wasabi exists as wasabi.\n",
    "* soy exists as soy_sauce.\n",
    "\n",
    "So maybe if a recipe contains all three ingredients: rice, wasabi, and soy_sauce, then we can confidently say that the recipe is a **Japanese** cuisine! Let's keep this in mind!\n",
    "\n",
    "----------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Data Preparation <a id=\"4\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DS0103EN/labs/images/lab3_fig5_flowchart_data_preparation.png\" width=500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "In this section, we will prepare data for the next stage in the data science methodology, which is modeling. This stage involves exploring the data further and making sure that it is in the right format for the machine learning algorithm that we selected in the analytic approach stage, which is decision trees."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "First, look at the data to see if it needs cleaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "recipes[\"country\"].value_counts() # frequency table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "By looking at the above table, we can make the following observations:\n",#addingnewfile
    "\n",
    "1. Cuisine column is labeled as Country, which is inaccurate.\n",
    "2. Cuisine names are not consistent as not all of them start with an uppercase first letter.\n",
    "3. Some cuisines are duplicated as variation of the country name, such as Vietnam and Vietnamese.\n",
   #lookingthenewtable
    "4. Some cuisines have very few recipes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Let's fixes these problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Fix the name of the column showing the cuisine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "column_names = recipes.columns.values\n",
    "column_names[0] = \"cuisine\"\n",
    "recipes.columns = column_names\n",
    "\n",
    "recipes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Make all the cuisine names lowercase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "recipes[\"cuisine\"] = recipes[\"cuisine\"].str.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Make the cuisine names consistent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "recipes.loc[recipes[\"cuisine\"] == \"austria\", \"cuisine\"] = \"austrian\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"belgium\", \"cuisine\"] = \"belgian\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"china\", \"cuisine\"] = \"chinese\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"canada\", \"cuisine\"] = \"canadian\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"netherlands\", \"cuisine\"] = \"dutch\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"france\", \"cuisine\"] = \"french\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"germany\", \"cuisine\"] = \"german\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"india\", \"cuisine\"] = \"indian\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"indonesia\", \"cuisine\"] = \"indonesian\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"iran\", \"cuisine\"] = \"iranian\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"italy\", \"cuisine\"] = \"italian\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"japan\", \"cuisine\"] = \"japanese\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"israel\", \"cuisine\"] = \"jewish\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"korea\", \"cuisine\"] = \"korean\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"lebanon\", \"cuisine\"] = \"lebanese\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"malaysia\", \"cuisine\"] = \"malaysian\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"mexico\", \"cuisine\"] = \"mexican\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"pakistan\", \"cuisine\"] = \"pakistani\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"philippines\", \"cuisine\"] = \"philippine\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"scandinavia\", \"cuisine\"] = \"scandinavian\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"spain\", \"cuisine\"] = \"spanish_portuguese\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"portugal\", \"cuisine\"] = \"spanish_portuguese\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"switzerland\", \"cuisine\"] = \"swiss\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"thailand\", \"cuisine\"] = \"thai\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"turkey\", \"cuisine\"] = \"turkish\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"vietnam\", \"cuisine\"] = \"vietnamese\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"uk-and-ireland\", \"cuisine\"] = \"uk-and-irish\"\n",
    "recipes.loc[recipes[\"cuisine\"] == \"irish\", \"cuisine\"] = \"uk-and-irish\"\n",
    "\n",
    "recipes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Remove cuisines with < 50 recipes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# get list of cuisines to keep\n",
    "recipes_counts = recipes[\"cuisine\"].value_counts()\n",
    "cuisines_indices = recipes_counts > 50\n",
    "\n",
    "cuisines_to_keep = list(np.array(recipes_counts.index.values)[np.array(cuisines_indices)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "rows_before = recipes.shape[0] # number of rows of original dataframe\n",
    "print(\"Number of rows of original dataframe is {}.\".format(rows_before))\n",
    "\n",
    "recipes = recipes.loc[recipes['cuisine'].isin(cuisines_to_keep)]\n",
    "\n",
    "rows_after = recipes.shape[0] # number of rows of processed dataframe\n",
    "print(\"Number of rows of processed dataframe is {}.\".format(rows_after))\n",
    "\n",
    "print(\"{} rows removed!\".format(rows_before - rows_after))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Convert all Yes's to 1's and the No's to 0's"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "recipes = recipes.replace(to_replace=\"Yes\", value=1)\n",
    "recipes = recipes.replace(to_replace=\"No\", value=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Let's analyze the data a little more in order to learn the data better and note any interesting preliminary observations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Run the following cell to get the recipes that contain **rice** *and* **soy** *and* **wasabi** *and* **seaweed**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "recipes.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "check_recipes = recipes.loc[\n",
    "    (recipes[\"rice\"] == 1) &\n",
    "    (recipes[\"soy_sauce\"] == 1) &\n",
    "    (recipes[\"wasabi\"] == 1) &\n",
    "    (recipes[\"seaweed\"] == 1)\n",
    "]\n",
    "\n",
    "check_recipes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Based on the results of the above code, can we classify all recipes that contain **rice** *and* **soy** *and* **wasabi** *and* **seaweed** as **Japanese** recipes? Why?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Your Answer:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "No, because other recipes such as Asian and East_Asian recipes also contain these ingredients.\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's count the ingredients across all recipes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# sum each column\n",
    "ing = recipes.iloc[:, 1:].sum(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# define each column as a pandas series\n",
    "ingredient = pd.Series(ing.index.values, index = np.arange(len(ing)))\n",
    "count = pd.Series(list(ing), index = np.arange(len(ing)))\n",
    "\n",
    "# create the dataframe\n",
    "ing_df = pd.DataFrame(dict(ingredient = ingredient, count = count))\n",
    "ing_df = ing_df[[\"ingredient\", \"count\"]]\n",
    "print(ing_df.to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now we have a dataframe of ingredients and their total counts across all recipes. Let's sort this dataframe in descending order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "ing_df.sort_values([\"count\"], ascending=False, inplace=True)\n",
    "ing_df.reset_index(inplace=True, drop=True)\n",
    "\n",
    "print(ing_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### What are the 3 most popular ingredients?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Your Answer:\n",
    "1.\n",
    "\n",
    "2.\n",
    "\n",
    "3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "// 1. Egg with 21,025 occurrences. \n",
    "// 2. Wheat with 20,781 occurrences. \n",
    "// 3. Butter with 20,719 occurrences.\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "However, note that there is a problem with the above table. There are ~40,000 American recipes in our dataset, which means that the data is biased towards American ingredients."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Therefore**, let's compute a more objective summary of the ingredients by looking at the ingredients per cuisine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Let's create a *profile* for each cuisine.\n",
    "\n",
    "In other words, let's try to find out what ingredients Chinese people typically use, and what is **Canadian** food for example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "cuisines = recipes.groupby(\"cuisine\").mean()\n",
    "cuisines.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "As shown above, we have just created a dataframe where each row is a cuisine and each column (except for the first column) is an ingredient, and the row values represent the percentage of each ingredient in the corresponding cuisine.\n",
    "\n",
    "**For example**:\n",
    "\n",
    "* *almond* is present across 15.65% of all of the **African** recipes.\n",
    "* *butter* is present across 38.11% of all of the **Canadian** recipes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's print out the profile for each cuisine by displaying the top four ingredients in each cuisine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "num_ingredients = 4 # define number of top ingredients to print\n",
    "\n",
    "# define a function that prints the top ingredients for each cuisine\n",
    "def print_top_ingredients(row):\n",
    "    print(row.name.upper())\n",
    "    row_sorted = row.sort_values(ascending=False)*100\n",
    "    top_ingredients = list(row_sorted.index.values)[0:num_ingredients]\n",
    "    row_sorted = list(row_sorted)[0:num_ingredients]\n",
    "\n",
    "    for ind, ingredient in enumerate(top_ingredients):\n",
    "        print(\"%s (%d%%)\" % (ingredient, row_sorted[ind]), end=' ')\n",
    "    print(\"\\n\")\n",
    "\n",
    "# apply function to cuisines dataframe\n",
    "create_cuisines_profiles = cuisines.apply(print_top_ingredients, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "At this point, we feel that we have understood the data well and the data is ready and is in the right format for modeling!\n",
    "\n",
    "-----------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Thank you for completing this lab!\n",
    "\n",
    "This notebook was created by [Alex Aklson](https://www.linkedin.com/in/aklson/). We hope you found this lab session interesting. Feel free to contact us if you have any questions!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "This notebook is part of a course on **Coursera** called *Data Science Methodology*. If you accessed this notebook outside the course, you can take this course, online by clicking [here](https://cocl.us/DS0103EN_Coursera_LAB3)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<hr>\n",
    "\n",
    "Copyright &copy; 2019 [Cognitive Class](https://cognitiveclass.ai/?utm_source=bducopyrightlink&utm_medium=dswb&utm_campaign=bdu). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
#from Understanding to preperation