neural-machine-translation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "NHzu9qskQJYS",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Assignment #4\n",
    "**CE4719: Deep Learing**\n",
    "\n",
    "\n",
    "*   Spring 2020\n",
    "*   http://ce.sharif.edu/courses/98-99/2/ce719-1/index.php\n",
    "\n",
    "<br/>\n",
    "\n",
    "\n",
    "**Please pay attention to these notes:**\n",
    "- the coding parts you have to complete are specified by:\n",
    "```\n",
    "    ################################################################################\n",
    "    # TODO:                                                                        #\n",
    "    ################################################################################\n",
    "    pass\n",
    "    ################################################################################\n",
    "    #                                 END OF YOUR CODE                             #\n",
    "    ################################################################################ \n",
    "```\n",
    "- We always recommend discussion in groups for assignments. However, each student should finish all of the questions by him/herself. \n",
    "- All submitted code will be compared against all student's codes using Stanford MOSS.\n",
    "- If you have any questions about this assignment, feel free to drop us a line. You may also post your questions on the course's forum page.\n",
    "- We HIGHLY encourage you to run this notebook on Google Colab.\n",
    "- **Before starting to work on the assignment, please fill your name in the next section *AND remember to RUN the cell.***\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2521,
     "status": "ok",
     "timestamp": 1587909672111,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "koeFly4WhvEZ",
    "new_sheet": false,
    "outputId": "c4635b07-6e3b-4e04-98a0-2fa909518727",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "your student id: 98208824\n",
      "your name: Sadroddin Barikbin\n"
     ]
    }
   ],
   "source": [
    "#@title Enter your information & \"RUN the cell!!\"\n",
    "student_id = \"98208824\" #@param {type:\"string\"}\n",
    "student_name = \"Sadroddin Barikbin\" #@param {type:\"string\"}\n",
    "\n",
    "print(\"your student id:\", student_id)\n",
    "print(\"your name:\", student_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "0urkLzHNDJOR",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## 3. Seq2Seq Machine Translation with Attention (41 pts + 5 Bonus pts )\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "ykQ7GalMOq1U",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "In this problem we will implement a neural network that translates sentences from French to English. This is made possible by the simple but powerful idea of the sequence to sequence network, in which two recurrent neural networks work together to transform one sequence to another. In its vanilla version, an encoder network condenses an input sequence into a vector, and a decoder network unfolds that vector into a new sequence. To improve upon this model, we’ll use an attention mechanism, which lets the decoder learn to focus over a specific range of the input sequence. <br/>\n",
    "The architecture of our network is inspired by (but not exactly similar to) the neural machine translation model proposed by [1]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "VEmGY0ueXv39",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.utils\n",
    "import torch.nn.functional as F\n",
    "from torch.nn.utils.rnn import pack_padded_sequence as pack\n",
    "from torch.nn.utils.rnn import pad_packed_sequence as unpack\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "import numpy as np\n",
    "from typing import List, Dict\n",
    "from pprint import pprint\n",
    "import unicodedata\n",
    "from itertools import chain\n",
    "from collections import Counter\n",
    "import random\n",
    "import json\n",
    "import re\n",
    "import time\n",
    "import math\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "Z0FojtjueMGk",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 3.1 Downloading and Reading the Data\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "HKhtPobmfEPc",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "The data for this problem is a set of many thousands of English to French translation pairs. First let's download the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 224
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 16588,
     "status": "ok",
     "timestamp": 1588284909993,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "oE6nWGOIeLZt",
    "new_sheet": false,
    "outputId": "2e71cead-fc15-4945-dd10-27fadaa0856f",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-04-30 22:14:58--  https://download.pytorch.org/tutorial/data.zip\n",
      "Resolving download.pytorch.org (download.pytorch.org)... 13.227.223.82, 13.227.223.75, 13.227.223.15, ...\n",
      "Connecting to download.pytorch.org (download.pytorch.org)|13.227.223.82|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2882130 (2.7M) [application/zip]\n",
      "Saving to: ‘data.zip’\n",
      "\n",
      "\r",
      "data.zip              0%[                    ]       0  --.-KB/s               \r",
      "data.zip              4%[                    ] 126.64K   523KB/s               \r",
      "data.zip             37%[======>             ]   1.04M  2.15MB/s               \r",
      "data.zip            100%[===================>]   2.75M  4.85MB/s    in 0.6s    \n",
      "\n",
      "2020-04-30 22:14:59 (4.85 MB/s) - ‘data.zip’ saved [2882130/2882130]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! wget https://download.pytorch.org/tutorial/data.zip\n",
    "! unzip -q data.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "F9IGe8h0ffCc",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Here is a sneak peek at our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 17957,
     "status": "ok",
     "timestamp": 1588284913918,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "3tFcjYuxfPB3",
    "new_sheet": false,
    "outputId": "794bcc67-626e-454e-cbce-808c25f8b5d2",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Go.\tVa !\n",
      "Run!\tCours !\n",
      "Run!\tCourez !\n",
      "Wow!\tÇa alors !\n",
      "Fire!\tAu feu !\n",
      "Help!\tÀ l'aide !\n",
      "Jump.\tSaute.\n",
      "Stop!\tÇa suffit !\n",
      "Stop!\tStop !\n",
      "Stop!\tArrête-toi !\n"
     ]
    }
   ],
   "source": [
    "DATA_PATH = 'data/eng-fra.txt'\n",
    "! head -n 10 $DATA_PATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "jI_t9Pszfjqy",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "As you can see, every line contains a sentence in English and its translated version in French (Sentences are sorted by their length, so the first few sentences are very short)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 15424,
     "status": "ok",
     "timestamp": 1588284913920,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "CMiPqhayYS2S",
    "new_sheet": false,
    "outputId": "2b7bb1ec-2403-4cc1-f1dc-4461d9324db1",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "# Setting the device\n",
    "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(DEVICE)\n",
    "\n",
    "# Feel free to use the following variables wherever you like throughout the notebook\n",
    "START_TOKEN = '<START>'\n",
    "END_TOKEN = '<END>'\n",
    "PAD_TOKEN = '<PAD>'\n",
    "\n",
    "PAD_IDX = 0\n",
    "START_IDX = 1\n",
    "END_IDX = 2\n",
    "\n",
    "\n",
    "# Function for setting the random seed for reproducibility\n",
    "def set_seed(seed):\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    if DEVICE == torch.device(\"cuda\"):\n",
    "        torch.cuda.manual_seed(seed)\n",
    "        torch.backends.cudnn.deterministic = True\n",
    "        torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "VHGg20qDMkdf",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's define some functions to read the data from file and preprocess it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "PNGgkIyGgSW4",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Turn a Unicode string to plain ASCII, thanks to\n",
    "# https://stackoverflow.com/a/518232/2809427\n",
    "def unicodeToAscii(s):\n",
    "    return ''.join(\n",
    "      c for c in unicodedata.normalize('NFD', s)\n",
    "      if unicodedata.category(c) != 'Mn'\n",
    "    )\n",
    "  \n",
    "# Lowercase, trim, and remove non-letter characters\n",
    "def normalizeString(s):\n",
    "    s = unicodeToAscii(s.lower().strip())\n",
    "    s = re.sub(r\"([.!?])\", r\" \\1\", s)\n",
    "    s = re.sub(r\"[^a-zA-Z.!?]+\", r\" \", s)\n",
    "    return s\n",
    "\n",
    "\n",
    "def read_corpus(path):\n",
    "    \"\"\"Reads the corpus from the file.\n",
    "\n",
    "    path: corpus file's path\n",
    "    \"\"\"\n",
    "    print(\"Reading the corpus...\")\n",
    "\n",
    "    # Read the file and split into lines\n",
    "    lines = open(path, encoding='utf-8').read().strip().split('\\n')\n",
    "\n",
    "    # Split every line into pairs and normalize\n",
    "    pairs = [[normalizeString(s) for s in l.split('\\t')] for l in lines]\n",
    "\n",
    "    src_sents = [s[1].split() for s in pairs]\n",
    "    # Marking the beginning and end of target sentences with <START> and <END>\n",
    "    tgt_sents = [[START_TOKEN] + s[0].split() + [END_TOKEN] for s in pairs]\n",
    "\n",
    "    return src_sents, tgt_sents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "4fZBL9vn27HH",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's read the corpus and print a couple of sentences of it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 14222,
     "status": "ok",
     "timestamp": 1588284918192,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "pfofaz073G1v",
    "new_sheet": false,
    "outputId": "326112cf-6e7a-48f8-92ba-ccd0fbff2baa",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading the corpus...\n",
      "[['va', '!'],\n",
      " ['cours', '!'],\n",
      " ['courez', '!'],\n",
      " ['ca', 'alors', '!'],\n",
      " ['au', 'feu', '!']]\n",
      "[['<START>', 'go', '.', '<END>'],\n",
      " ['<START>', 'run', '!', '<END>'],\n",
      " ['<START>', 'run', '!', '<END>'],\n",
      " ['<START>', 'wow', '!', '<END>'],\n",
      " ['<START>', 'fire', '!', '<END>']]\n",
      "\n",
      "Number of sentences 135842\n"
     ]
    }
   ],
   "source": [
    "src_sents, tgt_sents = read_corpus(DATA_PATH)\n",
    "pprint(src_sents[:5])\n",
    "pprint(tgt_sents[:5])\n",
    "\n",
    "print('\\nNumber of sentences', len(src_sents))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "BEXjU-VyfpjZ",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 3.2 Vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "bZ_iARJSqsqy",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "As we need the Vocab class you implemented in Problem 1 here, copy your implementaion in the following cell (or copy your code into a .py file and import it from there)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "aVKSzWaZfOku",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "class Vocab:\n",
    "    def __init__(self, word2id=None):\n",
    "        \"\"\"Constructor of Vocab\n",
    "\n",
    "        word2id: dictionary that maps tokens to their ids.\n",
    "        \"\"\"\n",
    "        self.pad_token = '<PAD>'\n",
    "        self.end_token = '<END>'\n",
    "        self.start_token = '<START>'\n",
    "        self.unk_token = '<UNK>'\n",
    "\n",
    "        if word2id is None:\n",
    "            self.word2id = {self.pad_token: 0,\n",
    "                            self.start_token: 1,\n",
    "                            self.end_token: 2,\n",
    "                            self.unk_token: 3}\n",
    "            self.size = 4\n",
    "        else:\n",
    "            self.word2id = word2id\n",
    "            self.size = len(self.word2id)\n",
    "\n",
    "        self.id2word = {v: k for (k, v) in self.word2id.items()}\n",
    "\n",
    "    def build(self, tokenized_corpus: List[List[str]], size=None, min_freq=None):\n",
    "        \"\"\"Builds the vocab from a tokenized corpus.\n",
    "\n",
    "        tokenized_corpus: corpus as a list of list of tokens (strings)\n",
    "        size: Final size of (number of unique tokens in) our vocab\n",
    "        min_freq: minimum frequency\n",
    "        \"\"\"\n",
    "        tokens2freq = Counter(chain(*tokenized_corpus))\n",
    "        frequent_tokens = []\n",
    "        frequent_tokens=map(lambda x:x[0],filter(lambda x: not min_freq or x[1]>=min_freq,tokens2freq.most_common(size)))\n",
    "        for token in frequent_tokens:\n",
    "            self.add_token(token)\n",
    "\n",
    "    def get_token_by_id(self, t_id: int) -> str:\n",
    "        \"\"\"Returns the token with the corresponding id in the vocab.\n",
    "        If the id is not valid, returns None.\n",
    "\n",
    "        t_id: token id\n",
    "        \"\"\"\n",
    "        return self.id2word.get(t_id, None)\n",
    "\n",
    "    def get_id_by_token(self, token: str) -> int:\n",
    "        \"\"\"Returns the id of the token in the vocab. If the token does not exist,\n",
    "        returns the id of <UNK> token.\n",
    "\n",
    "        token: token (as a string) for which the id should be returned.\n",
    "        \"\"\"\n",
    "        return self.word2id.get(token, self.word2id[self.unk_token])\n",
    "\n",
    "    def add_token(self, token: str):\n",
    "        \"\"\"Adds the token to the vocab's data structures\n",
    "        token: token as a string\n",
    "        \"\"\"\n",
    "        if not token in self.word2id:\n",
    "          self.word2id[token]=self.size\n",
    "          self.id2word[self.size]=token\n",
    "          self.size+=1\n",
    "\n",
    "    def tokens2ids(self, sents):\n",
    "        \"\"\"Convert list of words or list of sentences of tokens \n",
    "        into list or list of list of indices.\n",
    "\n",
    "        sents: input sentences as List[List[str]] (multiple sentences) or List[str]\n",
    "        (single sentence)\n",
    "        \"\"\"\n",
    "        if type(sents[0]) == list:\n",
    "            return [[self.get_id_by_token(token) for token in sent] for sent in sents]\n",
    "        else:\n",
    "            return [self.get_id_by_token(token) for token in sents]\n",
    "  \n",
    "    def to_tensor(self, sent: List[str]):\n",
    "        \"\"\"Converts a sentence as a list of tokens into a tensor of indices.\n",
    "\n",
    "        sent: a sentence as a list of strings (tokens)\n",
    "        \"\"\"\n",
    "        return torch.tensor(self.tokens2ids(sent),dtype=torch.long,device=DEVICE)\n",
    "\n",
    "    def pad_sents(self, sents: List[List[str]]) -> List[List[str]]:\n",
    "        \"\"\"Pads list of sentences according to the longest sentence.\n",
    "\n",
    "        sents: sentences as a list of list of tokens (strings).\n",
    "        \"\"\"\n",
    "        sents_padded = []\n",
    "        maxx=max(map(lambda sent:len(sent),sents))\n",
    "        sents_padded=[sent+['<PAD>']*(maxx-len(sent)) for sent in sents]\n",
    "        return sents_padded\n",
    "\n",
    "    def save(self, path: str):\n",
    "        \"\"\"Saves the vocab in a json file.\n",
    "\n",
    "        path: path to save the vocab in\n",
    "        \"\"\"\n",
    "        with open(path, 'w') as f:\n",
    "            json.dump(self.word2id, f)\n",
    "\n",
    "    @staticmethod\n",
    "    def load(path: str):\n",
    "        \"\"\"Loads vocab from a json file.\n",
    "\n",
    "        path: path to load the vocab from\n",
    "        \"\"\"\n",
    "        with open(path, 'r') as f:\n",
    "            word2id = json.load(f)\n",
    "\n",
    "        return Vocab(word2id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "s44fwNO6ftyh",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now let's create the vocabs using our source and target sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4743,
     "status": "ok",
     "timestamp": 1588284918194,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "20WsmLSifw-a",
    "new_sheet": false,
    "outputId": "41b0107c-d229-42ea-cdcf-5265b706041c",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "French Vocab Size: 21335\n",
      "English Vocab Size: 13044\n"
     ]
    }
   ],
   "source": [
    "src_vocab = Vocab()\n",
    "src_vocab.build(src_sents)\n",
    "tgt_vocab = Vocab()\n",
    "tgt_vocab.build(tgt_sents)\n",
    "\n",
    "# You could also set the size of the vocab and minimum frequency of the words if you'd like\n",
    "print('French Vocab Size:', src_vocab.size)\n",
    "print('English Vocab Size:', tgt_vocab.size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "xoD6TMSXXoiv",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 3.3 Creating Dataset and DataLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "xiLWkpjbXtZ1",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now Let's create datasets and dataloaders so that we can later use them for training our model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "rABkKu9JXvhA",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "class NMTData(Dataset):\n",
    "    def __init__(self, src_sents: List[List[str]], tgt_sents: List[List[str]], transform):\n",
    "        self.src_sents = src_sents\n",
    "        self.tgt_sents = tgt_sents\n",
    "        self.transform = transform\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.src_sents)\n",
    "\n",
    "    def __getitem__(self, item):\n",
    "        sents = {'src': self.src_sents[item], 'tgt': self.tgt_sents[item]}\n",
    "        return self.transform(sents)\n",
    "    \n",
    "\n",
    "# utility function for padding a tensor\n",
    "def pad_tensor(tensor: torch.Tensor, size: int, dim: int=-1):\n",
    "    \"\"\"\n",
    "    tensor: tensor to pad\n",
    "    size: the size to pad to\n",
    "    dim: dimension to pad\n",
    "\n",
    "    returns a new tensor padded to 'size' in dimension 'dim'\n",
    "    \"\"\"\n",
    "    pad_size = list(tensor.shape)\n",
    "    pad_size[dim] = size - tensor.size(dim)\n",
    "    pad = torch.full(pad_size, PAD_IDX, dtype=tensor.dtype, device=tensor.device)\n",
    "\n",
    "    return torch.cat([tensor, pad], dim=dim)\n",
    "\n",
    "\n",
    "# Thanks to https://discuss.pytorch.org/t/dataloader-for-various-length-of-data/6418/8\n",
    "class PadCollate:\n",
    "    \"\"\"\n",
    "    a variant of callate_fn that pads according to the longest sequence in\n",
    "    a batch of sequences\n",
    "    \"\"\"\n",
    "    def __init__(self, dim=-1):\n",
    "        \"\"\"\n",
    "        dim: the dimension to be padded (e.g., dimension of time in sequences)\n",
    "        \"\"\"\n",
    "        self.dim = dim\n",
    "\n",
    "    def pad_collate(self, batch):\n",
    "        \"\"\"\n",
    "        batch: a batch of data as a list of Dictionaries\n",
    "\n",
    "        returns:\n",
    "          src_sents: tensor of source sentences\n",
    "          tgt_sents: tensor of target sentences\n",
    "          src_lengths: lengths of source sentences before padding as a list \n",
    "        \"\"\"\n",
    "\n",
    "        # sort pairs based on the lengths of source sentences\n",
    "        batch = sorted(batch, key=lambda pair: pair['src'].size(0), reverse=True)\n",
    "        src_lengths = [pair['src'].size(0) for pair in batch]\n",
    "\n",
    "        # find longest sequence\n",
    "        max_len_src = max(map(lambda pair: pair['src'].shape[self.dim], batch))\n",
    "        max_len_tgt = max(map(lambda pair: pair['tgt'].shape[self.dim], batch))\n",
    "\n",
    "        # pad according to max_len\n",
    "        batch = list(map(lambda pair: (pad_tensor(pair['src'], size=max_len_src, dim=self.dim),\n",
    "                                       pad_tensor(pair['tgt'], size=max_len_tgt, dim=self.dim)),\n",
    "                         batch))\n",
    "\n",
    "        # stack all\n",
    "        src_sents = torch.stack([pair[0] for pair in batch], dim=-1)\n",
    "        tgt_sents = torch.stack([pair[1] for pair in batch], dim=-1)\n",
    "\n",
    "        return src_sents, tgt_sents, src_lengths\n",
    "\n",
    "    def __call__(self, batch):\n",
    "        return self.pad_collate(batch)\n",
    "\n",
    "\n",
    "class ToTensor(object):\n",
    "    \"\"\"A callable transform that converts sentences to Tensors.\n",
    "    \"\"\"\n",
    "    def __init__(self, src_vocab: Vocab, tgt_vocab: Vocab):\n",
    "        self.src_vocab = src_vocab\n",
    "        self.tgt_vocab = tgt_vocab\n",
    "\n",
    "    def __call__(self, sents: Dict):\n",
    "        src_sent, tgt_sent = sents['src'], sents['tgt']\n",
    "\n",
    "        return {'src': self.src_vocab.to_tensor(src_sent),\n",
    "                'tgt': self.tgt_vocab.to_tensor(tgt_sent)}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "8Y6ldOPRZ4kz",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now let's create a sample dataloader and read a batch of data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 561
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 10749,
     "status": "ok",
     "timestamp": 1588284931000,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "SuesGV3UZ-8H",
    "new_sheet": false,
    "outputId": "2d24884d-0b23-46f7-e9aa-e9e039a3ba26",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[   5,    5,   24, 1862,   89],\n",
      "        [  31,   15,   99,   14,  451],\n",
      "        [  33,   33,   76,  977, 4730],\n",
      "        [ 743,    9,  215,   40,    4],\n",
      "        [ 243, 1104,   40,  134,    0],\n",
      "        [   7,    7,   28,  263,    0],\n",
      "        [ 739,   20, 7563,    4,    0],\n",
      "        [   6,   13,    4,    0,    0],\n",
      "        [ 441,   17,    0,    0,    0],\n",
      "        [ 978,  162,    0,    0,    0],\n",
      "        [  16,    4,    0,    0,    0],\n",
      "        [ 428,    0,    0,    0,    0],\n",
      "        [ 107,    0,    0,    0,    0],\n",
      "        [   4,    0,    0,    0,    0]], device='cuda:0')\n",
      "\n",
      "tensor([[   1,    1,    1,    1,    1],\n",
      "        [   5,    5,   25,  101,   32],\n",
      "        [ 103,   35,   71,    8,  391],\n",
      "        [  65,   36, 7238,  942,   60],\n",
      "        [ 308,  138,    4,    7, 1610],\n",
      "        [  77,   26,    2,   32,    4],\n",
      "        [  15,    6,    0,  166,    2],\n",
      "        [   5,   71,    0,    4,    0],\n",
      "        [  91,  436,    0,    2,    0],\n",
      "        [ 485,    4,    0,    0,    0],\n",
      "        [   8,    2,    0,    0,    0],\n",
      "        [ 217,    0,    0,    0,    0],\n",
      "        [ 287,    0,    0,    0,    0],\n",
      "        [   4,    0,    0,    0,    0],\n",
      "        [   2,    0,    0,    0,    0]], device='cuda:0')\n",
      "\n",
      "[14, 11, 8, 7, 4]\n"
     ]
    }
   ],
   "source": [
    "set_seed(40719)\n",
    "dataset = NMTData(src_sents=src_sents, tgt_sents=tgt_sents, \n",
    "                  transform=ToTensor(src_vocab, tgt_vocab))\n",
    "dataloader = DataLoader(dataset, batch_size=5, shuffle=True, num_workers=0,\n",
    "                        collate_fn=PadCollate())\n",
    "\n",
    "src_sents_tensor, tgt_sents_tensor, src_lengths = next(iter(dataloader))\n",
    "print(src_sents_tensor, tgt_sents_tensor, src_lengths, sep='\\n\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "feCVz6vd5uyL",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now we are ready to code our Seq2Seq model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "paCGNCC25PpY",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 3.4 Seq2Seq with Attention Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "qsNJVJoOO8tC",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "In Machine Translation, our goal is to convert a sentence from the source language to the target language. In this assignment, we will implement a sequence-to-sequence (Seq2Seq) network with attention, to build a Neural Machine Translation (NMT) system. In this section, we describe the training procedure for the proposed NMT system:\n",
    "\n",
    "1. Given a sentence in the source language, we look up the word embeddings from the source embeddings matrix with the embedding dimension $e$.\n",
    "\n",
    "2. We feed these embeddings to the Encoder, yielding hidden states $h_{0}^{enc},...,h_{m}^{enc} \\in \\mathbb{R}^h$ where $h$ is the *hidden size* of the network and $m$ is the length of the source sequence.\n",
    "\n",
    "3. We then initialize the Decoder's first hidden state $h_0^{dec} \\in \\mathbb{R}^h$ by applying a linear projection (called *bridge*) on the Encoder's final hidden state.\n",
    "\n",
    "4. With the Decoder initialized, we must now feed it the matching sentence in the target language. On the $t$-th step, we look up the embedding for the $t$-th word (or token), $y_t \\in \\mathbb{R}^e$ where $e$ is the *embedding size*. Note that during the training procedure where we use teacher forcing, $y_t$ is the $t$-th word in the target sentence. However, in the test stage where we do not have a target sentence, the output word of the model at the previous time step will be used as $y_t$.\n",
    "\n",
    "5. We then concatenate $y_t$ with the combined-output vector from the previous timestep (the gray vector in the illustration below which is denoted by $\\tilde{h}$) to produce $\\hat{y}_{t} \\in \\mathbb{R}^{e+h}$. for the first target word (i.e. the start token), previous combined-output vector is a zero-vector. We then feed $\\hat{y}_{t}$ as input to the decoder.\n",
    "\n",
    "6. We use $h_t^{dec}$ to compute **dot attention** over $h_{0}^{enc},...,h_{m}^{enc}$ (please refer to the paper for the details about this type of attention). This will give us a combination of the Encoder's hidden states called the *context vector* (blue vector in the illustration).\n",
    "\n",
    "7. We concatenate the context vector with the decoder hidden state $h_{t}^{dec}$ at each step and apply a linear layer (called *attn_combine* in the code), Tanh, and Dropout to get the next combined-output vector.\n",
    "\n",
    "8. After doing the above procedure for all of the time steps (i.e. generating the combined outputs of all time steps), we will apply the target vocab projection layer to compute scores (logits) of producing different words in the target vocab at all of the time steps simultaneously. \n",
    "\n",
    "9. We then produce the log-probability **$P$**, using log-softmax function, over target words and sum the log probability of generating the correct word over all of the time steps. We then minimize negative value of this sum to train our network.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "FlKkEZH8sidK",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "The above algorithm will become clearer to you as we walk through their implementation step by step later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "vJ6OQ-2o6dIz",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "The following figure from the paper illustrates the attention mechanism we will implement. Note that without attention mechanism, the encoder creates a single vector which, in the ideal case, encodes the “meaning” of the input sequence into a single vector — a single point in some $h$ dimensional space of sentences. This vector will then be passed to the decoder to generate the desired output. This single vector carries the burden of encoding the entire sentence. But, with attention, the decoder network can “focus” on a different part of the encoder’s outputs for every step of the decoder’s own outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "0cS5iLuQZv-x",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<p align=\"center\">\n",
    "<img src=\"https://i.imgur.com/RUmR1CU.png\" height=400/>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "PN2oUAje86jr",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Here is another illustration of the training procedure of our Seq2Seq model. Note that this figure shows stacked (multi-layer) RNNs as the Encoder and the Decoder. We will use a single layer rnn in our implementation. Note that `<s>` and `</s>` are START and END tokens [4].\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "42g1NRKH9M89",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<p align=\"center\">\n",
    "<img src=\"https://imgur.com/download/Xf46eg0\" height=400/>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "MhC3bIYh0i4U",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's start coding the model. Here is the code of our encoder's class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "AiFB2kbXz9LS",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    def __init__(self, hidden_size, embed_size: int, src_vocab_size: int):\n",
    "        \"\"\"\n",
    "        hidden_size: hidden size to be used for the RNN\n",
    "        embed_size: embedding size (dimension)\n",
    "        src_vocab_size: source language vocab's size\n",
    "        \"\"\"\n",
    "        super(Encoder, self).__init__()\n",
    "        self.embed_size = embed_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.src_vocab_size = src_vocab_size\n",
    "        self.rnn = None\n",
    "        self.src_embedding = None\n",
    "        ################################################################################\n",
    "        # TODO: IMPORTANT: INSTANTIATE OBJECTS IN THE EXACT FOLLOWING ORDER:           #\n",
    "        #       1) Instantiate self.rnn using nn.GRU.                                  #\n",
    "        #       2) Instantiate src_embedding layer which will be used for embedding    #\n",
    "        #          source sentences. (Don't forget to set the padding index to PAD_IDX)#\n",
    "        ################################################################################\n",
    "        self.rnn=nn.GRU(self.embed_size,self.hidden_size)\n",
    "        self.src_embedding=nn.Embedding(self.src_vocab_size,self.embed_size,padding_idx=PAD_IDX)\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "\n",
    "    def forward(self, src: torch.Tensor, src_lengths: List[int]):\n",
    "        \"\"\"\n",
    "        src: tensor of source sentences containing token's indices in the vocab\n",
    "        src_lengths: lengths of source sentences before padding\n",
    "        \"\"\"\n",
    "        enc_hiddens, last_hidden, enc_hiddens_mask = None, None, None\n",
    "        ################################################################################\n",
    "        # TODO: 1) embed the source sentences using the embedding layer.               #\n",
    "        #       2) apply the encoder rnn to the tensor of source sentences             #      \n",
    "        #          and get the encoder hidden states (or outputs) and the              #\n",
    "        torch.zeros((batch_true_box_features.size(0)*2,self.hidden_size),dtype=torch.float,device=DEVICE)#          last hidden state. Save the hidden states (outputs) and last hidden #\n",
    "        #          state in enc_hiddens and last_hidden variables. (Don't forget to    #\n",
    "        #          pack and unpack!)                                                   #\n",
    "        #       3) In its default mode, the encoder's outputs will be of the shape     #\n",
    "        #          (seq_length, batch_size, embed_size). Later when we want to apply   #\n",
    "        #          attention, we want it to be (batch_size, seq_length, embed_size).   #\n",
    "        #          So use torch.permute to reshape it into the desired shape right     #\n",
    "        #          here.                                                               #\n",
    "        #       4) generate encoder hiddens mask using generate_enc_hiddens_mask()     #\n",
    "        #          method and save the mask in enc_hiddens_mask variable.              #\n",
    "        #          this mask will be used to mask <PAD> tokens in the source sentences #\n",
    "        #          when we apply the attention mechanism in the decoder.               #\n",
    "        ################################################################################\n",
    "        embedded=self.src_embedding(src)\n",
    "        enc_hiddens,last_hidden=self.rnn(pack(embedded,src_lengths))\n",
    "        enc_hiddens=unpack(enc_hiddens)[0]\n",
    "        enc_hiddens=enc_hiddens.permute(1,0,2)\n",
    "        enc_hiddens_mask=self.generate_enc_hiddens_mask(enc_hiddens,src_lengths)\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "        return enc_hiddens, last_hidden, enc_hiddens_mask\n",
    "\n",
    "    def generate_enc_hiddens_mask(self, enc_hiddens: torch.Tensor, lengths: List[int]) -> torch.Tensor:\n",
    "        \"\"\"Generates mask which masks the encoder hidden states corresponding to <PAD> tokens \n",
    "        in the source sentences\n",
    "        \"\"\"\n",
    "        enc_masks = torch.zeros(enc_hiddens.shape[0], enc_hiddens.size(1), dtype=torch.float)\n",
    "        for iterator, length in enumerate(lengths):\n",
    "            enc_masks[iterator, length:] = 1\n",
    "\n",
    "        return enc_masks.to(DEVICE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "GSmlbil70c3p",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Here is the code of our decoder's class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "XEq2afGM0X9f",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "class AttnDecoder(nn.Module):\n",
    "    def __init__(self, hidden_size: int, embed_size: int, tgt_vocab_size: int, dropout_rate=0.2):\n",
    "        \"\"\"\n",
    "        hidden_size: hidden size to be used for the RNN\n",
    "        embed_size: embedding size (dimension)\n",
    "        tgt_vocab_size: target language vocab size \n",
    "        dropout_rate: rate to be used for the dropout layer \n",
    "        \"\"\"\n",
    "        super(AttnDecoder, self).__init__()\n",
    "        self.embed_size = embed_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.tgt_vocab_size = tgt_vocab_size\n",
    "        self.rnn = None\n",
    "        self.dropout = None\n",
    "        self.tgt_vocab_projection = None  \n",
    "        self.attn_combine = None \n",
    "        self.tgt_embedding = None\n",
    "        ################################################################################\n",
    "        # TODO: IMPORTANT: INSTANTIATE OBJECTS IN THE EXACT FOLLOWING ORDER:           #\n",
    "        #       1) Instantiate the target embedding layer in self.tgt_embedding. Don't #\n",
    "        #          forget to set the padding_idx to PAD_IDX.                           #\n",
    "        #       2) Instantiate the decoder rnn (nn.GRUCell) (Try to figure out why we  #\n",
    "        #          use GRU for the encoder and GRUCell for the decoder.)               #\n",
    "        #       3) Instantiate the attn_combine layer, a Linear layer which will       #\n",
    "        #          be applied on the concatenation of context vector and decoder's     #\n",
    "        #          hidden state to get the combined output vector.                     # \n",
    "        #       4) Instantiate the tgt_vocab_projection layer which is a Linear layer  #\n",
    "        #          that will be used for projecting the combined output (denoted by    #\n",
    "        #          h_tilde in the paper) to the vocab's space.                         #\n",
    "        #       5) Instantiate the dropout layer in self.dropout                       #\n",
    "        #       IMPORTANT: SET THE BIAS OF ALL LINEAR PROJECTION LAYERS TO ZERO!       #\n",
    "        ################################################################################\n",
    "        self.tgt_embedding=nn.Embedding(self.tgt_vocab_size,self.embed_size,padding_idx=PAD_IDX)\n",
    "        self.rnn=nn.GRUCell(self.embed_size+self.hidden_size,self.hidden_size)\n",
    "        self.attn_combine=nn.Linear(2*self.hidden_size,self.hidden_size,bias=False)\n",
    "        self.tgt_vocab_projection=nn.Linear(self.hidden_size,self.tgt_vocab_size,bias=False)\n",
    "        self.dropout=nn.Dropout(dropout_rate)\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "\n",
    "    def forward(self, enc_hiddens: torch.Tensor, enc_hiddens_mask: torch.Tensor, \n",
    "                dec_init_state: torch.Tensor, tgt: torch.Tensor):\n",
    "        \"\"\"Runs the teacher forcing algorithm and returns the scores (logits) of different words \n",
    "        across all of the time steps. The logits shape should be (seq_len_tgt-1, batch_size, tgt_vocab_size).\n",
    "\n",
    "        tgt: tensor of target sentences with shape (seq_len_tgt, batch_size)\n",
    "        enc_hiddens: tensor of the encoder hidden states with shape (batch_size, seq_len_src, hidden_size)\n",
    "        enc_hiddens_mask: tensor of the encoder hidden states with shape (batch_size, seq_len_src)\n",
    "        dec_init_state: decoder's initial hidden state\n",
    "        \"\"\"\n",
    "        \n",
    "        # remove the <END> token from the longest sentence(s)\n",
    "        tgt = tgt[:-1]\n",
    "        \n",
    "        # Initialize the decoder state\n",
    "        dec_state = dec_init_state\n",
    "\n",
    "        # Initialize previous combined output vector as zero\n",
    "        batch_size = enc_hiddens.size(0)\n",
    "        combined_output_prev = torch.zeros(batch_size, self.hidden_size, device=DEVICE)\n",
    "\n",
    "        # Initialize a list we will use to collect the combined output o_t at each step\n",
    "        combined_outputs = []\n",
    "        logits = None\n",
    "        ################################################################################\n",
    "        # TODO: 1) Apply the embedding layer to the target sentences tensor            #\n",
    "        #       2) Use a for loop and implement teacher forcing algorithm. At each     # \n",
    "        #          step you have to call the step method. Append the combined output of#\n",
    "        #          each step to the combined_outputs list.                             #\n",
    "        #       3) Stack the combined_outputs and make a tensor from it using          #\n",
    "        #          torch.stack                                                         #\n",
    "        #          HINT: After stacking, combined_outputs shape should be              #\n",
    "        #          (seq_len_tgt-1, batch_size, hidden_size).                           #\n",
    "        #       4) Apply the vocab projection layer to project the combined outputs to #\n",
    "        #          vocab's space and get the scores of the producing every word in the # \n",
    "        #          vocab for all of the time steps.                                    #\n",
    "        #          of the time steps. Save the scores in logits variable.              #      \n",
    "        ################################################################################\n",
    "        embedded=self.tgt_embedding(tgt)\n",
    "        for t in range(tgt.size(0)):\n",
    "          ybar_t=torch.cat([embedded[t],combined_output_prev],dim=-1)\n",
    "          dec_state,combined_output_prev=self.step(ybar_t,dec_state,enc_hiddens,enc_hiddens_mask)\n",
    "          combined_outputs+=[combined_output_prev]\n",
    "        combined_outputs=torch.stack(combined_outputs)\n",
    "        logits=self.tgt_vocab_projection(combined_outputs)\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "        return logits\n",
    "\n",
    "    def step(self, ybar_t: torch.Tensor, dec_state: torch.Tensor, enc_hiddens: torch.Tensor, enc_hiddens_mask):\n",
    "        \"\"\"\n",
    "        ybar_t: the input to be fed to the decoder at the current step with shape\n",
    "        dec_state: decodor's previous hidden state \n",
    "        enc_hiddens: encoder hidden states (outputs)\n",
    "        enc_hiddens_mask: mask generated during the encoding phase that masks <PAD>s \n",
    "\n",
    "        returns the current hidden state and combined output\n",
    "        \"\"\"\n",
    "        attn_scores_t, dec_state_t = None, None\n",
    "        ################################################################################\n",
    "        # TODO: 1) Compute the decoder's current hidden state by applying the decoder  #\n",
    "        #          rnn on the current input and save it in dec_state_t variable.       #\n",
    "        #       2) Compute attention scores by multiplying the encoder's hidden states #\n",
    "        #          by the decoder's hidden state and save it in attn_scores_t.         #\n",
    "        #          HINT: use torch.bmm                                                 #\n",
    "        ################################################################################\n",
    "        dec_state_t=self.rnn(ybar_t,dec_state)\n",
    "        attn_scores_t=torch.bmm(enc_hiddens,dec_state_t.unsqueeze(-1)).squeeze(-1)\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "\n",
    "        # Set attn_scores_t to -inf where enc_hiddens_mask has 1, where there is a <PAD>\n",
    "        # in the source sentence.\n",
    "        if enc_hiddens_mask is not None:\n",
    "            attn_scores_t.data.masked_fill_(enc_hiddens_mask.bool(), -float('inf'))\n",
    "\n",
    "        combined_output_t = None\n",
    "        ################################################################################\n",
    "        # TODO: 1) Compute attention distribution (weights) by applying softmax        #\n",
    "        #          function on the attention scores.                                   #\n",
    "        #       2) Compute context vector by multiplying the encoder hidden states by  #\n",
    "        #          attention weights. Each hidden state should be multiplied by its    #\n",
    "        #          corresponding weight!                                               #\n",
    "        #          HINT: use torch.bmm.                                                #                \n",
    "        #       3) Compute the combined output (The gray vectors in the attention      #\n",
    "        #          illustration above) and save it in combined_output_t                #\n",
    "        #          variable.                                                           #\n",
    "        #          HINT: combined_output_t shape should be (batch_size, hidden_size).  #\n",
    "        #       4) First Apply tanh and then dropout layer on the combined_output_t.   #\n",
    "        ################################################################################\n",
    "        attn_weights=torch.softmax(input=attn_scores_t,dim=-1)\n",
    "        context_vect=torch.bmm(enc_hiddens.permute(0,2,1),attn_weights.unsqueeze(-1)).squeeze(-1)\n",
    "        combined_output_t=self.attn_combine(torch.cat([dec_state_t,context_vect],dim=1))\n",
    "        combined_output_t=self.dropout(torch.tanh(combined_output_t))\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "        return dec_state_t, combined_output_t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "Hwl7lCBVW22S",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "The following class is the whole Sequence-to-Sequence model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "ouguEZHiFhVS",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "class AttnSeq2Seq(nn.Module):\n",
    "    def __init__(self, hidden_size: int, embed_size: int, src_vocab_size: int, \n",
    "                 tgt_vocab_size: int, dropout_rate:float=0.2):\n",
    "        \"\"\"\n",
    "        hidden_size: hidden size (dimensionality) of the RNNs\n",
    "        embed_size: embedding size (dimensionality)\n",
    "        src_vocab_size: source sentences Vocabulary object\n",
    "        tgt_vocab_size: target sentences Vocabulary object\n",
    "        dropout_rate: dropout probability\n",
    "        \"\"\"\n",
    "        super(AttnSeq2Seq, self).__init__()\n",
    "        self.embed_size = embed_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.src_vocab_size, self.tgt_vocab_size = src_vocab_size, tgt_vocab_size\n",
    "        self.dropout_rate = dropout_rate\n",
    "        self.encoder, self.decoder, self.bridge = None, None, None\n",
    "        ################################################################################\n",
    "        # TODO: IMPORTANT: INSTANTIATE OBJECTS IN THE EXACT FOLLOWING ORDER:           #\n",
    "        #       1) Instantiate self.encoder using Encoder class.                       #\n",
    "        #       2) Instantie the bridge between encoder and decoder. Note that bridge  #\n",
    "        #          is a Linear layer used for initializing the decoder's first hidden  #\n",
    "        #          state from encoder's last hidden state.                             #\n",
    "        #       3) Instantiate self.decoder using AttnDecoder class.                   #\n",
    "        #       IMPORTANT: SET THE BIAS OF ALL LINEAR PROJECTION LAYERS TO ZERO!       #\n",
    "        ################################################################################\n",
    "        self.encoder=Encoder(hidden_size,embed_size,src_vocab_size)\n",
    "        self.bridge=nn.Linear(hidden_size,hidden_size,bias=False)\n",
    "        self.decoder=AttnDecoder(hidden_size,embed_size,tgt_vocab_size,dropout_rate=dropout_rate)\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "  \n",
    "    def init_weights(self, val: float):\n",
    "        ################################################################################\n",
    "        # TODO: Initialize all of the parameters of the model with uniform distribution#\n",
    "        #       between -val and +val.                                                 #\n",
    "        ################################################################################\n",
    "        for param in self.parameters():\n",
    "          nn.init.uniform_(param,a=-val,b=val)\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "\n",
    "    def forward(self, src_tensor: torch.Tensor, tgt_tensor: torch.Tensor, src_lengths: List[int]):\n",
    "        \"\"\" Takes a mini-batch of source and target sentences, compute the log-likelihood of\n",
    "        target sentences under the language models learned by the neural machine translation\n",
    "        system.\n",
    "\n",
    "        src_tensor: tensor of source sentences \n",
    "        tgt: tensor of target sentences\n",
    "        src_lengths: lengths of source sentences before padding\n",
    "        \"\"\"\n",
    "        log_probs = None\n",
    "        ################################################################################\n",
    "        # TODO: 1) Apply the encoder to the tensor of source sentences by calling      #\n",
    "        #          self.encoder() and get the encoder hidden states (or outputs)       #\n",
    "        #          and last hidden state and the mask.                                 #\n",
    "        #       2) Compute the decoder's intial hidden state using self.bridge         #\n",
    "        #          HINT: decoder's hidden state should be a 2d tensor!                 #\n",
    "        #       3) get the outputs scores (logits) of the target words from the decoder# \n",
    "        #       4) produce the log probability over the target words using             #\n",
    "        #          F.log_softmax.                                                      #\n",
    "        ################################################################################   \n",
    "        enc_hiddens,last_hidden,enc_hidden_masks=self.encoder(src_tensor,src_lengths)\n",
    "        dec_initial_state=self.bridge(last_hidden).squeeze(0)\n",
    "        logits=self.decoder(enc_hiddens,enc_hidden_masks,dec_initial_state,tgt_tensor)\n",
    "        log_probs=F.log_softmax(logits,dim=2)\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "        tgt_mask = (tgt_tensor != PAD_IDX).float()\n",
    "\n",
    "        # Compute log probability of generating true target words\n",
    "        tgt_gold_words_log_prob = torch.gather(log_probs, index=tgt_tensor[1:].unsqueeze(-1), dim=-1).squeeze(-1) * tgt_mask[1:]\n",
    "\n",
    "        # sum scores across different time steps of sequences (i.e. computing log probability of the sequences)\n",
    "        scores = tgt_gold_words_log_prob.sum(dim=0)\n",
    "\n",
    "        return scores\n",
    "\n",
    "    def greedy_decode(self, src_sent: torch.Tensor, tgt_vocab: Vocab, max_length: int = 50):\n",
    "        \"\"\"\n",
    "        src_sent: source sentence tensor with shape (seq_len, 1)\n",
    "        max_length: allowed maximum length of the target translation\n",
    "\n",
    "        returns decoded_sent: decoded translation of source sentence\n",
    "        \"\"\"\n",
    "        decoded_sent = []\n",
    "        ################################################################################\n",
    "        # TODO: Implement the greedy decoding algorithm.                               #\n",
    "        #       First you have to run the encoder on the sourec sentence. Then run the #\n",
    "        #       decoder step by step to generate the target sentence. Note that you    #\n",
    "        #       have to directly call the step method from here.                       #\n",
    "        #       Pass the <START> token as the first input and then use the output of   #\n",
    "        #       previous step. Don't forget to concatenate combined output of previous #\n",
    "        #       time step with the input. Use all-zeros vector as the first            #\n",
    "        #       combined output.                                                       #\n",
    "        #       Stop decooding when either one of the following cases occurs:          #\n",
    "        #         1) The decoder generates an <END> token.                             #\n",
    "        #         2) The decoder generates max_length words without generating an      #\n",
    "        #            <END> token.                                                      #\n",
    "        #       Use the tgt_vocab to convert indices into words and append the         #\n",
    "        #       generated word at each time step (including <END> and exluding         #\n",
    "        #       <START>) to the decoded_sent list.                                     # \n",
    "        ################################################################################ \n",
    "        combined_output = torch.zeros(1, self.hidden_size, device=DEVICE)\n",
    "        enc_hiddens,last_hidden,enc_hidden_masks=self.encoder(src_sent,[len(src_sent)])\n",
    "        dec_state=self.bridge(last_hidden).squeeze(0)\n",
    "        first_step=True\n",
    "        curr_len=0\n",
    "        while(curr_len<=max_length):\n",
    "          ybar=self.decoder.tgt_embedding(tgt_vocab.to_tensor(['<START>'])[0]).unsqueeze(0) if first_step else self.decoder.tgt_embedding(tgt_vocab.to_tensor([word])[0]).unsqueeze(0)\n",
    "          ybar_t=torch.cat([ybar,combined_output],dim=-1)\n",
    "          first_step=False\n",
    "          dec_state,combined_output=self.decoder.step(ybar_t, dec_state, enc_hiddens, enc_hidden_masks)\n",
    "          logits=self.decoder.tgt_vocab_projection(combined_output)\n",
    "          word=tgt_vocab.get_token_by_id(torch.argmax(logits).item())\n",
    "          curr_len+=1\n",
    "          decoded_sent+=[word]\n",
    "          if word=='<END>':\n",
    "            break\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "\n",
    "        return decoded_sent\n",
    "  \n",
    "    def count_parameters(self):\n",
    "        \"\"\"Counts the number of parameters of the model.\n",
    "        \"\"\"\n",
    "        return sum(p.numel() for p in self.parameters() if p.requires_grad)\n",
    "\n",
    "    @staticmethod\n",
    "    def load(model_path: str):\n",
    "        \"\"\"Loads the model from a file.\n",
    "\n",
    "        model_path: the file's path to load the model from\n",
    "        \"\"\"\n",
    "        print('loading from [%s]' % model_path)\n",
    "        params = torch.load(model_path, map_location=lambda storage, loc: storage)\n",
    "        args = params['args']\n",
    "        model = AttnSeq2Seq(**args)\n",
    "        model.load_state_dict(params['state_dict'])\n",
    "        return model\n",
    "\n",
    "    def save(self, path: str):\n",
    "        \"\"\"Saves the model's parameters to a file.\n",
    "\n",
    "        path: path for the model to be saved\n",
    "        \"\"\"\n",
    "        print(\"save model's parameters to [%s]\" % path)\n",
    "\n",
    "        params = {\n",
    "            'args': dict(hidden_size=self.hidden_size,\n",
    "                         embed_size=self.embed_size,  \n",
    "                         src_vocab_size=self.src_vocab_size,\n",
    "                         tgt_vocab_size=self.tgt_vocab_size,\n",
    "                         dropout_rate=self.dropout_rate),\n",
    "            'state_dict': self.state_dict()\n",
    "        }\n",
    "\n",
    "        torch.save(params, path)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "1BST0pn40Nms",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### 3.4.1 Initialization (6 pts)\n",
    "Implement the `__init__()` methods of the AttnSeq2Seq, Encoder, and AttnDecoder.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "Hn5-CJxw0jyj",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### 3.4.2 Enocder (5 pts)\n",
    "Implement `forward()` method of the Encoder class. Note that you also have to implement calling this method from the AttnSeq2Seq model too. Let's test your imeplementation of Encoder with the following simple test:\n",
    "\n",
    "**NOTE: If you do not pass the test, please double check the order of instantiations in the constructor of Encoder class!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1392,
     "status": "ok",
     "timestamp": 1588250526664,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "m7y_uIJA2a6A",
    "new_sheet": false,
    "outputId": "32614b01-e577-42eb-8121-4866494a657c",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "passed!\n"
     ]
    }
   ],
   "source": [
    "def test_encoder():\n",
    "    set_seed(40719)\n",
    "    test_encoder = Encoder(hidden_size=3, embed_size=5, src_vocab_size=10)\n",
    "    test_encoder.to(DEVICE)\n",
    "    test_input = torch.tensor([[9, 5, 6, 8, 2, 6],\n",
    "                               [6, 5, 5, 2, 9, 0],\n",
    "                               [6, 2, 3, 0, 0, 0],\n",
    "                               [2, 3, 0, 0, 0, 0]]).t().to(DEVICE)\n",
    "\n",
    "    test_lengths = [6, 5, 3, 2]\n",
    "\n",
    "    with torch.no_grad():\n",
    "        test_outputs = test_encoder(test_input, test_lengths)\n",
    "\n",
    "    enc_hiddens, last_hidden, enc_hiddens_mask = list(map(lambda x: x.cpu().numpy(), test_outputs))\n",
    "    assert last_hidden.shape == (1, 4, 3)\n",
    "    assert enc_hiddens.shape == (4, 6, 3)\n",
    "\n",
    "\n",
    "    assert np.linalg.norm(last_hidden - [[[-0.4525945,   0.54975474, -0.09157258],\n",
    "                                          [ 0.02271755,  0.39946806, -0.70311207],\n",
    "                                          [-0.59517485,  0.34378132, -0.02917046],\n",
    "                                          [-0.52983993,  0.38805154, -0.10685067]]]) < 1e-4\n",
    "\n",
    "\n",
    "    assert np.linalg.norm(enc_hiddens_mask - [[0., 0., 0., 0., 0., 0.],\n",
    "                                              [0., 0., 0., 0., 0., 1.],\n",
    "                                              [0., 0., 0., 1., 1., 1.],\n",
    "                                              [0., 0., 1., 1., 1., 1.]]) < 1e-8\n",
    "\n",
    "    print('passed!') \n",
    "\n",
    "\n",
    "test_encoder()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "1G2N4PON-qFy",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### 3.4.3 Decoder (10 pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "Hcn9R61Xk235",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Implement `forward()` method of the AttnDecoder class. Note that you also have to implement calling this method from the AttnSeq2Seq model too. Let's test your imeplementation of AttnDecoder with the following simple test:\n",
    "\n",
    "**NOTE: If you do not pass the test, please double check the order of instantiations in the constructor of AttnDecoder class!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1196,
     "status": "ok",
     "timestamp": 1588250530512,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "uxKJRDjJ-sAy",
    "new_sheet": false,
    "outputId": "66c2dd16-3cd4-4a00-d58b-8e9a8036f387",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "passed!\n"
     ]
    }
   ],
   "source": [
    "def test_decoder():\n",
    "    set_seed(40719)\n",
    "    test_decoder = AttnDecoder(hidden_size=3, embed_size=5, tgt_vocab_size=5)\n",
    "    test_decoder.to(DEVICE)\n",
    "\n",
    "    test_enc_hiddens = torch.randn(2, 6, 3).to(DEVICE)\n",
    "    test_dec_init_state = torch.randn(2, 3).to(DEVICE)\n",
    "    test_enc_hiddens_mask = torch.tensor([[0., 0., 0., 0., 0., 0.],\n",
    "                                          [0., 0., 1., 1., 1., 1.]]).to(DEVICE)\n",
    "  \n",
    "    test_tgt = torch.tensor([[1, 3, 4, 4, 2],\n",
    "                             [1, 3, 4, 2, 0]]).t().to(DEVICE)\n",
    "    with torch.no_grad():\n",
    "        logits = test_decoder(test_enc_hiddens, test_enc_hiddens_mask, \n",
    "                              test_dec_init_state, test_tgt)\n",
    "    \n",
    "    assert np.linalg.norm(logits.cpu().numpy() - [[[-0.06004265,  0.25477493,  0.3830248,   0.00774992,  0.17179054],\n",
    "                                                   [-0.18582398,  0.13055463,  0.08564769, -0.25669473,  0.12312719]],\n",
    "                                                  [[ 0.05147437, -0.04828914,  0.02907404,  0.04513214,  0.01152593],\n",
    "                                                   [ 0.0291394,  -0.06567273, -0.02651883,  0.00318674, -0.00192575]],\n",
    "                                                  [[ 0.15429737, -0.02719174,  0.00957732,  0.2634642,  -0.09186685],\n",
    "                                                   [ 0.06497972, -0.03984828,  0.011322,    0.08325085, -0.01648209]],\n",
    "                                                  [[ 0.15074214, -0.11919288, -0.13798426,  0.21574639, -0.14181782],\n",
    "                                                   [-0.07257012, -0.01131682, -0.04402388, -0.13441817,  0.02880232]]]) < 1e-4\n",
    "    print('passed!')\n",
    "\n",
    "test_decoder()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "FtUhXa9XtNJL",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### 3.4.4 AttnSeq2Seq (5 pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "iKyNIegcttlY",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's test your implementation of the `forward()` method of AttnSeq2Seq class:\n",
    "\n",
    "**NOTE: If you do not pass the test, please double check the order of instantiations in the constructors of Encoder, AttnDecoder, AttnSeq2Seq classes!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1411,
     "status": "ok",
     "timestamp": 1588250535773,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "LRTZX9EqtgwK",
    "new_sheet": false,
    "outputId": "6df07858-e1d9-409f-dd2d-fbbff1a45cec",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "passed!\n"
     ]
    }
   ],
   "source": [
    "def test_attn_seq2seq():\n",
    "    set_seed(40719)\n",
    "\n",
    "    test_attn_seq2seq = AttnSeq2Seq(hidden_size=3, embed_size=5, \n",
    "                                  src_vocab_size=src_vocab.size, tgt_vocab_size=5)\n",
    "    test_attn_seq2seq.to(DEVICE)\n",
    "\n",
    "    test_src = torch.tensor([[11, 5, 6, 100, 2, 14],\n",
    "                             [2, 3, 0, 0, 0, 0]]).t().to(DEVICE)\n",
    "    test_src_lengths = [6, 2]\n",
    "    test_tgt = torch.tensor([[1, 3, 4, 4, 2],\n",
    "                             [1, 3, 4, 2, 0]]).t().to(DEVICE)\n",
    "\n",
    "    with torch.no_grad():\n",
    "        scores = test_attn_seq2seq(test_src, test_tgt, test_src_lengths)\n",
    "  \n",
    "    assert np.linalg.norm(scores.cpu().numpy() - [-6.347041, -4.7729683]) < 1e-4\n",
    "  \n",
    "    print('passed!')\n",
    "\n",
    "test_attn_seq2seq()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "PTtLqWyDqmoF",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 3.5 Train (9 pts + 5 Bonus pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "3z9-sONeqpGR",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now let's implement the following function to train your network.\n",
    "\n",
    "**Note that**: \n",
    "1. you will first call the train function in the debug mode (by setting ```debug = True``` in the args) and sanity-check your network on the 100 first samples in the corpus. In this case, set the maximum number of epochs to 200. After running for this amount of epochs, you have to witness a training error very close to 0.\n",
    "\n",
    "2. take the mean of the scores returned by the network across a batch. Use the negative value of this mean as the loss of a batch. You can take a running training loss which is the mean of the losses of batches fed to the model every `log_every` iterations and print it. Note that this running loss should converges to zero during sanity check.\n",
    "\n",
    "3. When ```debug = True``` there is no need for evaluation.\n",
    "\n",
    "4. You have to report running training loss every `log_every` iteration.\n",
    "\n",
    "5. You have to evaluate your network every `val_niter` iterations (only when ```debug = False```). For evaluation, simply report the mean value of loss across all of the validation samples. DO NOT forget to set the model to eval mode!\n",
    "\n",
    "6. As well as printing train and validation losses, save them throughout training in two lists and plot them after training is finished!\n",
    "\n",
    "7. Using techniques such as gradient clipping, early stopping, and learning rate decay (use it wisely!) will have bonus points. \n",
    "\n",
    "8. Don't forget to set `dropout_rate` to zero for sanity checking.\n",
    "\n",
    "9. After doing sanity check for 200 epochs, save your model in a file (you will load it later for greedy decoding).\n",
    "\n",
    "\n",
    "After doing sanity check and making sure that your network works fine, you can train your model on the entire corpus. In that case, you have to report the loss on the validation set too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {},
    "colab_type": "code",
    "id": "06eUwPZ5nlE7",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "def train(args: Dict):\n",
    "    \"\"\"Trains the AttnSeq2Seq model.\n",
    "    args: arguments needed for training the model\n",
    "    \"\"\"\n",
    "    set_seed(40719)\n",
    "\n",
    "    batch_size = int(args['batch_size']) \n",
    "    log_every = int(args['log_every'])\n",
    "    debug = bool(args['debug'])\n",
    "\n",
    "    src_sents, tgt_sents = read_corpus(DATA_PATH)\n",
    "\n",
    "    if debug:\n",
    "        print('DEBUG MODE!')\n",
    "        # Use a tiny subset of sentences for sanity check\n",
    "        src_sents, tgt_sents = src_sents[:100], tgt_sents[:100]\n",
    "\n",
    "        train_dataset = NMTData(src_sents=src_sents, tgt_sents=tgt_sents, \n",
    "                                transform=ToTensor(src_vocab, tgt_vocab))\n",
    "\n",
    "        train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, \n",
    "                                      num_workers=0, collate_fn=PadCollate())\n",
    "    else:\n",
    "        print('TRAINING ON ENTIRE CORPUS!')\n",
    "        val_ratio = 0.01\n",
    "        val_size = int(len(src_sents) * val_ratio)\n",
    "        train_size = len(src_sents) - val_size\n",
    "\n",
    "        # Splitting the dataset to validation and training set\n",
    "        train_dataset, val_dataset = torch.utils.data.random_split(dataset, [train_size, val_size])\n",
    "\n",
    "        train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, \n",
    "                                      num_workers=0, collate_fn=PadCollate())\n",
    "        val_dataloader = DataLoader(val_dataset, batch_size=batch_size, shuffle=True, \n",
    "                                    num_workers=0, collate_fn=PadCollate())\n",
    "\n",
    "        val_niter = int(args['val_niter'])\n",
    "\n",
    "\n",
    "\n",
    "    model = AttnSeq2Seq(embed_size=int(args['embed_size']),\n",
    "                        hidden_size=int(args['hidden_size']),\n",
    "                        dropout_rate=float(args['dropout_rate']),\n",
    "                        src_vocab_size=src_vocab.size,\n",
    "                        tgt_vocab_size=tgt_vocab.size)\n",
    "  \n",
    "    model = model.to(DEVICE)\n",
    "    optimizer = torch.optim.Adam(model.parameters(), lr=float(args['lr']))\n",
    "\n",
    "    model.init_weights(0.1)\n",
    "    model.train()\n",
    "\n",
    "    print('Begin training')\n",
    "    ################################################################################\n",
    "    # TODO: Implement the train loop.                                              #\n",
    "    ################################################################################\n",
    "    train_loss_over_epochs=[]\n",
    "    val_loss_over_epochs=[]\n",
    "    for epoch in range(int(args['max_epoch'])):\n",
    "      running_loss=0.\n",
    "      for step,(src_batch,tgt_batch,src_lengths) in enumerate(train_dataloader):\n",
    "        scores=model(src_batch,tgt_batch,src_lengths)\n",
    "        loss=-torch.mean(scores)\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        running_loss=(running_loss*step+loss.item())/float(step+1)\n",
    "      train_loss_over_epochs+=[running_loss]\n",
    "      if (epoch+1)%log_every==0:\n",
    "        print('Epoch {}: Train Loss: {}'.format(epoch+1,running_loss))\n",
    "      if not debug and (epoch+1)%val_niter==0:\n",
    "        model.eval()\n",
    "        total_count=0\n",
    "        val_total_loss=0.\n",
    "        for src_batch,tgt_batch,src_lengths in val_dataloader:\n",
    "          scores=torch.sum(model(src_batch,tgt_batch,src_lengths))\n",
    "          val_total_loss+=torch.sum(scores).item()\n",
    "          totarunning_lossl_count+=scores.size(0)\n",
    "        loss=-val_total_loss/float(total_count)\n",
    "        val_loss_over_epochs+=[loss]\n",
    "        print('Evaluation : Validation loss: {}'.format(loss))\n",
    "        model.train()\n",
    "    torch.save(model.state_dict(),'./Seq2Seq')\n",
    "    plt.plot(train_loss_over_epochs,'-g', label='train')\n",
    "    plt.plot(val_loss_over_epochs,'-r', label='validation')\n",
    "    plt.xlabel('epoch')\n",
    "    plt.ylabel('loss')\n",
    "    plt.legend(loc='upper right')\n",
    "    plt.show()\n",
    "    ################################################################################\n",
    "    #                                 END OF YOUR CODE                             #\n",
    "    ################################################################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "hAN2wrtKsfaP",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now let's call the train function and train your network on a tiny subset of the data (100 first samples) with the following cell. As we want the network to overfit on a tiny dataset, there is no need for validation set and evaluation.\n",
    "\n",
    "The hyperparameters specified for you below would probably give good results. But Feel free to change them if they do not work for you. (Do not change `debug` and `dropout_rate` hyperparams! Note that we turn off any kind of regularization during sanity check.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 365
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 22394,
     "status": "ok",
     "timestamp": 1588250570210,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "xVtih06UrKmT",
    "new_sheet": false,
    "outputId": "685e1245-8f6a-4d3a-f089-14653b4a6f18",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading the corpus...\n",
      "DEBUG MODE!\n",
      "Begin training\n",
      "Epoch 100: Train Loss: 0.15177708864212036\n",
      "Epoch 200: Train Loss: 0.055169448256492615\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEHCAYAAACp9y31AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXxV9Z3/8dcnNzd7gAARkC0oLggqYEQq4lJqB5lad9Ff23ErVH+1lYe2P63a0XFqf3a0yq8zjlZHWzu1KkIVddwRt9YNFBFERTTsu4Yl+/L5/XFPYgIJBsi5J9z7fj4eeeTcc8/yycnN+37zved8j7k7IiKSPjKiLkBERJJLwS8ikmYU/CIiaUbBLyKSZhT8IiJpRsEvIpJmMsPasJnlAK8C2cF+Zrr7DWb2R+AEYEuw6IXuvmBX2+rdu7eXlJSEVaqISEqaP3/+Jncv3nF+aMEP1ADfdPftZhYHXjezZ4Lnfu7uMzu6oZKSEubNmxdKkSIiqcrMlrc1P7Tg98SVYduDh/HgS1eLiYhELNQ+fjOLmdkCYAPwgru/FTx1s5ktNLM7zCw7zBpERKS1UIPf3RvcfSQwABhjZiOAXwCHAkcDPYGr21rXzKaa2Twzm7dx48YwyxQRSSth9vE3c/dyM5sLTHT324LZNWb2B+Bn7axzD3APQGlpqbqIRFJEXV0dq1atorq6OupSUkZOTg4DBgwgHo93aPkwz+opBuqC0M8FTgZ+Y2b93H2tmRlwOrAorBpEpOtZtWoVhYWFlJSUkIgB2RvuzubNm1m1ahVDhgzp0Dphtvj7AQ+YWYxEl9IMd3/KzF4K3hQMWABcGmINItLFVFdXK/Q7kZnRq1cvdqdLPMyzehYCo9qY/82w9iki+waFfufa3eOZ0lfuPvXJU/zm9d9EXYaISJeS0sH/3KfPcevfb426DBHpYsrLy/nP//zP3V5v0qRJlJeXh1BRcqV08GdnZlNdrzMHRKS19oK/vr5+l+s9/fTT9OjRI6yykiYpp3NGJSczh5qGmqjLEJEu5pprrmHZsmWMHDmSeDxOTk4ORUVFfPTRR3zyySecfvrprFy5kurqaq644gqmTp0KfDV8zPbt2znllFM47rjj+Pvf/07//v2ZPXs2ubm5Ef9kHZPSwZ8dy6a+sZ6GxgZiGbGoyxGRHUx7dhoL1u1yjMbdNrLvSKZPnL7LZW655RYWLVrEggULePnll/nHf/xHFi1a1Hw65P3330/Pnj2pqqri6KOP5qyzzqJXr16ttrF06VIeeugh7r33Xs4991xmzZrF97///U79WcKS8l09gFr9IrJLY8aMaXUO/O9+9zuOPPJIxo4dy8qVK1m6dOlO6wwZMoSRI0cCcNRRR1FWVpascvdaSrf4czJzAKiuryYvnhdxNSKyo69rmSdLfn5+8/TLL7/Miy++yBtvvEFeXh4nnnhim1cZZ2d/NcxYLBajqqoqKbV2htRu8ceCFn+9Wvwi8pXCwkK2bdvW5nNbtmyhqKiIvLw8PvroI958880kVxe+tGnxi4g06dWrF+PGjWPEiBHk5ubSp0+f5ucmTpzI3XffzbBhwzjkkEMYO3ZshJWGI6WDX338ItKev/zlL23Oz87O5plnnmnzuaZ+/N69e7No0VfDjP3sZ22ONdllpXRXj1r8IiI7S+ngVx+/iMjOUjr41eIXEdlZSge/+vhFRHaW0sGvFr+IyM5SOvjVxy8isrOUDn61+EWkMxQUFACwZs0azj777DaXOfHEE5k3b94utzN9+nQqKyubH0c1zHNKB7/6+EWkM+2///7MnDlzj9ffMfijGuY5pYO/qcWvrh4Raemaa67hzjvvbH5844038qtf/YoJEyYwevRoDj/8cGbPnr3TemVlZYwYMQKAqqoqzjvvPIYNG8YZZ5zRaqyeyy67jNLSUoYPH84NN9wAJAZ+W7NmDSeddBInnXQSkBjmedOmTQDcfvvtjBgxghEjRjB9+vTm/Q0bNowpU6YwfPhwvv3tb3fKmECpfeVu0Mevrh6RLmraNFjQucMyM3IkTN/14G+TJ09m2rRp/PjHPwZgxowZPPfcc/z0pz+lW7dubNq0ibFjx/Ld73633fvZ3nXXXeTl5bFkyRIWLlzI6NGjm5+7+eab6dmzJw0NDUyYMIGFCxfy05/+lNtvv525c+fSu3fvVtuaP38+f/jDH3jrrbdwd4455hhOOOEEioqKQhn+ObQWv5nlmNnbZva+mS02s38J5g8xs7fM7FMze8TMssKqobnFr64eEWlh1KhRbNiwgTVr1vD+++9TVFRE3759ufbaazniiCP41re+xerVq1m/fn2723j11VebA/iII47giCOOaH5uxowZjB49mlGjRrF48WI+/PDDXdbz+uuvc8YZZ5Cfn09BQQFnnnkmr732GhDO8M9htvhrgG+6+3YziwOvm9kzwJXAHe7+sJndDVwC3BVGAZkZmRimFr9IV/U1LfMwnXPOOcycOZN169YxefJkHnzwQTZu3Mj8+fOJx+OUlJS0ORzz1/n888+57bbbeOeddygqKuLCCy/co+00CWP459Ba/J6wPXgYD74c+CbQ9OnIA8DpYdVgZmRnZquPX0R2MnnyZB5++GFmzpzJOeecw5YtW9hvv/2Ix+PMnTuX5cuX73L9448/vnmgt0WLFrFw4UIAtm7dSn5+Pt27d2f9+vWtBnxrbzjo8ePH8/jjj1NZWUlFRQWPPfYY48eP78SftrVQ+/jNLAbMB4YCdwLLgHJ3b7qj8Sqgf5g15GTmqMUvIjsZPnw427Zto3///vTr14/vfe97nHrqqRx++OGUlpZy6KGH7nL9yy67jIsuuohhw4YxbNgwjjrqKACOPPJIRo0axaGHHsrAgQMZN25c8zpTp05l4sSJ7L///sydO7d5/ujRo7nwwgsZM2YMAD/84Q8ZNWpUaHf1MncPZcOtdmLWA3gM+CXwR3cfGswfCDzj7iPaWGcqMBVg0KBBR33du297+t7Wl9MPPZ27v3P3npYvIp1oyZIlDBs2LOoyUk5bx9XM5rt76Y7LJuV0TncvB+YC3wB6mFnTfxoDgNXtrHOPu5e6e2lxcfEe71stfhGR1sI8q6c4aOljZrnAycASEm8ATZe+XQDsfLJsJ8rOzNZZPSIiLYTZx98PeCDo588AZrj7U2b2IfCwmf0KeA+4L8Qa1OIX6YLcvd3z42X37W6XfWjB7+4LgVFtzP8MGBPWfneUHdNZPSJdSU5ODps3b6ZXr14K/07g7mzevJmcnJwOr5PSV+6CWvwiXc2AAQNYtWoVGzdujLqUlJGTk8OAAQM6vHzKB392ZjaVdZVfv6CIJEU8HmfIkCFRl5HWUnqQNlCLX0RkRykf/OrjFxFpLeWDXy1+EZHWUj74s2M6j19EpKWUD361+EVEWkv54NfonCIiraV88Odk5qirR0SkhZQP/uxYNrUNtTR6Y9SliIh0Cakf/JmJu9fUNtRGXImISNeQ8sHfdN9dfcArIpKQ8sGfHUu0+PUBr4hIQsoHv1r8IiKtpXzwN/Xx68weEZGElA9+tfhFRFpL+eBXH7+ISGspH/xq8YuItJbywa8+fhGR1lI++NXiFxFpLeWDX338IiKthRb8ZjbQzOaa2YdmttjMrgjm32hmq81sQfA1KawaQC1+EZEdhXmz9XrgKnd/18wKgflm9kLw3B3ufluI+26mPn4RkdZCC353XwusDaa3mdkSoH9Y+2uPWvwiIq0lpY/fzEqAUcBbwazLzWyhmd1vZkXtrDPVzOaZ2byNGzfu8b6b+vgV/CIiCaEHv5kVALOAae6+FbgLOBAYSeI/gt+2tZ673+Pupe5eWlxcvMf7z4vnAVBZV7nH2xARSSWhBr+ZxUmE/oPu/lcAd1/v7g3u3gjcC4wJs4Z4LE5WLIuK2oowdyMiss8I86weA+4Dlrj77S3m92ux2BnAorBqaJIfz2d77fawdyMisk8I86yeccAPgA/MbEEw71rgfDMbCThQBvwoxBoAKMgqoKJOLX4REQj3rJ7XAWvjqafD2md78rPyFfwiIoGUv3IX1NUjItJSWgR/QVaBPtwVEQmkRfCrq0dE5CvpEfzq6hERaZYewZ+Vr64eEZFAWgR/QVync4qINEmL4M/PUlePiEiT9Aj+eD61DbXUN9ZHXYqISOTSIvgLsgoA1M8vIkKaBH9+Vj6AuntEREiX4I8ngl8f8IqIpEnwq6tHROQraRH86uoREflKegS/unpERJqlR/AHLX519YiIpEnwN/Xxq6tHRCRNgl9dPSIiX0mP4FdXj4hIs7QI/rx4HqCuHhERSJPgz7AM8uJ56uoRESHE4DezgWY218w+NLPFZnZFML+nmb1gZkuD70Vh1dBSflxj8ouIQLgt/nrgKnc/DBgL/NjMDgOuAea4+0HAnOBx6AqyCthep64eEZHQgt/d17r7u8H0NmAJ0B84DXggWOwB4PSwamhJd+ESEUlISh+/mZUAo4C3gD7uvjZ4ah3Qp511pprZPDObt3Hjxr2uQffdFRFJCD34zawAmAVMc/etLZ9zdwe8rfXc/R53L3X30uLi4r2uoyBLt18UEYGQg9/M4iRC/0F3/2swe72Z9Que7wdsCLOGJurqERFJCPOsHgPuA5a4++0tnnoCuCCYvgCYHVYNLamrR0QkITPEbY8DfgB8YGYLgnnXArcAM8zsEmA5cG6INTTLj+erq0dEhBCD391fB6ydpyeEtd/2FGQVqKtHRIQ0uXIXgvP4a7eT+DxZRCR9pU3wF2YX4ri6e0Qk7aVN8HfL7gbAtpptEVciIhKttAn+wqxCALbVKvhFJL2lT/BnJ4J/a83Wr1lSRCS1dSj4zewKM+tmCfeZ2btm9u2wi+tM6uoREUnoaIv/4mC4hW8DRSTOz78ltKpC0NTVoxa/iKS7jgZ/0/n4k4D/dvfFtH+OfpfU3OJXH7+IpLmOBv98M3ueRPA/Z2aFQGN4ZXU+9fGLiCR09MrdS4CRwGfuXmlmPYGLwiur86mPX0QkoaMt/m8AH7t7uZl9H7ge2BJeWZ0vNzOXDMtQV4+IpL2OBv9dQKWZHQlcBSwD/hRaVSEwMwqzCtXVIyJpr6PBXx/cNOU04D/c/U6gMLyywtEtu5ta/CKS9jrax7/NzH5B4jTO8WaWAcTDKyschdlq8YuIdLTFPxmoIXE+/zpgAHBraFWFpFt2N324KyJpr0PBH4T9g0B3M/sOUO3u+1QfP6A+fhEROj5kw7nA28A5JO6Y9ZaZnR1mYWFQH7+ISMf7+K8Djnb3DQBmVgy8CMwMq7AwFGYXqqtHRNJeR/v4M5pCP7B5N9btMtTVIyLS8Rb/s2b2HPBQ8Hgy8HQ4JYWnqavH3THbp4YaEhHpNB39cPfnwD3AEcHXPe5+9a7WMbP7zWyDmS1qMe9GM1ttZguCr0l7U/zuKswqpNEbqayrTOZuRUS6lI62+HH3WcCs3dj2H4H/YOcrfO9w99t2YzudpuUInflZ+VGUICISuV0Gv5ltA7ytpwB3927trevur5pZyV5V18lajtDZt6BvxNWIiERjl8Hv7mEMy3C5mf0TMA+4yt2/DGEfbdIInSIiyT8z5y7gQBJDPK8FftvegmY21czmmdm8jRs3dsrOdcN1EZEkB7+7r3f3BndvBO4Fxuxi2XvcvdTdS4uLiztl/00tfp3SKSLpLKnBb2b9Wjw8A1jU3rJhaOrjV1ePiKSzDp/Vs7vM7CHgRKC3ma0CbgBONLORJD4wLgN+FNb+26IbrouIhBj87n5+G7PvC2t/HaEbrouI7IPDLuyNvHge8Yw4mys3R12KiEhk0ir4zYxB3QexYuuKqEsREYlMWgU/wOAegykrL4u6DBGRyKRd8Jd0L1Hwi0haS7/g71HCuu3rqK6vjroUEZFIpGXwA6zYon5+EUlPaRf8g3sMBlB3j4ikrbQL/qYWv4JfRNJV2gX//oX7k5mRyfLy5VGXIiISibQL/syMTAZ2G0jZlrKoSxERiUTaBT/oXH4RSW9pGfwlPXQuv4ikr/QM/u4lrNm2Rufyi0haSsvgP7DngYDO7BGR9JSWwT+051AAPv3i04grERFJPgW/iEiaScvg75Xbi+7Z3RX8IpKW0jL4zYyhPYey7MtlUZciIpJ0aRn8kOjuUYtfRNJR2gb/gUUHUlZeRl1DXdSliIgkVWjBb2b3m9kGM1vUYl5PM3vBzJYG34vC2v/XGdpzKPWN9RqeWUTSTpgt/j8CE3eYdw0wx90PAuYEjyOhM3tEJF2FFvzu/irwxQ6zTwMeCKYfAE4Pa/9fR8EvIukq2X38fdx9bTC9DuiT5P0361vQl4KsAj7Y8EFUJYiIRCKyD3fd3QFv73kzm2pm88xs3saNGzt9/2bGxKETeeyjx6hvrO/07YuIdFXJDv71ZtYPIPi+ob0F3f0edy9199Li4uJQijlv+HlsqNjAy2Uvh7J9EZGuKNnB/wRwQTB9ATA7yftvZdJBkyjMKuShDx6KsgwRkaQK83TOh4A3gEPMbJWZXQLcApxsZkuBbwWPI5Mbz+WMYWcwa8ksKusqoyxFRCRpwjyr53x37+fucXcf4O73uftmd5/g7ge5+7fcfcezfpJuyugpbKnZwm///tuoSxERSYq0vXK3yXGDjuOsYWdxy99uYfXW1VGXIyISurQPfoBbT76VhsYGrnz+yqhLEREJnYIfGFI0hOuPv54Zi2fw1yV/jbocEZFQKfgDV4+7mlF9R3HZ/1ymLh8RSWkK/kA8FueB0x+gqq6Kk//7ZDZVboq6JBGRUCj4Wzi8z+E8ef6TfF7+OUffezRPffJU1CWJiHQ6Bf8OTig5gTn/NIe8eB6nPnQqcz+fG3VJIiKdSsHfhmMHHsu8KfPoW9CXW/4W6TVmIiKdTsHfjtx4LlcccwXPL3ue99a+F3U5IiKdRsG/C5eWXkphViH/8sq/kBhMVERk36fg34UeOT24bvx1zP54Nve/d3/U5YiIdIrMqAvo6n4+7ue88NkL/OSZn7ClZgsXjbyIotzIbhUsIrLX1OL/GhmWwZ/P/DNj+o/hquevos9tffjOX77Dmm1roi5NRGSPKPg7oG9BX16+8GXe+9F7TBs7jTmfz+HaOddGXZaIyB5RV89uGNl3JCP7jqShsYHpb03nF8f9gkN6HxJ1WSIiu0Ut/j1w9XFXk5uZyw0v3xB1KSIiu03Bvwf2y9+PaWOn8cjiR3h79dtRlyMislsU/Hvo6nFXs1/+flz53JU6x19E9ikK/j1UmF3Ir076FX9b+TfOeOQMXil7BQB3p9EbI65ORKR9+nB3L1w86mLKysv4/fzfM/vj2Zx2yGl8+sWnbKvdxrKfLiMzQ4dXRLqeSFr8ZlZmZh+Y2QIzmxdFDZ0hlhHj5gk3s+rKVdx04k08++mzbKrcxIotK1i4fmHU5YmItCnKrp6T3H2ku5dGWEOnyMnM4Zcn/JKKayt4Z8o7ALy2/LWIqxIRaZv6+DtRLCPGwO4DGdx9MK+tUPCLSNcUVfA78LyZzTezqRHVEJrxg8fz2orXuP+9+ym9p5Ty6vKoSxIRaRZV8B/n7qOBU4Afm9nxOy5gZlPNbJ6Zzdu4cWPyK9wLxw86ng0VG7j0qUuZv3Y+//a3f4u6JBGRZpEEv7uvDr5vAB4DxrSxzD3uXurupcXFxckuca+MHzweSNzMZeLQiUx/c7oGdRORLiPpwW9m+WZW2DQNfBtYlOw6wnRIr0M4+7Czue+793HnpDupb6znkicuobahNurSREQiafH3AV43s/eBt4H/cfdnI6gjNGbGo+c8ytmHnc0BRQdw56Q7efbTZzl/1vlU1lVGXZ6IpLmkX2Hk7p8BRyZ7v1GactQUKusqmfbcNBZvWMyMc2ZwRJ8joi5LRNKUTudMkivGXsGLP3iRrTVbOeORM9TyF5HIKPiTaMIBE/jLWX/hsy8/Y+qTUznzkTP511f+NeqyRCTNaDCZJDux5ESmjJ7Cve/eS1Ysi8c+eowhRUP4/hHfj7o0EUkTavFHYPrE6Txy9iOsu2odJww+gSlPTuH6l65nY8W+db2CiOybFPwRyIvnce7wcynKLeLRcx5l0kGT+PVrv2bovw/ljjfuoK6hLuoSRSSFKfgjVpxfzKxzZ7H4fy/m2IHHcuXzV3Lk3Udy+dOXM+7+cSxYtyDqEkUkxSj4u4hhxcN4+n89zZPnP0ltQy33vXcfC9Yt4GfP/yzq0kQkxSj4uxAz4zsHf4elP1nK1mu2cvM3b2bO53OY89mcqEsTkRSi4O+CzIx4LM6lpZcyoNsALn7iYu544w6q6qqiLk1EUoCCvwvLyczhwTMfpG9BX658/komz5xMQ2OD7ukrIntFwd/FHT/4eN764Vv8+yn/zpOfPMmo348i7+Y8bn71ZtydmR/OZOWWlVGXKSL7EF3AtY+4fMzlrNm2hllLZjG632iun3s9c8vmMufzOQztOZQ3LnmD3nm9oy5TRPYBavHvQ3494dd8fPnHvHTBSxy9/9HM+XwOU0ZPYeWWlZzy4Ck8tuQxDf0sIl9LLf59UE5mDs//4Hk+2fwJY/qPYeLQifzoqR9x5owzGd1vNA+f9TCbqzZzUM+D6JXXK+pyRaSLMXePuoavVVpa6vPmzYu6jC6tvrGeWR/OYsqTU9hWuw2AvgV9eeK8Jzi6/9ERVyciUTCz+e5euuN8tfhTRGZGJpNHTObIvkfy6OJHKelRwj+//M+Mu38cJx94MuMHjad7dndeKnuJ/oX9uemkm+iW3S3qskUkAmrxp7ANFRv4zeu/YfbHs1n25TIA+hX0Y33FegZ2G8gDpz/ACSUnRFyliISlvRa/gj9NbKvZxqbKTZT0KOGNVW9wweMXsOyLZZw57ExG9h3JucPP5eBeB0ddpoh0IgW/tFJRW8F1L13H4x89zootK3Cck0pO4tSDT2Vwj8Hkx/Pplt2N0f1Gk52Z3Wrd6vpq4hlxYhmxiKoXkY5Q8Eu71m9fzz3z7+GhRQ+xZNOSVs8N6j6Iq75xFccPPp7l5cuZtWQWMxbPoF9hP6445gouLb2UnMyciCoXkV3pUsFvZhOB/wfEgP9y91t2tbyCP3lWb13NpspNVNRVsGbbGm79+628vfrt5ucLswo5f8T5LNm0hNdWvMag7oOYNHQS2+u28/qK1+lb0JefjPkJBxQdwIotK1iycQnnDD+Hw4oP22lfDY0N1DXW6Y1DJCRdJvjNLAZ8ApwMrALeAc539w/bW0fBHx13Z+kXS5m3Zh4Dug3gmP7HNHf9zP18LtfPvZ5PNn9CPCPO2AFjWbh+YfMHyU0yLINvDPgG3XO6kxfPIy+eR6M38sKyF/iy+ku+OeSbVNdX09DYwM+P/Tn1jfWUlZdxzvBzGNBtwC7rK68up9Eb6ZnbM7RjILKv6krB/w3gRnf/h+DxLwDc/f+2t46Cf9/R0NjA26vfpry6nN55vRnYfSDT35zOm6vepKKugoraCirqKqhvrGfcwHH0K+jHs8uepSiniA0VG/i8/PPmbcUsRv9u/cnJzCE3M5fceC65mbnkZOaQk5nDttptvFz2Mg2NDYzqN4phvYdRWVfJog2LGNxjMCXdS2jwBrJiWeTF88jNzG1+48mL5zVvL5YRIzMjk8yMTGKWmG6at+PjuoY6Fq5fyMbKjc11tKwpN/7VdHYsm3gs3rzteMZX05kZmcRjcWobavl408dkZmQytOdQKuoqmLF4Bo9/9DhnDjuTi0ddTFYsiwzLIMMyMAwz65TfVVVdFSu3rqR/YX/ys/KprKvk6aVPs7VmK6cdclq7F/99UfUFW6q3MKj7IGIZMRq9keXly8nPyme//P2+dr/uzvIty6moreDQ3oc2f1ZU21BLZV0lFbUVVNVX0begL9mxbOavnc/7696nvLqcc4efy5CiITR6I+u2ryMrlkWv3F6tjom70+iN1DXWUd9YT019DTUNNeyXvx+ZGXt3BntFbeK12z2nO+XV5ZSVl7F221pG7DeCgd0H7nLdTZWbeH/d+1TWVTKseBhDew7dq1o6oisF/9nARHf/YfD4B8Ax7n55e+so+NNDbUMtj3/0OL3zejOg2wD+vPDPrNiygqr6Kqrrq6mqq6Kqvoqa+hqq66sxMyYeOJH8rHxeWf4Ky75YRlYsi8P7HN78BxnLiDUHSmVd5T4zsmlJjxLKysvafb7lGwEkhvJu+abQ1vSOy26t2YqT+PvPi+dRXV/dfHwyLIPsWDYNnhgNtuX2moYFyY5lk5+VT0VtBTUNNQAU5RQRj8Wb62wrX+oa6yivLm/eb2ZGJpV1ldQ31u+0bE5mDtX11c2PDaNHTg+21W5rXr4gqwDDqG+sbw77tmTFsuid15utNVsxjOzM7OY31aafr+W0ETwOpmsaalhevhzHyY5lN//MTXrk9GhuLGRYBrGMGDGLEcuI4e6tGjUAg7sPJpYRo6a+hrrGusQ6wfItp//r1P9i/ODxbf5MX2efu4DLzKYCUwEGDRoUcTWSDFmxLM4dfm7z45tOuqlTt+/u1DXWUVVX1fxGUFVfRUNjA/WN9TR48D143NY8gOH7DWdAtwHNb0DV9dXNb07Nj+uqqGmoad5OXUNd83TLgMqwDA7udTB1DXV89uVnFGYXMrrfaI7e/2he+vwl3l79Nk6iBdvyy91p8Ibmn8vx5pBta7op4Fsu2zO3JyU9Sli1dRVfVn9JTmYOE4ZMoEdOD2Z/PJuK2ormEAKa99unoA/dsrvxyeZPqKyrJC+ex8G9DmZ77XaWbl6605vrjq1xM2PEfiMoyCrg3bXvAok3gPx4fuJ7Vj7ZsWxWbl3J5srNjBs0jqP3P5oMy+BP7/+J9RXrKcgqaP4drNiyAjNr8z+reCxOPCNOdmY2ZeVlbKjYQPfs7kDiDaymoSbxcwXHZFfTmRmZXDTyIvLieazfvp4+BX0o6VFCcV4x7617j2VfLGt+o2xobKDBG5ofN3ojF4+6mGMHHkt+PJ83V73Jm6vfJDMjk6yMLOKxePN6jbRYv7EhlAst1dUjIpKi2mvxRzE65zvAQWY2xMyygPOAJyKoQ0QkLSW9q8fd683scuA5Eqdz3u/ui5Ndh4hIuoqkj9/dnwaejmLfIiLpTp3xOksAAAaCSURBVDdiERFJMwp+EZE0o+AXEUkzCn4RkTSj4BcRSTP7xLDMZrYRWL6Hq/cGNnViOZ2lq9YFXbc21bV7umpd0HVrS7W6Brt78Y4z94ng3xtmNq+tK9ei1lXrgq5bm+raPV21Lui6taVLXerqERFJMwp+EZE0kw7Bf0/UBbSjq9YFXbc21bV7umpd0HVrS4u6Ur6PX0REWkuHFr+IiLSQ0sFvZhPN7GMz+9TMromwjoFmNtfMPjSzxWZ2RTD/RjNbbWYLgq9JEdRWZmYfBPufF8zraWYvmNnS4HtRkms6pMUxWWBmW81sWlTHy8zuN7MNZraoxbw2j5El/C54zS00s9FJrutWM/so2PdjZtYjmF9iZlUtjt3dSa6r3d+dmf0iOF4fm9k/JLmuR1rUVGZmC4L5yTxe7eVDeK8xd0/JLxJDPi8DDgCygPeBwyKqpR8wOpguJHGz+cOAG4GfRXycyoDeO8z7N+CaYPoa4DcR/x7XAYOjOl7A8cBoYNHXHSNgEvAMYMBY4K0k1/VtIDOY/k2LukpaLhfB8Wrzdxf8HbwPZANDgr/ZWLLq2uH53wL/HMHxai8fQnuNpXKLfwzwqbt/5u61wMPAaVEU4u5r3f3dYHobsAToH0UtHXQa8EAw/QBweoS1TACWufueXsC319z9VeCLHWa3d4xOA/7kCW8CPcysX7Lqcvfn3b3pprNvAgPC2Pfu1rULpwEPu3uNu38OfEribzepdVni/pDnAg+Fse9d2UU+hPYaS+Xg7w+sbPF4FV0gbM2sBBgFvBXMujz4d+3+ZHepBBx43szmW+I+xwB93H1tML0O6BNBXU3Oo/UfY9THq0l7x6grve4uJtEybDLEzN4zs1fMbM/u3r132vrddZXjNR5Y7+5LW8xL+vHaIR9Ce42lcvB3OWZWAMwCprn7VuAu4EBgJLCWxL+ayXacu48GTgF+bGbHt3zSE/9bRnLqlyVuzfld4NFgVlc4XjuJ8hi1x8yuA+qBB4NZa4FB7j4KuBL4i5l1/l2829clf3ctnE/rBkbSj1cb+dCss19jqRz8q4GBLR4PCOZFwsziJH6pD7r7XwHcfb27N7h7I3AvIf2Luyvuvjr4vgF4LKhhfdO/jsH3DcmuK3AK8K67rw9qjPx4tdDeMYr8dWdmFwLfAb4XBAZBV8rmYHo+ib70g5NV0y5+d13heGUCZwKPNM1L9vFqKx8I8TWWysHfZW7qHvQf3gcscffbW8xv2S93BrBox3VDrivfzAqbpkl8MLiIxHG6IFjsAmB2MutqoVUrLOrjtYP2jtETwD8FZ16MBba0+Hc9dGY2Efg/wHfdvbLF/GIziwXTBwAHAZ8lsa72fndPAOeZWbaZDQnqejtZdQW+BXzk7quaZiTzeLWXD4T5GkvGp9ZRfZH49PsTEu/W10VYx3Ek/k1bCCwIviYB/w18EMx/AuiX5LoOIHFGxfvA4qZjBPQC5gBLgReBnhEcs3xgM9C9xbxIjheJN5+1QB2J/tRL2jtGJM60uDN4zX0AlCa5rk9J9P82vc7uDpY9K/gdLwDeBU5Ncl3t/u6A64Lj9TFwSjLrCub/Ebh0h2WTebzay4fQXmO6cldEJM2kclePiIi0QcEvIpJmFPwiImlGwS8ikmYU/CIiaUbBLxIyMzvRzJ6Kug6RJgp+EZE0o+AXCZjZ983s7WD89d+bWczMtpvZHcE46XPMrDhYdqSZvWlfjXvfNFb6UDN70czeN7N3zezAYPMFZjbTEmPlPxhcrSkSCQW/CGBmw4DJwDh3Hwk0AN8jcQXxPHcfDrwC3BCs8ifganc/gsTVk03zHwTudPcjgWNJXCkKiREXp5EYZ/0AYFzoP5RIOzKjLkCki5gAHAW8EzTGc0kMitXIV4N3/Rn4q5l1B3q4+yvB/AeAR4Nxj/q7+2MA7l4NEGzvbQ/GgrHEXZ5KgNfD/7FEdqbgF0kw4AF3/0WrmWa/3GG5PR3jpKbFdAP625MIqatHJGEOcLaZ7QfN9zsdTOJv5Oxgmf8FvO7uW4AvW9yc4wfAK564e9IqMzs92Ea2meUl9acQ6QC1OkQAd//QzK4ncTeyDBIjOP4YqADGBM9tIPE5ACSGyb07CPbPgIuC+T8Afm9mNwXbOCeJP4ZIh2h0TpFdMLPt7l4QdR0inUldPSIiaUYtfhGRNKMWv4hImlHwi4ikGQW/iEiaUfCLiKQZBb+ISJpR8IuIpJn/D5yjSEAdP4PnAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "DESCRIPTION OF ARGUMENTS:\n",
    "\n",
    "log_every: determines how often (based on iterations) training loss should be printed\n",
    "out.  \n",
    "\n",
    "val_niter: hyperparameter to determine how often evaluation on validation set \n",
    "should be done (used only when you want to train on the entire corpus).\n",
    "\n",
    "max_epoch: determines maximum number of epochs you want the model to be trained.\n",
    "When you reach this number of epochs, you should break the training loop.\n",
    "\n",
    "lr: learning rate\n",
    "\n",
    "uniform_init: will indicate the low and high values used for uniform initialization \n",
    "of the weights of the model (e.g. if it is 0.1, then initial weights would be\n",
    "distributed uniformly over (-0.1, +0.1)).\n",
    "\n",
    "debug: set it to True for sanity check and to False for training on entire \n",
    "corpus.\n",
    "\n",
    "This cell should run pretty fast on GPU. Training for 200 epochs on the first 100\n",
    "samples takes roughly 10 seconds for us.\n",
    "\"\"\"\n",
    "\n",
    "args = dict(batch_size=32, embed_size=256, hidden_size=256, dropout_rate=0.0,\n",
    "            uniform_init=0.1, lr=1e-3, log_every=100, max_epoch=200, \n",
    "            val_niter=None, debug=True)\n",
    "\n",
    "train(args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "t_5zo4XqyXcG",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 3.6 Greedy Decode and Evaluation (6 pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "s3jfZbR3yRyp",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Implement `greedy_decode()` method of AttnSeq2Seq. This method will generate the translation of a single sentence in the target langauge using the greedy decoding algorithm.\n",
    "\n",
    "Test your implementation with the following function which runs greedy decoding for the model you trained and saved on the first 100 samples and prints out the sentences for which your output does not match the ground truth output. As your model has overfitted on the first 100 sentences, ALMOST all of your outputs must match the gournd truth outputs and only a couple of sentences should be printed out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 156
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 7296,
     "status": "ok",
     "timestamp": 1588254141069,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "h4dRbMJKyhul",
    "new_sheet": false,
    "outputId": "b22376d4-509f-4ec5-bc98-892d1401b09b",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading the corpus...\n",
      "\n",
      "format of prints: [sample-index] [source-sentence] [your-translation] [ground-truth-translation]\n",
      "\n",
      "22 ['merci', '!'] ['<START>', 'thanks', '.', '<END>'] \t ['<START>', 'cheers', '!', '<END>']\n",
      "78 ['degage', '!'] ['<START>', 'go', 'away', '!', '<END>'] \t ['<START>', 'beat', 'it', '.', '<END>']\n",
      "88 ['allez', '!'] ['<START>', 'come', 'on', '!', '<END>'] \t ['<START>', 'come', 'on', '.', '<END>']\n"
     ]
    }
   ],
   "source": [
    "def evaluate_sanity():\n",
    "    ################################################################################\n",
    "    # TODO: 1) Load a pretrained model                                             #\n",
    "    ################################################################################\n",
    "    model=AttnSeq2Seq(256,256,src_vocab.size,tgt_vocab.size,dropout_rate=0.)\n",
    "    model.load_state_dict(torch.load('./Seq2Seq'))\n",
    "    model.eval()\n",
    "    ################################################################################\n",
    "    #                                 END OF YOUR CODE                             #\n",
    "    ################################################################################\n",
    "    src_sents, tgt_sents = read_corpus(DATA_PATH)\n",
    "    src_sents, tgt_sents = src_sents[:100], tgt_sents[:100]\n",
    "\n",
    "    model.to(DEVICE)\n",
    "    model.eval()\n",
    "    print('\\nformat of prints: [sample-index] [source-sentence] [your-translation] [ground-truth-translation]\\n')\n",
    "    for it, (src_sent, tgt_sent) in enumerate(zip(src_sents, tgt_sents)):\n",
    "        src_sent_tensor = src_vocab.to_tensor(src_sent).unsqueeze(1)\n",
    "        if ['<START>'] + model.greedy_decode(src_sent_tensor, tgt_vocab) != tgt_sent:\n",
    "            print(it+1, src_sent, ['<START>'] + model.greedy_decode(src_sent=src_sent_tensor, tgt_vocab=tgt_vocab), \n",
    "                  '\\t', tgt_sent)\n",
    "\n",
    "evaluate_sanity()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "2muncpfeynNI",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "After making sure that your network works fine, now let's set ```debug = False``` and train your network on the entire corpus. This time, you also need to report loss on the validation set. Before running the following cell, please pay attention to the notes mentioned in the text cell before the train loop. Don't forget to plot your results afterwards (feel free to return any values you need from the train function.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8550640,
     "status": "ok",
     "timestamp": 1588293566338,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "zIKDn4WdcRnm",
    "new_sheet": false,
    "outputId": "c4745cf7-cc83-40af-847e-8f30e8b35962",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading the corpus...\n",
      "TRAINING ON ENTIRE CORPUS!\n",
      "Begin training\n",
      "Epoch 3: Train Loss: 18.67498363378836\n",
      "Epoch 6: Train Loss: 9.314135504312842\n",
      "Epoch 9: Train Loss: 6.493255776597522\n",
      "Epoch 12: Train Loss: 5.0280823236182615\n",
      "Epoch 15: Train Loss: 4.153705217992397\n",
      "Epoch 18: Train Loss: 3.5040092232562743\n",
      "Epoch 21: Train Loss: 3.0039453869083537\n",
      "Epoch 24: Train Loss: 2.6336271808174616\n",
      "Epoch 27: Train Loss: 2.314423208454262\n",
      "Epoch 30: Train Loss: 2.0567763667596166\n",
      "Epoch 33: Train Loss: 1.8530357631893666\n",
      "Epoch 36: Train Loss: 1.659223595046272\n",
      "Epoch 39: Train Loss: 1.5380313564162744\n",
      "Epoch 42: Train Loss: 1.3663018588330813\n",
      "Epoch 45: Train Loss: 1.309730001728799\n",
      "Epoch 48: Train Loss: 1.219851287145578\n",
      "Epoch 51: Train Loss: 1.135070285189741\n",
      "Epoch 54: Train Loss: 1.0545198450523645\n",
      "Epoch 57: Train Loss: 1.0040763351400537\n",
      "Epoch 60: Train Loss: 0.9633072043553051\n",
      "Epoch 63: Train Loss: 0.9285024914451425\n",
      "Epoch 66: Train Loss: 0.8885124654371023\n",
      "Epoch 69: Train Loss: 0.8524377581737794\n",
      "Epoch 72: Train Loss: 0.831220681223126\n",
      "Epoch 75: Train Loss: 0.7804506504943616\n",
      "Epoch 78: Train Loss: 0.7764243320367182\n",
      "Epoch 81: Train Loss: 0.7848283276811753\n",
      "Epoch 84: Train Loss: 0.7695025391451762\n",
      "Epoch 87: Train Loss: 0.6859365533966528\n",
      "Epoch 90: Train Loss: 0.7427044456449298\n",
      "Epoch 93: Train Loss: 0.6909511882542657\n",
      "Epoch 96: Train Loss: 0.6862017700427386\n",
      "Epoch 99: Train Loss: 0.6807019891847677\n",
      "Epoch 102: Train Loss: 0.6833985024078718\n",
      "Epoch 105: Train Loss: 0.6721255136760016\n",
      "Epoch 108: Train Loss: 0.6327593784368537\n",
      "Epoch 111: Train Loss: 0.6598432178506379\n",
      "Epoch 114: Train Loss: 0.637933714988114\n",
      "Epoch 117: Train Loss: 0.6438222953575194\n",
      "Epoch 120: Train Loss: 0.6640191526920625\n",
      "Epoch 123: Train Loss: 0.591889499711446\n",
      "Epoch 126: Train Loss: 0.6352439373165026\n",
      "Epoch 129: Train Loss: 0.6190509560443604\n",
      "Epoch 132: Train Loss: 0.6140373623416449\n",
      "Epoch 135: Train Loss: 0.5894111735965821\n",
      "Epoch 138: Train Loss: 0.5933006535011549\n",
      "Epoch 141: Train Loss: 0.5817191618930254\n",
      "Epoch 144: Train Loss: 0.6060650701078627\n",
      "Epoch 147: Train Loss: 0.5826478188028805\n",
      "Epoch 150: Train Loss: 0.5965840517341406\n",
      "Epoch 153: Train Loss: 0.5772901381376578\n",
      "Epoch 156: Train Loss: 0.5677031889387857\n",
      "Epoch 159: Train Loss: 0.5917862079216049\n",
      "Epoch 162: Train Loss: 0.5826051099218796\n",
      "Epoch 165: Train Loss: 0.543289737783004\n",
      "Epoch 168: Train Loss: 0.5750011006235622\n",
      "Epoch 171: Train Loss: 0.5705953257392117\n",
      "Epoch 174: Train Loss: 0.5643686252175177\n",
      "Epoch 177: Train Loss: 0.5582005064297088\n",
      "Epoch 180: Train Loss: 0.5651674950530774\n",
      "Epoch 183: Train Loss: 0.5422215859472979\n",
      "Epoch 186: Train Loss: 0.5458851322928312\n",
      "Epoch 189: Train Loss: 0.5626870193182288\n",
      "Epoch 192: Train Loss: 0.5591211332567753\n",
      "Epoch 195: Train Loss: 0.5390662254036152\n",
      "Epoch 198: Train Loss: 0.530565933809534\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEICAYAAABYoZ8gAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de3RV9Z338fc3Jycn9xAgXAoqWG8IaMCUoYNQ74PSVi1e2tp5tHVKx2VXZc10WrXzTO2z6iz7VK3tqtXaUUtnrNZrdfnY2tZB0XoNcjGAd1FAIAEHEhJyO/k+f5ydkGASAmSfHbI/r7XOyj777H32Nzsnn/M7v7P3b5u7IyIi8ZETdQEiIpJdCn4RkZhR8IuIxIyCX0QkZhT8IiIxo+AXEYmZ0IPfzBJmtsLMHg/uTzazl8zsbTP7nZnlhV2DiIjsYWEfx29m/wRUAaXu/lkzux942N3vM7PbgVXuflt/zzF69GifNGlSqHWKiAw3y5cv3+buFXvPzw1zo2Y2EVgAXA/8k5kZcBrw5WCRJcB1QL/BP2nSJKqrq0OsVERk+DGz93ubH3ZXzy3Ad4CO4P4oYIe7twf3NwITQq5BRES6CS34zeyzQK27Lz/A9ReZWbWZVdfV1Q1ydSIi8RVmi38O8HkzWw/cR6aL56fACDPr7GKaCGzqbWV3v8Pdq9y9qqLiY11UIiJygELr43f3a4BrAMzsFODb7n6JmT0AXEDmzeBS4NGwahCRoaetrY2NGzfS3NwcdSnDRn5+PhMnTiSZTA5o+VC/3O3Dd4H7zOyHwArgzghqEJGIbNy4kZKSEiZNmkTmeA85GO7O9u3b2bhxI5MnTx7QOlkJfnd/Gng6mH4XmJWN7YrI0NPc3KzQH0RmxqhRo9if70J15q6IZJ1Cf3Dt7/4c1sH/+JuP86PnfhR1GSIiQ8qwDv4/vv1Hfvz8j6MuQ0SGmB07dvCLX/xiv9c755xz2LFjRwgVZdewDv68RB6t6daoyxCRIaav4G9vb+9l6T2eeOIJRowYEVZZWRPFUT1Zo+AXkd5cffXVvPPOO1RWVpJMJsnPz6e8vJzXX3+dN998k/POO48NGzbQ3NzMVVddxaJFi4A9w8fs2rWLs88+m5NPPpnnn3+eCRMm8Oijj1JQUBDxbzYwsQh+d9eXSSJD0OI/LmbllpWD+pyV4yq5Zf4t/S5zww03UFNTw8qVK3n66adZsGABNTU1XYdD3nXXXYwcOZLdu3fzqU99ioULFzJq1Kgez/HWW29x77338qtf/YqLLrqIhx56iK985SuD+ruEZdh39ThOe0f/H99EJN5mzZrV4xj4n/3sZ5x44onMnj2bDRs28NZbb31sncmTJ1NZWQnASSedxPr167NV7kEb9i1+gNZ0K8nEwM5oE5Hs2VfLPFuKioq6pp9++mn+8pe/8MILL1BYWMgpp5zS61nGqVSqazqRSLB79+6s1DoYhnWLP5XI/GHUzy8i3ZWUlNDQ0NDrYzt37qS8vJzCwkJef/11XnzxxSxXF77YtPhFRDqNGjWKOXPmMG3aNAoKChg7dmzXY/Pnz+f2229nypQpHHvsscyePTvCSsOh4BeRWPrtb3/b6/xUKsUf/vCHXh/r7McfPXo0NTU1XfO//e1vD3p9YRrWXT0KfhGRj4tF8LekWyKuRERk6IhF8KvFLyKyh4JfRCRmhnXwp3J1OKeIyN6GdfCrxS8i8nGhBb+Z5ZvZy2a2yszWmNkPgvm/NrP3zGxlcKsMqwYFv4gMhuLiYgA+/PBDLrjggl6XOeWUU6iuru73eW655Raampq67kc1zHOYLf4W4DR3PxGoBOabWeeZEP/i7pXBbXBHaOqm66iedh3VIyIH7xOf+AQPPvjgAa+/d/BHNcxzaMHvGbuCu8ng5mFtrzdq8YtIb66++mpuvfXWrvvXXXcdP/zhDzn99NOZOXMm06dP59FHH/3YeuvXr2fatGkA7N69my9+8YtMmTKF888/v8dYPVdccQVVVVVMnTqV73//+0Bm4LcPP/yQU089lVNPPRXIDPO8bds2AG6++WamTZvGtGnTuOWWW7q2N2XKFL7+9a8zdepUzjrrrEEZEyjUM3fNLAEsB44CbnX3l8zsCuB6M/s34CngancPpUmu4BcZ4hYvhpWD/KG/shJu6X/wt4svvpjFixdz5ZVXAnD//ffz5JNP8q1vfYvS0lK2bdvG7Nmz+fznP9/nkO633XYbhYWFrFu3jtWrVzNz5syux66//npGjhxJOp3m9NNPZ/Xq1XzrW9/i5ptvZunSpYwePbrHcy1fvpy7776bl156CXfnb/7mb/jMZz5DeXl5KMM/h/rlrrun3b0SmAjMMrNpwDXAccCngJHAd3tb18wWmVm1mVXvz9Xju9MgbSLSmxkzZlBbW8uHH37IqlWrKC8vZ9y4cVx77bWccMIJnHHGGWzatImtW7f2+RzLli3rCuATTjiBE044oeux+++/n5kzZzJjxgzWrFnD2rVr+63nueee4/zzz6eoqIji4mK+8IUv8OyzzwLhDP+clbF63H2HmS0F5rv7jcHsFjO7G+h1kAt3vwO4A6CqquqAuojU4hcZ4vbRMg/ThRdeyIMPPsiWLVu4+OKLueeee6irq2P58uUkk0kmTZrU63DM+/Lee+9x44038sorr1BeXs5ll112QM/TKYzhn8M8qqfCzEYE0wXAmcDrZjY+mGfAeUBN389ycBT8ItKXiy++mPvuu48HH3yQCy+8kJ07dzJmzBiSySRLly7l/fff73f9efPmdQ30VlNTw+rVqwGor6+nqKiIsrIytm7d2mPAt76Gg547dy6///3vaWpqorGxkUceeYS5c+cO4m/bU5gt/vHAkqCfPwe4390fN7P/NrMKwICVwD+GVYDG6hGRvkydOpWGhgYmTJjA+PHjueSSS/jc5z7H9OnTqaqq4rjjjut3/SuuuIKvfvWrTJkyhSlTpnDSSScBcOKJJzJjxgyOO+44DjvsMObMmdO1zqJFi5g/fz6f+MQnWLp0adf8mTNnctlllzFr1iwA/uEf/oEZM2aEdlUvc8/qgTYHpKqqyvd1fGxvWtpbyL8+n+tPu55r514bQmUisr/WrVvHlClToi5j2Oltv5rZcnev2nvZYX3mbuflFtXVIyKyx7AO/hzLIZmTVPCLiHQzrIMfMv38Cn6RoeVQ6GI+lOzv/lTwi0hW5efns337doX/IHF3tm/fTn5+/oDXGdbX3IVM8GusHpGhY+LEiWzcuJEDPTFTPi4/P5+JEycOePlYBH9rh1r8IkNFMplk8uTJUZcRa+rqERGJmWEf/KnclIJfRKSbYR/8avGLiPSk4BcRiZlYBL+O6hER2SMWwa8Wv4jIHgp+EZGYGfbBn0roqB4Rke6GffCrxS8i0pOCX0QkZmIR/LoCl4jIHmFeczffzF42s1VmtsbMfhDMn2xmL5nZ22b2OzPLC6sGUItfRGRvYbb4W4DT3P1EoBKYb2azgR8BP3H3o4D/AS4PsQYFv4jIXkILfs/YFdxNBjcHTgMeDOYvAc4LqwbQUT0iInsLtY/fzBJmthKoBf4MvAPscPf2YJGNwIQwa1CLX0Skp1CD393T7l4JTARmAccNdF0zW2Rm1WZWfTAXbMhL5NHe0U6Hdxzwc4iIDCdZOarH3XcAS4FPAyPMrPMCMBOBTX2sc4e7V7l7VUVFxQFvOy+R+e5YrX4RkYwwj+qpMLMRwXQBcCawjswbwAXBYpcCj4ZVAyj4RUT2FualF8cDS8wsQeYN5n53f9zM1gL3mdkPgRXAnSHWoOAXEdlLaMHv7quBGb3Mf5dMf39WpHJTgIJfRKRTLM7cBQW/iEgnBb+ISMzEJvh1FS4RkYzYBL9a/CIiGQp+EZGYGfbBn0roqB4Rke6GffCrxS8i0pOCX0QkZmIT/LoKl4hIRmyCXy1+EZEMBb+ISMwM++DXWD0iIj0N++BXi19EpKfYBL+GbBARyRj2wZ+fmw9Ac3tzxJWIiAwNwz74U4kUhtHU1hR1KSIiQ8KwD34zozBZqOAXEQmEec3dw8xsqZmtNbM1ZnZVMP86M9tkZiuD2zlh1dCpMFnI7vbdYW9GROSQEOY1d9uBf3b3V82sBFhuZn8OHvuJu98Y4rZ7KEgWqMUvIhII85q7m4HNwXSDma0DJoS1vf6oq0dEZI+s9PGb2SQyF15/KZj1TTNbbWZ3mVl52NtX8IuI7BF68JtZMfAQsNjd64HbgE8ClWQ+EdzUx3qLzKzazKrr6uoOqgYFv4jIHqEGv5klyYT+Pe7+MIC7b3X3tLt3AL8CZvW2rrvf4e5V7l5VUVFxUHXoy10RkT3CPKrHgDuBde5+c7f547stdj5QE1YNnQpy9eWuiEinMI/qmQP8PfCama0M5l0LfMnMKgEH1gPfCLEGQF09IiLdhXlUz3OA9fLQE2Ftsy8KfhGRPYb9mbug4BcR6S42wb+7TV/uiohAjIK/Jd1CuiMddSkiIpGLRfAX5BYA6JBOERFiEvyFyUIA9fOLiKDgFxGJHQW/iEjMxCr4dWSPiEhMgr8gmflyVy1+EZGYBL+6ekRE9lDwi4jEjIJfRCRmYhX8OoFLRCQmwd955q5a/CIiMQl+dfWIiOwRi+DPz80HFPwiIjDA4Dezq8ys1DLuNLNXzeyssIsbLGamMflFRAIDbfF/zd3rgbOAcjKXVLyhvxXM7DAzW2pma81sjZldFcwfaWZ/NrO3gp/lB/UbDJDG5BcRyRho8HdeQvEc4D/dfQ29X1axu3bgn939eGA2cKWZHQ9cDTzl7kcDTwX3Q1eQW0BTu1r8IiIDDf7lZvYnMsH/pJmVAB39reDum9391WC6AVgHTADOBZYEiy0BzjuQwveXunpERDIGerH1y4FK4F13bzKzkcBXB7oRM5sEzABeAsa6++bgoS3A2AFXexAU/CIiGQNt8X8aeMPdd5jZV4B/BXYOZEUzKwYeAhYH3xN0cXcHvI/1FplZtZlV19XVDbDMvin4RUQyBhr8twFNZnYi8M/AO8Bv9rWSmSXJhP497v5wMHurmY0PHh8P1Pa2rrvf4e5V7l5VUVExwDL7puAXEckYaPC3B63zc4Gfu/utQEl/K5iZAXcC69z95m4PPQZcGkxfCjy6fyUfmIJkgY7qERFh4H38DWZ2DZnDOOeaWQ6Q3Mc6c4LlXzOzlcG8a8kcBnq/mV0OvA9ctP9l7z+1+EVEMgYa/BcDXyZzPP8WMzsc+HF/K7j7c/R9yOfpAy9xcBTmKvhFRGCAXT3uvgW4Bygzs88Cze6+zz7+oUQtfhGRjIEO2XAR8DJwIZmumZfM7IIwCxtshclCGtsaoy5DRCRyA+3q+R7wKXevBTCzCuAvwINhFTbYSlIltKZbaU23kpfIi7ocEZHIDPSonpzO0A9s3491h4QR+SMA2Nk8oNMPRESGrYG2+P9oZk8C9wb3LwaeCKekcJSlygDY0byDiqKDPy9ARORQNaDgd/d/MbOFZA7RBLjD3R8Jr6zBV5afCf6dLWrxi0i8DbTFj7s/ROYs3EOSunpERDL6DX4za6D3sXSMzFA7paFUFYLuXT0iInHWb/C7e7/DMhxK1NUjIpJxSB2ZczDU1SMikhGb4C/Jy3x4UVePiMRdbII/kZOgJK9EXT0iEnuxCX7IdPco+EUk7mIV/GX5ZerqEZHYi1fwp8r05a6IxF6sgn9E/gi1+EUk9mIV/GX5ZerjF5HYCy34zewuM6s1s5pu864zs01mtjK4nRPW9nszIjVCXT0iEnthtvh/DczvZf5P3L0yuGV1hM/OL3cz140XEYmn0ILf3ZcBH4X1/AeiLFVG2tO6BKOIxFoUffzfNLPVQVdQeTY33DVsg/r5RSTGsh38twGfBCqBzcBNfS1oZovMrNrMquvq6gZl450DtenIHhGJs6wGv7tvdfe0u3cAvwJm9bPsHe5e5e5VFRWDc8WszqGZ9QWviMRZVoPfzMZ3u3s+UNPXsmFQV4+IyH5cgWt/mdm9wCnAaDPbCHwfOMXMKslc3GU98I2wtt8bdfWIiIQY/O7+pV5m3xnW9gZCXT0iIjE7c1ddPSIiMQv+wmQheYk8tjVti7oUEZHIxCr4zYwJJRPY1LAp6lJERCITq+AHmFg6kU31Cn4Ria/YBf+E0glsrN8YdRkiIpGJXfBPLJnIpoZNGqhNRGIrdsE/oXQCze3NfLR7SI0fJyKSNbEL/omlEwHU3SMisRXb4NeRPSISV7EL/gklEwC1+EUkvmIX/OOKx5FjOTqkU0RiK3bBn0wkGVs0Vi1+EYmt2AU/BCdxqY9fRGIqlsGvk7hEJM5iGfydJ3GJiMRRLIP/sLLD2NG8Q+Pyi0gsxTL4j684HoC1dWsjrkREJPtCC34zu8vMas2sptu8kWb2ZzN7K/hZHtb2+zNtzDQAamqzeslfEZEhIcwW/6+B+XvNuxp4yt2PBp4K7mfd4WWHU5xXzGu1r0WxeRGRSIUW/O6+DNh7JLRzgSXB9BLgvLC2358cy2FqxVS1+EUklrLdxz/W3TcH01uAsVnefpfpY6Yr+EUkliL7ctczA+L3OSi+mS0ys2ozq66rqxv07U8bM426pjpqG2sH/blFRIaybAf/VjMbDxD87DN13f0Od69y96qKiopBL6TzC97XtqqfX0TiJdvB/xhwaTB9KfBolrffRUf2iEhchXk4573AC8CxZrbRzC4HbgDONLO3gDOC+5EYUzSGMUVjeHXLq1GVICISidywntjdv9THQ6eHtc39YWacfPjJPPv+s1GXIiKSVbE8c7fTvMPn8d6O99iwc0PUpYiIZE2sg3/uEXMBePYDtfpFJD5iHfwnjj2RkrwSdfeISKzEOvgTOQnmHD6HZR8si7oUEZGsiXXwQ6aff23dWrbu2hp1KSIiWRH74F9wzAIAfv/67yOuREQkO2If/NPHTOfokUfz0LqHoi5FRCQrYh/8ZsbCKQv57/f+m+1N26MuR0QkdLEPfoALjr+AtKd57I3Hoi5FRCR0Cn5g5viZHFl+JEtWLdn3wiIihzgFP5nunm+c9A2eef8ZjdYpIsOegj9w+YzLyc/N5+cv/zzqUkREQqXgD4wqHMWXp32Z/3rtv/ho995XjBQRGT4U/N0snr2YprYmbnnxlqhLEREJjYK/m+ljp7NwykJ++tJP1eoXkWFLwb+X73/m+9S31HPT8zdFXYqISCgU/HuZPnY6l0y/hBtfuJE3tr0RdTkiIoMukuA3s/Vm9pqZrTSz6ihq6M9NZ91EYbKQRY8vosM7oi5HRGRQRdniP9XdK929KsIaejW2eCw3nnkjy95fpsM7RWTYUVdPH74242ssOHoB3/nzd1i9dXXU5YiIDJqogt+BP5nZcjNbFFEN/TIz7j73bsoLyll4/0JqG2ujLklEZFBEFfwnu/tM4GzgSjObt/cCZrbIzKrNrLquri77FQIVRRU8cvEjbKrfxDn3nEN9S30kdYiIDKZIgt/dNwU/a4FHgFm9LHOHu1e5e1VFRUW2S+wye+JsHrjwAVZtXcXZ95yt8BeRQ17Wg9/MisyspHMaOAuoyXYd+2PBMQu4b+F9vLzpZc74zRlsbtgcdUkiIgcsihb/WOA5M1sFvAz8P3f/YwR17JeFxy/k4YseZk3dGmb9xyxWbF4RdUkiIgck68Hv7u+6+4nBbaq7X5/tGg7U5479HH/92l8xjJPvPpkH1jwQdUkiIvtNh3Pup8pxlbz89Zc5YewJXPTgRXz9sa+zq3VX1GWJiAyYgv8AjCsexzOXPcM1J1/DnSvupPL2Sl7Y8ELUZYmIDIiC/wDlJfL499P/nWcue4b2jnb+9q6/5dLfX8qm+k1RlyYi0i8F/0Gae8RcVl+xmu/O+S731dzHMT8/hh88/QMaWxujLk1EpFcK/kFQmirlhjNu4PUrX2fB0Qu47pnrOPbnx/Kzl37GzuadUZcnItKDgn8QTS6fzP0X3s+zX32Ww8sO56o/XsWEmydwxeNXUFM7pE9VEJEYMXePuoZ9qqqq8urqITd68z5Vf1jNra/cyr2v3UtLuoV5R8zj8hmXs3DKQoryiqIuT0SGOTNb3tsIyAr+LNjWtI27VtzFL5f/knf/511K8kq48PgLueD4Czht8mmkclNRlygiw5CCfwhwd5794Fl+vfLXPLD2AXa17qI0VcqCoxfwhSlf4O8++XeUpEqiLlNEhgkF/xDT3N7MU+8+xSOvP8KjbzzKtqZtJCzBjPEzmHf4POYdMY+TDz+ZUYWjoi5VRA5RCv4hrL2jnb9+8Feeeu8plr2/jBc3vkhLugWAqRVTmXfEPOYePpcZ42dw1MijyM3JjbhiETkUKPgPIS3tLbzy4Ssse38Zy95fxl83/LVrWIi8RB7HjT6OaWOmMa1iGlPHTOWokUdxRNkR+sJYRHpQ8B/C2jvaWb11Na9tfY2a2hrW1K2hpraGDfUbupYxjGNGHcPM8TOZMW4GM8fP5LCywyjOK+665ZiO3hWJEwX/MLSzeSdr69ayfsd63tz+Jiu2rGDFlhV8sPODjy2bzEkyoXQCUyumcvTIoxlbPJYxRWMYUzSGsUV7pguSBRH8JiIShr6CX53Fh7Cy/DI+fdin+fRhn+4xf1vTNlZsXkFtYy27Wnexq3UX25q28UH9B6zasopn3n+mzxFFS/JK9rwhFI9lTOGYrvvFecWkclPkJfJIJVIUJAsYkT+CEfkjKEuVUZxXTH5uPmaWjV9fRA6Qgn8YGl04mjM/eWa/yzS2NlLXVEdtYy1bd22ltrE2M924Z/rtj97m+Q3Ps61pGx3eMaBtG0ZhspCivCKKkkUU5RVl7gfTyZwkuTm5JBNJipPFlOWXUZYq+9jyhcnCHsvm5uT2ed9xWtpbaEm3kMxJMqpwFO5OKjdFYbJwMHapyLCi4I+porxMyE4aMWmfy6Y70mzfvZ2mtiZa0620tLfQmm6lqa2JnS072dG8gx3NO2hsbaSxrXHPz27TTW1NbKrfRHtHO+0d7bSmW9nVuoudLTtpbm8O7fccVTCKEfkjKC8oZ2LpRBpaGtjUsIkPGz4kx3IoySuhJFVCUXLPG05BsoD83Hw6vIP2jnbSHWnSnibdkabDOyhMFlKcV0xRsojivGIAtu/ezvbd22lub6Ykr4TSVCllqTJGFoxkZMHIrvMz3B3H9/mzwzv2uQxAQW4BowpHMXnEZJKJJM3tzTS3N7O7bXfXdF4ij+K84q7n7fAOciwHwzAzmtubMazH90FFeUWkO9Jdz9Hc3kyHd1CUV9T1ya7zb9mWbsv87Gijwzu6Pg2mEikSOQkMw3F2te6iLd1GMpEkL5HX9cbd+RooTZVSmiqlILegq9bOfd79lvY0QFdDofMoN3fno90fdT1uZD55dn4CzbEcEpYgkZPo+pljOaQ70rR1tNGWbiM/N7/fT62dXePZ+lTb+TpI5CQG9XkjCX4zmw/8FEgA/+HuN0RRhwxMIifBmKIxoT1/S3tL15tD5xvF7rbdXWGyd7jsfd8wUrkpUokUrelWtu/eTo7l0NTWxIadG6hvrWd703be3P4mpalSplZM5cwjM5+IGlobaGhpoKmtiaa2JrY1baOprYnm9uYeAZGbk0vCEpgZu9t2s6t1Fw2tDV1dZqMKRjGqcBT5ufms37Ge+pZ6djbvpLFNo7SGrfONrfPvdrByc3IpTZWSsMTHXm9pT5OwBMV5xZSkSshL5HW9OaU93fVm1X2687F0RxrHybGcTGMkvxzHezQszKzrNdfY2khtYy1PfuVJTj/y9EHYU91+x0F9tgEwswRwK3AmsBF4xcwec/e12a5FhoZUbopUboqRBSOjLmXQtaZb+Wj3RzS0NACZlmJnS7u3nzmWs89lOn8CNLU1UddUx/od60l3pLs+reTn5lOQW0AqN/Nm2NDSkGnxdmuBu2da1ancFO5OY1tj13dCja2N5ObkkspN7WkFY12f4prbm8nNyf1Y11uO5dCSbun6xNH9E0pJXgnJRJK2dBttHW20pltp72jvauU3tDZQ31JPU1sTOZbT1ULvnO5+6/zdd7Xu6qq7ILeAiaUTSSaSXS3zzk9Gnb9rZ8B2D+ZETqLr00dLuoX6lnrqW+rp8I4eXYqdt/aO9q43/tZ0a1cDoXu9Pebl7HnMzOjwDnY07+Cj3R/1WDaRk8DdSXua9o52CnMLGVs8lomlEwf9dRlFi38W8La7vwtgZvcB5wIKfhl28hJ5jCsex7jicaE8f3lBORNKJ1A5rjKU55fhKYoDuycAG7rd3xjMExGRLBiyZ/SY2SIzqzaz6rq6uqjLEREZNqII/k3AYd3uTwzm9eDud7h7lbtXVVRUZK04EZHhLorgfwU42swmm1ke8EXgsQjqEBGJpax/uevu7Wb2TeBJModz3uXua7Jdh4hIXEVyHL+7PwE8EcW2RUTibsh+uSsiIuFQ8IuIxMwhMSyzmdUB7x/g6qOBbYNYzmAZqnXB0K1Nde2foVoXDN3ahltdR7j7xw6LPCSC/2CYWXVv41FHbajWBUO3NtW1f4ZqXTB0a4tLXerqERGJGQW/iEjMxCH474i6gD4M1bpg6NamuvbPUK0Lhm5tsahr2Pfxi4hIT3Fo8YuISDfDOvjNbL6ZvWFmb5vZ1RHWcZiZLTWztWa2xsyuCuZfZ2abzGxlcDsngtrWm9lrwfarg3kjzezPZvZW8LM8yzUd222frDSzejNbHNX+MrO7zKzWzGq6zet1H1nGz4LX3Gozm5nlun5sZq8H237EzEYE8yeZ2e5u++72LNfV59/OzK4J9tcbZvZ3Wa7rd91qWm9mK4P52dxffeVDeK8xdx+WNzLjAL0DHAnkAauA4yOqZTwwM5guAd4EjgeuA74d8X5aD4zea97/Ba4Opq8GfhTx33ELcERU+wuYB8wEava1j4BzgD8ABswGXspyXWcBucH0j7rVNan7chHsr17/dsH/wSogBUwO/mcT2aprr8dvAv4tgv3VVz6E9hobzi3+rit9uXsr0Hmlr6xz983u/mow3QCsY2hffOZcYEkwvQQ4L8JaTgfecfcDPYHvoLn7MuCjvWb3tY/OBX7jGS8CI8xsfLbqcvc/uXt7cPdFMsOeZ1Uf+6sv56dy0f4AAARdSURBVAL3uXuLu78HvE3mfzerdZmZARcB94ax7f70kw+hvcaGc/APySt9mdkkYAbwUjDrm8HHtbuy3aUScOBPZrbczBYF88a6++ZgegswNoK6On2Rnv+MUe+vTn3to6H0uvsamZZhp8lmtsLMnjGzuRHU09vfbqjsr7nAVnd/q9u8rO+vvfIhtNfYcA7+IcfMioGHgMXuXg/cBnwSqAQ2k/momW0nu/tM4GzgSjOb1/1Bz3y2jOTQL8tcr+HzwAPBrKGwvz4myn3UFzP7HtAO3BPM2gwc7u4zgH8CfmtmpVksaUj+7br5Ej0bGFnfX73kQ5fBfo0N5+Af0JW+ssXMkmT+qPe4+8MA7r7V3dPu3gH8ipA+4vbH3TcFP2uBR4IatnZ+dAx+1ma7rsDZwKvuvjWoMfL91U1f+yjy152ZXQZ8FrgkCAyCrpTtwfRyMn3px2Srpn7+dkNhf+UCXwB+1zkv2/urt3wgxNfYcA7+IXOlr6D/8E5gnbvf3G1+936584GavdcNua4iMyvpnCbzxWANmf10abDYpcCj2ayrmx6tsKj311762kePAf8rOPJiNrCz28f10JnZfOA7wOfdvanb/AozSwTTRwJHA+9msa6+/naPAV80s5SZTQ7qejlbdQXOAF53942dM7K5v/rKB8J8jWXjW+uobmS+/X6TzLv19yKs42QyH9NWAyuD2znAfwKvBfMfA8Znua4jyRxRsQpY07mPgFHAU8BbwF+AkRHssyJgO1DWbV4k+4vMm89moI1Mf+rlfe0jMkda3Bq85l4DqrJc19tk+n87X2e3B8suDP7GK4FXgc9lua4+/3bA94L99QZwdjbrCub/GvjHvZbN5v7qKx9Ce43pzF0RkZgZzl09IiLSCwW/iEjMKPhFRGJGwS8iEjMKfhGRmFHwi4TMzE4xs8ejrkOkk4JfRCRmFPwiATP7ipm9HIy//kszS5jZLjP7STBO+lNmVhEsW2lmL9qece87x0o/ysz+YmarzOxVM/tk8PTFZvagZcbKvyc4W1MkEgp+EcDMpgAXA3PcvRJIA5eQOYO42t2nAs8A3w9W+Q3wXXc/gczZk53z7wFudfcTgb8lc6YoZEZcXExmnPUjgTmh/1IifciNugCRIeJ04CTglaAxXkBmUKwO9gze9V/Aw2ZWBoxw92eC+UuAB4Jxjya4+yMA7t4MEDzfyx6MBWOZqzxNAp4L/9cS+TgFv0iGAUvc/ZoeM83+917LHegYJy3dptPof08ipK4ekYyngAvMbAx0Xe/0CDL/IxcEy3wZeM7ddwL/0+3iHH8PPOOZqydtNLPzgudImVlhVn8LkQFQq0MEcPe1ZvavZK5GlkNmBMcrgUZgVvBYLZnvASAzTO7tQbC/C3w1mP/3wC/N7P8Ez3FhFn8NkQHR6Jwi/TCzXe5eHHUdIoNJXT0iIjGjFr+ISMyoxS8iEjMKfhGRmFHwi4jEjIJfRCRmFPwiIjGj4BcRiZn/D5He5RHSJw/1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "args = dict(batch_size=512, embed_size=256, hidden_size=256, dropout_rate=0.2,\n",
    "            uniform_init=0.1, lr=1e-3, log_every=3, max_epoch=200, \n",
    "            val_niter=1000, debug=False)\n",
    "\n",
    "train(args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "otf1uVZEt9o_",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now let's use your trained model to generate translations of a random sample of validation sentences in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6256,
     "status": "ok",
     "timestamp": 1588293572628,
     "user": {
      "displayName": "sadra barikbein",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiCq0YqNHWfcZyly3I_AMTkvS8_tO8PZOQpGtkK=s64",
      "userId": "07513140462530453530"
     },
     "user_tz": -270
    },
    "id": "tq9JxChUuPoO",
    "new_sheet": false,
    "outputId": "3089c3cf-dc7c-48ec-ce96-f73d509aebbc",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading the corpus...\n",
      "\n",
      "format of prints: [source-sentence] [your-translation] [ground-truth-translation]\n",
      "\n",
      "source sentence:\n",
      "\t ['que', 'dites', 'vous', 'd', 'aller', 'voir', 'un', 'film', 'ce', 'soir', '?']\n",
      "Your translation:\n",
      "\t ['<START>', 'how', 'about', 'going', 'to', 'see', 'a', 'movie', 'tonight', '?', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'how', 'about', 'going', 'to', 'see', 'a', 'movie', 'tonight', '?', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['c', 'est', 'tout', 'ce', 'que', 'nous', 'faisons', 'ici', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'that', 's', 'all', 'we', 'do', 'here', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'that', 's', 'all', 'we', 'do', 'here', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['elle', 'passe', 'tout', 'son', 'temps', 'a', 'penser', 'aux', 'garcons', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'she', 'spends', 'all', 'her', 'time', 'thinking', 'about', 'boys', 'thinking', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'she', 'spends', 'all', 'her', 'time', 'thinking', 'about', 'boys', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['ce', 'n', 'est', 'pas', 'si', 'facile', 'd', 'apprendre', 'une', 'nouvelle', 'langue', 'apres', 'cinquante', 'ans', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'it', 'is', 'not', 'that', 'easy', 'to', 'learn', 'a', 'new', 'language', 'after', 'ten', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'it', 's', 'not', 'that', 'easy', 'to', 'learn', 'a', 'new', 'language', 'after', 'fifty', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['c', 'est', 'mon', 'affaire', 'a', 'moi', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'he', 'is', 'my', 'matter', 'at', 'me', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'that', 'is', 'my', 'own', 'affair', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['un', 'pub', 'est', 'un', 'endroit', 'ou', 'les', 'gens', 'se', 'rassemblent', 'pour', 'boire', 'de', 'la', 'biere', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'a', 'pub', 'for', 'people', 'or', 'they', 'd', 'collect', 'to', 'drink', 'beer', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'a', 'pub', 'is', 'a', 'popular', 'gathering', 'place', 'in', 'which', 'to', 'drink', 'beer', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['pekin', 'est', 'la', 'capitale', 'de', 'la', 'chine', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'beijing', 'is', 'the', 'capital', 'of', 'china', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'beijing', 'is', 'the', 'capital', 'of', 'china', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['je', 'prefere', 'les', 'trains', 'aux', 'bus', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'prefer', 'trains', 'to', 'buses', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'like', 'trains', 'better', 'than', 'buses', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['je', 'veux', 'm', 'en', 'debarrasser', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'want', 'to', 'get', 'rid', 'of', 'it', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'want', 'to', 'get', 'rid', 'of', 'it', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['il', 'eut', 'beau', 'essayer', 'mon', 'opinion', 'ne', 'changea', 'pas', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'no', 'matter', 'how', 'hard', 'i', 'tried', 'my', 'opinion', 'didn', 't', 'win', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'no', 'matter', 'how', 'hard', 'he', 'tried', 'my', 'opinion', 'didn', 't', 'change', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['je', 'veux', 'savoir', 'de', 'quoi', 'il', 's', 'agit', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'want', 'to', 'know', 'what', 'that', 'is', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'want', 'to', 'know', 'what', 'it', 'is', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['j', 'en', 'ferai', 'l', 'essai', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'll', 'give', 'it', 'up', 'for', 'the', 'attempt', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'll', 'give', 'it', 'a', 'try', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['il', 'n', 'a', 'pas', 'travaille', 'dimanche', 'soir', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'he', 'did', 'not', 'work', 'during', 'sunday', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'he', 'did', 'not', 'work', 'on', 'sunday', 'night', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['est', 'ce', 'tout', 'ce', 'qu', 'il', 'y', 'a', '?']\n",
      "Your translation:\n",
      "\t ['<START>', 'is', 'that', 'all', 'there', 'he', 'is', '?', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'is', 'this', 'all', 'there', 'is', '?', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['tu', 'seras', 'tout', 'a', 'fait', 'remis', 'dans', 'quelques', 'jours', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'you', 'will', 'be', 'all', 'right', 'again', 'in', 'a', 'couple', 'of', 'days', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'you', 'will', 'be', 'all', 'right', 'again', 'in', 'a', 'couple', 'of', 'days', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['aide', 'moi', 'rien', 'que', 'pour', 'une', 'minute', '!']\n",
      "Your translation:\n",
      "\t ['<START>', 'help', 'me', 'for', 'something', 'for', 'a', 'minute', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'help', 'me', 'for', 'just', 'a', 'minute', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['je', 'suis', 'un', 'etranger', 'ici', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'am', 'a', 'stranger', 'here', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'm', 'a', 'stranger', 'here', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['j', 'aime', 'vraiment', 'les', 'chiens', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'really', 'like', 'dogs', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'like', 'dogs', 'very', 'much', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['je', 'dois', 'prendre', 'rendez', 'vous', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'have', 'to', 'take', 'an', 'appointment', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'need', 'to', 'make', 'an', 'appointment', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['quelque', 'chose', 'n', 'est', 'pas', 'correct', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'something', 's', 'not', 'right', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'something', 'isn', 't', 'right', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['j', 'assiste', 'a', 'des', 'conferences', 'scientifiques', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'will', 'go', 'tank', 'of', 'conferences', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'attend', 'scientific', 'conferences', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['qu', 'ai', 'je', 'fait', 'de', 'mal', '?']\n",
      "Your translation:\n",
      "\t ['<START>', 'what', 'have', 'i', 'done', 'wrong', '?', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'what', 'did', 'i', 'do', 'wrong', '?', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['je', 'ne', 'voulais', 'pas', 'attendre', 'apres', 'quoi', 'que', 'ce', 'soit', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'didn', 't', 'want', 'to', 'wait', 'for', 'anything', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'didn', 't', 'want', 'to', 'wait', 'for', 'anything', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['tom', 'et', 'mary', 'etaient', 'de', 'bons', 'amis', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'tom', 'and', 'mary', 'were', 'good', 'friends', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'tom', 'and', 'mary', 'used', 'to', 'be', 'good', 'friends', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['l', 'anglais', 'est', 'enseigne', 'dans', 'la', 'plupart', 'des', 'pays', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'english', 'is', 'taught', 'in', 'most', 'of', 'the', 'country', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'english', 'is', 'taught', 'in', 'most', 'countries', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['c', 'est', 'parti', '!']\n",
      "Your translation:\n",
      "\t ['<START>', 'how', 'long', 'ago', '!', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'here', 'we', 'go', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['comment', 'est', 'ce', 'que', 'tu', 't', 'appelles', '?']\n",
      "Your translation:\n",
      "\t ['<START>', 'what', 's', 'your', 'name', '?', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'what', 'are', 'you', 'called', '?', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['je', 'vous', 'ai', 'achete', 'un', 'cadeau', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'bought', 'you', 'a', 'present', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'bought', 'you', 'a', 'present', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['je', 'ne', 'me', 'fais', 'pas', 'de', 'soucis', 'pour', 'l', 'argent', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'm', 'not', 'concerned', 'about', 'money', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'm', 'not', 'worried', 'about', 'money', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['je', 'ne', 'faisais', 'que', 'dire', 'mes', 'prieres', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'was', 'just', 'saying', 'my', 'one', 'to', 'be', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'was', 'only', 'saying', 'my', 'prayers', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['vous', 'm', 'oublierez', 'un', 'jour', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'you', 're', 'blocking', 'my', 'clock', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'you', 'll', 'forget', 'about', 'me', 'someday', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['allons', 'prendre', 'quelque', 'chose', 'a', 'manger', '!']\n",
      "Your translation:\n",
      "\t ['<START>', 'let', 's', 'hit', 'something', 'about', 'it', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'let', 's', 'get', 'something', 'to', 'eat', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['j', 'ai', 'couru', 'dehors', 'et', 'la', 'porte', 's', 'est', 'refermee', 'derriere', 'moi', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'ran', 'outside', 'and', 'the', 'door', 'took', 'out', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'ran', 'outside', 'and', 'the', 'door', 'locked', 'itself', 'behind', 'me', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['ils', 'viennent', 'juste', 'd', 'arriver', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'they', 'just', 'arrived', 'here', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'they', 'have', 'just', 'arrived', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['cette', 'maison', 'n', 'est', 'pas', 'a', 'vendre', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'this', 'house', 'is', 'not', 'for', 'sale', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'this', 'house', 'is', 'not', 'for', 'sale', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['nos', 'intentions', 'etaient', 'bonnes', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'our', 'intentions', 'were', 'good', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'we', 'meant', 'well', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['aujourd', 'hui', 'c', 'est', 'le', 'jour', 'le', 'plus', 'chaud', 'de', 'l', 'annee', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'today', 'is', 'the', 'warmer', 'day', 'today', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'today', 'is', 'the', 'hottest', 'day', 'this', 'year', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['lorsque', 'je', 'me', 'suis', 'reveille', 'ce', 'matin', 'je', 'me', 'sentais', 'malade', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'when', 'i', 'woke', 'up', 'this', 'morning', 'i', 'felt', 'sick', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'when', 'i', 'woke', 'up', 'this', 'morning', 'i', 'felt', 'sick', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['cette', 'maison', 'n', 'est', 'pas', 'a', 'vendre', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'this', 'house', 'is', 'not', 'for', 'sale', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'this', 'house', 'is', 'not', 'for', 'sale', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['c', 'est', 'tres', 'propre', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'it', 's', 'very', 'clean', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'it', 's', 'very', 'clean', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['croyez', 'en', 'moi', '!']\n",
      "Your translation:\n",
      "\t ['<START>', 'trust', 'me', 'but', 'me', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'trust', 'me', 'on', 'that', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['tous', 'les', 'mecs', 'm', 'ont', 'charriee', 'avec', 'ca', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'all', 'the', 'guys', 'have', 'left', 'to', 'me', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'all', 'the', 'guys', 'teased', 'me', 'about', 'it', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['je', 'prefere', 'le', 'vin', 'rouge', 'au', 'blanc', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'prefer', 'red', 'wine', 'to', 'white', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'prefer', 'red', 'wine', 'to', 'white', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['t', 'ai', 'je', 'entendue', 'parler', 'a', 'quelqu', 'un', '?']\n",
      "Your translation:\n",
      "\t ['<START>', 'did', 'i', 'hear', 'you', 'talking', 'to', 'someone', '?', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'did', 'i', 'hear', 'you', 'talking', 'to', 'someone', '?', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['son', 'travail', 'l', 'absorbe', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'her', 'job', 'does', 'not', 'live', 'up', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'he', 'is', 'absorbed', 'in', 'his', 'work', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['j', 'etais', 'tres', 'fatigue', 'mais', 'j', 'etais', 'quand', 'meme', 'incapable', 'de', 'dormir', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'i', 'was', 'very', 'tired', 'but', 'i', 'was', 'in', 'even', 'there', 'even', 'know', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'i', 'was', 'very', 'tired', 'but', 'i', 'was', 'nevertheless', 'unable', 'to', 'sleep', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['nous', 'ne', 'saurons', 'jamais', 'qui', 'il', 'est', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'we', 'll', 'never', 'know', 'who', 'he', 'is', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'we', 'll', 'never', 'know', 'who', 'he', 'is', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['ce', 'n', 'est', 'pas', 'si', 'facile', 'd', 'apprendre', 'une', 'nouvelle', 'langue', 'apres', 'cinquante', 'ans', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'it', 'is', 'not', 'that', 'easy', 'to', 'learn', 'a', 'new', 'language', 'after', 'ten', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'it', 's', 'not', 'that', 'easy', 'to', 'learn', 'a', 'new', 'language', 'after', 'fifty', '.', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['etaient', 'ils', 'hier', 'a', 'la', 'bibliotheque', '?']\n",
      "Your translation:\n",
      "\t ['<START>', 'were', 'they', 'at', 'the', 'library', 'yesterday', '?', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'were', 'they', 'in', 'the', 'library', 'yesterday', '?', '<END>'] \n",
      "\n",
      "\n",
      "source sentence:\n",
      "\t ['venez', 'a', 'nous', '.']\n",
      "Your translation:\n",
      "\t ['<START>', 'come', 'to', 'us', '.', '<END>']\n",
      "Ground truth translation:\n",
      "\t ['<START>', 'come', 'to', 'us', '.', '<END>'] \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def evaluate():\n",
    "    ################################################################################\n",
    "    # TODO: 1) Load a model pretrained on the entire corpus                        #\n",
    "    ################################################################################\n",
    "    model=AttnSeq2Seq(256,256,src_vocab.size,tgt_vocab.size,dropout_rate=0.2)\n",
    "    model.load_state_dict(torch.load('./Seq2Seq'))\n",
    "    model.eval()\n",
    "    ################################################################################\n",
    "    #                                 END OF YOUR CODE                             #\n",
    "    ################################################################################\n",
    "    model.to(DEVICE)\n",
    "    model.eval()\n",
    "\n",
    "    set_seed(40719)\n",
    "    src_sents, tgt_sents = read_corpus(DATA_PATH)\n",
    "    val_ratio = 0.01\n",
    "    val_size = int(len(src_sents) * val_ratio)\n",
    "    train_size = len(src_sents) - val_size\n",
    "\n",
    "    _, val_dataset = torch.utils.data.random_split(dataset, [train_size, val_size])\n",
    "\n",
    "    print('\\nformat of prints: [source-sentence] [your-translation] [ground-truth-translation]\\n')\n",
    "\n",
    "    idxs = np.random.randint(0, len(val_dataset), 50)\n",
    "\n",
    "    for idx in idxs:\n",
    "        src_sent_tensor, tgt_sent_tensor = val_dataset[idx]['src'], val_dataset[idx]['tgt']\n",
    "        src_sent = [src_vocab.get_token_by_id(el.item()) for el in src_sent_tensor]\n",
    "        tgt_sent = [tgt_vocab.get_token_by_id(el.item()) for el in tgt_sent_tensor]\n",
    "        print('source sentence:\\n\\t', src_sent)\n",
    "        print('Your translation:\\n\\t', ['<START>'] + model.greedy_decode(src_sent_tensor.unsqueeze(1), tgt_vocab))\n",
    "        print('Ground truth translation:\\n\\t', tgt_sent, '\\n\\n')\n",
    "\n",
    "evaluate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "sH-PhLGEsieu",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Note**: If you have implemented everything right, You should see that your network generates some of the translations correctly, some of its outputs mismatch in only a couple of words with the ground truth output, and some of them are even worse than that! <br/>\n",
    "You will witness that sometimes your network repeats generating a single word consecutively for multiple time steps and then after that the translation goes completely in the wrong way! In fact, these are all drawbacks of greedy decoding algorithm. As the network gets its own output, once one output is wrong, the network gets lost and can can generate garbage translation afterwards! There is an algorithm called **beam search** which amis to fix this issue and can generate much better outputs. But, it is outside the scope of our class. Interested students can search and read more about this algorithm on the Internet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "bPnw_vPjIiD3",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**One final point**: Note that there are some advanced packages out there such as, spaCy and torchtext, that can help you automate many of the tasks we did above manullay (e.g. preprocessing, masking, reading the sequences, and etc.). Now that you have learned what's under the hood of training a Seq2Seq model, you can use these packages to make developing your Seq2Seq model easier from this point onwards! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MG0_Hr56pJj8"
   },
   "source": [
    "### 3.7 Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "t3r5rMBbqNpz"
   },
   "source": [
    "- Check and review your answers. Make sure all cells' output are what you have planned.\n",
    "- Select File > Save.\n",
    "- To download the notebook, select File > Download .ipynb.\n",
    "- Create an archive of all notebooks (P1.ipynb, P2.ipynb, and P3.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "e392uwzbj29j",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### 3.8 References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "colab_type": "text",
    "id": "2FponxifRlYJ",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "[1] Minh-Thang Luong, Hieu Pham, Christopher D. Manning, \"Effective Approaches to Attention-based Neural Machine Translation\", CoRR 2015 https://arxiv.org/abs/1508.04025.\n",
    "\n",
    "[2] PyTorch Sequence to Sequence Machine Translation Tutorial https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html\n",
    "\n",
    "[3] CS224N, Stanford University, Assignment 4\n",
    "\n",
    "[4] https://colab.research.google.com/drive/1uFJBO1pgsiFwCGIJwZlhUzaJ2srDbtw-"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "HW4_Part3.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}