07_GAN1.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "0e7d24fe-a15b-4d78-aaac-f029bd76fa82",
   "metadata": {},
   "source": [
    "---\n",
    "execute:\n",
    "  eval: false\n",
    "skip_exec: true\n",
    "skip_showdoc: true\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b60ab1e-44d1-4bac-815e-8ffa0764ca40",
   "metadata": {},
   "source": [
    "# Lesson 7: GANs Part 1 - GAN Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8178eb6c-8741-4d13-8308-1ee757a44588",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO video intro"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5e92aad-f063-4fa6-83de-8f993609fabd",
   "metadata": {},
   "source": [
    "## GANs - Theory\n",
    "\n",
    "The motivation for GANs was the desire for a generative model that could generate plausible new images (or any kind of data for that matter) that look like they could have come from some domain or training set. In the previous lesson we saw how Variational Auto-Encoders learn how to encode an input into some latent representation and then decode it back into an output, and how the decoder part of an auto-encoder can be used to generate new images.\n",
    "\n",
    "GANs achieve a similar result but tend to peform far better than auto-encoder based systems at generating new, novel outputs.\n",
    "\n",
    "They do this by training two separate sub-networks, a generator network and a discriminator network. They are trained in parallel. The goal of the discriminator is to tell whether a given image is real (aka from the training data) or fake (aka generated by the generator model). The generator tries to create plausible outputs to fool the discriminator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "447a6cf2-a970-42c8-82e6-e912ea09587f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO images"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b582b007-bb5d-44b7-9e66-514f61377b69",
   "metadata": {},
   "source": [
    "A typical GAN training loop might look something like the following:\n",
    "\n",
    "for batch in data:\n",
    "\n",
    "*   generate a batch of fake images\n",
    "*   feed both the fake images and the batch of training images into the discriminator\n",
    "*   calculate the discriminator loss and update the discriminator weights to improve it's accuracy\n",
    "*   update the weights of the generator to better fool the discriminator (i.e. increase the discriminator loss)\n",
    "\n",
    "This core idea is very simple, and yet GAN training can fail in some unexpected ways. For example, a GAN that perfectly memorizes the training data can fool the discriminator perfectly, but might not be able to generate any new data. Even worse: it can get away with memorizing only a subset of the training data, a situation called 'mode collapse'.\n",
    "\n",
    "Because of these quirks, GAN training is viewed by many as some sort of dark art! But in this lesson we're going to face it bravely, exploring a basic GAN implementation and seeing for ourselves what training looks like. In the next lesson, we'll introduce some additional ideas and a few of the tricks people use to get better results with GANs. And in the bonus notebook [coming some time] we'll look at an idea called NO-GAN. \n",
    "\n",
    "\"Wait, I just want to train a good GAN right now!\" you say? Fair enough - in that case skip this simple demo and jump straight to something like [LightWeightGAN](https://github.com/lucidrains/lightweight-gan) which has ready-to-use training scripts, or look around for one of the many StyleGAN tutorials. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d8a191b-3483-49ed-8a5d-a4e3ff024042",
   "metadata": {},
   "source": [
    "## DC-GAN From Scratch\n",
    "\n",
    "Much of the code here comes from the PyTorch docs [TODO link]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b470b6d-9336-4d31-be1d-6b951f4c8828",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|hide\n",
    "import torch\n",
    "import torchvision\n",
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import transforms\n",
    "from datasets import load_dataset\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8861030-1b10-489b-bd06-264c8dbdac8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "xb = next(iter(train_dataloader))['images'].to(device)[:8]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f302a5e-483d-4fd5-abe2-1b8d017828d8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration huggan--smithsonian_butterflies_subset-7665b1021a37404c\n",
      "Reusing dataset parquet (/home/jonathan/.cache/huggingface/datasets/huggan___parquet/huggan--smithsonian_butterflies_subset-7665b1021a37404c/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n"
     ]
    }
   ],
   "source": [
    "#|output:false\n",
    "dataset = load_dataset(\"huggan/smithsonian_butterflies_subset\", split=\"train\")\n",
    "\n",
    "image_size = 32\n",
    "batch_size = 32\n",
    "\n",
    "preprocess = transforms.Compose(\n",
    "    [\n",
    "        transforms.Resize((image_size, image_size)),\n",
    "        transforms.RandomHorizontalFlip(),\n",
    "        transforms.ToTensor(),\n",
    "        transforms.Normalize([0.5], [0.5]),\n",
    "    ]\n",
    ")\n",
    "\n",
    "def transform(examples):\n",
    "    images = [preprocess(image.convert(\"RGB\")) for image in examples[\"image\"]]\n",
    "    return {\"images\": images}\n",
    "\n",
    "dataset.set_transform(transform)\n",
    "\n",
    "train_dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a8a5632-facb-426b-b5ca-a3b5aa841ec7",
   "metadata": {},
   "source": [
    "## A few improvements\n",
    "\n",
    "- Add logging to training loop\n",
    "- Add a few things like pixelshuffle/blur\n",
    "- Leave space to tweak loss functions and such\n",
    "- Export a bunch of components for next notebook\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6d33697-96bd-4e3d-b996-af56e49abb2c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7b588511-3851-46be-b2ab-8e95c7a07919",
   "metadata": {},
   "source": [
    "## Playing with an existing GAN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2df61cc4-7e6f-41d6-afa4-35a199700006",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load a lightweight-GAN? Or a model trained with the above code (wrapped in a script) for a little longer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac5e2dc5-248f-480c-a1df-9a7aef3fba09",
   "metadata": {},
   "outputs": [],
   "source": [
    "# latent walk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b7afdb8-7b92-42fd-ae16-af7dc687db9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Guide with a loss (demo one from generators and losses notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a57955aa-0416-4f1b-b49d-2b7afa24ee82",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Link to butterflyGAN demo on HF spaces"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "811ded8c-3589-43f2-841c-f5e942f6ec72",
   "metadata": {},
   "source": [
    "Page stats: Total Hits: [![HitCount](https://hits.dwyl.com/johnowhitaker/tglcourse.svg?style=flat-square&show=unique)](http://hits.dwyl.com/johnowhitaker/tglcourse)\n",
    "Page visitors:\n",
    "![visitor badge](https://page-views.glitch.me/badge?page_id=tglcourse.l07)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}