Part 1 - Tensors in PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Deep Learning with PyTorch\n",
    "\n",
    "In this notebook, you'll get introduced to [PyTorch](http://pytorch.org/), a framework for building and training neural networks. PyTorch in a lot of ways behaves like the arrays you love from Numpy. These Numpy arrays, after all, are just tensors. PyTorch takes these tensors and makes it simple to move them to GPUs for the faster processing needed when training neural networks. It also provides a module that automatically calculates gradients (for backpropagation!) and another module specifically for building neural networks. All together, PyTorch ends up being more coherent with Python and the Numpy/Scipy stack compared to TensorFlow and other frameworks.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Networks\n",
    "\n",
    "Deep Learning is based on artificial neural networks which have been around in some form since the late 1950s. The networks are built from individual parts approximating neurons, typically called units or simply \"neurons.\" Each unit has some number of weighted inputs. These weighted inputs are summed together (a linear combination) then passed through an activation function to get the unit's output.\n",
    "\n",
    "<img src=\"assets/simple_neuron.png\" width=400px>\n",
    "\n",
    "Mathematically this looks like: \n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "y &= f(w_1 x_1 + w_2 x_2 + b) \\\\\n",
    "y &= f\\left(\\sum_i w_i x_i \\right)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "With vectors this is the dot/inner product of two vectors:\n",
    "\n",
    "$$\n",
    "h = \\begin{bmatrix}\n",
    "x_1 \\, x_2 \\cdots  x_n\n",
    "\\end{bmatrix}\n",
    "\\cdot \n",
    "\\begin{bmatrix}\n",
    "           w_1 \\\\\n",
    "           w_2 \\\\\n",
    "           \\vdots \\\\\n",
    "           w_n\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stack them up!\n",
    "\n",
    "We can assemble these unit neurons into layers and stacks, into a network of neurons. The output of one layer of neurons becomes the input for the next layer. With multiple input units and output units, we now need to express the weights as a matrix.\n",
    "\n",
    "<img src='assets/multilayer_diagram_weights.png' width=450px>\n",
    "\n",
    "We can express this mathematically with matrices again and use matrix multiplication to get linear combinations for each unit in one operation. For example, the hidden layer ($h_1$ and $h_2$ here) can be calculated \n",
    "\n",
    "$$\n",
    "\\vec{h} = [h_1 \\, h_2] = \n",
    "\\begin{bmatrix}\n",
    "x_1 \\, x_2 \\cdots \\, x_n\n",
    "\\end{bmatrix}\n",
    "\\cdot \n",
    "\\begin{bmatrix}\n",
    "           w_{11} & w_{12} \\\\\n",
    "           w_{21} &w_{22} \\\\\n",
    "           \\vdots &\\vdots \\\\\n",
    "           w_{n1} &w_{n2}\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "The output for this small network is found by treating the hidden layer as inputs for the output unit. The network output is expressed simply\n",
    "\n",
    "$$\n",
    "y =  f_2 \\! \\left(\\, f_1 \\! \\left(\\vec{x} \\, \\mathbf{W_1}\\right) \\mathbf{W_2} \\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensors\n",
    "\n",
    "It turns out neural network computations are just a bunch of linear algebra operations on *tensors*, a generalization of matrices. A vector is a 1-dimensional tensor, a matrix is a 2-dimensional tensor, an array with three indices is a 3-dimensional tensor (RGB color images for example). The fundamental data structure for neural networks are tensors and PyTorch (as well as pretty much every other deep learning framework) is built around tensors.\n",
    "\n",
    "<img src=\"assets/tensor_examples.svg\" width=600px>\n",
    "\n",
    "With the basics covered, it's time to explore how we can use PyTorch to build a simple neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "import helper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's see how we work with PyTorch tensors. These are the fundamental data structures of neural networks and PyTorch, so it's imporatant to understand how these work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.9107  0.7433\n",
       " 0.7535  0.6031\n",
       " 0.7666  0.8878\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.rand(3, 2)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 1  1\n",
       " 1  1\n",
       " 1  1\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = torch.ones(x.size())\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 1.9107  1.7433\n",
       " 1.7535  1.6031\n",
       " 1.7666  1.8878\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = x + y\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general PyTorch tensors behave similar to Numpy arrays. They are zero indexed and support slicing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 1.9107\n",
       " 1.7433\n",
       "[torch.FloatTensor of size 2]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 1.7433\n",
       " 1.6031\n",
       " 1.8878\n",
       "[torch.FloatTensor of size 3x1]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z[:, 1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensors typically have two forms of methods, one method that returns another tensor and another method that performs the operation in place. That is, the values in memory for that tensor are changed without creating a new tensor. In-place functions are always followed by an underscore, for example `z.add()` and `z.add_()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 2.9107  2.7433\n",
       " 2.7535  2.6031\n",
       " 2.7666  2.8878\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Return a new tensor z + 1\n",
    "z.add(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 1.9107  1.7433\n",
       " 1.7535  1.6031\n",
       " 1.7666  1.8878\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# z tensor is unchanged\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 2.9107  2.7433\n",
       " 2.7535  2.6031\n",
       " 2.7666  2.8878\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add 1 and update z tensor in-place\n",
    "z.add_(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 2.9107  2.7433\n",
       " 2.7535  2.6031\n",
       " 2.7666  2.8878\n",
       "[torch.FloatTensor of size 3x2]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# z has been updated\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reshaping\n",
    "\n",
    "Reshaping tensors is a really common operation. First to get the size and shape of a tensor use `.size()`. Then, to reshape a tensor, use `.resize_()`. Notice the underscore, reshaping is an in-place operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 2])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 2.9107  2.7433  2.7535\n",
       " 2.6031  2.7666  2.8878\n",
       "[torch.FloatTensor of size 2x3]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.resize_(2, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 2.9107  2.7433  2.7535\n",
       " 2.6031  2.7666  2.8878\n",
       "[torch.FloatTensor of size 2x3]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numpy to Torch and back\n",
    "\n",
    "Converting between Numpy arrays and Torch tensors is super simple and useful. To create a tensor from a Numpy array, use `torch.from_numpy()`. To convert a tensor to a Numpy array, use the `.numpy()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.33669496,  0.59531562,  0.65433944],\n",
       "       [ 0.86531224,  0.59945364,  0.28043973],\n",
       "       [ 0.48409303,  0.98357622,  0.33884284],\n",
       "       [ 0.25591391,  0.51081783,  0.39986403]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.random.rand(4,3)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.3367  0.5953  0.6543\n",
       " 0.8653  0.5995  0.2804\n",
       " 0.4841  0.9836  0.3388\n",
       " 0.2559  0.5108  0.3999\n",
       "[torch.DoubleTensor of size 4x3]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = torch.from_numpy(a)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.33669496,  0.59531562,  0.65433944],\n",
       "       [ 0.86531224,  0.59945364,  0.28043973],\n",
       "       [ 0.48409303,  0.98357622,  0.33884284],\n",
       "       [ 0.25591391,  0.51081783,  0.39986403]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The memory is shared between the Numpy array and Torch tensor, so if you change the values in-place of one object, the other will change as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.6734  1.1906  1.3087\n",
       " 1.7306  1.1989  0.5609\n",
       " 0.9682  1.9672  0.6777\n",
       " 0.5118  1.0216  0.7997\n",
       "[torch.DoubleTensor of size 4x3]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Multiply PyTorch Tensor by 2, in place\n",
    "b.mul_(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.67338991,  1.19063124,  1.30867888],\n",
       "       [ 1.73062448,  1.19890728,  0.56087946],\n",
       "       [ 0.96818606,  1.96715243,  0.67768568],\n",
       "       [ 0.51182782,  1.02163565,  0.79972807]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Numpy array matches new values from Tensor\n",
    "a"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}