W1_1_Pandas_Intro_student.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "W1.1_Pandas_Intro_student.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "sDKBErWbSFAI",
        "10YQkXh1Smjm",
        "-54odkZ5VsQw",
        "C_GMWdaPWK3P",
        "QrsnQQ33XIxD",
        "J9CIkU51YdET",
        "leTpYZAXa442",
        "bAVfBKEJbYxc",
        "L57zEGqtenG5",
        "3nI-tc-igqYV",
        "6uJo8zUKhjt7",
        "pqSmI4JXhx1Y",
        "8ryJN1THiIDO",
        "bJzFRnRVjQUo",
        "h5t9M5SNu3nb",
        "w9XGVvHUv72L"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.5"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hpnightowl/owlsbooks/blob/master/W1_1_Pandas_Intro_student.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v8JFOvfT4Yak",
        "colab_type": "text"
      },
      "source": [
        "# Week 1 - Topic 1\n",
        "**Authors:** Saravanan JaiChandaran **|** Julián Darío Miranda **|** Manish Kumar Chaudhary\n",
        "# `Pandas`, Data Analysis library\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wrkhz-y-4Yax",
        "colab_type": "text"
      },
      "source": [
        "`Pandas` is a data analysis library that provides a variety of data structures and data manipulation methods that allows to perform complex tasks with simple one-line commands.\n",
        "\n",
        "In the following sections, we will be working on the `Pandas` library and its uses. We will be reviewing how the objects are defined by `pandas`, operating their values and executing a series of important operations for data analysis. Let us begin!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sDKBErWbSFAI"
      },
      "source": [
        "## 1. Installing and Using Pandas"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6MFFlPwB4YbA",
        "colab_type": "text"
      },
      "source": [
        "The downloading and installation of `pandas` library process can be done through the `pip` standard package-management system, by executing in your local environment's console this single line:\n",
        "\n",
        "`pip install pandas`\n",
        "\n",
        "Once installed, we can import it in the following way:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "9YdVFFU9R-rB",
        "outputId": "6345bc61-65ca-444e-8487-9daeeef2d0b3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "import pandas\n",
        "pandas.__version__"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'1.0.3'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w90rwwtb4Yb2",
        "colab_type": "text"
      },
      "source": [
        "It is good practice to create the alias `pd` for `pandas`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "eAPiNAf1SU0E",
        "colab": {}
      },
      "source": [
        "import pandas as pd"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "10YQkXh1Smjm"
      },
      "source": [
        "## 2. Reviewing Pandas Objects"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WTZfSc5y4YcJ",
        "colab_type": "text"
      },
      "source": [
        "At the core of the `` pandas`` library there are two fundamental data structures/objects:\n",
        "1. **`Series`**: stores single column data along with an **index**. An index is just a way to \"number\" the `Series` object.\n",
        "2. **`DataFrame`**: is a two-dimensional tabular data structure with labeled axes. It is conceptually useful to think of a `DataFrame` object as a collection of `Series` objects. That is, think of each column in a DataFrame as a single object in the `Series`, where each of these objects in the `Series` shares a common index: the index of the `DataFrame` object.\n",
        "\n",
        "Let's import the necessary libraries and we will delve into each of these `pandas` concepts:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "z-e4958eShTL",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qZmjwNm4Tv3n"
      },
      "source": [
        "### 2.1. Pandas `Series` Object\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RQKyqlDMT6CW"
      },
      "source": [
        "A Pandas ``Series`` is a one-dimensional array of indexed data.\n",
        "It can be created from a list or array as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "K9HLemZBTp8P",
        "outputId": "72ee936e-bbcc-413a-e280-579a3d0bb3a2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "data = pd.Series([0.25, 0.5, 0.75, 1.0])\n",
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    0.25\n",
              "1    0.50\n",
              "2    0.75\n",
              "3    1.00\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "c-RKCZxZUFc7"
      },
      "source": [
        "As we see in the output, the `Series` wraps both a sequence of values and a sequence of indices starting from 0 to the number of values added through the list, which we can access with the `values` and `index` attributes. The values are simply a familiar `NumPy` array:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "cBrmv_0nUE3n",
        "outputId": "42fdd9be-7c23-4453-d852-38870c71a9fc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "data.values"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.25, 0.5 , 0.75, 1.  ])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "gOQTX-vbUOoJ"
      },
      "source": [
        "The index is an array-like object of type `pd.Index`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "js6OTpRsUMUF",
        "outputId": "bc674112-a9a4-47ce-fefa-1758a38ae8e5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "data.index"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RangeIndex(start=0, stop=4, step=1)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3iq6Q7VqUUzH"
      },
      "source": [
        "Like with a NumPy array, data can be accessed by the associated index via the familiar Python square-bracket slicing notation:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "qr241H-CURiv",
        "outputId": "2e11fe8c-9aea-46aa-d5ca-de4a64bd634c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "data[1]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.5"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "P9C3vWHdUXh3",
        "outputId": "7930917f-f52d-4b39-ccf4-7f2c31c54350",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "data[1:3]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1    0.50\n",
              "2    0.75\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "uBmCmljvUw3n"
      },
      "source": [
        "As we will see, the Pandas ``Series`` is much more general and flexible than the one-dimensional NumPy array that it emulates."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-54odkZ5VsQw"
      },
      "source": [
        "#### 2.1.1. `Series` as generalized NumPy array"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "j-cCX95-V1Vx"
      },
      "source": [
        "`Numpy` Array has an implicitly defined integer index used to access the values. The Pandas `Series` has an explicitly defined `index` associated with the values."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_vqQfEH6V8QS"
      },
      "source": [
        "This explicit `index` definition gives the `Series` object additional capabilities. The `index` does not need an integer value mandatory, but can consist of values of any desired type, as we can see in the following example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "-n6zOXzwUY70",
        "outputId": "62901f0b-ccaa-4d2c-a216-d27edcf892db",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "data = pd.Series([0.25, 0.5, 0.75, 1.0],\n",
        "                 index=['a', 'b', 'c', 'd'])\n",
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    0.25\n",
              "b    0.50\n",
              "c    0.75\n",
              "d    1.00\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-ecFAE3o4YeE",
        "colab_type": "text"
      },
      "source": [
        "As we can see, we have defined the indices as characters of the English alphabet. One important fact is that the number of defined indices corresponds to the number of added values in the `Series`. When accessing the values associated with the indexes, we must consider the new index nomenclature and access the values with the common slicing:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "TVDgDkSsV0WJ",
        "outputId": "e982bfc7-6eb6-41df-bfb0-fdca83e31225",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "data['b']"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.5"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "my2sw_TJWEVd"
      },
      "source": [
        "We can even use non-contiguous or non-sequential indices (index), considering the number of defined indices corresponds to the number of added values in the `Series`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PgtEPrZmWCPd",
        "outputId": "d66195ae-ec76-4de8-84b1-dc94d8607a23",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "data = pd.Series([0.25, 0.5, 0.75, 1.0],\n",
        "                 index=[2, 5, 3, 7])\n",
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2    0.25\n",
              "5    0.50\n",
              "3    0.75\n",
              "7    1.00\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HZFMAd0V4YeZ",
        "colab_type": "text"
      },
      "source": [
        "And we access the values by slicing the indices such as:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "l7ZzVQ0NWG0_",
        "outputId": "32f052b3-7a63-4935-e676-b822c2c3134d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "data[5]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.5"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "C_GMWdaPWK3P"
      },
      "source": [
        "#### 2.1.2 `Series` as specialized dictionaries"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "FvcZnAdGWf5S"
      },
      "source": [
        "A dictionary is a structure that maps arbitrary keys to a set of arbitrary values, and a `Series` is a structure which maps typed keys to a set of typed values. We can take advantage of these similarities to create a `Series` from a dictionary, where the `keys` are the `indices` of the `Series` and the `values` are those associated with these `indices`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7redH09qWIUm",
        "outputId": "10f5c620-d940-473c-b47c-3f559c94a0c7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "population_dict = {'California': 38332521,\n",
        "                   'Texas': 26448193,\n",
        "                   'New York': 19651127,\n",
        "                   'Florida': 19552860,\n",
        "                   'Illinois': 12882135}\n",
        "population = pd.Series(population_dict)\n",
        "population"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "California    38332521\n",
              "Texas         26448193\n",
              "New York      19651127\n",
              "Florida       19552860\n",
              "Illinois      12882135\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "PX2-yPcwWp-g"
      },
      "source": [
        "From here, typical dictionary-style item access can be performed:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "J_1kNBEHWjU8",
        "outputId": "6baf6696-7905-4d83-9aff-eb63207aba07",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "population['California']"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "38332521"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "vq38Yb8lW6q1"
      },
      "source": [
        "Unlike a dictionary, the ``Series`` also supports array-style operations such as slicing:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7g1hqM_JWrsQ",
        "outputId": "2e26514f-1b9c-49e6-b151-5e49532e7fdb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "population['California':'Illinois']"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "California    38332521\n",
              "Texas         26448193\n",
              "New York      19651127\n",
              "Florida       19552860\n",
              "Illinois      12882135\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QrsnQQ33XIxD"
      },
      "source": [
        "### 2.2. Pandas `DataFrame` Object"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "csNIAbs0XWuA"
      },
      "source": [
        "A ``DataFrame`` is an analog of a two-dimensional array with both flexible row indices and flexible column names.\n",
        "Just as you might think of a two-dimensional array as an ordered sequence of aligned one-dimensional columns, you can think of a ``DataFrame`` as a sequence of aligned ``Series`` objects.\n",
        "Here, by \"aligned\" we mean that they share the same index.\n",
        "\n",
        "To demonstrate this, let's first construct a new ``Series`` listing the area of each of the five states discussed in the previous section:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "yxihQyz5W8s6",
        "outputId": "81b73001-a7a8-467a-e279-bcd7d4697f3b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "area_dict = {'California': 423967, 'Texas': 695662, 'New York': 141297,\n",
        "             'Florida': 170312, 'Illinois': 149995}\n",
        "area = pd.Series(area_dict)\n",
        "area"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "California    423967\n",
              "Texas         695662\n",
              "New York      141297\n",
              "Florida       170312\n",
              "Illinois      149995\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YxznvrIRXj_F"
      },
      "source": [
        "Now that we have this along with the ``population`` Series from before, we can use a dictionary to construct a single two-dimensional object containing this information:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "4gIwq7f7XaUo",
        "outputId": "a24b692c-8c48-49aa-9709-c83ccc402b38",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "states = pd.DataFrame({'population': population,\n",
        "                       'area': area})\n",
        "states"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>population</th>\n",
              "      <th>area</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>California</th>\n",
              "      <td>38332521</td>\n",
              "      <td>423967</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Texas</th>\n",
              "      <td>26448193</td>\n",
              "      <td>695662</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>New York</th>\n",
              "      <td>19651127</td>\n",
              "      <td>141297</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Florida</th>\n",
              "      <td>19552860</td>\n",
              "      <td>170312</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Illinois</th>\n",
              "      <td>12882135</td>\n",
              "      <td>149995</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            population    area\n",
              "California    38332521  423967\n",
              "Texas         26448193  695662\n",
              "New York      19651127  141297\n",
              "Florida       19552860  170312\n",
              "Illinois      12882135  149995"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aZPy1KcoXo2x"
      },
      "source": [
        "Like the ``Series`` object, the ``DataFrame`` has an ``index`` attribute that gives access to the index labels:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Yvymj7g4Xl03",
        "outputId": "4f0d57a7-26b2-405f-8e9e-f9ef1753ff54",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "states.index"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['California', 'Texas', 'New York', 'Florida', 'Illinois'], dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AvgpnWh1Xtq_"
      },
      "source": [
        "Additionally, the ``DataFrame`` has a ``columns`` attribute, which is an ``Index`` object holding the column labels:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Pf6XodkhXqW9",
        "outputId": "f8f58919-e311-4229-ec01-31d0bdd585e9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "states.columns"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['population', 'area'], dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3cXAlI-rX1je"
      },
      "source": [
        "Thus the ``DataFrame`` can be thought of as a generalization of a two-dimensional NumPy array, where both the rows and columns have a generalized index for accessing the data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IihmEYCs4Yf6",
        "colab_type": "text"
      },
      "source": [
        "There are some functions and attributes that allow us to observe basic information about the data stored in a `DataFrame` object:\n",
        "\n",
        "1. `DataFrame.head()` -> returns the content of the first 5 rows, by default\n",
        "2. `DataFrame.tail()` -> returns the content of the last 5 rows, by default\n",
        "3. `DataFrame.shape` -> returns a tuple of the form (num_rows, num_columns)\n",
        "4. `DataFrame.columns` -> returns the name of the columns\n",
        "5. `DataFrame.index` -> returns the index of the rows\n",
        "\n",
        "Using `` `data.head ()` `` and `` `data.tail ()` `` we can see the content of the data. Unless otherwise specified, `DataFrame` and `Series` objects have indexes starting from 0 and incrementing monotonically and incrementally as integers."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nZpzlJvr4Yf8",
        "colab_type": "code",
        "outputId": "7c898304-9ddb-4654-a649-0076838a839a",
        "colab": {}
      },
      "source": [
        "states.head(3) # The first three rows"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>population</th>\n",
              "      <th>area</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>California</th>\n",
              "      <td>38332521</td>\n",
              "      <td>423967</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Texas</th>\n",
              "      <td>26448193</td>\n",
              "      <td>695662</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>New York</th>\n",
              "      <td>19651127</td>\n",
              "      <td>141297</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            population    area\n",
              "California    38332521  423967\n",
              "Texas         26448193  695662\n",
              "New York      19651127  141297"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dGbPhrX-4YgE",
        "colab_type": "code",
        "outputId": "a99370d1-c00d-4624-ce98-a8d762d0e832",
        "colab": {}
      },
      "source": [
        "states.tail(3) # The last three rows"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>population</th>\n",
              "      <th>area</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>New York</th>\n",
              "      <td>19651127</td>\n",
              "      <td>141297</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Florida</th>\n",
              "      <td>19552860</td>\n",
              "      <td>170312</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Illinois</th>\n",
              "      <td>12882135</td>\n",
              "      <td>149995</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "          population    area\n",
              "New York    19651127  141297\n",
              "Florida     19552860  170312\n",
              "Illinois    12882135  149995"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "J9CIkU51YdET"
      },
      "source": [
        "### 2.3. Pandas `Index` Object\n",
        "\n",
        "This ``Index`` object is an interesting structure itself, and it can be thought as an *immutable array*:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "VBFPXJ_1Xwyj",
        "outputId": "732f535e-89d3-4a90-8604-d70f58e65ecf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "ind = pd.Index([2, 3, 5, 7, 11])\n",
        "ind"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Int64Index([2, 3, 5, 7, 11], dtype='int64')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bcpiAVRj4YgS",
        "colab_type": "text"
      },
      "source": [
        "This index object can be sliced as a list or a numpy array:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "YL0A6HoiYno4",
        "outputId": "fb3fe240-90eb-445b-e442-dc92ef39eda5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "ind[1] # Accessing element in position 1"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "_w6Gkc1nYpB1",
        "outputId": "79c8e54f-47ea-4ef8-d9ba-ee457477c3f3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "ind[::2] # Accessing elements starting from position 0 through all the elements two by two."
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Int64Index([2, 5, 11], dtype='int64')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "STd6x0Zw4Ygf",
        "colab_type": "text"
      },
      "source": [
        "Some common attributes with `Numpy` arrays:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "SdPm35JMYqju",
        "outputId": "4643ce78-6d2d-42b6-a3fe-c9d6c6de7d66",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(' Size:',ind.size,'\\n',\n",
        "      'Shape:',ind.shape,'\\n',\n",
        "      'Dimension:',ind.ndim,'\\n', \n",
        "      'Data type:',ind.dtype)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " Size: 5 \n",
            " Shape: (5,) \n",
            " Dimension: 1 \n",
            " Data type: int64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AOB8_VkqYyB8"
      },
      "source": [
        "One difference between ``Index`` objects and `Numpy` arrays is that indices are **immutable**. That is, they cannot be modified via the normal means, which will cause an error:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "HFy9E1v1YtEY",
        "outputId": "b73c3c18-823c-43af-d574-95c802de056e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        }
      },
      "source": [
        "ind[1] = 0"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "Index does not support mutable operations",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-50-906a9fa1424c>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mind\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[1;32m~\\Anaconda3\\envs\\DS_Practice_Python\\lib\\site-packages\\pandas\\core\\indexes\\base.py\u001b[0m in \u001b[0;36m__setitem__\u001b[1;34m(self, key, value)\u001b[0m\n\u001b[0;32m   4258\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   4259\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m__setitem__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 4260\u001b[1;33m         \u001b[1;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Index does not support mutable operations\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   4261\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   4262\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m__getitem__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;31mTypeError\u001b[0m: Index does not support mutable operations"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "NlKz6ms_azwG"
      },
      "source": [
        "## 3. Data Indexing and Selection"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QwSgdcxm4Ygw",
        "colab_type": "text"
      },
      "source": [
        "Let's see in detail how to access the elements of the `Series` and `DataFrames` objects."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "leTpYZAXa442"
      },
      "source": [
        "### 3.1. Data Selection in `Series`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a7uKXu1W4Yg1",
        "colab_type": "text"
      },
      "source": [
        "Let's redefine a `Series` object for explanatory purposes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IwqXVDfyYz0S",
        "outputId": "e7cd844a-f0bf-4a4f-9265-1264a4186fdb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "data = pd.Series([0.25, 0.5, 0.75, 1.0],\n",
        "                 index=['a', 'b', 'c', 'd'])\n",
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    0.25\n",
              "b    0.50\n",
              "c    0.75\n",
              "d    1.00\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E2rAGgy_4Yg7",
        "colab_type": "text"
      },
      "source": [
        "In addition to accessing the elements of a `Series` through slicing the indexes, we can update the values associated with them as:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N4GEiF3Q4Yg8",
        "colab_type": "code",
        "outputId": "53b9997a-9c3c-4788-9a94-537a88564720",
        "colab": {}
      },
      "source": [
        "data['d'] = 0.95 # updating the value associated with 'd' index in Series object\n",
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    0.25\n",
              "b    0.50\n",
              "c    0.75\n",
              "d    0.95\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wbDAa1WJ4YhA",
        "colab_type": "text"
      },
      "source": [
        "We can also add a new value using the same procedure, just as we would do with a dictionary:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "LI3ZI3DibAc8",
        "outputId": "bbf5da64-71a5-4108-d922-9e5aed41f7be",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "data['e'] = 1.25 # New value added with 'e' index in Series object\n",
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    0.25\n",
              "b    0.50\n",
              "c    0.75\n",
              "d    0.95\n",
              "e    1.25\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sG5WWUqB4YhJ",
        "colab_type": "text"
      },
      "source": [
        "Slicing access can be done *explicitly* or *implicitly* through a range, such that:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "z0-Eh-lebFL8",
        "outputId": "e5eebbbf-6500-4235-c372-0d18f37c19eb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "# slicing by explicit index\n",
        "data['a':'c']"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    0.25\n",
              "b    0.50\n",
              "c    0.75\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "h8uwmr0HbLvh",
        "outputId": "2a943e1b-2820-4d0d-80a3-5554d8b0a7eb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# slicing by implicit integer index\n",
        "data[0:2]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    0.25\n",
              "b    0.50\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RhaWB93v4YhZ",
        "colab_type": "text"
      },
      "source": [
        "We can access a group of specific indices by including these indices in a list, in such a way that access is done as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "2960iQtybM2v",
        "outputId": "81177db6-8bfb-470e-e240-c590d0c754eb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# express indexing\n",
        "data[['a', 'e']]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    0.25\n",
              "e    1.25\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ERgbiCCSbVhp"
      },
      "source": [
        "Notice that when slicing with an explicit index (i.e., `data ['a':'c']`), the final index is included in the slice, while when slicing with an implicit index (i.e., `data[0:2]`), the final index is excluded from the slice. When we slicing through a list (i.e., `data [['a', 'e']]`), all indices are equally accessed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bAVfBKEJbYxc"
      },
      "source": [
        "### 3.2. Indexers: loc, iloc for `Series`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y3heiMk34Yhh",
        "colab_type": "text"
      },
      "source": [
        "We are going to review two ways to access the elements of a `Series`, using two attributes determined for this: `.loc[]` and `.iloc[]`. Let's first define a `Series` of three string elements and character indices:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Zsue-7dcbPn8",
        "outputId": "fd1eb40d-e5c7-431b-e3bd-bcfbde1c76e7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "data = pd.Series(['Hello', 'DPhi', 'world'], index=['a', 'b', 'c'])\n",
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    Hello\n",
              "b     DPhi\n",
              "c    world\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DE-cLr7C4Yho",
        "colab_type": "text"
      },
      "source": [
        "#### 3.2.1. `loc` attribute"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0On5fbdLdUs_"
      },
      "source": [
        "The ``loc`` attribute allows indexing and slicing that always references the explicit index (the explicit name of the index):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ug4ownUTdQHn",
        "outputId": "38e6b546-b41f-436c-bb06-0414864f466b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "data.loc['a']"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Hello'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xxEkJjcqdYoA",
        "outputId": "d81d9e80-8b41-416d-aa02-c7406cbcfa32",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "data.loc['a':'c']"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    Hello\n",
              "b     DPhi\n",
              "c    world\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 63
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TjRjxJMc4Yh0",
        "colab_type": "text"
      },
      "source": [
        "#### 3.2.2. `iloc` attribute"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "e7fxH_KIddJM"
      },
      "source": [
        "The ``iloc`` attribute allows indexing and slicing that always references the implicit Python-style index (the direct row number of the values monotonically increasing from 0 to 3):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "8H3LFd1VdZuQ",
        "outputId": "b538fedf-c955-4d27-f548-7214bac9d380",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "data.iloc[1]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'DPhi'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "0nS4PzRqdfDk",
        "outputId": "168e9cae-73e7-4c5e-978a-de417d732ce5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "data.iloc[1:3]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "b     DPhi\n",
              "c    world\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wf7N69xM4YiA",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 1\n",
        "\n",
        "Consider the following lists:\n",
        "```\n",
        "lst1 = [1, 2, 3, 5, 8]\n",
        "lst2 = [8, 5, 3, 2, 1]\n",
        "```\n",
        "\n",
        "1. Create and display two individual `Series` objects `s1` and `s2` from the data available on each list.\n",
        "\n",
        "\n",
        "2. Perform the following operations with the two series (element-wise):\n",
        "    1. Add `s1` and `s2` and store the result in a new variable `s3_add`\n",
        "    2. Subtract `s2` from `s1` and store the result in a new variable `s3_sub`\n",
        "    3. Multiply `s1` and `s2` and store the result in a new variable `s3_mul`\n",
        "    4. Divide `s1` by `s2` and store the result in a new variable `s3_div`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2AWgOpXA4YiB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "s1 = pd.Series([1,2,3,5,8])\n",
        "s2 = pd.Series([8,5,3,2,1])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mNqlJvJL4YiG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "s3_add = s2.add(s1)\n",
        "s3_sub  = s2.sub(s1)\n",
        "s3_mul = s2.mul(s1)\n",
        "s3_div = s2.div(s1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q2k--y9l4YiK",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 2\n",
        "\n",
        "Consider the following `Series` object:\n",
        "```\n",
        "0    45000\n",
        "1    37872\n",
        "2    57923\n",
        "3    68979\n",
        "4    78934\n",
        "5    69897\n",
        "6    56701\n",
        "Name: Amazon_Reviews, dtype: int64\n",
        "```\n",
        "\n",
        "1. Create and display the `Amazon_Reviews` Series.\n",
        "\n",
        "2. Get the last three values from `Amazon_Reviews` using negative indexing."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tIR8kNzT4YiK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 156
        },
        "outputId": "78dd972c-f1a1-4a61-9061-9535c7e893a0"
      },
      "source": [
        "data = pd.Series([45000,37872,57923,68979,78934,69897,56701],name='Amazon_reviews',index =[0, 1, 2, 3, 4, 5,6])\n",
        "data"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    45000\n",
              "1    37872\n",
              "2    57923\n",
              "3    68979\n",
              "4    78934\n",
              "5    69897\n",
              "6    56701\n",
              "Name: Amazon_reviews, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mivpP0G-4YiQ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "071408d2-6bb4-4f9c-a367-92daab0880de"
      },
      "source": [
        "data[6:3:-1]"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "6    56701\n",
              "5    69897\n",
              "4    78934\n",
              "Name: Amazon_reviews, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UUJmepUT4YiV",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 3\n",
        "\n",
        "Consider the following dictionary which is relating the area in sq units of some USA states: \n",
        "```\n",
        "    area_dict = {'California': 423967, 'Texas': 695662, 'New York': 141297,\n",
        "             'Florida': 170312, 'Illinois': 149995}\n",
        "```\n",
        "\n",
        "1. Create a `Series` using the given dictionary\n",
        "2. Extract areas for 'Texas', 'New York',  and 'Florida' from the created series"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rFwk8IFW4YiX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data = {'California': 423967, 'Texas': 695662, 'New York': 141297,\n",
        "             'Florida': 170312, 'Illinois': 149995}\n",
        "area_dict = pd.Series(data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jqrxzuw_4Yid",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "339aa2ce-2025-4a4b-c505-f4a48fb4cc93"
      },
      "source": [
        "area_dict[1:4]"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Texas       695662\n",
              "New York    141297\n",
              "Florida     170312\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "L57zEGqtenG5"
      },
      "source": [
        "### 3.3. Data Selection in `DataFrame`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vRAutQdn4Yiq",
        "colab_type": "text"
      },
      "source": [
        "Let's see in detail how to access the elements of the `DataFrame` objects, redefine a `DataFrame` object for explanatory purposes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "x9ax5WMldgDc",
        "outputId": "90f0e65e-0b0c-40d6-ad6c-9bc343b159d2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "area = pd.Series({'California': 423967, 'Texas': 695662,\n",
        "                  'New York': 141297, 'Florida': 170312,\n",
        "                  'Illinois': 149995})\n",
        "pop = pd.Series({'California': 38332521, 'Texas': 26448193,\n",
        "                 'New York': 19651127, 'Florida': 19552860,\n",
        "                 'Illinois': 12882135})\n",
        "data = pd.DataFrame({'area':area, 'pop':pop})\n",
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>area</th>\n",
              "      <th>pop</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>California</th>\n",
              "      <td>423967</td>\n",
              "      <td>38332521</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Texas</th>\n",
              "      <td>695662</td>\n",
              "      <td>26448193</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>New York</th>\n",
              "      <td>141297</td>\n",
              "      <td>19651127</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Florida</th>\n",
              "      <td>170312</td>\n",
              "      <td>19552860</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Illinois</th>\n",
              "      <td>149995</td>\n",
              "      <td>12882135</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              area       pop\n",
              "California  423967  38332521\n",
              "Texas       695662  26448193\n",
              "New York    141297  19651127\n",
              "Florida     170312  19552860\n",
              "Illinois    149995  12882135"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Y6V7L6dFesh8"
      },
      "source": [
        "The individual ``Series`` that make up the columns of the ``DataFrame`` can be accessed via dictionary-style indexing of the column name:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "eGABmVWSeprS",
        "outputId": "7e36b468-e301-4286-978a-5be8444855f8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "data['area'] # Accessing the 'area' column of the data DataFrame"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "California    423967\n",
              "Texas         695662\n",
              "New York      141297\n",
              "Florida       170312\n",
              "Illinois      149995\n",
              "Name: area, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MAusC4y14Yi0",
        "colab_type": "code",
        "outputId": "f04f7a23-08ff-4c2d-c18c-7a0488b7b2f4",
        "colab": {}
      },
      "source": [
        "data['pop'] # Accessing the 'pop' column of the data DataFrame"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "California    38332521\n",
              "Texas         26448193\n",
              "New York      19651127\n",
              "Florida       19552860\n",
              "Illinois      12882135\n",
              "Name: pop, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BZUAgmMUe0LX"
      },
      "source": [
        "Equivalently, we can use attribute-style access with column names that are strings, which will result in the exact same `Series` output:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "TVe3LNpLerhv",
        "outputId": "5c16a56a-a382-4af3-f4f5-4dd353fa71bb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "data.area # Equivalent to data['area']"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "California    423967\n",
              "Texas         695662\n",
              "New York      141297\n",
              "Florida       170312\n",
              "Illinois      149995\n",
              "Name: area, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dY3Is2CQe6aT"
      },
      "source": [
        "Like with the ``Series`` objects discussed earlier, this dictionary-style syntax can also be used to modify the object, in this case adding a new column:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "QlxFQv-ae16T",
        "outputId": "0abb7e29-5454-457d-e4b9-a82bd58a4947",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "data['density'] = data['pop']/data['area']\n",
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>area</th>\n",
              "      <th>pop</th>\n",
              "      <th>density</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>California</th>\n",
              "      <td>423967</td>\n",
              "      <td>38332521</td>\n",
              "      <td>90.413926</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Texas</th>\n",
              "      <td>695662</td>\n",
              "      <td>26448193</td>\n",
              "      <td>38.018740</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>New York</th>\n",
              "      <td>141297</td>\n",
              "      <td>19651127</td>\n",
              "      <td>139.076746</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Florida</th>\n",
              "      <td>170312</td>\n",
              "      <td>19552860</td>\n",
              "      <td>114.806121</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Illinois</th>\n",
              "      <td>149995</td>\n",
              "      <td>12882135</td>\n",
              "      <td>85.883763</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              area       pop     density\n",
              "California  423967  38332521   90.413926\n",
              "Texas       695662  26448193   38.018740\n",
              "New York    141297  19651127  139.076746\n",
              "Florida     170312  19552860  114.806121\n",
              "Illinois    149995  12882135   85.883763"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 94
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_MOeJGiv4YjG",
        "colab_type": "text"
      },
      "source": [
        "As you can see, when the `Series` 'pop' is accessed and divided over the `Series` 'area', the arithmetic operation becomes element-wise and the result is assigned to the new `Series` 'density', which becomes the third column of the `DataFrame` `data`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YyAP8ug0fKHw"
      },
      "source": [
        "We can also view the ``DataFrame`` as an enhanced two-dimensional array.\n",
        "We can examine the raw underlying data array using the ``values`` attribute, which will return a two-dimensional array in which each row corresponds to a row of `DataFrame` values:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "i0APM5L9e8YT",
        "outputId": "00d06299-5730-4148-9ff5-dda3b68641a1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "data.values"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[4.23967000e+05, 3.83325210e+07, 9.04139261e+01],\n",
              "       [6.95662000e+05, 2.64481930e+07, 3.80187404e+01],\n",
              "       [1.41297000e+05, 1.96511270e+07, 1.39076746e+02],\n",
              "       [1.70312000e+05, 1.95528600e+07, 1.14806121e+02],\n",
              "       [1.49995000e+05, 1.28821350e+07, 8.58837628e+01]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "j03HjPyJ4YjP"
      },
      "source": [
        "### 3.4. Indexers: loc, iloc for `DataFrame`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RxpcQ-Bm4YjQ",
        "colab_type": "text"
      },
      "source": [
        "We are going to review two ways to access the elements of a `DataFrame`, using two attributes determined for this: `.loc[]` and `.iloc[]`:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XGws49kM4YjQ",
        "colab_type": "text"
      },
      "source": [
        "#### 3.4.1. `loc` attribute"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ssjo7pr04YjR"
      },
      "source": [
        "The ``loc`` attribute allows indexing and slicing that always references the explicit index (the explicit name of the index):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "c8kPXGCCfNX1",
        "outputId": "d2320b0c-30b1-43f2-bdde-3a465d3a079b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "data.loc[:'Illinois', :'pop']"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>area</th>\n",
              "      <th>pop</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>California</th>\n",
              "      <td>423967</td>\n",
              "      <td>38332521</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Texas</th>\n",
              "      <td>695662</td>\n",
              "      <td>26448193</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>New York</th>\n",
              "      <td>141297</td>\n",
              "      <td>19651127</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Florida</th>\n",
              "      <td>170312</td>\n",
              "      <td>19552860</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Illinois</th>\n",
              "      <td>149995</td>\n",
              "      <td>12882135</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              area       pop\n",
              "California  423967  38332521\n",
              "Texas       695662  26448193\n",
              "New York    141297  19651127\n",
              "Florida     170312  19552860\n",
              "Illinois    149995  12882135"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j-nMK9ym4YjW",
        "colab_type": "text"
      },
      "source": [
        "In this example we are slicing on the indices starting with the first by default and ending in 'Illinois', as well as by the columns starting with the first by default, and ending in 'pop'."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8t64Og0V4YjW",
        "colab_type": "text"
      },
      "source": [
        "#### 3.4.2. `iloc` attribute"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "eyIhQv7b4YjX"
      },
      "source": [
        "The ``iloc`` attribute allows indexing and slicing that always references the implicit Python-style index (the direct row number of the values monotonically increasing from 0 to 3):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "2B-P38C7e_86",
        "outputId": "e15b461e-428a-4886-a54a-c6945138a8d2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        }
      },
      "source": [
        "data.iloc[:3, :2]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>area</th>\n",
              "      <th>pop</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>California</th>\n",
              "      <td>423967</td>\n",
              "      <td>38332521</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Texas</th>\n",
              "      <td>695662</td>\n",
              "      <td>26448193</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>New York</th>\n",
              "      <td>141297</td>\n",
              "      <td>19651127</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              area       pop\n",
              "California  423967  38332521\n",
              "Texas       695662  26448193\n",
              "New York    141297  19651127"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3lvk2LdE4Yjb",
        "colab_type": "text"
      },
      "source": [
        "In this example we are slicing the `DataFrame` from index 0 by default to 3 exlusive, and from column 0 by default to 2 exlusive."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bW8WtyPf4Yjc",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 4\n",
        "\n",
        "Consider below DPhi Bootcamp's information about different batches:\n",
        "\n",
        "```\n",
        "Total_Candidates = {'absolute_beginners': 785, 'beginners': 825, 'intermediat_advanced': 602} # this is true data\n",
        "Active_Candidates = {'absolute_beginners': 500, 'beginners': 825, 'intermediat_advanced': 602}  # this is hypothetical data\n",
        "```\n",
        "    \n",
        "1. Create a Pandas `DataFrame` using above information (name your Dataframe as `DPhi`)\n",
        "2. Get all the columns in DPhi.\n",
        "3. Get the information of total candidates present in each batches using dictionary-style indexing.\n",
        "4. Find the number of candidates for each batches who are not active and add this information to the dataframe DPhi.\n",
        "5. Also, find the percent of candidates that are active in each batches and add this information to the `DPhi` dataframe (hint: $percent = (active / total)* 100$)\n",
        "6. Get all the batches where percentage of active candidates are greater than 60%"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bo9YmD0k4Yjc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "outputId": "b5ab8a77-f6d4-4d10-d483-df6becc70cf9"
      },
      "source": [
        "Total_Candidates = {'absolute_beginners': 785, 'beginners': 825, 'intermediat_advanced': 602}\n",
        "Active_Candidates ={'absolute_beginners': 500, 'beginners': 825, 'intermediat_advanced': 602}\n",
        "Dphi = pd.DataFrame([Total_Candidates,Active_Candidates],index=['Total_Candidates','Active_Candidates'])\n",
        "Dphi"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>absolute_beginners</th>\n",
              "      <th>beginners</th>\n",
              "      <th>intermediat_advanced</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Total_Candidates</th>\n",
              "      <td>785</td>\n",
              "      <td>825</td>\n",
              "      <td>602</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Active_Candidates</th>\n",
              "      <td>500</td>\n",
              "      <td>825</td>\n",
              "      <td>602</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                   absolute_beginners  beginners  intermediat_advanced\n",
              "Total_Candidates                  785        825                   602\n",
              "Active_Candidates                 500        825                   602"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RQTdS_Gz4Yjg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "597f003e-9946-4568-954b-a3135ffd306b"
      },
      "source": [
        "data.columns"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['absolute_beginners', 'beginners', 'intermediat_advanced'], dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9ncJeYFD4Yjm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "b64e2937-d4a7-4060-b5a8-7387d0af4b37"
      },
      "source": [
        "Dphi.loc['Total_Candidates']"
      ],
      "execution_count": 87,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "absolute_beginners      785.0\n",
              "beginners               825.0\n",
              "intermediat_advanced    602.0\n",
              "Name: Total_Candidates, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 87
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AQxVWqRg4Yjr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "outputId": "0d4ec870-804f-41ec-a976-0cc1c872b62b"
      },
      "source": [
        "Dphi.loc['Not active'] = Dphi.loc['Total_Candidates']-Dphi.loc['Active_Candidates']\n",
        "Dphi"
      ],
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>absolute_beginners</th>\n",
              "      <th>beginners</th>\n",
              "      <th>intermediat_advanced</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Total_Candidates</th>\n",
              "      <td>785.0</td>\n",
              "      <td>825.0</td>\n",
              "      <td>602.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Active_Candidates</th>\n",
              "      <td>500.0</td>\n",
              "      <td>825.0</td>\n",
              "      <td>602.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Not active</th>\n",
              "      <td>285.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                   absolute_beginners  beginners  intermediat_advanced\n",
              "Total_Candidates                785.0      825.0                 602.0\n",
              "Active_Candidates               500.0      825.0                 602.0\n",
              "Not active                      285.0        0.0                   0.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PcRQNPqP4Yjv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "outputId": "6c16dadd-e8a5-4f94-b62f-a49555979a48"
      },
      "source": [
        "Dphi.loc['Percent'] = (Dphi.loc['Active_Candidates']/Dphi.loc['Total_Candidates'])*100\n",
        "Dphi"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>absolute_beginners</th>\n",
              "      <th>beginners</th>\n",
              "      <th>intermediat_advanced</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Total_Candidates</th>\n",
              "      <td>785.000000</td>\n",
              "      <td>825.0</td>\n",
              "      <td>602.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Active_Candidates</th>\n",
              "      <td>500.000000</td>\n",
              "      <td>825.0</td>\n",
              "      <td>602.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Not active</th>\n",
              "      <td>285.000000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Percent</th>\n",
              "      <td>63.694268</td>\n",
              "      <td>100.0</td>\n",
              "      <td>100.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                   absolute_beginners  beginners  intermediat_advanced\n",
              "Total_Candidates           785.000000      825.0                 602.0\n",
              "Active_Candidates          500.000000      825.0                 602.0\n",
              "Not active                 285.000000        0.0                   0.0\n",
              "Percent                     63.694268      100.0                 100.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iM2NQ_wD4Yjy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "b931f91c-4be8-4726-8e7d-ff79da0a4a45"
      },
      "source": [
        "Dphi.loc['Percent'] >60"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "absolute_beginners      True\n",
              "beginners               True\n",
              "intermediat_advanced    True\n",
              "Name: Percent, dtype: bool"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hvWWojeT4Yj1",
        "colab_type": "text"
      },
      "source": [
        "### 3.5. Subsetting a `Dataframe`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dcYHSBCg4Yj1",
        "colab_type": "text"
      },
      "source": [
        "**Subsetting** a `DataFrame` is a way of filtering which allows to extract portions of interest. Subsetting can be done using comparison operators and logical operators inside a pair of square brackets `[]` as shown in the following example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "V4Wd7z9VfpTU",
        "outputId": "4ddbd90a-8015-41c9-8f57-0cd3b1b4c7ca",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 111
        }
      },
      "source": [
        "data[data['density'] > 100]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>area</th>\n",
              "      <th>pop</th>\n",
              "      <th>density</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>New York</th>\n",
              "      <td>141297</td>\n",
              "      <td>19651127</td>\n",
              "      <td>139.076746</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Florida</th>\n",
              "      <td>170312</td>\n",
              "      <td>19552860</td>\n",
              "      <td>114.806121</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            area       pop     density\n",
              "New York  141297  19651127  139.076746\n",
              "Florida   170312  19552860  114.806121"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 95
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ovv6OdAM4Yj9",
        "colab_type": "text"
      },
      "source": [
        "In the above example, we are extracting the rows for which it is `True` that the population density is greater than 100 units. For more clarification on the logical operation concept, take a look to the following extract of the above example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ECH4sISH4Yj-",
        "colab_type": "code",
        "outputId": "f5933cd3-28e0-4b5e-a890-bf883543a93a",
        "colab": {}
      },
      "source": [
        "data['density'] > 100"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "California    False\n",
              "Texas         False\n",
              "New York       True\n",
              "Florida        True\n",
              "Illinois      False\n",
              "Name: density, dtype: bool"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 101
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fP3mrwdL4YkQ",
        "colab_type": "text"
      },
      "source": [
        "Since there are only two rows where the density is greater than 100, the result will be a boolean-values `DataFrame` in which the `True` values correspond to the rows that accomplish the locigal expression, and the `False` values to the ones that do not.\n",
        "\n",
        "Let's see an example in which we include a second logical operation to select those rows in which the population density is greater than 100 units `data['density'] > 100` and (`&`) the area is less than 150,000 units `data['area'] < 150000`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CIqUWhFJ4YkQ",
        "colab_type": "code",
        "outputId": "b8242008-7baa-4f10-ccc5-46e01868ab5c",
        "colab": {}
      },
      "source": [
        "data[(data['density'] > 100) & (data['area'] < 150000)]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>area</th>\n",
              "      <th>pop</th>\n",
              "      <th>density</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>New York</th>\n",
              "      <td>141297</td>\n",
              "      <td>19651127</td>\n",
              "      <td>139.076746</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            area       pop     density\n",
              "New York  141297  19651127  139.076746"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0JDphJAh4Ykf",
        "colab_type": "text"
      },
      "source": [
        "We could also select those records in which the population density is less than 90 units `data['density'] < 90` or (`|`) greater than 120 `data['density'] > 120`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tXg6SXwl4Ykg",
        "colab_type": "code",
        "outputId": "2a6ae244-3107-4006-a4f4-4c23c07a49d0",
        "colab": {}
      },
      "source": [
        "data[(data['density'] < 90) | (data['density'] > 120)]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>area</th>\n",
              "      <th>pop</th>\n",
              "      <th>density</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Texas</th>\n",
              "      <td>695662</td>\n",
              "      <td>26448193</td>\n",
              "      <td>38.018740</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>New York</th>\n",
              "      <td>141297</td>\n",
              "      <td>19651127</td>\n",
              "      <td>139.076746</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Illinois</th>\n",
              "      <td>149995</td>\n",
              "      <td>12882135</td>\n",
              "      <td>85.883763</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            area       pop     density\n",
              "Texas     695662  26448193   38.018740\n",
              "New York  141297  19651127  139.076746\n",
              "Illinois  149995  12882135   85.883763"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 98
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8v4Qip0V4Ykj",
        "colab_type": "text"
      },
      "source": [
        "The previous example can be rewritten to express the negation of the above condition, select those records that do not have a population density greater than or equal to 90 units and less than or equal to 120 units `~((data['density'] >= 90) & (data['density'] <= 120))`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hjgao8sS4Ykm",
        "colab_type": "code",
        "outputId": "2c1f0c5d-af31-44b3-9732-5595a3010105",
        "colab": {}
      },
      "source": [
        "data[~((data['density'] >= 90) & (data['density'] <= 120))]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>area</th>\n",
              "      <th>pop</th>\n",
              "      <th>density</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Texas</th>\n",
              "      <td>695662</td>\n",
              "      <td>26448193</td>\n",
              "      <td>38.018740</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>New York</th>\n",
              "      <td>141297</td>\n",
              "      <td>19651127</td>\n",
              "      <td>139.076746</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Illinois</th>\n",
              "      <td>149995</td>\n",
              "      <td>12882135</td>\n",
              "      <td>85.883763</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            area       pop     density\n",
              "Texas     695662  26448193   38.018740\n",
              "New York  141297  19651127  139.076746\n",
              "Illinois  149995  12882135   85.883763"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 100
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bSoPEfV-4Ykw",
        "colab_type": "text"
      },
      "source": [
        "As we can see, the results of the last two examples are the same, since we are expressing the same condition in different ways."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IetLY6DI4Yky",
        "colab_type": "text"
      },
      "source": [
        "## 4. Data Wrangling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GlFIkoE0hWFO"
      },
      "source": [
        "The difference between data found in many tutorials and data from the real world is that real-world data is rarely clean and homogeneous. In particular, many interesting datasets will have some amount of data missing. To make matters even more complicated, different data sources may indicate missing data in different ways. \n",
        "\n",
        "In this way, we need to define methods that allow us to structure, clean and enrich the data acquired from the real world, which are the main steps for **Data Wrangling**. Before continuing, let's see what is the difference between these three steps and expand their definition:\n",
        "\n",
        "**1. Data structuring:** \n",
        "\n",
        "The first step in the data wrangling process is to separate the relevant data into multiple columns, so that the analysis can be run grouping by common values in a separate way. In turn, if there are columns that are not desired or that will not be relevant to the analysis, this is the phase to filter the data or mix together some of their columns.\n",
        "\n",
        "**2. Data Cleaning**\n",
        "\n",
        "In this step, the data is cleaned up for high-quality analysis. `Null values` are handled, and the data format is standardized. We will enter this process in the following weeks.\n",
        "\n",
        "**3. Data Enriching**\n",
        "\n",
        "After cleaning, the data is enriched by increasing some variables in what is known as *Data Augmentation* and using additional sources to enrich them for the following stages of processing.\n",
        "\n",
        "For now, we will review how to handle missing values, a fundamental step for data cleaning."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3nI-tc-igqYV"
      },
      "source": [
        "## 5. Handling Missing Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BifKbk8y4Yk0",
        "colab_type": "text"
      },
      "source": [
        "This is a fundamental step in data cleaning. It is common that during the data acquisition processes, there are lost records, either due to the difficulties of acquiring them, due to errors in the source or destination, or because we simply could not acquire the data. There are three types of missing data:\n",
        "\n",
        "- Missing completely at random (MCAR): when the fact that the data is missing is independent of the observed and unobserved data.\n",
        "- Missing at random (MAR): when the fact that the data is missing is systematically related to the observed but not the unobserved data.\n",
        "- Missing not at random (MNAR): when the missingness of data is related to events or factors which are not measured by the researcher.\n",
        "\n",
        "We will go into these types in detail later. For now, we'll look at the fundamentals of handling missing data in pandas:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6uJo8zUKhjt7"
      },
      "source": [
        "### 5.1. `NaN` and `None` in Pandas\n",
        "\n",
        "Missing data is handled in Pandas as `NaN` values placeholders. `NaN` value is a IEEE 754 floating point representation of *Not a Number (NaN)*. One of the main reasons to handle missing data as `NaN` rather than `Null` in Pandas is that `NaN` (from `np.nan`) allows for vectorized operations, since it is a float value. `None`, by definition, forces object type, which basically disables all efficiency in Numpy and Pandas.\n",
        "\n",
        "``NaN`` and ``None`` are handled nearly interchangeably by `Pandas`, converting between them where appropriate:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "DldHcdFlhn4H",
        "outputId": "7eb2775a-0ba3-401d-db87-776e92c09657",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "pd.Series([1, np.nan, 2, None])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    1.0\n",
              "1    NaN\n",
              "2    2.0\n",
              "3    NaN\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pqSmI4JXhx1Y"
      },
      "source": [
        "### 5.2. Operations on Missing Values\n",
        "\n",
        "There are several useful methods for detecting, removing, and replacing missing values in Pandas data structures:\n",
        "\n",
        "- ``isnull()``: generates a boolean mask indicating missing values\n",
        "- ``notnull()``: generates a boolean mask of non-missing values. Is the opposite of ``isnull()``.\n",
        "- ``dropna()``: returns a filtered version of the data, without missing values.\n",
        "- ``fillna()``: returns a copy of the data with missing values filled or imputed with a desired strategy.\n",
        "\n",
        "Let's review some examples of the first two functions `isnull()` and `notnull()`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "slqzJonWhoS_",
        "outputId": "3895b095-5b77-4c64-d715-3958f3843aa4",
        "colab": {}
      },
      "source": [
        "data = pd.Series([1, np.nan, 'hello', None])\n",
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0        1\n",
              "1      NaN\n",
              "2    hello\n",
              "3     None\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 103
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "rGO3h34qh97l",
        "outputId": "47bff3a5-1f8a-4cad-ff04-db4328dd6116",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "data.isnull()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    False\n",
              "1     True\n",
              "2    False\n",
              "3     True\n",
              "dtype: bool"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u_4Vt-7k4YlR",
        "colab_type": "code",
        "outputId": "5bd37e97-333c-464c-c74a-00249c8154ab",
        "colab": {}
      },
      "source": [
        "data.notnull()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0     True\n",
              "1    False\n",
              "2     True\n",
              "3    False\n",
              "dtype: bool"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 104
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tbqGpJzV4Ylf",
        "colab_type": "text"
      },
      "source": [
        "As you can see, the `.isnull()` function returns a Dataframe with boolean values, where `False` denotes a present value and `True` denotes a missing value. Conversely, the `.notnull()` function returns a Dataframe with boolean values, where `True` denotes a present value and `False` denotes a missing value.\n",
        "\n",
        "With this boolean result we can make a subsetting to filter those missing values:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "utfaW2xoh_0Z",
        "outputId": "335092e8-85c7-4103-dbc5-3d228bde811e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "data[data.notnull()]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0        1\n",
              "2    hello\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 105
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S3PVa2NN4Ylj",
        "colab_type": "code",
        "outputId": "ceb1c7d6-e5e1-43a0-f896-707f1e382ce6",
        "colab": {}
      },
      "source": [
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0        1\n",
              "1      NaN\n",
              "2    hello\n",
              "3     None\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 106
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rxSv_Aqe4Ylm",
        "colab_type": "text"
      },
      "source": [
        "Although we have filtered the missing data, if we review the content of the variable again, we will see that the missing data persists. This is because no subsetting operation is done inplace. Now let's see how we remove missing data from our dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8ryJN1THiIDO"
      },
      "source": [
        "### 5.3. Dropping missing values\n",
        "\n",
        "The basic function to remove any missing values from a `Series` object is as follows, although the function is not executed inplace:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "vbGhWLdxiESs",
        "outputId": "fdc6d942-8c49-4a3f-9bac-696c67875703",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "data.dropna()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0        1\n",
              "2    hello\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JCYW3K1i4Ylr",
        "colab_type": "code",
        "outputId": "67cff150-5086-47b0-a94c-a0307580fbd5",
        "colab": {}
      },
      "source": [
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0        1\n",
              "1      NaN\n",
              "2    hello\n",
              "3     None\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 107
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MDS7T1AYiUoS"
      },
      "source": [
        "For a ``DataFrame``, there are more options.\n",
        "Consider the following ``DataFrame``:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "z0JKzK1piP3D",
        "outputId": "30b9ac69-1e5c-41d0-9275-cce33cece442",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        }
      },
      "source": [
        "df = pd.DataFrame([[1,      np.nan, 2],\n",
        "                   [2,      3,      5],\n",
        "                   [np.nan, 4,      6]])\n",
        "df"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     0    1  2\n",
              "0  1.0  NaN  2\n",
              "1  2.0  3.0  5\n",
              "2  NaN  4.0  6"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "G_m-crjHia3f"
      },
      "source": [
        "We cannot drop single values from a ``DataFrame``; we can only drop full rows or full columns.\n",
        "Depending on the application, you might want one or the other, so ``dropna()`` gives a number of options for a ``DataFrame``.\n",
        "\n",
        "By default, ``dropna()`` will drop all rows in which *any* missing value is present:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Kk9cwOY8iWtf",
        "outputId": "b5aa9503-da17-4df7-dcb7-0a98b69b985e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        }
      },
      "source": [
        "df.dropna()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     0    1  2\n",
              "1  2.0  3.0  5"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5c-N5KhgigFR"
      },
      "source": [
        "Alternatively, you can drop missing values along a different axis; `axis=1` drops all columns containing a missing value:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RccLI6QZic5h",
        "outputId": "81e405ab-2cf7-43cc-fb73-7e3c8f272222",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        }
      },
      "source": [
        "df.dropna(axis='columns')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   2\n",
              "0  2\n",
              "1  5\n",
              "2  6"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "F1AZmZXAiqU3"
      },
      "source": [
        "But this drops some good data as well; you might rather be interested in dropping rows or columns with *all* `NaN` values, or a majority of `NaN` values. This can be specified through the ``how`` or ``thresh`` parameters, which allow fine control of the number of nulls to allow through.\n",
        "\n",
        "The default is ``how='any'``, such that any row or column (depending on the ``axis`` keyword) containing a null value will be dropped. You can also specify ``how='all'``, which will only drop rows/columns that are *all* null values:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "jKYG2VHniiUN",
        "outputId": "83dbbb53-b0f2-42ca-e711-faeb013f26c9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        }
      },
      "source": [
        "df = pd.DataFrame([[1,      np.nan, 2, np.nan],\n",
        "                   [2,      3,      5, np.nan],\n",
        "                   [np.nan, 4,      6, np.nan]])\n",
        "df"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     0    1  2   3\n",
              "0  1.0  NaN  2 NaN\n",
              "1  2.0  3.0  5 NaN\n",
              "2  NaN  4.0  6 NaN"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 108
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kAeoH8l6iwZg",
        "outputId": "9dea1397-72dd-4771-e7ef-8b0c8e7aef7f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        }
      },
      "source": [
        "df.dropna(axis='columns', how='all')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     0    1  2\n",
              "0  1.0  NaN  2\n",
              "1  2.0  3.0  5\n",
              "2  NaN  4.0  6"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bJzFRnRVjQUo"
      },
      "source": [
        "### 5.4. Filling null values\n",
        "\n",
        "Sometimes rather than dropping `NaN` values, you'd rather replace them with a valid value.\n",
        "This value might be a single number like zero, or it might be some sort of imputation or interpolation from the good values.\n",
        "\n",
        "There are four types of treatment that can be given, in that order, to unwanted non-existent or missing data:\n",
        "\n",
        "1. **Treatment 1:** Ignore the missing or unwanted data in some columns, considering that in other columns of the same rows there are important or relevant data for the study.\n",
        "2. **Treatment 2:** Replace the missing or unwanted data with values that represent an indicator of nullity.\n",
        "3. **Treatment 3:** Replace the missing, nonexistent or unwanted data with interpolated values that are related to the trend of the data that is present.\n",
        "4. **Treatment 4:** Delete the missing data, with the certainty that valuable information will not be lost when analyzing the data.\n",
        "\n",
        "You can apply **Treatment 2** and **Treatment 3** in-place using the ``isnull()`` method as a mask, but because it is such a common operation `Pandas` provides the ``fillna()`` method, which returns a copy of the array with the missing values replaced.\n",
        "\n",
        "Consider the following ``Series``:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "brtx-2TFi5-H",
        "outputId": "e832cdc7-b47a-4154-c076-58c47356d53e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "data = pd.Series([1, np.nan, 2, None, 3], index=list('abcde'))\n",
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    1.0\n",
              "b    NaN\n",
              "c    2.0\n",
              "d    NaN\n",
              "e    3.0\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8nmvNbMI4YmM",
        "colab_type": "text"
      },
      "source": [
        "To replace the missing values of the Series with a null value, we can do the following:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ZE4gEWacjXcf",
        "outputId": "1f40d1be-ecf3-43a5-9271-188cda002169",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "data.fillna(0)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    1.0\n",
              "b    0.0\n",
              "c    2.0\n",
              "d    0.0\n",
              "e    3.0\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wV8dHLAcjfHq"
      },
      "source": [
        "We can specify a forward-fill to propagate the previous value forward:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "JaSxaabNjaT7",
        "outputId": "36fdf9d5-658c-4c67-b6d3-ad8c0da0ed6f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "# forward-fill\n",
        "data.fillna(method='ffill')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    1.0\n",
              "b    1.0\n",
              "c    2.0\n",
              "d    2.0\n",
              "e    3.0\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 61
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "e89lgDwnjj5d"
      },
      "source": [
        "Or we can specify a back-fill to propagate the next values backward:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ecKI6evKjdG_",
        "outputId": "625664a1-3c04-4d7c-fe1b-a420e96111cb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "# back-fill\n",
        "data.fillna(method='bfill')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "a    1.0\n",
              "b    2.0\n",
              "c    2.0\n",
              "d    3.0\n",
              "e    3.0\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "emsOW59kjyAX"
      },
      "source": [
        "For ``DataFrame`` objects the options are similar, but we can also specify an ``axis`` along which the fills take place:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gwy6t-xEjlw8",
        "outputId": "bc5c1fd9-291f-4092-decd-82921b4db131",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        }
      },
      "source": [
        "df"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     0    1  2   3\n",
              "0  1.0  NaN  2 NaN\n",
              "1  2.0  3.0  5 NaN\n",
              "2  NaN  4.0  6 NaN"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 63
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "mu29vP-7jzku",
        "outputId": "26584781-b7ad-4320-a341-caf5a71af777",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        }
      },
      "source": [
        "df.fillna(method='ffill', axis=1) # forward-fill along the columns"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>6.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     0    1    2    3\n",
              "0  1.0  1.0  2.0  2.0\n",
              "1  2.0  3.0  5.0  5.0\n",
              "2  NaN  4.0  6.0  6.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U0tvtejJ4Ymr",
        "colab_type": "code",
        "outputId": "5dee27bd-7e7d-4cba-a573-149c22d67616",
        "colab": {}
      },
      "source": [
        "df.fillna(method='ffill', axis=0) # forward-fill along the rows"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     0    1  2   3\n",
              "0  1.0  NaN  2 NaN\n",
              "1  2.0  3.0  5 NaN\n",
              "2  2.0  4.0  6 NaN"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 110
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "SkWQkZvIj4Ux"
      },
      "source": [
        "Notice that if a previous value is not available during a forward fill, the `NaN` value remains."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "h5t9M5SNu3nb"
      },
      "source": [
        "## 6. `Pandas` String Operations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bo3gO8WV4Ymx",
        "colab_type": "text"
      },
      "source": [
        "When a `Pandas` object stores string data, `Pandas` provides certain operations to facilitate its manipulation. Let's see what would happen if a classic data storage structure like a list had missing data and a string operation was executed. Firstly, we define a list with four string values:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xasAeBYDj0x_",
        "outputId": "305b1118-f458-4403-f1d5-f6b15b677286",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "data = ['peter', 'Paul', 'MARY', 'gUIDO']\n",
        "[s.capitalize() for s in data]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Peter', 'Paul', 'Mary', 'Guido']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RXk_o8GDvF7D"
      },
      "source": [
        "This is perhaps sufficient to work with some data, but it will break if there are any missing values:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "BSdSRFSXvDg7",
        "outputId": "2d2002af-29b7-4bea-a904-fa7043d4f505",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 249
        }
      },
      "source": [
        "data = ['peter', 'Paul', None, 'MARY', 'gUIDO'] #this has a missing value None hence the error below\n",
        "[s.capitalize() for s in data]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "AttributeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-6-3b0264c38d59>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'peter'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Paul'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'MARY'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'gUIDO'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;34m[\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcapitalize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0ms\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m<ipython-input-6-3b0264c38d59>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'peter'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Paul'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'MARY'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'gUIDO'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;34m[\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcapitalize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0ms\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'capitalize'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2qwp6ZUL4Ym6",
        "colab_type": "text"
      },
      "source": [
        "Now let's look at a Pandas `Series`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PJs2RvhkvHYH",
        "outputId": "cf9538b4-4ddd-416d-94c1-c40622257d7d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "import pandas as pd\n",
        "names = pd.Series(data)\n",
        "names"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0        1\n",
              "1      NaN\n",
              "2    hello\n",
              "3     None\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 111
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "IDgsV07tvM0v"
      },
      "source": [
        "We can now call a single method that will capitalize all the entries, while skipping over any missing values or non-string values:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "zqyNJLWBvKQG",
        "outputId": "4aa3eb48-8347-410e-9373-cf257c740eb9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "names.str.capitalize()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0      NaN\n",
              "1      NaN\n",
              "2    Hello\n",
              "3      NaN\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 112
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "53WRpcIB4YnA",
        "colab_type": "text"
      },
      "source": [
        "We have accessed the `str` attribute that parses the values stored in the `Series` to string."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sqCVYZFevoFa"
      },
      "source": [
        "### 6.1. String Methods"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8sfY_mCMvlFz"
      },
      "source": [
        "Here is a list of Pandas ``str`` methods that mirror Python string methods:\n",
        "\n",
        "|             |                  |                  |                  |\n",
        "|-------------|------------------|------------------|------------------|\n",
        "|``len()``    | ``lower()``      | ``translate()``  | ``islower()``    | \n",
        "|``ljust()``  | ``upper()``      | ``startswith()`` | ``isupper()``    | \n",
        "|``rjust()``  | ``find()``       | ``endswith()``   | ``isnumeric()``  | \n",
        "|``center()`` | ``rfind()``      | ``isalnum()``    | ``isdecimal()``  | \n",
        "|``zfill()``  | ``index()``      | ``isalpha()``    | ``split()``      | \n",
        "|``strip()``  | ``rindex()``     | ``isdigit()``    | ``rsplit()``     | \n",
        "|``rstrip()`` | ``capitalize()`` | ``isspace()``    | ``partition()``  | \n",
        "|``lstrip()`` |  ``swapcase()``  |  ``istitle()``   | ``rpartition()`` |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rHqEu4e24YnD",
        "colab_type": "text"
      },
      "source": [
        "Let's see some examples of string methods for Pandas `Series` with the `monte` Series:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "LyevXtCovOj7",
        "colab": {}
      },
      "source": [
        "monte = pd.Series(['Graham Chapman', 'John Cleese', 'Terry Gilliam',\n",
        "                   'Eric Idle', 'Terry Jones', 'Michael Palin'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "zM8-rywYvUj0",
        "outputId": "b6934639-d542-418a-cc57-6ea3b9ca84dc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "monte.str.lower() # Parse values to string and transform all characters to lowercase"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    graham chapman\n",
              "1       john cleese\n",
              "2     terry gilliam\n",
              "3         eric idle\n",
              "4       terry jones\n",
              "5     michael palin\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 114
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3pWQ-VWkvYBB",
        "outputId": "9382773c-5a18-4175-ca60-d8dcefc4ab42",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "monte.str.len() # Parse values to string and calculates their length"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    14\n",
              "1    11\n",
              "2    13\n",
              "3     9\n",
              "4    11\n",
              "5    13\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 115
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "TVgUpU8hvaF_",
        "outputId": "f0665ff9-e669-44e0-ec51-b13d1a31f62d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "# Parse values to string and calculates a mask of string values starting by 'T'\n",
        "monte.str.startswith('T')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    False\n",
              "1    False\n",
              "2     True\n",
              "3    False\n",
              "4     True\n",
              "5    False\n",
              "dtype: bool"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 116
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "L5vyaS90vcBc",
        "outputId": "375a1086-6e9d-4f33-e66d-948a51f1c3de",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "monte.str.split() # Parse values to string and splits them by ' ' character, by default"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    [Graham, Chapman]\n",
              "1       [John, Cleese]\n",
              "2     [Terry, Gilliam]\n",
              "3         [Eric, Idle]\n",
              "4       [Terry, Jones]\n",
              "5     [Michael, Palin]\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bYuKRLP04Ynn",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 5\n",
        "\n",
        "Consider the following lists:\n",
        "\n",
        "```\n",
        "country = ['Netherland', 'Germany', 'Peru', 'Israel', 'Madagascar']\n",
        "year = [2002, 2002, 1957, 2007, 1967]\n",
        "population = [16122830.0, np.nan, 9146100.0, 6426679.0, 6334556.0]\n",
        "continent = ['Europe', 'europe', 'Americas', 'asia', 'Africa']\n",
        "```\n",
        "\n",
        "1. Create a Dataframe object which contains all the lists values as Series. The final DataFrame should be named as `country_info`, containing 4 columns and 5 rows.\n",
        "2. Delete the rows which contains missing values\n",
        "3. Capitalize all the continents in continent column.\n",
        "4. Get the length of each country's names."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dJFC4dMf4Ynn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "4bda1a5b-b058-4481-ff2b-b8ecd731057a"
      },
      "source": [
        "country = ['Netherland', 'Germany', 'Peru', 'Israel', 'Madagascar']\n",
        "year = [2002, 2002, 1957, 2007, 1967]\n",
        "population = [16122830.0, np.nan, 9146100.0, 6426679.0, 6334556.0]\n",
        "continent = ['Europe', 'europe', 'Americas', 'asia', 'Africa']\n",
        "info = {'country': country,'year':year,'population':population,'continent': continent}\n",
        "country_info = pd.DataFrame(info)\n",
        "country_info"
      ],
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>country</th>\n",
              "      <th>year</th>\n",
              "      <th>population</th>\n",
              "      <th>continent</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Netherland</td>\n",
              "      <td>2002</td>\n",
              "      <td>16122830.0</td>\n",
              "      <td>Europe</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Germany</td>\n",
              "      <td>2002</td>\n",
              "      <td>NaN</td>\n",
              "      <td>europe</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Peru</td>\n",
              "      <td>1957</td>\n",
              "      <td>9146100.0</td>\n",
              "      <td>Americas</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Israel</td>\n",
              "      <td>2007</td>\n",
              "      <td>6426679.0</td>\n",
              "      <td>asia</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Madagascar</td>\n",
              "      <td>1967</td>\n",
              "      <td>6334556.0</td>\n",
              "      <td>Africa</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      country  year  population continent\n",
              "0  Netherland  2002  16122830.0    Europe\n",
              "1     Germany  2002         NaN    europe\n",
              "2        Peru  1957   9146100.0  Americas\n",
              "3      Israel  2007   6426679.0      asia\n",
              "4  Madagascar  1967   6334556.0    Africa"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ICuJOMpi4Ynp",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "outputId": "372cf73a-2f73-49df-f276-27be239714ac"
      },
      "source": [
        "country_info.dropna(axis=0)"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>country</th>\n",
              "      <th>year</th>\n",
              "      <th>population</th>\n",
              "      <th>continent</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Netherland</td>\n",
              "      <td>2002</td>\n",
              "      <td>16122830.0</td>\n",
              "      <td>Europe</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Peru</td>\n",
              "      <td>1957</td>\n",
              "      <td>9146100.0</td>\n",
              "      <td>Americas</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Israel</td>\n",
              "      <td>2007</td>\n",
              "      <td>6426679.0</td>\n",
              "      <td>asia</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Madagascar</td>\n",
              "      <td>1967</td>\n",
              "      <td>6334556.0</td>\n",
              "      <td>Africa</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      country  year  population continent\n",
              "0  Netherland  2002  16122830.0    Europe\n",
              "2        Peru  1957   9146100.0  Americas\n",
              "3      Israel  2007   6426679.0      asia\n",
              "4  Madagascar  1967   6334556.0    Africa"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q84vnGv14Ynr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        },
        "outputId": "832feccd-47ff-4d56-f3b2-be8da0bd0064"
      },
      "source": [
        "country_info['continent'] = country_info['continent'].str.upper()\n",
        "country_info['continent']"
      ],
      "execution_count": 86,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0      EUROPE\n",
              "1      EUROPE\n",
              "2    AMERICAS\n",
              "3        ASIA\n",
              "4      AFRICA\n",
              "Name: continent, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 86
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DbVMFCGg4Ynt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        },
        "outputId": "65f5aa7d-caa6-4a70-dbf7-cd81b68bccc6"
      },
      "source": [
        "country_info['country'].str.len()"
      ],
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    10\n",
              "1     7\n",
              "2     4\n",
              "3     6\n",
              "4    10\n",
              "Name: country, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 88
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "w9XGVvHUv72L"
      },
      "source": [
        "## 7. Concatenate `Series`\n",
        "\n",
        "Here we'll take a look at simple concatenation of `Series` and `DataFrame` objects with the `pd.concat()` function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "UhPWz7G9veUr",
        "outputId": "66f94037-9706-4afa-b71c-d2cdb53d312f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "ser1 = pd.Series(['A', 'B', 'C'], index=[1, 2, 3])\n",
        "ser2 = pd.Series(['D', 'E', 'F'], index=[4, 5, 6])\n",
        "pd.concat([ser1, ser2], axis=0)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1    A\n",
              "2    B\n",
              "3    C\n",
              "4    D\n",
              "5    E\n",
              "6    F\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 119
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y_3FrZpP4Yn9",
        "colab_type": "code",
        "outputId": "6ad4126b-9aed-449e-82ec-2cbfd4f8c594",
        "colab": {}
      },
      "source": [
        "pd.concat([ser1, ser2], axis=1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>B</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>C</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>NaN</td>\n",
              "      <td>D</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>NaN</td>\n",
              "      <td>E</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>NaN</td>\n",
              "      <td>F</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     0    1\n",
              "1    A  NaN\n",
              "2    B  NaN\n",
              "3    C  NaN\n",
              "4  NaN    D\n",
              "5  NaN    E\n",
              "6  NaN    F"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 120
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yW_23Rto4YoA",
        "colab_type": "text"
      },
      "source": [
        "By default, the concatenation takes place row-wise within the ``DataFrame`` (i.e., ``axis=0``). However, as you can see, the concatenation of `Series` in a `DataFrame` can be done in contiguous rows or columns by specifying the `axis` parameter. In the case where they are columns, care must be taken to define the same index values, so that the columns are placed contiguously without `NaN` values."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "07hetTOW4YoB",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 6\n",
        "\n",
        "Consider the following lists:\n",
        "\n",
        "```\n",
        "country = ['Netherland', 'Germany', 'Peru', 'Israel', 'Madagascar']\n",
        "gdp_per_cap = [33724.757780, 30035.801980, 4245.256698, 25523.277100, 1634.047282]\n",
        "```\n",
        "\n",
        "1. Create a Dataframe object which contains all the lists values as Series. The final DataFrame should be named as `country_gdp`, containing 2 columns and 5 rows.\n",
        "2. Concatenate the two dataframes: `country_info` and `country_gdp` with `axis=0` and name it `concat_data`\n",
        "3. Check if there are any null values in `concat_data`\n",
        "4. Find total numer of missing values in each column. *hint: Use `.isnull()` and `.sum()` functions*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Eb10iyS9Bdh",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 156
        },
        "outputId": "d1e8b88e-6015-4657-81c5-4a7f599abc22"
      },
      "source": [
        "country = ['Netherland', 'Germany', 'Peru', 'Israel', 'Madagascar']\n",
        "gdp_per_cap = [33724.757780, 30035.801980, 4245.256698, 25523.277100, 1634.047282]\n",
        "\n",
        "data = {'country': country , 'gdp_per_cap': gdp_per_cap}\n",
        "\n",
        "country_gdp = pd.DataFrame(data)\n",
        "print(country_gdp)\n",
        "print(\"\\nShape\", country_gdp.shape)"
      ],
      "execution_count": 94,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "      country   gdp_per_cap\n",
            "0  Netherland  33724.757780\n",
            "1     Germany  30035.801980\n",
            "2        Peru   4245.256698\n",
            "3      Israel  25523.277100\n",
            "4  Madagascar   1634.047282\n",
            "\n",
            "Shape (5, 2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ks1yCkmy9D1f",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        },
        "outputId": "869bf78c-c3a9-4f64-af43-e94d1fdf5d8d"
      },
      "source": [
        "concat_data = pd.concat([country_info,country_gdp],axis =0)\n",
        "concat_data"
      ],
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>country</th>\n",
              "      <th>year</th>\n",
              "      <th>population</th>\n",
              "      <th>continent</th>\n",
              "      <th>gdp_per_cap</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Netherland</td>\n",
              "      <td>2002.0</td>\n",
              "      <td>16122830.0</td>\n",
              "      <td>EUROPE</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Germany</td>\n",
              "      <td>2002.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>EUROPE</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Peru</td>\n",
              "      <td>1957.0</td>\n",
              "      <td>9146100.0</td>\n",
              "      <td>AMERICAS</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Israel</td>\n",
              "      <td>2007.0</td>\n",
              "      <td>6426679.0</td>\n",
              "      <td>ASIA</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Madagascar</td>\n",
              "      <td>1967.0</td>\n",
              "      <td>6334556.0</td>\n",
              "      <td>AFRICA</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Netherland</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>33724.757780</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Germany</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>30035.801980</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Peru</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4245.256698</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Israel</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>25523.277100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Madagascar</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1634.047282</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      country    year  population continent   gdp_per_cap\n",
              "0  Netherland  2002.0  16122830.0    EUROPE           NaN\n",
              "1     Germany  2002.0         NaN    EUROPE           NaN\n",
              "2        Peru  1957.0   9146100.0  AMERICAS           NaN\n",
              "3      Israel  2007.0   6426679.0      ASIA           NaN\n",
              "4  Madagascar  1967.0   6334556.0    AFRICA           NaN\n",
              "0  Netherland     NaN         NaN       NaN  33724.757780\n",
              "1     Germany     NaN         NaN       NaN  30035.801980\n",
              "2        Peru     NaN         NaN       NaN   4245.256698\n",
              "3      Israel     NaN         NaN       NaN  25523.277100\n",
              "4  Madagascar     NaN         NaN       NaN   1634.047282"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 96
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZsIq27Oy9E3X",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        },
        "outputId": "86cb83a5-7065-431e-d1e2-b34abdc4c5c6"
      },
      "source": [
        "concat_data.isnull()"
      ],
      "execution_count": 97,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>country</th>\n",
              "      <th>year</th>\n",
              "      <th>population</th>\n",
              "      <th>continent</th>\n",
              "      <th>gdp_per_cap</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   country   year  population  continent  gdp_per_cap\n",
              "0    False  False       False      False         True\n",
              "1    False  False        True      False         True\n",
              "2    False  False       False      False         True\n",
              "3    False  False       False      False         True\n",
              "4    False  False       False      False         True\n",
              "0    False   True        True       True        False\n",
              "1    False   True        True       True        False\n",
              "2    False   True        True       True        False\n",
              "3    False   True        True       True        False\n",
              "4    False   True        True       True        False"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SUWP8Wj29Fg3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        },
        "outputId": "de5f4eff-d073-41d4-9509-90551dc87274"
      },
      "source": [
        "concat_data.isnull().sum()"
      ],
      "execution_count": 99,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "country        0\n",
              "year           5\n",
              "population     6\n",
              "continent      5\n",
              "gdp_per_cap    5\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_uwL4BbE4YoB",
        "colab_type": "text"
      },
      "source": [
        "## 8. `DataFrame` fancy table printing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P_yOZvv84YoC",
        "colab_type": "text"
      },
      "source": [
        "In the next two cells we are going to define a fancy way to visualize the data of multiple `DataFrame` objects. Let's first use the `IPython.display` library, which allows us to view the contents of `DataFrame` objects individually, in a table fancy way:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EAnbkqF34YoC",
        "colab_type": "code",
        "outputId": "804c6f16-2a31-4cec-a2e3-b507d0c15e45",
        "colab": {}
      },
      "source": [
        "from IPython.display import display, HTML\n",
        "\n",
        "display(pd.concat([ser1, ser2], axis=1))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>B</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>C</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>NaN</td>\n",
              "      <td>D</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>NaN</td>\n",
              "      <td>E</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>NaN</td>\n",
              "      <td>F</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     0    1\n",
              "1    A  NaN\n",
              "2    B  NaN\n",
              "3    C  NaN\n",
              "4  NaN    D\n",
              "5  NaN    E\n",
              "6  NaN    F"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Yw_Dp2o4YoG",
        "colab_type": "text"
      },
      "source": [
        "Now let's look at a function defined by us, which allows us to evidence the data of multiple `DataFrame` objects. It is not necessary for now to have a complete understanding of the following function. What is important at this time is knowing how to use it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "W5eEZIU8wqlj",
        "colab": {}
      },
      "source": [
        "class display(object):\n",
        "    \"\"\"Display HTML representation of multiple objects\"\"\"\n",
        "    template = \"\"\"<div style=\"float: left; padding: 10px;\">\n",
        "    <p style='font-family:\"Courier New\", Courier, monospace'>{0}</p>{1}\n",
        "    </div>\"\"\"\n",
        "    def __init__(self, *args):\n",
        "        self.args = args\n",
        "        \n",
        "    def _repr_html_(self):\n",
        "        return '\\n'.join(self.template.format(a, eval(a)._repr_html_())\n",
        "                         for a in self.args)\n",
        "    \n",
        "    def __repr__(self):\n",
        "        return '\\n\\n'.join(a + '\\n' + repr(eval(a))\n",
        "                           for a in self.args)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8TSJlkZ94YoJ",
        "colab_type": "text"
      },
      "source": [
        "Let's also define a user function that allows us to create a `DataFrame` quickly using *Dictionary Comprehension*. This function transforms an input string into a square array that is later converted to a `DataFrame` where the column names are each of the characters in the string:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "TBMdkEUuwF5Q",
        "outputId": "63ffaec3-edbe-42ad-aa06-ba4f9cd67cf9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        }
      },
      "source": [
        "def make_df(cols, ind):\n",
        "    \"\"\"Quickly make a DataFrame\"\"\"\n",
        "    data = {c: [str(c) + str(i) for i in ind]\n",
        "            for c in cols}\n",
        "    return pd.DataFrame(data, ind)\n",
        "\n",
        "# example DataFrame\n",
        "make_df('ABC', range(3))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>A0</td>\n",
              "      <td>B0</td>\n",
              "      <td>C0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A1</td>\n",
              "      <td>B1</td>\n",
              "      <td>C1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A2</td>\n",
              "      <td>B2</td>\n",
              "      <td>C2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    A   B   C\n",
              "0  A0  B0  C0\n",
              "1  A1  B1  C1\n",
              "2  A2  B2  C2"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 124
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dJlt7pLAwbJV"
      },
      "source": [
        "The function also allows us to concatenate higher-dimensional objects, such as ``DataFrame`` objects:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "byfgkY19wYCP",
        "outputId": "5ac87b7b-3b29-4169-e63f-8cb82c8674c5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222
        }
      },
      "source": [
        "df1 = make_df('AB', [1, 2])\n",
        "df2 = make_df('AB', [3, 4])\n",
        "display('df1', 'df2', 'pd.concat([df1, df2])')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div style=\"float: left; padding: 10px;\">\n",
              "    <p style='font-family:\"Courier New\", Courier, monospace'>df1</p><div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A1</td>\n",
              "      <td>B1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A2</td>\n",
              "      <td>B2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    </div>\n",
              "<div style=\"float: left; padding: 10px;\">\n",
              "    <p style='font-family:\"Courier New\", Courier, monospace'>df2</p><div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A3</td>\n",
              "      <td>B3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A4</td>\n",
              "      <td>B4</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    </div>\n",
              "<div style=\"float: left; padding: 10px;\">\n",
              "    <p style='font-family:\"Courier New\", Courier, monospace'>pd.concat([df1, df2])</p><div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A1</td>\n",
              "      <td>B1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A2</td>\n",
              "      <td>B2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A3</td>\n",
              "      <td>B3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A4</td>\n",
              "      <td>B4</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    </div>"
            ],
            "text/plain": [
              "df1\n",
              "    A   B\n",
              "1  A1  B1\n",
              "2  A2  B2\n",
              "\n",
              "df2\n",
              "    A   B\n",
              "3  A3  B3\n",
              "4  A4  B4\n",
              "\n",
              "pd.concat([df1, df2])\n",
              "    A   B\n",
              "1  A1  B1\n",
              "2  A2  B2\n",
              "3  A3  B3\n",
              "4  A4  B4"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RULRT3vewaji",
        "outputId": "4c0dc9ed-5f2b-4491-8d51-bd226f428e11",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 160
        }
      },
      "source": [
        "df3 = make_df('AB', [0, 1])\n",
        "df4 = make_df('CD', [0, 1])\n",
        "display('df3', 'df4', \"pd.concat([df3, df4], axis=1)\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div style=\"float: left; padding: 10px;\">\n",
              "    <p style='font-family:\"Courier New\", Courier, monospace'>df3</p><div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>A0</td>\n",
              "      <td>B0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A1</td>\n",
              "      <td>B1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    </div>\n",
              "<div style=\"float: left; padding: 10px;\">\n",
              "    <p style='font-family:\"Courier New\", Courier, monospace'>df4</p><div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>C</th>\n",
              "      <th>D</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>C0</td>\n",
              "      <td>D0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>C1</td>\n",
              "      <td>D1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    </div>\n",
              "<div style=\"float: left; padding: 10px;\">\n",
              "    <p style='font-family:\"Courier New\", Courier, monospace'>pd.concat([df3, df4], axis=1)</p><div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "      <th>D</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>A0</td>\n",
              "      <td>B0</td>\n",
              "      <td>C0</td>\n",
              "      <td>D0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A1</td>\n",
              "      <td>B1</td>\n",
              "      <td>C1</td>\n",
              "      <td>D1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    </div>"
            ],
            "text/plain": [
              "df3\n",
              "    A   B\n",
              "0  A0  B0\n",
              "1  A1  B1\n",
              "\n",
              "df4\n",
              "    C   D\n",
              "0  C0  D0\n",
              "1  C1  D1\n",
              "\n",
              "pd.concat([df3, df4], axis=1)\n",
              "    A   B   C   D\n",
              "0  A0  B0  C0  D0\n",
              "1  A1  B1  C1  D1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 82
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S2kEJlDN4YoU",
        "colab_type": "text"
      },
      "source": [
        "## Conclusions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oL4uLFAN4YoW",
        "colab_type": "text"
      },
      "source": [
        "We have learned the fundamentals of the `pandas` library for data analysis, which allows us to execute operations with stored data efficiently, add records and handle missing data. We also highlighted some of the most important functions of Pandas' `Series` and `DataFrame` objects, as well as following a Data Wrangling procedure.\n",
        "\n",
        "In the next case study, we will review the fundamentals of data visualization libraries, so that we can identify visual patterns in our data."
      ]
    }
  ]
}