run_ner_spanberta.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ludwigwittgenstein2/Research/blob/master/run_ner_spanberta.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CfVkuWAW5r36"
      },
      "source": [
        "# SpanBERTa - Part II: Named Entity Recognition with Transformers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R0rpgtxmkET4"
      },
      "source": [
        "# Introduction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LhKZ3vItVBzi"
      },
      "source": [
        "- [Part I: How We Trained RoBERTa Language Model for Spanish from Scratch](https://chriskhanhtran.github.io/posts/spanberta-bert-for-spanish-from-scratch/)\n",
        "\n",
        "In my previous blog post, we have discussed how my team pretrained SpanBERTa, a transformer language model for Spanish, on a big corpus from scratch. The model has shown to be able to predict correctly masked words in a sequence based on its context. In this blog post, to really leverage the power of transformer models, we will fine-tune SpanBERTa for a named-entity recognition task.\n",
        "\n",
        "According to its definition on [Wikipedia](https://en.wikipedia.org/wiki/Named-entity_recognition), Named-entity recognition (NER) (also known as entity identification, entity chunking and entity extraction) is a subtask of information extraction that seeks to locate and classify named entity mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.\n",
        "\n",
        "We will use the script [`run_ner.py`](https://github.com/huggingface/transformers/blob/master/examples/ner/run_ner.py) by Hugging Face and [CoNLL-2002 dataset](https://www.kaggle.com/nltkdata/conll-corpora) to fine-tune SpanBERTa."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SfBvd7zVIbDc"
      },
      "source": [
        "# Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "utGQ7w2Jjdi6"
      },
      "source": [
        "Download `transformers` and install required packages."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "klEBSPAuIaDj"
      },
      "source": [
        "%%capture\n",
        "!git clone https://github.com/huggingface/transformers\n",
        "%cd transformers\n",
        "!pip install .\n",
        "!pip install -r ./examples/requirements.txt\n",
        "%cd .."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GU8R8FeAIT2C"
      },
      "source": [
        "# Data\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JELuvmOKJaq6"
      },
      "source": [
        "## 1. Download Datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NQXQAFDg7ZpP"
      },
      "source": [
        "The below command will download and unzip the dataset. The files contain the train and test data for three parts of the [CoNLL-2002](https://www.clips.uantwerpen.be/conll2002/ner/) shared task:\n",
        "   - esp.testa: Spanish test data for the development stage\n",
        "   - esp.testb: Spanish test data\n",
        "   - esp.train: Spanish train data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bZSnJqZ2Sz4B"
      },
      "source": [
        "%%capture\n",
        "!wget -O 'conll2002.zip' 'https://drive.google.com/uc?export=download&id=1Wrl1b39ZXgKqCeAFNM9EoXtA1kzwNhCe'\n",
        "!unzip 'conll2002.zip'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p__tzmLA8geA"
      },
      "source": [
        "The size of each dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R10opzwVRVks",
        "outputId": "6d338bd4-69f2-419f-b2cf-3d6d9870d315",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "!wc -l conll2002/esp.train\n",
        "!wc -l conll2002/esp.testa\n",
        "!wc -l conll2002/esp.testb"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "273038 conll2002/esp.train\n",
            "54838 conll2002/esp.testa\n",
            "53050 conll2002/esp.testb\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CNHyXIP28q93"
      },
      "source": [
        "All data files has three columns: words, associated part-of-speech tags and named entity tags in the IOB2 format. Sentence breaks are encoded by empty lines."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MsN1VWUlIz5i",
        "outputId": "6d7fbc4a-c32f-4448-9d74-4d108b490efa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        }
      },
      "source": [
        "!head -n20 conll2002/esp.train"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Melbourne NP B-LOC\n",
            "( Fpa O\n",
            "Australia NP B-LOC\n",
            ") Fpt O\n",
            ", Fc O\n",
            "25 Z O\n",
            "may NC O\n",
            "( Fpa O\n",
            "EFE NC B-ORG\n",
            ") Fpt O\n",
            ". Fp O\n",
            "\n",
            "- Fg O\n",
            "\n",
            "El DA O\n",
            "Abogado NC B-PER\n",
            "General AQ I-PER\n",
            "del SP I-PER\n",
            "Estado NC I-PER\n",
            ", Fc O\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R3InVzK79kXH"
      },
      "source": [
        "We will only keep the word column and the named entity tag column for our train, dev and test datasets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7YVsTqo-rUXY"
      },
      "source": [
        "!cat conll2002/esp.train | cut -d \" \" -f 1,3 > train_temp.txt\n",
        "!cat conll2002/esp.testa | cut -d \" \" -f 1,3 > dev_temp.txt\n",
        "!cat conll2002/esp.testb | cut -d \" \" -f 1,3 > test_temp.txt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b3c4v9IKJVjL"
      },
      "source": [
        "## 2. Preprocessing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5gtQ-tVQWgWe"
      },
      "source": [
        "Let's define some variables that we need for further pre-processing steps and training the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z4Wl3iR0UBZv"
      },
      "source": [
        "MAX_LENGTH = 120 #@param {type: \"integer\"}\n",
        "MODEL = \"chriskhanhtran/spanberta\" #@param [\"chriskhanhtran/spanberta\", \"bert-base-multilingual-cased\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hSfMYC7w_UpX"
      },
      "source": [
        "The script below will split sentences longer than `MAX_LENGTH` (in terms of tokens) into small ones. Otherwise, long sentences will be truncated when tokenized, causing the loss of training data and some tokens in the test set not being predicted."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3gmazeojxn80"
      },
      "source": [
        "%%capture\n",
        "!wget \"https://raw.githubusercontent.com/stefan-it/fine-tuned-berts-seq/master/scripts/preprocess.py\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qn10UkLQxNg3",
        "outputId": "c7c231f0-56eb-4d9b-d95c-c921f17b5fd8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "!python3 preprocess.py train_temp.txt $MODEL $MAX_LENGTH > train.txt\n",
        "!python3 preprocess.py dev_temp.txt $MODEL $MAX_LENGTH > dev.txt\n",
        "!python3 preprocess.py test_temp.txt $MODEL $MAX_LENGTH > test.txt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-04-22 23:02:05.747294: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "Downloading: 100% 1.03k/1.03k [00:00<00:00, 704kB/s]\n",
            "Downloading: 100% 954k/954k [00:00<00:00, 1.89MB/s]\n",
            "Downloading: 100% 512k/512k [00:00<00:00, 1.19MB/s]\n",
            "Downloading: 100% 16.0/16.0 [00:00<00:00, 12.6kB/s]\n",
            "2020-04-22 23:02:23.409488: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-04-22 23:02:31.168967: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_SJQJHNlUNMP"
      },
      "source": [
        "## 3. Labels\n",
        "\n",
        "In CoNLL-2002/2003 datasets, there are have 9 classes of NER tags:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d-DlGoiJLqAR"
      },
      "source": [
        "- O, Outside of a named entity\n",
        "- B-MIS, Beginning of a miscellaneous entity right after another miscellaneous entity\n",
        "- I-MIS, Miscellaneous entity\n",
        "- B-PER, Beginning of a person’s name right after another person’s name\n",
        "- I-PER, Person’s name\n",
        "- B-ORG, Beginning of an organisation right after another organisation\n",
        "- I-ORG, Organisation\n",
        "- B-LOC, Beginning of a location right after another location\n",
        "- I-LOC, Location"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ee3mlWe2BQPZ"
      },
      "source": [
        "If your dataset has different labels or more labels than CoNLL-2002/2003 datasets, run the line below to get unique labels from your data and save them into `labels.txt`. This file will be used when we start fine-tuning our model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mEtntk5-Wnvu"
      },
      "source": [
        "!cat train.txt dev.txt test.txt | cut -d \" \" -f 2 | grep -v \"^$\"| sort | uniq > labels.txt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G80k0O2aWwzy"
      },
      "source": [
        "# Fine-tuning Model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Qy9XdmlJXqE"
      },
      "source": [
        "These are the example scripts from `transformers`'s repo that we will use to fine-tune our model for NER. After 04/21/2020, Hugging Face has updated their example scripts to use a new `Trainer` class. To avoid any future conflict, let's use the version before they made these updates."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1NoEUI-EGDQp"
      },
      "source": [
        "%%capture\n",
        "!wget \"https://raw.githubusercontent.com/chriskhanhtran/spanish-bert/master/ner/run_ner.py\"\n",
        "!wget \"https://raw.githubusercontent.com/chriskhanhtran/spanish-bert/master/ner/utils_ner.py\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xefzt_b1IiQb"
      },
      "source": [
        "Now it's time for transfer learning. In my [previous blog post](https://chriskhanhtran.github.io/posts/spanberta-bert-for-spanish-from-scratch/), I have pretrained a RoBERTa language model on a very large Spanish corpus to predict masked words based on the context they are in. By doing that, the model has learned inherent properties of the language. I have uploaded the pretrained model to Hugging Face's server. Now we will load the model and start fine-tuning it for the NER task.\n",
        "\n",
        "Below are our training hyperparameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rR7lmegy5Pzq"
      },
      "source": [
        "MAX_LENGTH = 128 #@param {type: \"integer\"}\n",
        "MODEL = \"chriskhanhtran/spanberta\" #@param [\"chriskhanhtran/spanberta\", \"bert-base-multilingual-cased\"]\n",
        "OUTPUT_DIR = \"spanberta-ner\" #@param [\"spanberta-ner\", \"bert-base-ml-ner\"]\n",
        "BATCH_SIZE = 32 #@param {type: \"integer\"}\n",
        "NUM_EPOCHS = 3 #@param {type: \"integer\"}\n",
        "SAVE_STEPS = 100 #@param {type: \"integer\"}\n",
        "LOGGING_STEPS = 100 #@param {type: \"integer\"}\n",
        "SEED = 42 #@param {type: \"integer\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jYATwvIjG-WN"
      },
      "source": [
        "Let's start training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AG-f9zzGWzWz"
      },
      "source": [
        "!python3 run_ner.py \\\n",
        "  --data_dir ./ \\\n",
        "  --model_type bert \\\n",
        "  --labels ./labels.txt \\\n",
        "  --model_name_or_path $MODEL \\\n",
        "  --output_dir $OUTPUT_DIR \\\n",
        "  --max_seq_length  $MAX_LENGTH \\\n",
        "  --num_train_epochs $NUM_EPOCHS \\\n",
        "  --per_gpu_train_batch_size $BATCH_SIZE \\\n",
        "  --save_steps $SAVE_STEPS \\\n",
        "  --logging_steps $LOGGING_STEPS \\\n",
        "  --seed $SEED \\\n",
        "  --do_train \\\n",
        "  --do_eval \\\n",
        "  --do_predict \\\n",
        "  --overwrite_output_dir"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Go3OZkoeQ0yf"
      },
      "source": [
        "Performance on the dev set:\n",
        "```\n",
        "04/21/2020 02:24:31 - INFO - __main__ -   ***** Eval results  *****\n",
        "04/21/2020 02:24:31 - INFO - __main__ -     f1 = 0.831027443864822\n",
        "04/21/2020 02:24:31 - INFO - __main__ -     loss = 0.1004064822183894\n",
        "04/21/2020 02:24:31 - INFO - __main__ -     precision = 0.8207885304659498\n",
        "04/21/2020 02:24:31 - INFO - __main__ -     recall = 0.8415250344510795\n",
        "```\n",
        "Performance on the test set:\n",
        "```\n",
        "04/21/2020 02:24:48 - INFO - __main__ -   ***** Eval results  *****\n",
        "04/21/2020 02:24:48 - INFO - __main__ -     f1 = 0.8559533721898419\n",
        "04/21/2020 02:24:48 - INFO - __main__ -     loss = 0.06848683688204177\n",
        "04/21/2020 02:24:48 - INFO - __main__ -     precision = 0.845858475041141\n",
        "04/21/2020 02:24:48 - INFO - __main__ -     recall = 0.8662921348314607\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ah2Of3XckcVm"
      },
      "source": [
        "Here are the tensorboards of fine-tuning [spanberta](https://tensorboard.dev/experiment/Ggs7aCjWQ0exU2Nbp3pPlQ/#scalars&_smoothingWeight=0.265) and [bert-base-multilingual-cased](https://tensorboard.dev/experiment/M9AXw2lORjeRzFZzEJOxkA/#scalars) for 5 epoches. We can see that the models overfit the training data after 3 epoches.\n",
        "\n",
        "![](https://raw.githubusercontent.com/chriskhanhtran/spanish-bert/master/img/spanberta-ner-tb-5.JPG)\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8uAzVHU1WjaD"
      },
      "source": [
        "**Classification Report**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t380TBzoqUvc"
      },
      "source": [
        "To understand how well our model actually performs, let's load its predictions and examine the classification report."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lqU8mvjIS_4D"
      },
      "source": [
        "def read_examples_from_file(file_path):\n",
        "    \"\"\"Read words and labels from a CoNLL-2002/2003 data file.\n",
        "    \n",
        "    Args:\n",
        "      file_path (str): path to NER data file.\n",
        "\n",
        "    Returns:\n",
        "      examples (dict): a dictionary with two keys: `words` (list of lists)\n",
        "        holding words in each sequence, and `labels` (list of lists) holding\n",
        "        corresponding labels.\n",
        "    \"\"\"\n",
        "    with open(file_path, encoding=\"utf-8\") as f:\n",
        "        examples = {\"words\": [], \"labels\": []}\n",
        "        words = []\n",
        "        labels = []\n",
        "        for line in f:\n",
        "            if line.startswith(\"-DOCSTART-\") or line == \"\" or line == \"\\n\":\n",
        "                if words:\n",
        "                    examples[\"words\"].append(words)\n",
        "                    examples[\"labels\"].append(labels)\n",
        "                    words = []\n",
        "                    labels = []\n",
        "            else:\n",
        "                splits = line.split(\" \")\n",
        "                words.append(splits[0])\n",
        "                if len(splits) > 1:\n",
        "                    labels.append(splits[-1].replace(\"\\n\", \"\"))\n",
        "                else:\n",
        "                    # Examples could have no label for mode = \"test\"\n",
        "                    labels.append(\"O\")\n",
        "    return examples"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hizogs3MCpFp"
      },
      "source": [
        "Read data and labels from the raw text files:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V9lVfi_A_Eln"
      },
      "source": [
        "y_true = read_examples_from_file(\"test.txt\")[\"labels\"]\n",
        "y_pred = read_examples_from_file(\"spanberta-ner/test_predictions.txt\")[\"labels\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "59B5FFb_CvBp"
      },
      "source": [
        "Print the classification report:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lP98_LWEa5x1",
        "outputId": "a2cff8d8-be36-48df-a118-b962ab07b5c1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        }
      },
      "source": [
        "from seqeval.metrics import classification_report as classification_report_seqeval\n",
        "\n",
        "print(classification_report_seqeval(y_true, y_pred))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "           precision    recall  f1-score   support\n",
            "\n",
            "      LOC       0.87      0.84      0.85      1084\n",
            "      ORG       0.82      0.87      0.85      1401\n",
            "     MISC       0.63      0.66      0.65       340\n",
            "      PER       0.94      0.96      0.95       735\n",
            "\n",
            "micro avg       0.84      0.86      0.85      3560\n",
            "macro avg       0.84      0.86      0.85      3560\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QyTHymw2ClDq"
      },
      "source": [
        "The metrics we are seeing in this report are designed specifically for NLP tasks such as NER and POS tagging, in which all words of an entity need to be predicted correctly to be counted as one correct prediction. Therefore, the metrics in this classification report are much lower than in [scikit-learn's classification report](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Li66swRquHmm",
        "outputId": "81bbb0e6-ba7f-4ab9-8c99-ecb847bb7622",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        }
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.metrics import classification_report\n",
        "\n",
        "print(classification_report(np.concatenate(y_true), np.concatenate(y_pred)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "       B-LOC       0.88      0.85      0.86      1084\n",
            "      B-MISC       0.73      0.73      0.73       339\n",
            "       B-ORG       0.87      0.91      0.89      1400\n",
            "       B-PER       0.95      0.96      0.95       735\n",
            "       I-LOC       0.82      0.81      0.81       325\n",
            "      I-MISC       0.85      0.76      0.80       557\n",
            "       I-ORG       0.89      0.87      0.88      1104\n",
            "       I-PER       0.98      0.98      0.98       634\n",
            "           O       1.00      1.00      1.00     45355\n",
            "\n",
            "    accuracy                           0.98     51533\n",
            "   macro avg       0.89      0.87      0.88     51533\n",
            "weighted avg       0.98      0.98      0.98     51533\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VLSmTE9Wrmv1"
      },
      "source": [
        "From above reports, our model has a good performance in predicting person, location and organization. We will need more data for `MISC` entities to improve our model's performance on these entities."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mff0GDpwELBF"
      },
      "source": [
        "# Pipeline "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PE2K4xCGLTwG"
      },
      "source": [
        "After fine-tuning our models, we can share them with the community by following the tutorial in this [page](https://huggingface.co/transformers/model_sharing.html). Now we can start loading the fine-tuned model from Hugging Face's server and use it to predict named entities in Spanish documents."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s7z9SwmTJtOW"
      },
      "source": [
        "from transformers import pipeline, AutoModelForTokenClassification, AutoTokenizer\n",
        "\n",
        "model = AutoModelForTokenClassification.from_pretrained(\"skimai/spanberta-base-cased-ner-conll02\")\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"skimai/spanberta-base-cased-ner-conll02\")\n",
        "\n",
        "ner_model = pipeline('ner', model=model, tokenizer=tokenizer)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FC6k_dgeMvnz"
      },
      "source": [
        "The example below is obtained from [La Opinión](https://laopinion.com/2020/04/19/secretario-del-tesoro-advierte-que-la-economia-de-estados-unidos-tardara-meses-en-recuperarse-tras-coronavirus/) and means \"*The economic recovery of the United States after the coronavirus pandemic will be a matter of months, said Treasury Secretary Steven Mnuchin.*\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VmRt4o22KXdm",
        "outputId": "5d723474-819f-4449-8592-ae9827f531a6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "sequence = \"La recuperación económica de los Estados Unidos después de la \" \\\n",
        "           \"pandemia del coronavirus será cuestión de meses, afirmó el \" \\\n",
        "           \"Secretario del Tesoro, Steven Mnuchin.\"\n",
        "ner_model(sequence)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'entity': 'B-ORG', 'score': 0.9155661463737488, 'word': 'ĠEstados'},\n",
              " {'entity': 'I-ORG', 'score': 0.800682544708252, 'word': 'ĠUnidos'},\n",
              " {'entity': 'I-MISC', 'score': 0.5006815791130066, 'word': 'Ġcorona'},\n",
              " {'entity': 'I-MISC', 'score': 0.510674774646759, 'word': 'virus'},\n",
              " {'entity': 'B-PER', 'score': 0.5558510422706604, 'word': 'ĠSecretario'},\n",
              " {'entity': 'I-PER', 'score': 0.7758238315582275, 'word': 'Ġdel'},\n",
              " {'entity': 'I-PER', 'score': 0.7096233367919922, 'word': 'ĠTesoro'},\n",
              " {'entity': 'B-PER', 'score': 0.9940345883369446, 'word': 'ĠSteven'},\n",
              " {'entity': 'I-PER', 'score': 0.9962581992149353, 'word': 'ĠM'},\n",
              " {'entity': 'I-PER', 'score': 0.9918380379676819, 'word': 'n'},\n",
              " {'entity': 'I-PER', 'score': 0.9848328828811646, 'word': 'uch'},\n",
              " {'entity': 'I-PER', 'score': 0.8513168096542358, 'word': 'in'}]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zmpiw8fmL2GE"
      },
      "source": [
        "Looks great! The fine-tuned model successfully recognizes all entities in our example, and even recognizes \"corona virus.\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0uOEdMAJj1_8"
      },
      "source": [
        "# Conclusion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KNOu4iT6x5l-"
      },
      "source": [
        "Named-entity recognition can help us quickly extract important information from texts. Therefore, its application in business can have a direct impact on improving human's productivity in reading contracts and documents. However, it is a challenging NLP task because NER requires accurate classification at the word level, making simple approaches such as bag-of-word impossible to deal with this task.\n",
        "\n",
        "We have walked through how we can leverage a pretrained BERT model to quickly gain an excellent performance on the NER task for Spanish. The pretrained SpanBERTa model can also be fine-tuned for other tasks such as document classification. I have written a detailed tutorial to finetune BERT for sequence classification and sentiment analysis.\n",
        "\n",
        "- [Fine-tuning BERT for Sentiment Analysis](https://chriskhanhtran.github.io/posts/bert-for-sentiment-analysis/)\n",
        "\n",
        "Next in this series, we will discuss ELECTRA, a more efficient pre-training approach for transformer models which can quickly achieve state-of-the-art performance. Stay tuned!"
      ]
    }
  ]
}