challenge.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Linkit Challenge SS 2023 Hand Gesture Detection - Team X"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 1. Setup\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, you usually create a virtual environment where you can keep track of all package versions.\n",
    "Create the environment with the following command in Terminal (Mac)/ CMD (Windows)\n",
    "\n",
    "python3 -m venv venv\n",
    "\n",
    "\n",
    "Then activate it with\n",
    "- in CMD: venv\\Scripts\\activate\n",
    "- in Terminal: . venv/bin/activate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# only run once to setup your environment\n",
    "first_run = False\n",
    "\n",
    "if first_run:\n",
    "    # install reqiurements.txt\n",
    "    !pip3 install -r requirements.txt\n",
    "    # clone the git repo of ultralytics/yolov5\n",
    "    !git clone https://github.com/ultralytics/yolov5.git\n",
    "    # install dependencies for yolo\n",
    "    !pip3 install -r yolov5/requirements.txt\n",
    "    # or from website\n",
    "    #!pip3 install -qr https://raw.githubusercontent.com/ultralytics/yolov5/master/requirements.txt"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 1.1. Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import pytorch_lightning as pl # PyTorch Lightning for easier training and evaluation of models\n",
    "import torch # PyTorch\n",
    "import cv2  # OpenCV for image processing\n",
    "import matplotlib.pyplot as plt # for plotting\n",
    "import matplotlib.patches as patches  # for plotting bounding boxes\n",
    "%matplotlib inline\n",
    "import uuid   # Unique identifier\n",
    "import os # File system operations\n",
    "import time # Time operations\n",
    "import subprocess # running shell commands\n",
    "import pandas as pd\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 2. Annotation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Capture Images\n",
    "\n",
    "Select the number of how many images you want to take per class and choose the path were these images will be stored"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = ['rock', 'paper', 'scissor']\n",
    "number_imgs = 1\n",
    "IMAGES_PATH = os.path.join('datasets', 'gestures', 'images')\n",
    "LABELS_PATH = os.path.join('datasets', 'gestures', 'labels')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the directory (works for all operation systems)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dir(path):\n",
    "    if not os.path.exists(path):\n",
    "        !mkdir {path}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for structure in [IMAGES_PATH,LABELS_PATH]:\n",
    "    create_dir(structure)\n",
    "    # create subfolder\n",
    "    for folder in ['test-dev','train','val']:\n",
    "        create_dir(os.path.join(structure, folder))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Capture the images! You can adjust the time between the different frames to have more time of switching between your hand gestures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for label in labels:\n",
    "    cap = cv2.VideoCapture(0)\n",
    "    print('Collecting images for {}'.format(label))\n",
    "    time.sleep(7) #time between gestures\n",
    "    for imgnum in range(number_imgs):\n",
    "        print('Collecting image {}'.format(imgnum))\n",
    "        ret, frame = cap.read()\n",
    "        imgname = os.path.join(IMAGES_PATH,'train',label+'.'+'{}.jpg'.format(str(uuid.uuid1())))\n",
    "        cv2.imwrite(imgname, frame)\n",
    "        cv2.imshow('frame', frame)\n",
    "        time.sleep(2) #time between frames\n",
    "\n",
    "        if cv2.waitKey(1) & 0xFF == ord('q'):\n",
    "            break\n",
    "cap.release()\n",
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go to https://www.makesense.ai and label your data there. First create the label rock, then paper and then scissor to ensure that they are labeled as 0, 1, 2.\n",
    "\n",
    "**Export the labels in yolo format**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understanding the bounding box:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "def draw_bounding_box(image_path, label_path):\n",
    "    # Load the image\n",
    "    image = cv2.imread(image_path)\n",
    "    image_height, image_width, _ = image.shape\n",
    "    \n",
    "    # Extract the first bounding box coordinates\n",
    "    coordinates =pd.read_csv(label_path,sep=' ',header=None).iloc[0,1:]\n",
    "    x_rel = coordinates.iloc[0]\n",
    "    y_rel = coordinates.iloc[1]\n",
    "    width_rel = coordinates.iloc[2]\n",
    "    height_rel = coordinates.iloc[3]\n",
    "    \n",
    "    # Calculate the absolute coordinates of the bounding box\n",
    "    x = int(x_rel * image_width)\n",
    "    y = int(y_rel * image_height)\n",
    "    width = int(width_rel * image_width)\n",
    "    height = int(height_rel * image_height)\n",
    "    \n",
    "    # Calculate the coordinates of the bounding box\n",
    "    x_min = int(x - width / 2)\n",
    "    y_min = int(y - height / 2)\n",
    "    x_max = int(x + width / 2)\n",
    "    y_max = int(y + height / 2)\n",
    "    \n",
    "    # Draw the bounding box rectangle on the image\n",
    "    cv2.rectangle(image, (x_min, y_min), (x_max, y_max), (0, 255, 0), 2)\n",
    "    \n",
    "    # Display the image with bounding box\n",
    "    cv2.imshow('Image with Bounding Box', image)\n",
    "    cv2.waitKey(0)\n",
    "    cv2.destroyAllWindows()\n",
    "\n",
    "def draw_bounding_box_fast_access(subfolder, img_name):\n",
    "    image_path = os.path.join(IMAGES_PATH,subfolder,img_name+'.jpg')\n",
    "    label_path = os.path.join(LABELS_PATH,subfolder,img_name+'.txt')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_path = os.path.join(IMAGES_PATH,'train','c.jpg')\n",
    "label_path = os.path.join(LABELS_PATH,'train','paper.c1e51c9a-fe04-11ed-a540-161b77bd6551.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "draw_bounding_box_fast_access('train','paper.c1e51c9a-fe04-11ed-a540-161b77bd6551')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 3. Training"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 3.2. Load and Test Pretrained Model\n",
    "In Computer Vision, we usually use pretrained Models, to reduce the number of samples required for training. In this case, we use a pretrained YOLOv5 model, which was trained on the COCO dataset. The COCO dataset contains 80 different classes, which are not relevant for our task. For now, let's just test the model on a random image.\n",
    "\n",
    "#### Tasks:\n",
    "_**Task 3.1:**_ Can you find better models for our task?\n",
    "_**Task 3.2:**_ Can you find better pretrained weights for our task?\n",
    "_Note:_ You can find possible models and weights on:\n",
    "1. huggingface [Model Hub](https://huggingface.co/models?pipeline_tag=object-detection&sort=downloads).\n",
    "2. pytorch [Model Zoo](https://pytorch.org/docs/stable/torchvision/models.html).\n",
    "3. pytorch [Model Hub](https://pytorch.org/hub/).\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 3.3 Adapt the Model for our Task\n",
    "We need to adapt the model for our task. Therefore, we need to remove the last layer of the model, and replace it with a new layer, which only contains 3 classes (one for each hand gesture).\n",
    "Since we use the [yolov5 implementation of ultralytics](https://github.com/ultralytics/yolov5), we can use their provided training script to train our model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class yolov5:\n",
    "    \n",
    "    def __init__(self, img: int=640, conf: float=0.4):\n",
    "        self.img = str(img)\n",
    "        self.conf = conf\n",
    "        # preparing command\n",
    "        self.__algo = 'yolov5'\n",
    "        self.__exePre = 'python3 '+self.__algo+'/'\n",
    "\n",
    "    \n",
    "    def getLastWeights(self) -> str:\n",
    "        weightsPath = os.path.join(self.__algo,'runs','train')\n",
    "        if os.path.exists(os.path.join(weightsPath,'exp','weights','last.pt')):\n",
    "            # look at all directories in weightsPath\n",
    "            expFolders =os.listdir(weightsPath)\n",
    "            # choose the highest exp number\n",
    "            highestExpFolder = max([s[3:] for s in expFolders])\n",
    "            # return the last.pt of the higherst exp folder\n",
    "            return self.__algo+'/runs/train/exp'+highestExpFolder+'/weights/last.pt'\n",
    "        else:\n",
    "            # use the pretrained coco weights\n",
    "            return self.__algo+'yolov5s.pt'\n",
    "        \n",
    "    \n",
    "    def train(self, data: str, weights: str=None, batch: int=16, epochs: int=3, workers: int=0, save_period: int=1):\n",
    "        if weights == None:\n",
    "            weights = self.getLastWeights()\n",
    "        exeStr = self.__exePre+'train.py --img '+str(img)+' --batch '+str(batch)+' --epochs '+str(epochs)+' --data '+data+' --weights '+weights+' --cache --workers '+str(workers)+' --save-period '+str(save_period)\n",
    "        print(exeStr)\n",
    "        subprocess.run(exeStr, shell=True)\n",
    "\n",
    "    def liveCameraPrediction(self, weights: str=None, conf: float=None):\n",
    "        if conf==None:\n",
    "            conf = self.conf\n",
    "        if weights==None:\n",
    "            weights = self.getLastWeights()\n",
    "        exeStr = self.__exePre+'detect.py --source 0 --weights '+weights+' --conf '+str(conf)\n",
    "        print(exeStr)\n",
    "        subprocess.run(exeStr, shell=True)\n",
    "\n",
    "    def predict(self, dataYAML: str, weights: str=None, conf=None):\n",
    "        if conf==None:\n",
    "            conf = 0\n",
    "        if weights==None:\n",
    "            weights = self.getLastWeights()\n",
    "        exeStr = self.__exePre+'val.py --weights '+weights+' --data '+dataYAML+' --img '+self.img+' --conf '+str(conf)\n",
    "        print(exeStr)\n",
    "        subprocess.run(exeStr, shell=True)\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "First we define our 3 label names, and the path to the images and labels.\n",
    "\n",
    "This is done in the dataset.yaml file.\n",
    "Here we have to define the 3 classes, and the path to the images and labels.\n",
    "Make sure to use the correct mapping between classID and label\n",
    "1. 0 -> \"rock\"\n",
    "2. 1 -> \"paper\"\n",
    "3. 2 -> \"scissors\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we train our model.\n",
    "we use the dataset.yml file to define the path to the images and labels, and the number of classes.\n",
    "\n",
    "For different Parameter Configurations refer to:\n",
    "https://github.com/ultralytics/yolov5/wiki/Train-Custom-Data\n",
    "\n",
    "### Monitoring\n",
    "It is important to monitor the training process to ensure that the model is training properly.\n",
    "To do so, we recommend [Weights and Biases](https://wandb.ai/) (or [tensorboard](https://www.tensorflow.org/tensorboard)).\n",
    "Both tools keep track of the training process and automatically log the results.\n",
    "#### Weights and Biases\n",
    "1. Create an account on [Weights and Biases](https://wandb.ai/)\n",
    "2. Install the wandb package `pip install wandb`\n",
    "3. Login to your account `wandb login`\n",
    "4. Run the training script with the `--project` flag `python train.py --project <project_name>`\n",
    "5. Go to your [Weights and Biases](https://wandb.ai/) dashboard to view the results\n",
    "\n",
    "For more information on the YOLOV5 integration with Weights and Biases, refer to [here](https://docs.wandb.ai/guides/integrations/yolov5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "## with usual commands:\n",
    "# Train YOLOv5s on COCO128 for 3 epochs\n",
    "#!cd yolov5 && python train.py --img 640 --batch 16 --epochs 3 --data ../dataset.yaml --weights yolov5s.pt --cache --workers 0 --save-period 1\n",
    "# if you want to train on a previous trained model run e.g.\n",
    "#!cd yolov5 && python train.py --img 640 --batch 16 --epochs 2 --data ../dataset.yaml --weights runs/train/exp3/weights/last.pt --cache --workers 0 --save-period 1\n",
    "\n",
    "## with our class\n",
    "# usually we need to train more epochs than 3!\n",
    "model = yolov5()\n",
    "model.train('dataset.yaml')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 4. Inference\n",
    "Now that we have trained our model, we can use it to detect hand gestures in images.\n",
    "We can use the `detect.py` script to detect hand gestures in images.\n",
    "\n",
    "1. The `--source 0` argument specifies that we want to use the webcam as input.\n",
    "2. The `--weights path.pt` argument specifies the path to the weights of the model.\n",
    "3. The `--conf 0.X` argument specifies the confidence threshold.\n",
    "\n",
    "The confidence threshold determines the minimum confidence score for a bounding box to be considered as a detection. If the confidence score is below the threshold, the bounding box will be ignored. This is useful to filter out false positives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test your model with real time predictions of your camera \n",
    "#  (you need to navigate to the python symbol that is popping up)\n",
    "model.liveCameraPrediction()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predict on your test-dev data and look at the mAP\n",
    "#  remember: don't set any conf value! \n",
    "#            We want the mean of many conf values, which is enabled only with conf = 0\n",
    "model.predict('eval.yaml')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send us the ipynb notebook and your best yolo-weight vector (.pt file)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then predict your score on a test set that is unknown to you."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venvExChal",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}