SignPose: A Dynamic Sign Language Detection Game

A LSTM Keypoint Model deployed on Flask that detects real time dynamic sign language (ASL animals) on browser.

Explore AlphaSign, the static version of our sign language game here.

Web Application Home Page

Animal Dynamic Sign Language Detection

1. Cow

2. Elephant

3. Butterfly

4. Gorilla

5. Bird

Video Demo of Web App

signpose_video_demo.mp4

Exploratory Data Analysis

Breakdown of Train Test Split using Stratified Sampling (to ensure even distribution of train and test data)

LSTM Model Architecture

# LSTM Sequential Model using Keras
model = Sequential()
model.add(LSTM(64, return_sequences=True, activation='relu', input_shape=(30,258))) #each video has input shape of 30 frames of 258 keypoints: X.shape (Handpose + Bodypose)
model.add(Dropout(0.2))
model.add(LSTM(128, return_sequences=True, activation='relu'))
model.add(Dropout(0.2))
model.add(LSTM(64, return_sequences=False, activation='relu')) #next layer is a dense layer so we do not return sequences here
model.add(Dense(64, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(actions.shape[0], activation='softmax'))

# Compile defines the loss function, the optimizer and the metrics. 
model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics=['categorical_accuracy'])

Model Evaluation

Model Training Graphs

1. Train & Test Categorical Accuracy over epochs

2. Train & Test Loss over epochs (early stopping implemented to prevent overfitting)

Confusion Matrix

1. Train Confusion Matrix

2. Test Confusion Matrix

Installation in command prompt

1. Clone Repo

git clone https://github.com/ngzhili/LSTM_Keypoint_Sign_Language_Detector.git

2. Create your virtual environment (venv) using conda

conda create -n <VENV_NAME> python=3.8

3. Activate your virtual environment (venv) using conda

conda activate <VENV_NAME>

4. Install dependencies in venv

pip install -r requirements.txt

5. Change directory to Sign-Language-Image-Recognition

cd <PATH/TO/Sign-Language-Image-Recognition>

6. Run Application in same directory as app.py

flask run