Skip to content

Latest commit

 

History

History
70 lines (40 loc) · 4.16 KB

README.md

File metadata and controls

70 lines (40 loc) · 4.16 KB

LearnSign: A Static Sign Language Alphabet Game

framework hosting libraries

Won 1st place AI/ML, overall 2nd place in Anaconda's Data Science Expo 2023 supported by AI Singapore

A tensorflow.js web application that utilies TF2 object detection models to recognize real-time static American Sign Language (ASL) via web browser. This web application comes in a form of a game that recognises ASL alphabets via the user's web cam. Try the live demo at https://learnsign.vercel.app.

Web Application Home Page

image image

Academic Poster

image

Object Detection Models

Using the concept of transfer learning, we finetuned TensorFlow 2 Detection Model Zoo SSD MobileNetv2 FPNLite 320x320 model weights via the TF2 Object Detection API in Google Colab. As the computer vision models are running on tensorflow.js, inference is carried out on cilent side and no video/image data from the user is sent to the website hosting server.

1) Baseline Model

image

  • 4 Classes: A, B, C, D.
  • Finetuned on SSD MobileNetv2 FPNLite 320x320 pre-trained on COCO 2017 dataset.

2) Extended Model

image

  • 24 Classes: A, B, C, D, E, F, G, H, I, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y.
  • J and Z are excluded as they are both dynamic sign langauges involving movement.
  • Finetuned on SSD MobileNetv2 FPNLite 320x320 pre-trained on COCO 2017 dataset.

Exploratory Data Analysis

image image

Evaluation Results @11,000 steps

  • Test mean average precision (mAP) and average recall image

  • Test Confusion Matrix @0.5IOU image

  • Test Precision Recall Plot @0.5IOU image

  • Test Precision Plot @0.5IOU image

  • Test Recall Plot @0.5IOU image

  • Inference on Test Image (Groundtruth bounding box in blue with IOU Score of prediction result) image

Video Inference

video_inference_output.mp4