Author: jk4e | Last Update: August 2024

This is a collection of demos as a Streamlit app showcasing the capabilities of custom-trained HGR models on freely available SDKs and frameworks for various HGR tasks. Select a model configuration from the sidebar to try out the different models and tasks. Have fun exploring!

• Support for Images🖼️, Videos🎞️, and Webcam📷 Photos and Video Streams
• Custom YOLOv8 detection model for hand detection task (HGR)
• Custom YOLOv8 classification model for hand classification task (HGR)
• Custom MediaPipe hand gesture recognition task (HGR)

Note on Webcam Support for Video Stream:
Webcam real-time video stream is not yet supported on the WebApp. This functionality is not natively possible on Streamlit Cloud, and would require a custom component such as streamlit-webrtc. Currently, there is no fast and easy way to integrate local webcam streams in the WebApp. (see 1, 2, 3)

Streamlit app (MediapipeYolo-HGR) with MediaPipe and YOLOv8 hand gesture recognition demos using custom trained models. The Streamlit Community Cloud is used to deploy the app. Check it out!

LINK TO APP: https://my-hgr.streamlit.app

Note: You can change via settings the app theme between Light or Dark (default is Use system setting)

These screenshots show some app pages.

This app uses custom-trained hand gesture models created with YOLOv8, MediaPipe Gesture Recognition Task, and Streamlit as the user interface.

For image annotation, i used CVAT (Computer Vision Annotation Tool). There are several free alternatives available in the market, such as those offered by Roboflow Annotate.

Freely available datasets for training can be found on platforms like:

• Roboflow
• Kaggle
• GitHub
• Google Dataset Search

Alternatively, you can generate your own dataset using a webcam on your PC or notebook.

For training, it's recommended to have access to:

• A powerful NVIDIA GPU with CUDA support, or
• Cloud compute instances (e.g., Google Colab Free Plan for beginners offers a cloud-based Jupyter notebook environment)
  • Google Colab provides access to GPU resources and pre-installed libraries

This app is deployed on Streamlit Community Cloud, which is free to use. Other deployment options include:

• Private deployment on Streamlit Community Cloud
• Local usage
• Non-GUI options like Jupyter Notebook
• Custom GUI apps using packages like Tkinter or PyQt

• Ultralytics YOLO Docs
• Model Training with Ultralytics YOLO
• Object Detection
• Image Classification

• Gesture recognition task guide
• Hand gesture recognition model customization guide

As of August 2024:

• The latest version of MediaPipe Model Maker (current version 0.2.1.4) cannot be installed on Windows or Silicon-based macOS.
• For more details, see issues: How to install mediapipe_model_maker 0.2.1.4 in Windows? #5545 or Windows Support #1206

1. Use a machine with Linux
2. Set up WSL2 (Windows Subsystem for Linux) on a Windows machine
   • See Install WSL for instructions
3. Use Google Colab, a free cloud-based Jupyter notebook environment
   • Google Colab provides access to GPU resources and pre-installed libraries

• This package is required if you want to train custom models with your own dataset.
• If you don't need to train or customize a model, you can simply install the MediaPipe package
  • See the Guide for Python for installation instructions.

• https://streamlit.io/
• Live Inference with Streamlit Application using Ultralytics YOLOv8

This project requires Python 3.11. (Currently, only Python 3.11 is tested.)

# Create a new environment named 'my_hgr_env'
conda create -n my_hgr_env python=3.11 pip

# Activate the environment
conda activate my_hgr_env

Alternatively, you can use other virtual environment tools like venv or virtualenv.

git clone https://github.com/jk4e/my-hgr
cd my-hgr

pip install -r requirements.txt

This process may take several minutes depending on your internet connection and system performance.

Note for beginners: If these instructions are unfamiliar, consider searching for "Python beginner setup guide" or "How to use virtual environments in Python" for more detailed explanations. You can also use AI assistants like ChatGPT or Claude for step-by-step guidance tailored to your operating system.

Ensure you're in the project directory and your virtual environment is activated, then run:

streamlit run streamlit-app.py

This command will start the Streamlit server and automatically open the application in your default web browser. If it doesn't open automatically, you can manually navigate to the URL displayed in the terminal (usually http://localhost:8501).

• Fix bugs and errors
• Refactor codebase
• Optimize the performance, response time, and design of GUI

• Add new YOLOv8 tasks (Instance Segmentation or Pose Estimation)
• Add support for additional media input formats, e.g., YouTube videos
• Add support for user-uploaded models
• Collect, annotate, and preprocess new datasets for model training
• Train models with higher accuracy for better results (access to better hardware required)

This project is for educational purposes only.