Hand Gesture Recognition for Human Computer Interaction

This was a final project I participated at Computer Vision class at the University of California, Santa Cruz. The goal of this project is to use OpenCV in Python to implement hand tracking and gesture recognition to control mouse events.

In the following video, I use hand gestures to manipulate (such as rotate, zoom, and pan) WebGL 3D objects.

The following is draft flow chart of project program:

Process Pipeline

The source code of this project:
https://github.com/RobinCPC/CE264-Computer_Vision

Some material for each subtasks

Row Image Input
- GUI feature in OpenCV
Blur, Skin Detection, Morphology, Background Subtraction
Hand Contour Extraction
- Draw Contour and Find Convex Hull or Convexity Defects
Gesture Recognition
- Find out how many fingers show up
Hand Position Tracking
- Dynamically change center of ROI

Input Control

Note: PyAutoGUI work well in Windows system for me. Ubuntu: got issue (invisible cursor move)

For install PyAutoGUI on Linux (for python 2.x):

# install x-lib and scrot (pip not work for me)
$ sudo apt-get install  python-xlib
$ sudo apt-get install scrot

# skip install python-tk and python2.x-dev (already had)
# install Pyautogui
$ sudo pip install pyautogui

For install PyAutoGui on Window (for python 2.x):
Open cmd.exe

1
2
3

# Install pillow first (PIL module may cause error during install)
C:\Python27\Scripts\pip.exe install pillow
C:\Python27\Scripts\pip.exe install pyautogui

Turn off Windows UAC

Reference

Hui-Shyong Yeo; Byung-Gook Lee; Hyotaek Lim, “Hand tracking and gesture recognition system for
human-computer interaction using low-cost hardware”, Multimedia Tools and Applications, Vol. 74,
Issue 8, Apr. 2015, On page(s) 2687-2715.
http://vipulsharma20.blogspot.com/2015/03/gesture-recognition-using-opencv-python.html