-2

Object:- How can I implement web cam mouse using visible light.

Details:- there will be a web cam seeing my right hand in visible light as follows i.stack.imgur.com/6ztqf.jpg :-

I want to make a system where I will control the mouse pointer with my (right) hand in the same way as there is an actual mouse in there. I will put my hand on the table and act like there is an actual mouse in my hand and will do all the actions that I would do as if there was a mouse in my hand. And there will be a normal web cam (may be good resolution to make accuracy high) watching my hand at certain angle from above.

Suppose I have magical Machine Learning system to do that. (I can accept all sort of constraints like, fixed camera position all the time, mouse move in limited rectangle all the time etc...)

In training phase, In the training phase I will use my actual mouse as I normally would and generate a large number of data. Data will consists of large numbers of images as inputs and, mouse position, clicks as outputs. I am flexible in everything like I can add two microphone to detect the sound of clicks to make sure that there is a click.(but sound of mouse click from actual mouse is different from sound when I hit the finger on table to mimic mouse click with my invisible mouse, but presence of a sound will make sure that there is a click. And two microphones will make sure that it is a left click or right click.) Now with all this data my system will learn an association between image of my hand and mouse actions.(may be not very accurate in this first version).

In running phase I will remove the mouse and put my hand at that same position as I would if there was a mouse and will start using this system to convert my hand images into mouse actions. I will move my have in 2D plane of table and move fingers like I am right/left clicking. My system should take images in real time and should convert those in mouse action with help of all the data that I collected in training phase.

Please help me how can I build this, I have average knowledge of Computer Vision and Machine Learning see this image(my web cam will see these types of images)

enter image description here

gre_gor
  • 6,669
  • 9
  • 47
  • 52
Hacklavya
  • 1,275
  • 8
  • 8

1 Answers1

0

Do not try to use ML algorithm to learn association between images and mouse actions. Better to define several key points within your hand in training images (for example, center of your hand or tip of your finger). Than train a regressor to search for such keypoint within image. Example where similar task is solved (search for facial keypoints, like eye or nose tip): http://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/

Than you just hardcode actions using the keypoints. For example 'The mouse pointer should be located at same position as center of hand'.

Finally, you use your trained regressor in 'combat mode' - it recognizes keypoints in image and the needed actions are performed as coded.

Maksim Khaitovich
  • 4,742
  • 7
  • 39
  • 70
  • Thanks a lot for answering. May I please talk/chat to you, I need some more clarifications. It will be not efficient and will take long time if I write what I want to say. Thanks. – Hacklavya Oct 05 '15 at 16:35
  • Sorry, was out sick. The while idea of stackoverflow is to post answers so that other fellow developers could find these answers if they face the same issue :) Edit your question to provide more insight into what you want to clarify and I will edit my answer to address your questions. – Maksim Khaitovich Oct 08 '15 at 17:38