For our ECE 4760 final project, we designed and built a pointing device with webcam-color-tracking based movement control. Our implementation allows the user to wear a set of finger-sleeves complete with buttons for clicking and scrolling and a red LED for color-tracking cursor control to interface with a computer as a normal mouse would. Users have the freedom to control their computer’s cursor movements and functions simply by moving a hand in the air without the need for a desk.
High Level Design
Rationale and Project Conception
For our final project, we wanted to do something that we thought would be interesting and could have practical functionality in day-to-day life. After some brainstorming and browsing of projects from previous years, we decided to do our own take on a pointing device that would track a user’s movements and act as a replacement to the mouse on a computer. Past groups have shown that hands-free mouse control can be achieved through the use of accelerometers to coordinate mouse movement. As such, we hoped to explore some other means of mouse position generation that haven’t been acted upon in previous years.
Initially, we explored two different ways to control cursor movement. The first was to use an array of infrared LEDs and infrared receivers to track intensity and transfer these relative changes to changes in mouse position. The second was to use a webcam and color-tracking to detect an object and track its movements. After discussing the tradeoffs of each design and some experimenting, we decided to move forward with the webcam design. Ultimately, our intent for this project was design a hand-motion based mouse that would be built around a different design than ones that are available in the market and ones that have been done in the past by other groups.
Another design choice was to use finger sleeves instead of gloves to house the buttons and red tape for color tracking. The finger sleeves allowed for a more natural typing experience since the finger tips were left exposed whereas the gloves felt weird and unnatural.
At a high level, our project consists of five main components: a webcam, three finger sleeves, a microcontroller, an on-off switch, and a Matlab interface. Each finger sleeve is fixed with a button and one of these sleeves is fitted with red tape for color tracking. The buttons provide left click, right click, and scrolling functionality and are wired to the microcontroller. Operation of the device begins with the finger sleeves and the webcam. The webcam is mounted in a convenient location and is connected to one’s computer. A user wearing the finger sleeves moves the finger with the red tape in the field of the webcam. The webcam tracks the centroid coordinates (x- and y-) of the red tape as it moves in its field of view through a Matlab interface and sends these coordinates to the microcontroller via serial communication. Upon receipt of this data, the microcontroller scales the coordinates to the resolution of the computer screen and encapsulates these with the button presses into a packet sent back serially to Matlab. Finally, Matlab unpacks the packet and interfaces with the computer with cursor movement, button clicks, and scrolling as one would expect of normal mouse. The on-off switch is for turning cursor movement off if the user wants to type on the keyboard without the annoyance of a moving cursor. All button functionality is still on even if the switch is turned off.
We found the use of the finger sleeves fitted with the push buttons to be refreshingly natural to use for both mouse control and typing compared the use of gloves fitted with the push buttons, which was our initial idea. The finger sleeves leave the finger tips exposed, allowing for a more natural-feeling when typing while the covered finger tips when using the gloves made typing awkward and uncomfortable. Additionally, the finger sleeves can easily be moved to different fingers to allow for a more personalized experience that better fits your needs and preferences in terms of button functionality.
From project conception, we knew that there were a couple of routes that we could take when it came to interfacing our webcam mouse with a computer. Simply put, one way involved the use of additional hardware in the form of a USB HID-compatible microcontroller and the second way involved a software interface using Java libraries in Matlab. After some discussion, we ultimately decided to take the software-based approach for mouse control and control cursor movements and clicks through Matlab. Intuitively this made sense to us, since Matlab would already be used for image acquisition from the webcam. Additionally, taking this approach allowed us to avoid the complexities of incorporating an additional microcontroller and incurring the costs associated with its implementation. The software approach proved to be a good decision.
When discussing the approach we wanted to take in controlling the mouse, we quickly realized that we could do so with either hardware or software. In using hardware, we would have to track the user’s movements and communicate this to the PC with an additional microcontroller. More importantly, after researching this implementation we chose not to do so due to complexity of implementation. Because we were interested in a webcam-based interface, we chose instead to utilize a software implementation to track the user which doesn’t require that communication. This would make calculation of the coordinates much easier, but also makes the image processing mandated by the webcam realizable. Given the small processing power and memory on the microcontroller, we didn’t believe it would be feasible to try to do all of these calculations onboard. Instead, we used the microcontroller’s ability to register user inputs on the order of microseconds to record the button presses.
Initially, when discussing webcam object tracking, we were posed with the design choice of tracking an object based via color or infrared. The former option would be easier to implement but be prone to tracking errors while the latter option would provide more robust object tracking at the expense of complexity. Given our time constraints for the project and the sufficiency of color tracking, we made the decision to go with color tracking over infrared tracking. Due to benefits of IR tracking, we discuss this at a later time in our conclusions as a possible extension.
Since our device would have to be recognized by a computer as a human interface device, we initially planned to use the USB HID protocol and thus would have to abide by the USB HID protocol specifications. However, we realized that interfacing the microcontroller with the computer would be made simpler if we used serial communication to send and receive data. By choosing serial communication, we were able to define our own packet structure and implement mouse control through the help of the Robot object in the Java library. The Matlab image hardware acquisition (imaqhwinfo) Toolbox was used for image acquisition from the webcam, which adheres to all standards for video communication and processing. We also adhere to the UART protocol by using prescribed 9600 baud rate as detailed on the microcontroller specification sheet. We use the UART code provided by the ECE 4760 course website linked in the Appendix.
Copyrights and Acknowledgements
Although we consulted past projects when initially trying to decide which project to do, we did not consult any of the projects for any portion of our own design. That being said, we did find and consult a very helpful open source Matlab project by A. Bhargav Anand on Matlab’s MatlabCentral File Exchange regarding red-colored-object tracking and a real-time processing in Matlab example by a man named Jerome Lecoq on Matlabtips.com. We were able to build off of these two code examples for our image acquisition and processing. Links to these two resources are provided in the Appendix.
|Part||Unit Cost||Vendor||Quantity||Total Cost|
|Atmega 1284||$5||Lab Stock||1||$5.00|
|White Board||$6||Lab Stock||1||$6.00|
|Header Pins||$0.05||Lab Stock||24||$1.20|
|Push Buttons||$0||Lab Stock||3||$0.00|
|Header-Pin Wires||$0.05||Lab Stock||27||$1.35|
|Wires and Switch||$0||Lab Stock||3||$0.00|
For more detail: The Webcam Mouse Using Atmega1284