It is becoming increasingly difficult for users to interact with the slew of portable gadgets they carry, especially in the area of text entry. Although miniature displays and keyboards make some portable devices, such as cell phones and PDAs, amazingly small, users hands do not shrink accordingly.
To solve this problem, we proposed a Virtual Keyboard. This device will replace a physical keypad with a customizable keyboard printed on a standard A3 size paper whose keystrokes are read and translated to real input. This virtual keyboard can be placed on any flat surface, such as desktops, airplane tray tables, kitchen counters, etc. and can theoretically be interfaced with any computing device that requires text entry. This would eliminate the need to carry anything around and also prevent any chance of mechanical damage to the keypad in harsh environments if a simple lamination is used to protect the paper. In addition, buttons on this device can be reconfigured on-the-fly to give a new keyboard layout using a GUI we built in Java and then transferring that data to the device using a computers serial port.
High Level Design
The Virtual Keyboard has three main components: the laser, camera, and printed keyboard. The laser beam is simply a conventional off-the-shelf red laser with a line-generating diffractive optical element attached to it. This assembly generates an invisible plane of red light hovering a few millimeters above the typing surface. When a finger passes through this plane, it shines bright red in that region.
he CMOS camera continuously captures images of the region where the printed keyboard is supposed to be placed and checks these images for red color data above a specified threshold. The threshold concept works in this case because the laser shining on a typical human finger generates saturating values of red color data, which is very easily distinguishable from its surroundings.
Lastly, the printed keyboard is simply a standard A3 size paper that contains a custom keyboard layout. After rigorous testing, we decided to use a black background and blue letters for the printed keyboard because our device doesnt use its own light source. Therefore, proper contrast is necessary to distinguish the typing finger from the surrounding area in various lighting conditions. The actual programming of the printed keyboard layout into the device can be done using a serial port and a GUI we developed in Java. This GUI basically gives the user a blank grid of buttons and the user can choose to assign any button to any letter or number he/she desires.
The software component was split into 5 main components:
- Implementing the I2C protocol to read and write registers from camera
- Reading values from camera to obtain 6 frames every second
- Processing the images to obtain a pressed key
- Converting the pressed key into a scan code which is then transmitted using the PS/2 protocol
- Sending serial data from a java application to update the array of scan codes in the Mega32
At first we initialize PORTA on the Mega32 to take UV input from the camera and PORTC to communicate with the camera over the I2C interface. The baud rate is set to 19,200bps for serial communication. We then run the calibrate function on the camera, which looks at a black keyboard to determine a distinguishable value for red color threshold. Then we call a function called “init_cam” which performs a soft reset on the camera before writing the required values to corresponding camera registers. These registers change the frame size to 176×144, turn on auto white balance, set the frame rate to 6 fps, and set the output format to 16-bit on the Y/UV mode with Y=G G G G and UV = B R B R. The code then enters an infinite loop which checks for the status of the PS2 transmitting queue and tries to process the next captured frame if the queue is empty. If not, the queue is updated and the PS2 transmission is allowed to continue.
The getRedIndex function captures rows of data from the camera and processes each of them. We first wait until a neg edge on the VSYNC, which indicates the arrival of new frame data on the UV and Y lines. We then wait for one HREF to go by since the first row of data is invalid. At this point, we can clock in 176 pixels of data for a given vertical line in the Bayer format.
In the mode where the UV line receives BR data, the output is given by: B11 R22 B13 R24 and so on. Since we only needed red data, we stored an array of 88 values in which we captured the data on the UV line every 2 PCLKS. The OV6630 also repeats the same set of pixels for consecutive rows and thus 2 vertical lines processed would have data about the same pixels. We considered optimizing this by completely dropping data about the even rows, but this was not going to save us anything since all our processing could be done between one neg edge and a pos edge (when data becomes valid again) of HREF.
Since we dont have enough memory to store entire frames of data to process, we do the processing after each vertical line. After each vertical line of valid data, HREF stays negative for about 0.8ms and the camera data becomes invalid; this gives us ample time to process one line worth of data. After each vertical line was captured, we looped through each pixel to check if it exceeded the red threshold found during calibration. For every pixel that met this threshold, we then checked if the pixel was part of a contiguous line of red pixels, which would indicate the presence of a key press. If such a pixel was found, we then mapped this pixel to a scan code by binary searching through an array of x, y values. If this scan code was found to be valid, we debounced the key by checking for 4 continuous presses, and then added the detected key to the queue of keys to send to the PC.
A very big part of our challenge was to figure out the correct configuration to use to capture and process the images from the camera. The communication protocol was not easy to work with and there were a total of about 92 registers we could use to set up the camera. At first we considered using the TWI interface provided by CodeVision to communicate with the camera, but we were unable to do so. Thus, we decided to modify and use a version developed by Ruibing Wang, which uses a lot of the TWI settings provided on the Mega32. The protocol uses a 2-wire communication scheme, which is activated by a 10kOhm pull-up resistor. The clock signal to the camera is provided by the SCL line, and the frequency is given by: 16MHz / (16 + 2 x (TWBR)(4TWPS)). We decided the optimal solution would be to satisfy the minimum requirement which was to set the bit rate register (TWBR) to 72 and the status register (TWSR) to 0. The rest of the code just followed the standard protocol defined by Philips. The camera registers could be written by writing a start bit, followed by a target register address and then the target data. We had no need to read from the camera registers except in the initial phase when we had to make sure we had the protocol working properly.
We decided to use a resolution of 176×144 since that was the minimum required to detect an entire A3 size paper on which the keyboard would be printed. At this resolution, we could capture at most 6 frames of color images per second. The camera output format was set to capture 16 bit UV/Y data, where UV had BRBR data and Y had GGGG data. The Y data was completely ignored.
Programming the EEPROM
Since we wanted to be able to change the key assignments on the fly, we stored the array of scan codes corresponding to each key in EEPROM and turned on the RS-232 receive interrupt. We also wrote a java applet that was a simple GUI where the user can enter scan codes of the keys they desire and transmit it to the microcontroller through a standard COM port on the PC.
Keyboard Output (PS/2)
The code was structured using two timer compare interrupts where timer1 compare was used to start transmissions of each data byte and timer2 compare was used to reset the waiting. Since the protocol allows a range of frequencies that a computer would understand, we decided to use a clock time of 250 and wait time of 700. When the timer1 interrupt is fired, it transmits the bits in the following order when the clock is set to high: start bit(0), data bits, parity bit(xor of all bits), and a stop bit(1). If not, the clock state is updated. The rest of the code simply maintains a queue which would hold the elements to transmit as characters. The queue has a get and put method that updates the 2 pointers in an array.
|RS-232 Serial Port||$1.00||ECE 476 Digital Lab|
|Red LED||—||ECE 476 Digital Lab|
|Jumpers||—||ECE 476 Digital Lab|
|Surface mount capacitors, resistors||—||ECE 476 Digital Lab|
|16 MHz Crystal Oscillator||—||ECE 476 Digital Lab|
|LM340T5 Voltage Regulator||—||Sampled|
|Slide Switch||—||ECE 476 Digital Lab|
|Atmel ATMega32||$8.00||ECE 476 Digital Lab|
|40 pin DIP Socket||$2.00||ECE 476 Digital Lab|
|8 pin DIP Socket||$0.40||ECE 476 Digital Lab|
|9V Power supply||—||ECE 476 Digital Lab|
|OV6630 CMOS Camera||$25.03||Electronics123|
|PS/2 cable and USB adapter||—||Previously owned|
For more detail : Virtual Keyboard Using Atmega32