Human Tetris — Video object tracking Using Atmega644

We have created a real-time video object tracking / shape recognition device, and a fun game library to demonstrate its abilities.
For our project, we wanted to push the video sampling and processing capabilities of the ATmega644 8-bit microcontroller. Using a high-speed analog-to-digital converter as an input device, we were able to sample a reasonably high-resolution grayscale image from a color camera’s video output. Using this grayscale image, we are able to track objects and recognize shapes that stood out from the background by a customizable threshold.

We created a game called Human Tetris to show off the system’s shape recognition capability. In this game, players must contort their bodies into shapes displayed on screen in a given amount of time. To demonstrate the potential for our device to expand to a larger game library, we also implemented a port of Brick Breaker and another ineractive game called Whack-a-Mole. Brick Breaker shows off the object tracking capability, where players must physically interact with the bouncing ball to keep it on screen and break bricks. Whack-a-Mole requires a player to flail around on the screen to hit all of the moles before they disappear.

High Level Design

Almost all major gaming consoles are moving to user input systems involving tracking all or part of a player’s body (see Nintendo Wii Remote, PlayStation Move, and XBox Project Natal). Our project setup is similar to Project Natal, which Microsoft claims to be a “controller-free gaming and entertainment experience”, using a multiple-sensor device (with at least camera and microphone) placed by the TV. We have demonstrated the feasibility of a basic motion-controlled gaming system using low-cost parts.
The system is based off of an ATmega644 microcontroller with two peripherals: a high-speed flash analog-to-digital converter, and an onscreen display device for video overlay. With just 4 kilobytes of RAM and a 20MHz clock speed, the MCU needs these peripherals to take some of the load off of its processing capabilities. Since we wanted to provide a real-time camera feed on the display (like a “mirror” for the player), and since the ATmega644 is not fast enough to generate a full color NTSC signal, we decided early on that we would use a color video camera module which outputs an NTSC signal directly.
The system accepts a color NTSC video signal, filters out the DC component, and samples the video stream at real-time speed. This sample is stored as a grayscale image in the MCU’s memory which can be processed by the application. The onscreen display overlays shape information and game data for user interaction.
Based on this system, we have implemented a video arcade game called Human Tetris, based on a Japanese game show event. In the game, contestants are faced with an oncoming solid wall with a shape cutout. The players must fit through the shape, or else they are thrown back into a pool of slime.
Our game displays a shape on the screen which the players must fit into within a short amount of time. If they succeed, they are faced with more difficult shapes and shorter amounts of time until they are inevitably “slimed”. The game provides the player with real-time feedback by displaying the video on the screen with the device’s interpretation of their shape.
Our project involves two standards: NTSC and SPI. NTSC is well-defined and is used as our input and output medium. SPI is more of an agreement than a standard, and our project uses it internally to the specifications of the microcontroller and on-screen display device.
Tetris is copyrighted and trademarked thoroughly, so attempts to turn this project into a product would have to be preceded by licensing talks or altering the aesthetics. “Human Tetris” is a commonly used description for the game show our game is based on, but the actual game play is not that of Tetris.

Hardware

Prototyping

Our finished product is neatly tucked onto a 2.5″ x 3.8″ custom printed circuit board, but for most of the design stage, we prototyped on an STK500, a solderless breadboard, and the ATmega644 custom target board. Our earliest prototyping simply involved working with the ATmega644 on the STK500, with the surface-mount peripherals soldered to breakout boards on a solderless breadboard, attached to the STK with long, loopy wires. We also used the large protoboards as an external power source to feed our substantial power requirements. Slowly, we weaned off our dependence on the STK and moved to using just the solderless breadboard and the target board, with an on-board power circuit. Once the PCB was set up, everything was contained directly on the board.

Design & Components

At the start, we decided in our high-level design that it was essential for the user to have real-time feedback of their position on the screen. This allows the player to correct their position on screen or move to an object. Since our microprocessor is not capable of outputting color NTSC, we decided to use a camera which outputs an NTSC signal directly. Then, we could split the signal for display and processing.

We chose the CM-26N color video camera because it was moderately priced and simple to use: simply plug in power, and it outputs a video signal. It also has a few features we were not aware of when we bought it, such as auto-brightness. One thing that we did not realize until setting up the system was that a camera pointed at the player would not produce a “mirror” effect like desired; instead, if a player moves to the right, he will move left on the screen. To remedy this, we used a low-tech solution: we point the camera at a mirror to flip the image it sees and displays on the screen, so that the game feels more natural to the player.
The signal from the camera goes through an RC input filter which biases up the signal swing to about 0.5V to 2.0V. This is based on the MAX7456 datasheet’s “Typical Operating Circuit”, which specifies a 75 Ohm load resistor to ground and a 0.1uF input coupling capacitor. Originally we had intended to create two separate circuits for display and processing using current buffers, and a shift/gain op-amp for input into the analog-to-digital converter. However, it was simpler and worked just as well to use the same signal as an input to both devices; it did not produce any detrimental effects.
The signal from the camera went into the analog input of the TLC5540 flash ADC. Since this signal swung from about 0.5-2.0V, we set up a bottom reference voltage of 0V and a top reference voltage of 2.0V, using a combination of the ADC’s internal voltage divider and a parallel resistor. This provided a good level of resolution for our application.

We set up a clock output from the MCU to go into the sampling clock port of the ADC, so that we could control the clock speed through software. The 8-bit parallel output from the ADC went to the 8 pins on the MCU’s port C. During prototyping, we experienced a lot of noise on these high-frequency digital signals, which produced some bad reads and unexpected behavior. The custom PCB brought these components must closer together with good traces between the two devices, which essentially eliminated all of these problems.
The MAX7456 on-screen display (OSD) was used as our primary output device, to overlay characters and data on top of the video signal going from the camera to the TV. The SPI interface between the MCU and the OSD allowed high-speed communication for user feedback. It also took a lot of processing demands off of our MCU, since we could just tell the OSD what to do over SPI quickly, and the OSD would take care of the video overlay and timing issues.

The OSD also provided some incredibly useful synchronization pulses for vertical sync and horizontal sync in the NTSC signal. We used these outputs to tell our MCU when to sample the video. We had a lot of trouble during the prototyping stage with bad writes to the OSD, most likely due to noise in the high-frequency communication on the SPI wires connecting the devices. Similar to the case with the ADC, when these components were carefully placed on the custom PCB, all of these issues disappeared, and the SPI communication could be cranked up to several MHz.
While we were originally planning on using the ATmega644 MCU, we wanted to sample the part, and we could only get samples of the ATmega644PA. These microprocessors are virtually identical (the ‘P’ is a low-power version of the chip, and the ‘A’ represents a manufacturing change). Virtually none of the hardware or software needed to be changed to accommodate this new chip.

We opted to run the chip at the maximum clock speed of 20MHz for highest performance, but this is still below the threshold for direct video capturing or generation, which is why we had to use the two peripherals. We were also limited to the 4K of RAM, which limited the amount of pixels from the sampled image we could store. Otherwise, the MCU acted a central hub for the entire circuit: everything was controlled by the MCU, and it implemented all of the game and user interface logic.
We determined that the current drawn by the total circuit was close to 0.6A (camera ~120mA, OSD ~430mA, flash ADC ~20mA, atmega644pa ~12mA), so we used a power adapter and created a power circuit that supplied up to 1 Amp of current. We used a 12V DC power supply, a 1A rectifier (1N4001), and a 5V voltage regulator (LM340T5). We also included a power switch to break the connection from the supply to the power circuit.
Our finished product contains an on-board power circuit with power switch, reset and user interface buttons, blinking LED heartbeat (with jumper to turn off), camera and program headers, exposed RX/TX ports for serial debugging, three surface mount chips (ADC, MCU, OSD), two crystals, several resistors and capacitors, a ground plane to reduce noise/interference, and an RCA jack for video output to a TV.

Custom PCB

About half way through the design stage, we realized that a custom PCB could be very beneficial to our project. After getting approval from Bruce Land, we decided that we would attempt to design and build a custom PCB after we had finalized our hardware design, budget and time permitting.

Parts List:

For more detail: Human Tetris — Video object tracking Using Atmega644