The Spatial Estimation Engine finds the best estimation of the current macroblock by using neighboring pixel values within the same image. There are a total of nine intra 4x4 luma modes, four intra 16x16 luma modes and four intra chroma modes defined in the H.264 specification.  The Spatial Estimation Engine determines the best luma and chroma modes to be used for the current macroblock. The best modes together with the corresponding estimated macroblock coding cost are sent to the Mode Decision Module. The Mode Decision Module compares the macroblock coding costs and determines the best prediction to be used. This prediction can either be inter or intra prediction.

The best intra mode is sent to the Intra Prediction Engine. The Intra Prediction Engine uses neighboring pixel values to generate the intra prediction for the current macroblock.

Based on the Mode Decision result, the Transform and Quantization Engine subtracts the prediction values from the current macroblock. Then, it performs transformation and quantization to generate the quantized coefficients. The quantized coefficients and other macroblock information such as motion vector, inter prediction mode, and intra prediction mode is collected by the Macroblock Info Collector.

The collected information is sent to the CABAC Module to generate the final bit stream. This is done by using context-based adaptive binary arithmetic coding. The resulting bit stream is then sent to the frame buffer.

The reference images used by the Motion Estimation Engine and the Motion Compensation Engine are generated by three steps: inverse quantization, inverse transformation, and deblock.  The Deblock Module reduces blocky artifacts from the image. The processed images are then sent to Frame Buffer and are used as reference images in the next frame period.

Design Details: Camera Sensor Front-End Module
The camera sensor used in this design is a MT9P031, a 5MP sensor from Aptina. This camera sensor is programmed to send 1920x1080 video images to the FPGA at 30 frames per second.  The video data format used is 12-bit RGB Bayer pattern. The data transfer clock is set to 100MHz.

Figure 4. The Camera Sensor Front-End Block Diagram

Figure 4 shows CFA Interpolation used to convert the Bayer RGB pattern to a regular RGB image. The color matrix is used to adjust RGB color to make it match with the real world. RGB to YC conversion is done next. It generates YC 4:2:2 video data that is then saved to the Frame Buffer using the DMA Engine. 

Other Design Details
An embedded processor is used for programming the various registers within the different modules as well as to run the TCP/IP stack for streaming the compressed video. Working with the Ethernet MAC module, the embedded processor runs the lwIP (a lightweight implementation of the TCP/IP stack), a streaming application, and a web server application. 

The multi-port Frame Buffer connects to two DDR2-SDRAM chips with a 32-bit data bus and runs at 150MHz. This gives a maximum of 1.2GB per second memory bandwidth.