SOPC-Based Cooperative Awareness Nodes in Smart Multimedia Sensor Networks

Yi Zhou1, Huiping Li2, Chunlin Wan2, Zhentao Hu1, Yong Jin1 1School of Computer and Information Engineering, Henan University, Kaifeng 475004, China

2Basic Experimental & Teaching Center, Henan University, Kaifeng 475004, China E-mail: {zhouyi, lihp, wancl, hzt, jy}@henu.edu.cn

Abstract—The design mind of SOPC (System on Programmable Chip), hardware & software co-design, will bring the system optimization with larger degree of freedom. It will establish the foundation on realizing the miniaturization and more efficient local preprocessing in smart cooperative sensor networks. In this contribution, a solution is proposed for smart multimedia sensor nodes based on the SOPC framework, aiming at improving the on-board processing capability and the degree of freedom in development. This paper mainly presents the structure and design method of the smart cooperative multimedia sensor node based on SOPC. The on-board feature extraction is realized and some correlative experimental data and results are presented as well. The experimental results evaluate the feasibility and efficiency of the proposed node design method.

Keyword: SOPC; Multimedia Sensor Networks; Cooperative Awareness;on-board processing; feature extraction

I. INTRODUCTION With the rapid development of embedded computing and

distributed cooperation technology [1-5], the traditional information processing pattern with the mass storage system and high-performance server is no longer the only solution. Building a harmonious man-machine environment, where it is easy of real-time access to the real-world data stream, i.e., a pervasive computing concept. The strong computing power can be realized through cooperative awareness nodes in wireless sensor networks (WSN), which also provides a strong support for the development of the information sensing technology in internet of things (IoT).

Using the sensor embedded computing and distributed cooperative processing, WSN can share various information acquired by sensors through wireless networks, and transmit these information to the remote control center for more analyzing and decision-making. WSN present wider application outlook from military to industry and agriculture, from medical treatment to environment monitoring, so that many universities and research institutes at home and abroad have devoted themselves into developing various WSN. Especially, more researches have been carrying on for wireless multimedia sensor networks recently, which will bring WSN applying a new breakthrough and broader space [6-7]. To implement more accurate and fine-grained environment monitoring, it is necessary to introduce image or video

multimedia data into Wireless Sensor Networks, and there come into being smart multimedia sensor networks with higher awareness ability. In [8], a scalable architecture for video sensor networks was built in a StrongARM-based embedded Linux board, and the IEEE 802.11 was used for the network communication. In [9], a new mote was proposed for distributed image sensor networks, where 32-bit ARM7TDMI CPU Core with IEEE 802.15.4 standard was used. In [10], an image sensor node was designed based on ARM7 and FPGA, where FPGA was used as the coprocessor. In [11-12], a multi-tier camera sensor network was proposed, where sensor nodes own different processing capability in different layer. Atmega128 MCU, OKI ARM, and XScale PXA255 were used to design the nodes with different sensing layer. In [13], a distributed camera event recognition network was presented as a scalable solution for tracking through wireless sensor nodes with embedded low-power image sensors.

As the demand and application of IoT awareness increasingly rise, the academic circles have been making active research on wireless multimedia sensor networks. However, there exist some limitations in the nodes based on traditional microprocessors, such as limited processing capability, low degree of freedom in development, and weak transplantation performance. Aiming at these problems, the design scheme of SOPC-based wireless multimedia sensor nodes is proposed in this contribution, and a hardware development prototype of the smart multimedia sensor node is set up. This paper presents a solution for the design of smart multimedia sensor nodes based on SOPC framework. The design mind of SOPC, hardware & software co-design, will bring the system optimization with larger degree of freedom, and will establish the foundation on realizing the miniaturization and more efficient local processing of smart cooperative sensor nodes. This paper mainly analyzes the structure and method of the smart cooperative multimedia sensor node with SOPC. Some correlative experimental data and results are presented as well.

The organization of this paper is as follows. First, the design structure and working state of the proposed awareness node are presented, followed by an analysis of the intellectual property (IP) core design method for the image feature extraction. Then the node system based on SOPC and relevant experimental results are presented, followed by a conclusion of this paper.

This work is supported by the Nature Science Foundation of Henan University (No. 2011YBZR033).

2012 7th International ICST Conference on Communications and Networking in China (CHINACOM)

978-1-4673-2699-5/12/$31.00 © 2012 IEEE71

II. NODE STRUCTURE DESIGN Compared with traditional wireless sensor nodes, smart

multimedia sensor nodes require higher on-board processing capability, greater storage capacity, and higher real-time performance etc. Using the SOPC technology, the 32-bit Nios II soft core processor is embedded in the FPGA chip. Owing to the high-speed parallel performance in the FPGA and the advanced digital signal processing (DSP) capability, the node system integration, performance and stability are further improved combined with the IP core multiplexing technology.

Figure 1. The state transition diagram of the awareness node

As depicted in Fig. 1, the main state transition for the multimedia sensor nodes includes initialization, receiving, transmission, information collection, information processing and compression. The initialization state mainly conducts self-configuration and self-checking of each node. The receiving and transmission state take charge the system commands and target information communications. The information collection state gains scene information through various types of sensors. The information processing state carries out the information feature extraction and eliminates redundant information. The information compression state encodes the processed interested information for improving the transmission efficiency.

Avalon

Bus

Figure 2. The framework diagram of the awareness node

The general design idea of the awareness node is as follows: Nios II, a 32-bit configurable soft-core processor, is embedded into a FPGA chip. The on-chip processor and peripherals are connected through the Avalon bus switch architecture, which sets up a single-chip embedded SOPC system. Some time-consuming processing units, such as image acquisition controller, discrete cosine transform (DCT) etc., can be realized by the hardware logic units in FPGA. Then the node system

can handle various tasks in parallel to enhance the real-time performance of the system. In addition, the image edge detection, JPEG compression and other complex processing algorithms can be modeled through the DSP Builder, and then be transferred into the HDL modules for the download and evaluation in FPGA.

As depicted in Fig. 2, the node structure mainly include the image acquisition, image processing, feature extraction, image compression and radio communication module. To evaluate the system performance more directly during the debugging process, a TFT LCD Module is also added in the node system. Moreover, according to different application requirements, the expansion interfaces are reserved for other types of sensors.

III. FEATURE EXTRACTION IP CORE SOPC-based hardware & software co-design supports more

flexible system development. A major feature of the SOPC technology is to reuse pre-designed and verified IP core, which is also a very effective solution. Generally the IP core should be characterized with testability, scalability, adaptability and standardization.

According to the design specifications, the IP module function is designed through the register transfer level (RTL) program using the hardware description language (HDL). The RTL file is a soft core, i.e., soft IP. Soft core is provided in the form of source code, so it owns a higher flexibility, which has nothing to do with the specific realization process. In this contribution, the image feature extraction was realized based on the Sobel operator through the SOPC technology, which is also evaluated in the proposed multimedia sensor hardware platform.

There are three main classical first-order edge detection operators, including Roberts operator, Prewitt operator and Sobel operator. Roberts operator is simple and intuitive, owning better effect in detecting the horizontal and vertical edges. Though it can provide the high positioning accuracy, but Roberts operator is susceptible to the noise. Prewitt operator owns better effect in detecting oblique step edges, and it has a smoothing effect that can filter some noises. However it also smoothes the real edges while removing part of the pseudo edges, and the positioning accuracy is not very high. Comparatively, Sobel operator can provide the most accurate edge direction estimation.

Sobel operator calculates the gradient using a 3×3 neighborhood template, where aiming at every pixel of an input image the weighted difference is inspected between the neighbor gray up and down, left and right. Sobel operator can be expressed as,

( ), x yS i j S S= + (1)

where Sx and Sy can be calculated using the 3×3 template as follows,

1 0 12 0 21 0 1

xS−⎛ ⎞

⎜ ⎟= −⎜ ⎟⎜ ⎟−⎝ ⎠

1 2 10 0 01 2 1

yS⎛ ⎞⎜ ⎟= ⎜ ⎟⎜ ⎟− − −⎝ ⎠ (2)

72

If we define f (x,y) as the pixel of an input image, the bidirectional gradient operator can be expressed as,

( ) ( ), ,H xG x y f x y S= ⊗ (3)

( ) ( ), ,V yG x y f x y S= ⊗ (4)

The gradient magnitude is,

( ) ( ) ( )2 2, , ,H V H VG x y G G G x y G x y= + = + (5)

Thus the edge extracted is,

( ) ( )1 ,,

0 OthersG x y T

E x y>⎧

= ⎨⎩

，

， (6)

where T is the predefined gradient threshold. Considering the hardware realization, the following transformation is conducted,

( ) ( ) ( ) ( )( ) ( ) ( )

, 1, 1 2 1, 1, 1

1, 1 2 1, 1, 1HG x y f x y f x y f x y

f x y f x y f x y

= − − + − + − + −

+ − − + − + + (7)

( ) ( ) ( ) ( )( ) ( ) ( )

, 1, 1 2 , 1 1, 1

1, 1 2 , 1 1, 1VG x y f x y f x y f x y

f x y f x y f x y

= − + + + + + + −

− − − − − + − (8)

Figure 3. The IP core of the feature extraction

According to the edge detection principle of Sobel operator, the gradient operator and threshold comparison can be realized in FPGA in parallel. to achieve based on the Sobel operator is. As depicted in Fig. 3, the IP core based on the Sobel operator is designed for the image edge detection. The module Image_Line_Buffer plays the data buffering effect, where the input data are operated using the delay shift to wait for further processing. The module Sobel_detect_v is a vertical Sobel operator filter, and the module Sobel_detect_h is a Sobel operator level filter. These two filters run in parallel pipeline. The module Combine_Sobel_v_h then merges the output of the horizontal and vertical filter. After the threshold comparison, the image edge feature can be obtained.

In the SOPC-based multimedia sensor node system, to use the image edge detection IP core, the Avalon bus interface

should be designed for the IP core. The Avalon bus interface standard can control the data exchange between the peripherals and inner logic units. The embedded Nios II processor first loads the image data stored in SDRAM through DMA, and then the Edge_Sobel edge detection coprocessor processes the data. The extracted edge image can be displayed or transmitted through the SRAM cache. As depicted in Fig. 4, the edge detection IP core is packaged through the Avalon bus interface co-processor model.

Figure 4. The IP core packaging with Avalon interface

IV. SOPC-BASED SYSTEM EVALUATION The design conception based on SOPC architecture is

mainly reflected in the system customizability. This flexible method allows us to create highly customized system on programmable chip in a shorter time. Using SOPC Builder, the complex system components, such as Nios II soft core processor, IP cores, memory interface, bus bridges and custom peripherals, can be quickly integrated into the high-density FPGA. Then a Nios II-based embedded system is generated. The SOPC Builder can generate the software development environment, including the header files and peripheral drivers. Any hardware changes will be immediately reflected in the software development environment. The idea of this hardware & software co-design has greatly improved the efficiency of the system design.

Figure 5. The design realization of the SOPC system

As depicted in Fig. 5, the entire awareness node system is realized through the SOPC Builder. The Nios II processor, flash memory, PIO, interval timer, SDRAM Controller, and Avalon tristate bridge are the SOPC standardized components, while other components are the custom ones. The TFT LCD

73

controller (for debugging), video input module and I2C master module are designed and generated by the Verilog HDL. The edge detector is designed through the DSP Builder modeling and transformed into the HDL module.

As depicted in Fig. 6, the simulation is conducted for the edge detection IP core with the Avalon bus interface. When the signal filter_data_valid, allowing post_thresh into the FIFO, is a high level, writedata begins to write into a write FIFO in 65ns. At the same time the post_thresh begins to write into a read FIFO. After four clock delay, both the signal dataavailable and read_next_pixel are effective, in pix_value we can find the pixel signal output. After five clock delay, the processed signal is outputted.

Figure 6. The simulation sequence diagram of Avalon signals

As depicted in Fig. 7, the final experimental evaluation results are presented. Fig. 7 (a) and (b) are the experimental background image and the target image separately. Fig. 7 (c) and (d) are respectively the simulation results in the DSP Builder. Fig. 7 (e) is the simulation result after the removal of background. Fig. 7 (f) presents the on-board processing result and TFT-LCD display in the SOPC-based multimedia sensor node prototype.

Figure 7. The experimental evaluation of the system

Finally through the whole hardware & software compilation, the inner resource utilization situation in FPGA is analyzed in Fig. 8. It can be found that the internal resources have been

effectively used, and there are remaining resources available for system extensions.

Figure 8. The analysis diagram of the inner logic routing in FPGA

V. CONCLUSIONS Smart cooperative awareness is a challenging technology.

There still exist a lot of bottleneck problems to be solved, but it has a high research value and application prospect. In this contribution, using the embedded hardware & software co-design method, the smart multimedia sensor node is designed based on SOPC architecture. The future work is to conduct a more in-depth validation and evaluation, especially in the improvement of the low power consumption and the feature extraction algorithm.

REFERENCES [1] Y. Zhou and Z.G. Pan, “Impact of LPF Mismatch on I/Q Imbalance in

Direct Conversion Receivers”, IEEE Transactions on Wireless Communications, vol. 10, issue 4, pp. 1702-1708, Jun. 2011.

[2] Y. Zhou and T.S. Ng, “Performance analysis on MIMO-OFCDM systems with Multi-code Transmission”, IEEE Transactions on Wireless Communications, vol. 8, issue 9, pp. 4426-4433, Sept. 2009.

[3] Y. Zhou, J. Wang, T.S. Ng, K. Higuchi and M. Sawahashi, “OFCDM: a promising broadband wireless access technique”, IEEE Communications Magazine, vol. 46, pp. 39-49, March 2008.

[4] Y. Zhou, J. Wang, and M. Sawahashi, “Downlink transmission of broadband OFCDM systems---Part II: Effect of Doppler Shift”, IEEE Trans. Commun., vol. COM-54, no. 6, pp. 1097-1108, June 2006.

[5] Y. Zhou, J. Wang, and M. Sawahashi, “Downlink transmission of broadband OFCDM systems---Part I: Hybrid Detection”, IEEE Trans. Commun., vol. COM-53, no. 4, pp. 718-729, April 2005.

[6] H. Ma, D. Tao, “Multimedia Sensor Network and Its Research Progresses,” Journal of Software, 2006, 17(9): 2013-2028.

[7] D. Tao, H. Ma, Y. Liu, “Energy-Efficient cooperative image processing in video sensor networks,” In Proc. of the Pacific-Rim Conf. on Multimedia. Berlin: Springer-Verlag, 2005, 572-583.

74

[8] W. Feng, B. Code, E. Kaiser, “Panoptes: A scalable architecture for video sensor networking applications,” In Proc. of the ACM Int’l Conf. on Multimedia, New York: ACM Press, 2003, 151-167.

[9] I. Downes, L. Baghaei Rad, H. Aghajan, “Development of a Mote for Wireless Image Sensor Networks,” In Proc. of Cognitive Systems and Interactive Sensors (COGIS 2006). Paris: Mar. 2006.

[10] Z. Cao, Z. Ji, M. Hu, “An image sensor node for wireless sensor networks,” International Conference on Information Technology: Coding and Computing (ITCC 2005), Apr. 2005, 740-745.

[11] P. Kulkarni, D. Ganesan, P. Shenoy, Lu QF, “SensEye: A multi-tier camera sensor network,” In Proc. of the 13th Annual ACM international Conference on Multimedia. New York: ACM Press, 2005, 229-238.

[12] P. Kulkarni, D. Ganesan, P. Shenoy, “The Case for Multi-tier Camera Sensor Networks,” In NOSSDAV’05. Washington: June, 2005.

[13] Teresa H. Ko, Nina M. Berry, “On Scaling Distributed Low-Power Wireless Image Sensors,” In Proc. of the 39th Hawaii International Conference on System Sciences. Hawaii: June 2006.

75