Session T3C-3 AUREOLE: A CAPSTONE PROJECT INTEGRATING …aseegsw.com/past...

Proceedings of the 2011 ASEE Gulf-Southwest Annual Conference University of Houston

Copyright © 2011, American Society for Engineering Education

Session T3C-3

AUREOLE: A CAPSTONE PROJECT INTEGRATING HARDWARE AND SOFTWARE TO BUILD AN

OMNIDIRECTIONAL MOBILE VIDEOCONFERENCING CAMERA

William Reading, Sejun Song, Troy Kensinger, Kyle Mays, George Wright Electronics Engineering Technology, Texas A&M University, College Station, TX

[email protected], [email protected], [email protected], [email protected], [email protected]

Abstract

A small scale, low-cost, highly mobile 360º video conferencing platform would improve the remote video conferencing experience by providing the virtual presence for multiple remote users in high definition. Virtual presence can be achieved by enabling the remote users to freely select a desired view and zoom angle without frequent help from a local camera controller. The device is capable of joining both wired and wireless networks, and powered from battery. Potential applications for this device include remote distance education, video conferencing, and surveillance, to name a few. In this paper, we discuss the problem statement, design, and implementation of such a device, as well as end-user software required to interact with the device on both desktop web-browser and common smart-phone environments, such as iOS and Android. We also present preliminary result of the system development and demo.

Introduction

The rapid growth of Internet has been changing the normal work pattern. Whenever the Internet connection is available, people in remote can collaborate via email, instant messaging, video conferencing and regular or cell phones. Although the remote collaboration is inevitable and may represent the future of work, others argue that it cannot overcome the face-to-face communication of the physical workplace environment that can have synergies from the impromptu meetings or casual hall way chatting. Despite the technical advancement, there still are various problems on the tele-conferencing system. As the practical example, if people want to have a face-to-face meeting with the remote workers, they need to find a room with video conferencing equipment and set a meeting time. The camera cannot be controlled by the remote users or only can be controlled by one at a time. Normally, the video conferencing system is expensive and not mobile.

To foster the remote collaboration, there is a need for a low-cost, highly mobile video conferencing platform that provides virtual presence for multiple remote users. Virtual presence means that you never have to get someone to move the camera to see what you want. Every remote user can independently zoom and turn the camera about, without disturbing others watching. In this paper, we present AUREOLE that provides a 360º video-streaming portal into a room accessible over standard TCP/IP networks. Once configured, it can be accessed remotely



with no interaction required from the remote end. We discuss the architecture and implementation of the AUREOLE physical device, as well as the client software required to interact with the device on both desktop web-browser and common smart-phone environments, such as iOS and Android. We also present preliminary result of the system development and demo.

The proposed virtual presence system, AUREOLE, is unique comparing with other existing projects. For example, [1] is different that they focus on larger spaces than a conference room, and their application is tuned accordingly. They also use multiple cameras. [2] is different that it is lower resolution and uses interlacing for video output. AUREOLE is also smaller and takes advantage of a limited field of vision needed for conference rooms). [3] uses low cost omni-directional sensors, but applies them for use with machine vision, not teleconferencing. [4] uses a combined pan/tilt camera and a cadadioptric to move around. AUREOLE moves in software because we get a high-resolution image out of the camera instead of trying to use a separate PTZ. [7] requires the use of the bloggie hardware, not a real-time conferencing. [9] is from ours because more expensive and stationary device. Usage Scenarios AUREOLE in Multiple Conference Rooms: George is a people manager and engineering lead at a major software company. He manages an offshore team located in India. George is having his weekly team meeting with both onshore and offshore teams. George ships an AUREOLE to the offshore team, but before doing so, he plugs the AUREOLE into his local switch, opens up the “My Computer” window on his desktop to discover local devices on his computer, finds the AUREOLE, and double clicks on it to open the web interface. From the web interface, he logs in using the default username and password of admin/admin and configures the device to use the SSID of the offshore team (“offshore”) and to join it with the key of “offshore” and use DHCP settings. As the device is now configured, he ships it off to the team in India. Upon arriving in India, the offshore team puts it in the conference room and turns it on. At the next scheduled meeting time, George simply puts in the IP address of the AUREOLE into his web browser, logs in again, and can see and hear the rest of the team. Since George has already configured another AUREOLE in the local conference room, the offshore team can also see and hear him. Bringing Along a Remote Worker to a Meeting: Jay is a sales representative at a hardware company, in a role that requires him to travel to meet with customers frequently. He frequently defers to sales engineers in other parts of the company for technical backup. Jay is selling an embedded microcontroller to a hardware solutions company located in Texas. Due to travel budget cuts as a result of the economy, his sales engineer is no longer able to accompany him on all of the sales visits for potential customers. Jay brings along an AUREOLE to his meeting and a 3G smartphone that supports tethering over WiFi. Before the meeting, Jay configures the AUREOLE to use the tethering capability directly from the phone and places the AUREOLE in the middle of the conference room. Jay calls the engineer over a traditional telephone network via his smartphone or the conference room’s phone and presses the “mute” button on the device to avoid any echo from the remote conference room. The remote engineer is able to connect to the AUREOLE to see everyone in the room.

Remote Connection without Prompting: Sejun is a software engineer that works remotely for a major networking devices company. Sejun is a remote worker for a networking company.



Because he is remote, it is difficult for him to jump into meetings. Fortunately, the networking company has put the AUREOLE device all of their conference rooms. At the scheduled time for a meeting, he connects to the AUREOLE, but he sees that a meeting is running over. He waits until he sees his colleagues enter the room and then dials the conference phone for that room over the company’s VoIP network so that they can hear his voice. He mutes the plugin on his local browser so that there is no echo.

Security Camera Application: Joseph is a tenured professor. He is collaborating with colleagues at another institution. Joseph is paranoid about his student workers misusing equipment in his flagship lab, and would like to be able to monitor everything going on in the lab at all times. A limitation of traditional security cameras is that he needs multiple security cameras to capture all views of the lab room at all times. While on a trip abroad, he opens up the lab camera using a smartphone and notices something written on the whiteboard. He digitally zooms onto the whiteboard and sees writing. He rotates around the camera virtually to see a student worker nearby.

AUREOLE Architecture

As illustrated in Fig 1, AUREOLE consists of two major components: the physical device and the client software.

Fig. 1: AUREOLE Concept Block Diagram

Physical Device The AUREOLE physical device is a small portable device housing the main circuit board, the communication modules, the camera and lens, and the power management subsystems as shown in fig 2. The device records the 360 degree video and audio in the conference room and transmits the data through the local area network via wired Ethernet (802.3) or toggled wireless (802.11) medium to the end device. The device gets a 360º strip of the room through the use of a catadioptic lens. Because the output of the lens is distorted, the device un-distorts the output of



the lens to expose it as a panoramic strip. It has a secondary local network so that it can be configured the remote client devices from a PC or smartphone joined to that local network.

Fig. 2: AUREOLE Software Architecture

Client Software On the remote end, the client software is available to configure, receive, and manipulate the real-time video stream sent from the AUREOLE physical device. The end users will have the stream accessible through a web browser, iOS device, or an Android device as shown in fig 3 and fig 4.

Fig. 3: Client Software Architecture (iOS & Android)



Fig. 4: Web-browser Client Software Architecture

It displays the entire panoramic strip taken by the device and scrolls through and digitally zoom into any interesting part of the strip. The client enables the device accessible via both wired and wireless networks. It can configure the device network setting such as DHCP or Static IP. It also supports security features such as WPA2+AES for joining a wireless network. For legacy networks and connecting remote sites together, especially behind NAT, it will support the ability to join Cisco IPSec based VPNs.

AUREOLE Implementation

Physical Device AUREOLE is a fully functional computer that is comparable to older model desktop computers. This powerful system can carry out all the processes needed for the AUREOLE conferencing system. At the core of the AUREOLE Software is a Linux operating system. The onboard Linux system runs with Debian Linux OS configured for no GUI (console UI only) and bootstrapped to image stored in an onboard 2GB SD card. AUREOLE has been implemented based upon the functional block diagram illustrated in fig 5. Centering on TI’s Beagle board with the OMAP3530 microcontroller, the device and peripherals will be housed in a single enclosure. The Beagle board will be powered by a wall 110VAC plug that is regulated to 5VDC. Should wall power not be available, a backup 5VDC lithium-ion battery will power the device for a minimum of one hour. A hardware switch will control the power to the device as well. The external audio will be recorded to the device through the ALSA Audio Driver and the peripheral camera will connect via USB2.0. Both the Ethernet (IEEE 802.3) and wireless (IEEE 802.11) network modules will be connected via USB2.0 as well. To facilitate the user, several hardware buttons and LEDs will be added as well and connected to the Beagle board through open GPIOs on the OMAP3530 microcontroller. Specifically, Wi-Fi Enable and Local Mute buttons will be added as well as activity LEDs to signal Power, Wi-Fi Enable, and Mute Enable. The entirety of this device will be housed in a single enclosure following the footprint guidelines. The followings summarize the detailed implementation of each functional block.



Fig. 5: AUREOLE Functional Block Diagram

Microcontroller: Although the Freescale i.MX51 microcontroller possesses most of the capabilities deemed necessary, the overall complexity and cost of the development kit were too great. Therefore, the TI’s Beagle board with OMAP3530 was chosen for this application. With a relatively simple reference design, built-in audio capture and USB, h.264 encoding capability, and embedded Linux support, the Beagle board will be able to handle all of the necessary capabilities of Project AUREOLE. 360º Camera Technology: Although camera stitching and a fisheye lens could work, the catadioptric lens is the best option for providing the full 360 degree, panoramic view required for Project AUREOLE. Video Codecs: When considering video codecs for Project AUREOLE, the possible resolution and data rate of video transfer were the major factors. In the case of MJPEG encoding, the bandwidth would be too great for the high resolution video required and h.263 encoding would not be capable of delivering the quality of video required. Using h.264 encoding, both the high quality and low bandwidth requirements can be met for Project AUREOLE. Camera Interface: Although the hardware is not as simple as other options, USB2.0 was chosen as the camera interface. Considering the resolution requirements of Project AUREOLE, a USB2.0 camera interface will meet the high bandwidth requests. Streaming Protocol: Since end user-devices can manipulate the video stream through a web browser interface, any HTTP streaming protocol would be beneficial. However, given the large amount of current devices using RTP and the general standardization of the protocol, RTP was chosen as the streaming protocol for Project AUREOLE. It buffers and streams H264/MPEG-4 AVC video, hosts multiple sessions, and streams both video and audio synchronously.



Network Interfaces: Per the network requirements of AUREOLE device, two modules will be needed as network interfaces. An Ethernet module (IEEE 802.3) as well as a wireless module (IEEE 802.11) with toggle on/off capabilities will be selected for Project AUREOLE. It also uses Universal Plug and Play (UPnP) to automatically generate a list of all network devices connected to AUREOLE. Once a list is generated, the AUREOLE software can automatically choose a preferred network or the best available network for conferencing. Power: AUREOLE’s two power sources (110VAC wall outlet jack and 5VDC Lithium-Ion battery ) go hand in hand for efficient management of electric resources and for system process enhancements. Power management features include 1) Automatic switch to battery power when wall outlet power is removed from device and 2) Configurable sleep and idle modes when running on battery power. Footprint: AUREOLE was designed on the concept of mobility. To achieve maximum portability, AUREOLE incorporates features including 1) Small enough to fit in a laptop case: <10” X 10” X 3”, 2) Very lightweight: < 1lb, 3) Must be able to view entire body of conference members from table top height, and 4) Rugged design with drop shock proof up to 4 feet.

Client Software Users of the AUREOLE device have the freedom of choice as to what type of terminal they wish to utilize Aureole. Whether the viewing platform is mobile or stationary, several options are available for the user to choose from. Internet Explorer 9 web interface, Android 2.2 (Froyo) conference application, and iOS 4 conference application are supported. The AUREOLE conference application and web UI has a wide range features that extends user control and functionality beyond just viewing video data. AUREOLE can be configured and controlled from the client user interfaces. The properties including network connection and selection, initiation of video streaming, RTSP server control and setup, video conference access and security, and power control and monitoring can be manipulated remotely from any of the conference user interfaces. Because the 360-degree video is so large, navigation of the video pane is available so that the user can virtually adjust what he or she is looking at. All the video view options available to the user includes dual video feed panes (full 360-degree or magnified segment view), vertical & horizontal video image adjustment, virtual video pane adjustment (Depth control with Zoom IN/OUT and vertical & horizontal control with X and Y axis). AUREOLE software throws error exceptions, error handlers, and conference notification messages to alert the user for any issues or events that AUREOLE experiences. The types of network status, hardware status, and software/application status messages such as disconnection & connection alert, network timeout error, invalid password or username, unable to connect to device, device low battery alert, and action confirmation messages.

AUREOLE Demo

In this paper, we offer to demo the preliminary result of the AUREOLE system. As the physical device board is currently under production, we would present with a regular PC with the AUREOLE camera (360 degree) connected via USB cable as shown in Fig 6. As illustrated in Fig 7 an AUREOLE device (PC) takes the 360 degree video and transforms it to an unwrapped video image. Source Element provides the video to the system. It can interface to video for linux to get the driver from the AUREOLE camera (360 degree), which uses the UVC linux driver. We



leverage the linux driver to get the video YUV formatted in a buffer. Transform Element does the transformation in real-time. It receives video in a packed YUV color space, transforms the video into the proper place through the use of a pre-computed nearest-neighbor lookup table and then feeds it to the next element in the pipeline. Sink Element takes the video stream and displays it or feeds it elsewhere. Skype can act as a sink element to stream the video over the network to the next place. The real-time video process example is shown in Fig 8. The sampled and unwrapped video will be sent to the remote device. The remote device users can focus on a part of the video and can zoom into the screen for the detailed display.

Fig. 6: PC based Demo Configuration

Fig. 7: Sample Pipeline Process



Fig. 8: Real-time Video Unwrapping Screen Shot

Conclusion

In this paper, we presented a small scale, low-cost, highly mobile 360º video conferencing platform, AUREOLE, to improve the remote video conferencing experience by providing the virtual presence for multiple remote users in high definition. We discussed the architecture and implementation of the physical device and client software. We also proposed a demo of the preliminary result of the system.

References

[1] Wai-Kwan Tang; Tien-Tsin Wong; Heng, P.-A., "A system for real-time panorama generation and display in tele-immersive applications," IEEE Transactions on Multimedia, vol.7, no.2, pp. 280- 292, April 2005

[2] Scotti, G.; Marcenaro, L.; Coelho, C.; Selvaggi, F.; Regazzoni, C.S., "Dual camera intelligent sensor for high definition 360 degrees surveillance," Vision, Image and Signal Processing, IEEE Proceedings , vol.152, no.2, pp. 250- 257, 8 April 2005

[3] Hiroshi Ishiguro, “Development of Low-Cost Compact Omnidirectional Vision Sensors and their applications”, Panoramic Vision, chapter 3, 1998, 433—439, Springer

[4] Tarhan, Metin; Altuğ, Erdinç, “A Catadioptric and Pan-Tilt-Zoom Camera Pair Object Tracking System for UAVs”, Journal of Intelligent & Robotic Systems, Springer Netherlands, November 2010

[5] Qimin, P.[Peng], Yunde, J.[Jia], “Wavelet-Based Resolution Enhancement of Omnidirectional Images”, VI02(125), PDF Version. 0208

[6] Klette, R.[Reinhard], Gimel'farb, G.[Georgy], Wei, S.K.[Shou Kang], Huang, F.[Fay], Scheibe, K.[Karsten], Scheele, M.[Martin],Börner, A.[Anko], Reulke, R.[Ralf], “On Design and Applications of Cylindrical Panoramas”, CAIP03(1-8), WWW Version. 0311

[7] Flong Project(Golan Levin at CMU) http://www.flong.com/blog/2010/open-source-panoramic-video-bloggie-openframeworks-processing/



[8] S.K. Nayar and V. Peri, "Folded Catadioptric Cameras," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol.2, pp.217-223, Jun, 1999, http://www.cs.columbia.edu/CAVE/projects/cat_cam_360/

[9] Panoguide, http://www.panoguide.com/products/

Session T3C-3 AUREOLE: A CAPSTONE PROJECT INTEGRATING …aseegsw.com/past...

Documents

Transcript of Session T3C-3 AUREOLE: A CAPSTONE PROJECT INTEGRATING …aseegsw.com/past...