SPIE 6957 March, 2008 1

Enhanced & Synthetic Vision System (ESVS) flight demonstration

John N. Sanders-Reed*, Ken Bernier†, Jeff Güell‡

ABSTRACT

Boeing has developed and flight demonstrated a distributed aperture enhanced and synthetic vision system for integrated situational awareness. The system includes 10 sensors, 2 simultaneous users with head mounted displays (one via a wireless remote link), and intelligent agents for hostile fire detection, ground moving target detection and tracking, and stationary personnel and vehicle detection. Flight demonstrations were performed in 2006 and 2007 on a MD-530 “Little Bird” helicopter. Keywords: Enhanced vision, synthetic vision, head tracking, head mounted display, brown-out, situational awareness, distributed aperture, tele-operation, all weather vision landing guidance

1. INTRODUCTION Situational awareness can be improved for crew members in both air and ground vehicles through a combination of distributed aperture enhanced and synthetic vision. The use of a Distributed Aperture System (DAS) with fixed mounted sensors, coupled with head tracking and see through Head Mounted Displays (HMDs), allows multiple crew members to simultaneously look in different directions using the same sensors. The sensors can be arranged around the vehicle to provide vision in any direction desired, such as behind, through the floor, or complete 360° around the vehicle. Sensors can be selected to provide vision at night or through obscurants such as fog, rain, snow, and dust. While fixed mounted sensors provide good wide area coverage for situational awareness and pilotage, turreted Narrow Field Of View (NFOV) sensors can provide directional, high resolution coverage of particular items of interest. Synthetic Vision augments the live sensor imagery with a larger context and a view in directions not covered by live sensors, as well as symbology and cueing. The flight test and associated ground tests described in this paper were intended to demonstrate the maturity and integration of components for a multi-user, distributed aperture Enhanced and Synthetic Vision System (ESVS). The system needed to provide 360 degree vision as well as down looking “through the floor” capability to at least 2 crew members (and be easily expandable to more) with independent Lines Of Sight (LOS). It also needed to support both fixed mounted and turreted sensors, integrated on the same platform. Adequate visibility and situational awareness in reduced visibility from dust (brown-out) as well fog, rain, or snow were also required. While sufficient for use in a single aircraft, static environment, a simple synthetic vision or even “see and remember” capability was deemed inadequate to support dynamic operations of multiple aircraft in a reduced visibility environment. As a result, the flight test was designed to demonstrate integration of a wide range of different modality sensors, both to demonstrate that the architecture could support them, and to begin evaluations of the different modalities under various conditions. Turreted sensors which are currently in widespread use, provide vision in a single, steerable direction with no simultaneous vision capability (for other user/operators) in other directions. In contrast, the fixed mounted sensors of a distributed aperture system provide simultaneous vision in multiple different directions. This means that even while the user is looking in one direction, the imagery in all other directions covered by sensors, is still being generated and could be utilized. The current set of demonstrations utilized several “Intelligent Agents” to process imagery and extract features to cue the pilot to various threats and opportunities. For the flight test we integrated a flash detection Hostile Fire Indicator (HFI), Ground Moving Target Detection & Tracking (GMTD&T), and vehicle and personnel detection *Boeing-SVS, Inc. 4411 The 25 Way NE, Albuquerque, NM 87109 *The Boeing Company, PO Box 516, MC S064-2374, St. Louis, MO 63166 †The Boeing Company, 2401 E. Wardlow Rd, MC C76-226, Long Beach, CA 90807

SPIE 6957 March, 2008 2

intelligent agents. Other possible intelligent agents include wire detection and runway outlining and incursion detection among others. While the basic system is designed to support on-board, hard-wired users, we also demonstrated a remote user connected over a wireless connection. The key to this is that all of the image assembly is performed on-board, thus reducing the required video bandwidth to a single (compressed) video stream for the instantaneous Line Of Sight. The central concepts of distributed aperture imaging systems with presentation on head mounted displays have been around for many years [1,2,3]. However, development of algorithms and processing hardware capable of performing the number of computations with sufficiently low latencies to be useful in a dynamic vehicle environment has required considerable effort. The basic algorithms required for image assembly and presentation include image stitching to form a seamless panoramic image using common modality sensors, image blending or fusion between different modality sensors, image warping for projection onto complex curved helmet mounted displays, and electronic zoom. In order to provide a useful system in a highly dynamic environment, the generally accepted latency requirement is around 40 msec from the time a photon enters a sensor until a photon is emitted to the display. This translates into a 16 msec processing latency requirement to perform all of the described operations [4,5]. In order to provide a flexible architecture which can be scaled to different platforms and applications we utilized a sensor independent, modular, scalable approach, described later.

2. SYSTEM DESCRIPTION 2.1 Requirements and Architecture

The goals of the flight test were to demonstrate:

• Integration of multiple different modality sensors (visible, infra-red, millimeter wave radar, and 3D lidar) • Sensor format independence (CameraLink, RS-170, etc) • Integration of up to 10 different sensors • Integration of live (enhanced) and synthetic vision • 360 degree vision, with additional downward “through the floor” vision, and rear-view mirror capability • 2 or more simultaneous users with independent lines of sight • Different types of displays including mono-chrome and color, bi-ocular with separate left/right warping or

single unwarped display • Demonstrate low latency image assembly algorithms and processing for image stitching, blending and fusion,

electronic zoom, and warping • Display Highway In The Sky (HITS), terrain impact warning, pop-up keep-out zones, and other navigation

symbology • Integration of Intelligent Agents: Ground Moving Target Detection & Tracking, Personnel & Vehicle detection,

Hostile Fire detection, and Passive Wire Detection, demonstrating different modes of integration • Wireless remote user with low latency compression

These requirements were flowed down to sensor selection, displays, algorithms, and system architecture. 2.2 Sensors

Most of the sensors were arranged in a primary sensor pod mounted underneath the chin of the helicopter. A 3D lidar was mounted on a plank extending to the right side of the helicopter to provide a different perspective. In addition, dedicated sensors for hostile fire detection were mounted on the right side plank extending from the helicopter. The sensor pod contained 10 sensors as shown in figure 1 below. The sensors included both CameraLink and RS-170 output sensors to show sensor format independence in the system design. In the forward looking direction we included Long Wave Infra-Red (LWIR), visible, and active 94 GHz Millimeter Wave (MMW) radar to demonstrate integration of various different modality sensors and to provide sensors for image blending and fusion algorithms. The MMW radar (Sierra Nevada Corporation, Waveband Division) provided an RS-170 C-scope output: a 2 dimensional “image”

SPIE 6957 March, 2008 3

consisting of azimuth and range, as compared to azimuth and elevation normally obtained from passive imaging sensors. A separate Narrow Field Of View (NFOV) LWIR camera was dedicated to passive wire detection. A downward facing LWIR camera provided “through the floor” imaging while the rear facing Low Light Level TV (LLLTV) provided both a rear looking sensor and the Rear View Mirror (RVM) functionality. All of the LWIR cameras operated in the 8-12 um waveband. The use of different types of sensors to provide the 360 degree vision demonstrated our ability to stitch different types of sensors into a single panoramic image.

Figure 1. Sensor Pod: Schematic showing sensor layout and relative fields of view (left) and photo (right)

Sensor Brand Pixels FOV (deg) Wavelength Output

1,2,3 I2Tech 640x480 64x48 LWIR RS-1704 I2Tech 640x480 32x24 LWIR RS-170

5,6 DRS 640x480 90x68 LWIR RS-1707 I2Tech 640x480 9x7 LWIR CameraLink8 Basler 1004x1004 23x23 Visible CameraLink9 Selex 576x488 26x20 LLLTV RS-17010 Sierra Nevada MMW Radar RS-170

Riegl Lidar near IR EthernetRadiance MWIR Ethernet

The 3D lidar was a Riegl LMS-Z210ii operating in the near IR and providing programmable scans such as 2 seconds for a 10x10 degree scan or 5 seconds for a 20x20 degree scan. Range is given by the manufacturer as between 200 m and 650 m depending on material. Accuracy is estimated at 15 mm range with an angular resolution of 0.005°. The lidar output was over Ethernet, demonstrating yet another sensor data format.

Figure 2. Riegl lidar mounted under plank on helicopter

The Hostile Fire Indication (HFI) system was provided by Radiance Technologies. For this demonstration it consisted of 2 Medium Wave Infra-Red (MWIR) high frame rate cameras (3-5 um) mounted on the right side plank of the helicopter,

SPIE 6957 March, 2008 4

one facing forward and one facing aft. Radiance also provided a processing unit which processed the high speed imagery to perform flash detection and characterization. Detections were output over Ethernet to the ESVS processing system. A GPS/INU provided helicopter position and orientation input to the synthetic vision system at 100 Hz data refresh rate. While all of the system sensors provided monochrome, grey scale output, a color camera was mounted in the cockpit to provide a record of what the pilot saw with normal human vision. This camera was not part of the ESVS processing and display system. 2.3 Displays

Two different displays and head trackers were used for the 2007 flight test: A BAE Gen IV helmet with optical head tracker and a pair of eMagin color Virtual Reality (VR) goggles with an Intersense inertial head tracker. The helmet was used for the hardwired, on-board user while the VR goggles were used for the off-board wireless remote user.

Figure 3. BAE Gen IV helmet and eMagin VR goggles with optical head tracker and Intersense inertial tracker

The BAE helmet is a monochrome bi-ocular device meaning that it requires separate imagery for each eye. The imagery must be warped (separately for each eye) for projection onto the complex curved visor so that it will appear conformal with the external world. Each eye uses a 1280x1024 pixel monochrome display updated at an 85 Hz frame rate. The helmet uses an integrated optical tracker with two imagers mounted in the cockpit measuring helmet orientation and operating at 240 Hz update rates. We selected an optical tracker for cockpit applications because it is not as sensitive to the cockpit environment as magnetic head trackers and it does not exhibit the drift associated with inertial trackers. The BAE helmet is a see-through display/visor providing “augmented reality” capabilities. The eMagin goggles provide a color display operating at 60 Hz with a resolution of 800x600 pixels. The same imagery is displayed to both the left and right eye with no warping. The use of color VR (virtual reality or non-see-through) goggles allowed us to demonstrate different display types and to compare monochrome with color displays. The Intersense InertiaCube3 is an inertial tracker operating at 180 Hz update rate. While an inertial tracker exhibits some drift this is not critical since the user was remote from the helicopter and not looking through the windscreen. This meant we did not need to worry about alignment between the display imagery and the external world as we did with the on-

SPIE 6957 March, 2008 5

board user. The inertial tracker also allowed the remote user to turn completely around to look “directly” through the rear facing sensors and not just through the rear view mirror. 2.4 Algorithms & Processing

The processor architecture was required to be modular and scalable to accommodate essentially any number of sensors and users. In addition, it was required to be sensor and display format independent so we could easily switch sensors and displays without changing hardware. To achieve this, a basic building block processor board was designed using a 3U PMC form factor with Field Programmable Gate Array (FPGA) processing and fiber-optic transceivers. All high volume I/O was sent over high speed serial fiber optic cables. The FPGA processor cards are called Modular Enhanced Vision System (MEVS) cards. In order to convert sensor output from the sensor format to the serial fiber format we developed transition modules for RS-170 to fiber and CameraLink to fiber. Others will be developed as needed. We also developed a fiber to DVI transition module for the output video stream. The result of this architecture is a standard building block processor card using a standard high speed interface, isolated by transition modules from variations in sensor and display formats. The fiber optic cables also provide Electro-Magnetic Interference (EMI) immunity and long transmission range. The overall system architecture is shown in figure 4. Output of the various sensors is sent through sensor to fiber transition modules. The resulting serial fiber cables go to a Fiber Distribution Box where they are split and sent to each user (in this case 2). The output of the Fiber Distribution Box is a bundle of fibers (one per sensor) routed to each user processor module with one user processor module per user. Note that depending on functions supported by the sensors, the fibers can be bi-directional allowing the processor to send sensor control settings back to the sensors. The user processor modules also require GPS/INU data input for the synthetic vision system. GPS/INU, Lidar data and Hostile Fire detections from the Radiance Hostile Fire Indication system are sent to the synthetic vision system over Ethernet. The processor module generates the output video stream from the input sensor data and an input serial port head tracker data stream. The user can control the processing options and display using a touch panel control GUI.

Figure 4. ESVS overall system architecture and details of single user processor architecture

SPIE 6957 March, 2008 6

For the hardwired, on-board user, the output video stream is sent out over a standard fiber optic serial cable and enters a fiber to DVI transition module which converts the serial fiber optic signal into a standard DVI signal which is sent to the helmet for display. For the remote user, two different wireless video options were supported: A bi-directional 802.11 wireless link and a downlink only Q-band link. Using only the 802.11 system, the output video stream is sent through a low latency JPEG compression module (20 msec) and then routed over an Ethernet connection to a wireless access point where it is transmitted to the remote user. The remote user station consists of a laptop running the control GUI and JPEG decompression algorithm. The wireless signal is received by a wireless access point and sent over Ethernet to the laptop. After decompression, the video signal is sent out the VGA port to the VR goggles. Head tracker data comes in a serial port on the laptop where it is packetized and along with any GUI commands, sent back out over the Ethernet cable to the wireless router and onto the remote user processor module on the aircraft. We assumed a useful bandwidth of 10 Mbit/sec for the 802.11 system. Note that the remote user concept is feasible and can be implemented with moderate bandwidth requirements because only a single video stream is sent over the wireless connection. All of the image assembly from the multiple sensors is performed in the user processor module on the aircraft, resulting in a single video stream (which changes dynamically depending on user head pointing as reported by the head tracker) to be sent back out over the wireless connection. Other off-board data, such as Blue Force Tracker data and Pop-up Keep Out Zones can be received by the remote user laptop from an external network and sent up to the on-board processor for rendering by the synthetic vision system. The Q-band system has a bandwidth of 1.45 Gbit/sec and can handle uncompressed video. For the Q-band system, uncompressed video output is sent through a fiber to DVI converter just as for the hardwired user. The DVI signal is then sent to the Q-band transmitter for downlink to the ground receiver which in turn sends the DVI signal into the ground computer. Since the Q-band system is downlink only, when using the Q-band system the User 2 Processor Module must either be slaved to the on-board user head tracker (making the remote user a purely passive observer), or the 802.11 system must be used to uplink remote user head tracker data. Figure 4 also shows the details of a user processor module, consisting of a number of MEVS processor cards and a Single Board Computer (SBC) running the Synthetic Vision System. The MEVS processor cards and algorithms were designed to support 1024x1024 pixel imagers operating at 60 Hz frame rates with no more than 16 msec processing latency in order to support a total 40 msec latency from photon in at a sensor, to photon out to a display. The architecture has sufficient processing power to support all required image stitching, blending and fusion algorithms, image warping, and related functions to generate 1280x1024 pixel independent left and right imagery at 85 Hz, or 800x600 pixel color imagery at 60 Hz, as required by the displays. Imagery enters the MEVS boards via the serial fiber-optic ports. MEVS boards are daisy chained together via fiber optic links. The MEVS boards assemble live and synthetic imagery with the output either over a serial fiber cable, or if compression is used, over Ethernet. In order to assemble imagery, the MEVS boards need the current head tracker Line Of Sight (LOS) data which enters the synthetic vision SBC over a serial port and is then distributed to the first MEVS board over Ethernet. From there it is transmitted to the other MEVS boards over the fiber daisy chain. The synthetic vision system generates a conformal 3D view in the current LOS direction from Digital Terrain Elevation Data (DTED), map data, and overhead stored imagery. It generates two DVI video outputs, one is the synthetic vision imagery while the other is a graphic overlay containing symbology such as a pitch ladder, system settings, intelligent agent results, and off-board data. The two synthetic vision output streams are sent through a DVI to fiber transition module and enter the MEVS boards as another video source. The 3D lidar data also enters the system through the synthetic vision system where it is treated as a high resolution update to the DTED data and used to render a higher fidelity terrain map in the region scanned. One MEVS board is sandwiched with a Power PC (PPC) processor card to provide system control functions. The architecture described allows one to add additional users by simply adding more user processor modules. Additional sensors can be accommodated by adding additional MEVS boards in each user processor module. Since each MEVS board can accept multiple input sensors, it is not always necessary to add MEVS cards to add one more sensor. The core functions performed by the MEVS processing are stitching imagery together to form a seamless panoramic view as the user turns their head, to perform image blending or (Principal Component Analysis) fusion [6] between different modality sensors resulting in a single composite image, perform any required image warping so the imagery

SPIE 6957 March, 2008 7

displayed on a curved visor is conformal with the outside world, perform an electronic zoom if desired by the user, and perform low latency compression on the output video stream if necessary. The system also combines the live and synthetic imagery. These functions are shown in figure 5. It is important to note that while we show these as sequential steps, they are in actuality performed as a single integrated operation [4,5] in order to meet the latency requirements. While it is possible to purchase third party algorithms or processor boxes that perform many of these functions, they typically buffer the image on the input and the output resulting in significant throughput latency. The sensor imagery is rendered on a surface (typically a sphere) in locations to be conformal with the external world. In addition, the rear facing sensor is mapped as a Rear View Mirror above the other live sensors and to the right, in the location of an automotive rear view mirror.

Figure 5. Basic processing functions to assemble and present live and synthetic imagery.

The Synthetic Vision Toolkit (SVTk) generates basic synthetic imagery using the DTED imagery textured with map or overhead imagery from the database, or a simple grid. The imagery can be generated as 3D perspective (as seen from the user’s point of view), or as a 2D moving map. When in 2D moving map mode, the synthetic vision imagery is rendered in a window below the live image sensors and to the left of the center line. When turned on, 3D synthetic imagery is rendered everywhere there is no live sensor, and in locations with a live sensor, the user can adjust the blending between live and synthetic. The synthetic vision system also generates a number of graphic overlays which can be toggled on or off. These include a Highway In The Sky (HITS), as well as results from the various intelligent agents. Based on experience from pervious generations of our system, special care was taken to avoid aliasing artifacts in the graphic overlays resulting from the image warping for helmet displays. We integrated three different intelligent agents for the current flight test: Ground Moving Target Detection & Tracking (GMTD&T, also known as Ground Moving Target Indication or GMTI), the HRL SwarmVision personnel and vehicle detection algorithms [7,8], and the Radiance Technology Hostile Fire Indication (HFI) system. We also developed a passive wire detection system which was tested separately but we ran out of time to integrate it for the current flight tests. The GMTD&T and SwarmVision were both C code which ran in a Power PC (PPC) sandwiched with one of the MEVS cards. Imagery entering the MEVS card was passed to the PPC where the GMTD&T or the SwarmVision algorithm could be run and the results were passed via Ethernet to the Synthetic Vision Toolkit for rendering as overlays. This demonstrated integration of third party algorithms supplied as algorithms, C code, or a compiled library. The HFI system consisted of its own sensor and processor with only detections passed via Ethernet to the synthetic vision system, demonstrating yet another way to integrate third party intelligent agents. While the core image assembly pipeline is tightly integrated due to the latency requirements, the intelligent agents do not share the same stringent latency requirements and thus we envision the current architecture as an open system available to plug in third party intelligent agents. From our perspective the key attribute of an intelligent agent is that it provides low volume information to be rendered as graphic overlays by the synthetic vision system. An intelligent agent can utilize its own sensor (as in the case

SPIE 6957 March, 2008 8

of the HFI system) or take imagery from one of the ESVS sensors. The intelligent agent can run in its own black box processor or as a software module in one of our processors. We also note that most intelligent agents only need to run in one user processor module and the results can be sent to other user processor modules for rendering. Thus one could run the GMTD&T algorithm on sensor data in one processor module and the SwarmVision algorithm on sensor data in the other processor module. The head mounted display provides a movable view window into the virtual image sphere. As the user moves their head, the view window is moved around the image sphere. Since the user only sees his current instantaneous view window at any one time, it is instructive to show the larger portion of the image sphere in order to understand the overall image layout (figure 6). In this view we see the stitched live imagery with a fused center image from 2 different sensors. The rear facing sensor is mapped as the Rear View Mirror and a 2D moving map display is present. The user’s current view window is shown highlighted with various symbology, including a green pitch ladder, purple Highway In The Sky (HITS), and an arrow pointing to a hostile fire event (the star on the ground outside the current view window). The 2 green nested squares indicate the current view window position within the larger image sphere.

Figure 6. A portion of the image sphere showing a user’s instantaneous view window.

2.5 Vehicles & System Integration

Flight demonstrations were performed using an MD-530 “Little Bird” helicopter. In addition to the ESVS equipment the flight system included digital video recorders and power supplies for the various components. As shown in figure 7, the main sensor pod was mounted under the chin of the aircraft. The 3D lidar and Radiance HFI detectors were mounted on the right side plank while the 802.11 omni-directional antenna and the Q-band directional antenna were mounted on the left side. The ESVS equipment plus power supplies and data recording systems were mounted inside the helicopter. A touch screen computer running the control GUI for the on-board user was mounted in front of the user. The ESVS evaluator/user was in the left seat and the aircraft was piloted from the right seat. In addition to the flight tests, an ESVS Mobile Test Van (MTV) was used to gain preliminary system integration and ground vehicle experience. The MTV utilized the same processors and displays but the sensors were individually mounted in the test van, rather than in the sensor pod. The MTV provides a different sensor geometry with the sensor more spatially distributed.

SPIE 6957 March, 2008 9

Figure 7. ESVS installed on Little Bird helicopter

3. FLIGHT DEMONSTRATION

Flight demonstrations were conducted in both 2006 and 2007. The 2007 system was configured as described above. The 2006 system used a COTS PC based system which supported a single onboard user with VR goggles (the PC based system was unable to perform image warping for the helmet left and right eyes) and fewer sensors. Our test site (used both years) included various terrain including a canyon and a town. In addition, a brown-out landing site was configured including man made obstacles (figure 8) and low relief dirt mounds which provide hard to see landing hazards. The “landing zone” was in a region of loose dirt allowing us to generate significant dust clouds.

Figure 8. Landing zone dust and obstacle site

The 2006 flight demonstration was highly successful within the bounds of the hardware configuration and provided both good imagery and solid baseline experience. The 2007 flight demonstration utilized much more capable hardware and demonstrated significant new functionality but also experienced a number of problems. Aircraft power conditioning problems caused the 3D lidar system and one of two digital video recording systems to fail in flight. The 3D lidar was successfully integrated and operated on the aircraft while on ground power but upon transition to aircraft power it failed. Failure of the digital video recording system while on aircraft power meant that we were able to record extensive raw sensor imagery but only limited integrated output imagery during the 2007 flight test.

SPIE 6957 March, 2008 10

During the 2007 flight test, the GPS/INU failed to operate during most of the flight tests. Since this is required for synthetic vision and for the Q-band directional antenna, these two sub-systems were non-operational for much of the flight tests. We did generate one flight with operational GPS/INU in which we were able to utilize both the synthetic vision and the Q-band down-link. During this flight, the Q-band system down-linked good quality, uncompressed color video. Post-mission analysis indicated that a GPS/INU software configuration setting had been changed shortly before flight test on both our primary and back-up units. Range of the 802.11 wireless link was significantly less than expected. The system provided good remote video with up-linked head tracker data operating in a closed loop manner at very close range but failed shortly after take-off as the range increased. We believe this was due to poor traffic control on our internal network, resulting in a “self jamming” situation. In addition to the ESVS components on the Ethernet network as shown in figure 5, the functional video recorder placed its remote desktop video display output on the Ethernet further increasing the traffic volume. Due to the above problems, we ran out of flight time to demonstrate the GMTD&T and SwarmVision intelligent agents operating in flight. We were however able to post-process the flight recorded imagery on the ground to evaluate GMTD&T and SwarmVision performance. Despite the above challenges, we were able to demonstrate all 10 sensors integrated with synthetic vision and the capability of supporting 2 simultaneous users with independent head mounted displays and lines of sight. We did not observe vibration induced image smear during any of our flights. The image assembly included image stitching, image blending and image fusion, and blending of synthetic and live imagery. The Rear View Mirror and “through the floor” imaging functions also worked well. Figure 9 shows the ESVS equipped helicopter in flight through a canyon and flying through dust at our landing zone obstacle course.

Figure 9. Little Bird flight test under various conditions. Flight through canyon terrain (left) and brown-out landing

zone obstacle course (right) We were also able to successfully detect hostile fire events (AK-47), with detections sent from the Radiance system over Ethernet and displayed as graphic overlays in the synthetic vision system.

4. RESULTS The processing system was able to stitch and fuse the live imagery, blend the live imagery with synthetic imagery and apply graphic overlays as shown in figures 10, 11. The left side of figure 10 shows Principal Component Analysis (PCA) image fusion between visible and LWIR sensors. The right hand image is live imagery recorded in flight, showing synthetic vision graphic overlays. Highway In The Sky (HITS) symbology is shown in purple while Pop-up keep out zones are shown as gridded half domes. A pitch ladder is also displayed and an indicator of current view window position within the overall image sphere. In figure 11, the left hand image shows imagery from a flight through a canyon. At the limits of the live imagery, the synthetic imagery can be seen as brown textured terrain. The synthetic vision is also blended with live imagery to add the color visible in the canyon walls. Synthetic vision terrain can be textured with map

SPIE 6957 March, 2008 11

data, overhead image data, or a grid pattern which is useful for highlighting small terrain variations in a landing zone. The right hand image in figure 11 shows the synthetic vision grid pattern blended with the live imagery.

Figure 10. ESVS displays showing image fusion (left) and graphic overlays (both), including Highway In The Sky, Pop-

Up Keep Out zones, home base symbols, pitch ladder, and line of sight indicator.

Figure 11. Combined live and synthetic imagery during canyon flight (left). Image fusion with synthetic vision grid

texture (right). Figure 12 shows examples of the synthetic imagery operating in 2D map mode, with a map data texture on the left, and overhead imagery texture on the right. One can clearly see the purple Highway In The Sky in each, and the blue “home base” dome. The three yellow dots are small “Keep Out” zones. During the 2006 flight test, the onboard test pilot utilized color VR goggles (due to the processing limitations of the PC based system), while for the 2007 flight test he used the bi-ocular, high resolution monochrome helmet display. Both displays were difficult to see during daytime flights (the helmet had a sun visor which was not supplied to us for the flight tests which may have helped). Not surprisingly, the pilot had a strong preference for the color display of the VR goggles, finding that the color information significantly aided his understanding and interpretation. Further, with the monochrome helmet display, the addition of synthetic vision often degraded the overall image quality. During the 2006 flight demonstration the control interface was difficult to use with the test pilot reporting that he could not really use the control system while wearing the helmet display. For the 2007 flight test we had a touch screen tablet

SPIE 6957 March, 2008 12

PC mounted in front of the ESVS test pilot. Because of the 2006 experience we limited the number of user controls. Due to the improved physical interface, the pilot found the control interface easy to use and requested additional controls to be able to optimize sensor settings.

Figure 12. 2D Moving map display showing map (left) and overhead imagery (right) textures. Purple in Highway In The

Sky. Blue circle is home base. As indicated above, the hostile fire intelligent agent worked well in flight, rendering detection symbology on the display. While the GMTD&T and SwarmVision intelligent agents were successfully integrated, we ran out of time to test them in flight. However, we post-processed flight video. Figure 13 shows 2 examples of GMTD&T results and one of SwarmVision. The left image is from the down looking LWIR sensor and shows detection and tracking of resolved moving targets, while the center image shows visible band detection of unresolved moving targets. Finally, the right image shows SwarmVision detection of a person using visible band imagery.

Figure 13. Intelligent Agents. Left and center are Ground Moving Target Detection and Tracking of both resolved and

unresolved targets in LWIR and visible imagery. Right image is SwarmVision detection of a person.

Figure 14. Fixed and turreted sensor integration

Following the flight test, a final laboratory demonstration showed integration of a turreted sensor with fixed mounted sensors. The flight tests utilized exclusively fix mounted sensors so this was considered an important capability extension. In figure 14, the fixed sensors are labeled A and B and are stitched. The turreted sensor C is tied to the user’s

SPIE 6957 March, 2008 13

head motion and moves across the fixed sensors. In reality, the turreted sensor is always in the center of the field of view, but since only 2 fixed sensors were used, the region to the left and right of the fixed sensors was blank and clipped to save space.

5. SUMMARY A modular, scalable architecture has been developed to provide distributed aperture enhanced and synthetic vision using essentially any number and format imaging sensors (sensor agnostic) and displayed on multiple head mounted displays. The system met the stringent low latency processing requirements for head mounted pilotage systems while performing panoramic image stitching, multi-sensor image fusion, warping, and combining live and synthetic imagery. The system has been flight demonstrated in a rotorcraft environment. We also demonstrated integration of multiple different intelligent agents to perform moving target detection, hostile fire detection, and vehicle or personnel detection. The overall system utilizes an open architecture to allow sensors, displays, and intelligent agents from various sources to be easily integrated. We also demonstrated the concept of a wireless remote user, although better design is required to achieve an operationally useful range. Finally, we demonstrated integration of fixed and turreted sensors. While the core functionality has been demonstrated and some flight experience gained, additional flight experience will be required to fully evaluate and characterize the system. The lidar system, while integrated, remains to be successfully flight tested. With the lidar system operational we will be able to simultaneously evaluate MMW, 3D lidar, LWIR, and synthetic vision for brown-out and other reduced visibility operations. Additional intelligent agents will be added to the system and various different configurations are under development for various platforms and applications.

ACKNOWLEDGEMENTS The flight demonstration described herein required cooperation from team members representing many Boeing organizations and geographic locations. Space limitations prevent us from listing all of the important contributors to this effort. However, in addition to the authors, key team members included Linda Goyne lead developer of the synthetic vision system, Dennis Yelton lead algorithm developer, Scot McDermott electronics design and integration, Chris Witt FPGA implementation, and Mark Hardesty flight engineer for the Little Bird. We would also like to thank Radiance Technologies for the loan of their Hostile Fire Indication system and support personnel and HRL for their support with the SwarmVision intelligent agent.

REFERENCES

1. J. Güell, S. Schroeder, R. Leitch, N. Marstad, M. DeVogel, “Exterior Aircraft Vision System Using A Helmet-Mounted Display”, US Patent, 7,180,476, Feb 20, 2007.

2. J. Güell, “Flying Infrared for Low-level Operation (FLILO) Systems Description and Capabilities”, Proc 18th Digital Avionics Systems Conference, 1998.

3. C.W. Jennings, K.W. Alter, A.K. Barrows, K.L. Bernier, J.J. Güell, “Synthetic Vision as an Integrated Element of an Enhanced Vision System”, Proc SPIE, 4713, April, 2002.

4. D.J. Yelton, K.L. Bernier, J.N. Sanders-Reed, “Processing System for an Enhanced Vision System”, Proc SPIE, 5424, April 2004.

5. D.J. Yelton, K.L. Bernier, “Systems and methods for providing enhanced vision imaging with decreased latency”, US Patent, 7,148,861, Dec 12, 2006.

6. R.K. Sharma, M. Pavel, T.K. Leen; “Multi-stream video fusion using local principal components analysis”, Proc SPIE, 3436, 1998.

7. Y. Owechko, S. Medasani, “Object recognition system incorporating swarming domain classifiers”, US Patent application, 20050196047, Sept 8, 2005.

8. Y. Owechko, S. Medasani, “Graph-based cognitive swarms for object group recognition”, US Patent application, 20070183670, August 9, 2007.