VOLUME 14NUMBER 1

Spring 2016

A D V A N C I N G C O L L A B O R A T I O N & C O M M U N I C A T I O N

COMBINING TECHNOLOGIES TO CREATE INNOVATIVE SOLUTIONS

WHY META-CONVERGENCE

MATTERS

STANDARDIZATION LEADS TO UTILIZATIONEnbridge renovates meeting rooms at HQ in Canada

SUPERBOOK IS A GAME CHANGEREngineering a spectacular display for Westgate Las Vegas

COLORADO DOT EXPANDS COMMAND & CONTROL SYSTEMMonitoring from inside the Hanging Lake Tunnels

VIEWPOINTThe nature of huddle rooms

18 IT/AV Report

Monitoring from inside the Hanging Lake Tunnels.

gove

rnm

ent

Carlos Rogers, PE, is CEO and Senior Engineer, Novate Solutions (www.novate.com).

Colorado DOT Expands Command & Control SystemBy Carlos Rogers, PE

Deep underground, within a system of mountain tunnels connecting the final leg and a particularly hazardous section of Colorado’s Interstate 70, is one of the most technologically advanced command and control centers in the transportation industry. The tunnels lie within Glenwood Canyon, an extreme-ly narrow, environmentally sensitive and spectacularly beautiful 12.5-mile road connecting bridges, viaducts and tunnels. The command and control facility is located within the Hanging Lake Tunnels, a 4000-foot, two-bore tunnel through the mountainside in the canyon.

Multiyear ProjectDesign, engineering and integration firm Novate Solutions recently com-

pleted an upgrade and expansion of the command and control facility. The mul-tiyear project encompassed the control system modernization and replacement of network infrastructure covering more than 30 miles of partially elevated highway and tunnels. The expansion included the installation of a videowall and connecting the control to the existing analog video system.

In addition to being a critical and direct conduit of transportation through the state, the multi-award-winning Glenwood Canyon project was also recog-

19Spring 2016www.ITAVReport.comwww.ITAVReport.com

government

A view of Hanging Lake Tunnels looking to the west. Signs visible

here at the bore entry direct traffic and warn drivers of potential

hazards ahead. The sign messages are displayed automatically

throughout the bores when the operator selects any number of

traffic operational modes.

Colorado DOT Expands Command & Control System

The DCS server room houses an array of redundant mission-critical gear. Locating the servers, switches, power supplies and other equipment in this room removes virtually all noise from the operators’ work area.

SERVING YOU SINCE 1984www.hallresearch.com

714-641-6607A New Wave in Connectivity

Extracts audio from HDMI input

Provides 3.5mm audio input that can be mixed with audio from HDMI

Volume can be controlled externally via simple rotary encoder, RS232, and IP

Built-in graphical web interface for control of audio mix and volume

Provides 2nd RS232 output to control other devices such as projectors

Directly connects to 8 ohm speakers, and features variable line-level output

EMX-I-AMP

4K HDMI Audio Extractor with 50 watt Ampliier

Come visit us at booth # N1033

20 IT/AV Report

nized as the “last link,” completing the original US interstate highway system when the tunnel opened in 1992.

The tunnel’s command and control facility was first established in 1991, just prior to the Glenwood Canyon proj-ect’s completion, to provide manned 24/7 oversight and first-responder capability to this remote stretch of road frequently beset by rockslides and heavy snowfall. In 2012, Novate Solu-tions was engaged to update the control system that, back then, consisted of an antiquated combination of TVs, cam-eras and monitors.

System OverhaulIn that initial project, Novate over-

hauled the control system, deploying an utterly failsafe integrated Distrib-uted Control System (DCS) designed to provide better situational awareness and visual detail to the facility’s com-mand and control operators, as well as its first responders.

Spotting and responding to roughly 450 incidents each year, ranging from simple fender benders to catastrophic fuel spills or rockslides, puts a signifi-cant strain on personnel. Not only are lives at stake, but a road closure means a 3.5-hour detour, or being stuck on the road in dangerous weather condi-tions. The remote location of the tunnel

requires crews to live onsite during their shifts, working three days on and three days off.

The facility had to provide failsafe operation in a highly dynamic envi-ronment, seamless communication between all devices and built-in redun-dancy and reliability without interrup-tion of operation.

Recent ExpansionThe product selection criteria was

based on the requirement to switch from proprietary equipment that was no longer being supported to off-the-shelf process automation equipment that uses Common Industrial Protocol (CIP). Equipment had to be robust, highly secure and easily interoper-able with other devices. These criteria specifically drove the use of Cisco and Rockwell equipment, both of which are well-established industrial brands that use CIP with proven track records for robustness and interoperability.

A significant portion of the equip-ment budget was spent on Jupiter Systems [now InFocus], given that the project was about visualization and the implementation of a videowall. The use of InFocus/Jupiter’s solutions was based around scalability and fault toler-ance. InFocus/Jupiter PixelNet was the only solution on the market at the time

that would support multiple failures and result in no more than a 20% reduc-tion in performance. InFocus/Jupiter’s Canvas software offers security fea-tures that were also instrumental in the overall package selection of InFocus/Jupiter as a key equipment vendor for this project.

Improving ResponseIntegrated information and visual

detail helps the control room operators more accurately monitor incidents and assess the proper response, thereby improving response times and reduc-ing operator stress. The DCS system network infrastructure uses more than 80km of fiber (OM1and SM1) cover- ing thousands of I/O points that are monitored and controlled on a 10- millisecond basis.

With this backbone in place, Novate installed a 27'x9' seamless digital wall of 28 Planar Matrix 55-inch remote-power 50,000 lumen screens to replace the individual computer monitors. The videowall, driven by a network of Pixel-Net video input and output nodes from InFocus/Jupiter, integrates live camera feeds from an analog video system with the DCS. PixelNet delivers high-resolution, real-time video from any camera or data source to any screen. Any information source can be shown

The control room, located above one of the tunnel bores, features five operator workstations. The video selection is designed to provide a sequential view of the highway, organized to reduce workload. Critical life-safety alarms are audibly reported to operators so the operators can remain focused on key tasks.

gove

rnm

ent

government

on any display as a window on a single display, or as a window spanning mul-tiple display devices in a display wall.

The Hanging Lake Tunnels video-wall allows operators to view multiple locations from multiple video feeds simultaneously, as well as to instantly identify the exact location of any spot-ted incident for precise communica-tion to first responders. The DCS also enables operators to broadcast text to the 42 roadside message signs and 82 lane-use signal signs (indicating open and closed lanes) when safety warrants such notices.

CDOT Office of Information and Technology Services Lead Kim De-Coursey explained that, although the facility’s primary focus is on life safety control and unobstructed traffic flow, the department also has a respon-sibility to Colorado taxpayers. “The upgrade, completed five years ago, cost effectively shifted the technology from a proprietary to an open architecture. Simultaneously, each new system com-

public safety without issue or interrup-tion of service.”

Hanging Lake Tunnels set the bar for technology management of tunnel traf-fic five years ago and has maintained that status through this flexible, future-friendly infrastructure.

Going ForwardNovate is now engaged in a compre-

hensive update of that equipment and technology designed to further improve response times through more granular/visual data views, while simultaneously adding two-way communication flow and the ability to selectively communi-cate information to the public and other CDOT facilities.

Novate is switching all analog feeds to IP, adding a multi-honed 1Gbps-speed network on top of 25-year-old OM1 62.5/125µm fiber originally designed to support only 256kbps. The network carries fire and safety monitor-ing (CO levels, heat detection, etc.),

TIPS>> Structure failure mode detection to provide clear root cause determination.

>> Remove equipment fan noise from the operators’ area in order to reduce fatigue and stress.

>> Engage with every level of user to incorporate different demands.

>> Provide recovery modes for nontechnical staff to use during middle-of-the-night recovery.

ponent was evaluated based on reliabil-ity and cost-effective performance. The resulting infrastructure and integrated system makes it possible to continue upgrading the command and control system over time to incorporate new and more powerful devices to enhance

22 IT/AV Report

gove

rnm

ent

operation of fire suppression pumps, 2400HP fan operation and traffic control data.

Analog cameras are being replaced with streaming high-definition cameras, along with FLIR thermal imaging cam-eras providing both visual and thermal data. Thermal cameras will replace obsolete linear heat-detection systems and will set trip points to notify the control room of high heat or fire. When this happens, the videowall automati-cally changes to a fire layout displaying video images from the FLIR cameras. All images are produced and delivered to the DCS through Rockwell/Allen-Bradley Process Automation Control-lers (over Common Industrial Protocol) while securely pushing the H.264 stream through a multiple-failure-point-tolerant, Cisco IE4000-based, fully quad-redundant fiber-ring IP network. From the top of this network, video data is passed through a pair of redun-dant Cisco 3850XS Layer 3 switches to a core based on a Vivotek VMS system.

Thermal, HD CamerasThe thermal and HD cameras sig-

nificantly improve the quality of visual information provided to facility com-mand and control operators, along with more precise coordinates for locating the incident. “Enhancing the visual acu-ity of our cameras, along with stream-ing views, enables us to more quickly spot a pending incident and helps us respond with the right equipment at the right place,” said Todd Anselman, Hanging Lake Tunnels Electronic Specialist IV. “We have 33 extraordi-narily dedicated operators here, and in so many situations, seconds matter. The more information we can provide to augment their significant emergency response experience, the better service we can provide to the public and, in turn, reduce stress in our personnel in quick-response situations.”

Bidirectional Video Facility management also wanted

support for bidirectional video feeds so first responders could see a broader spectrum of visual information on the incident and, in turn, transmit addi-tional incident information back to the videowall from their phone or onboard truck computer. In addition, facility

management wanted to be able to share and continuously provide selected video feeds to remote CDOT offices for assistance or collaboration.

To accomplish this video sharing, Novate installed Canvas collaborative visualization software from InFocus/Ju-piter. A Vivotek VMS system manages video streams from both the legacy analog cameras and the high-resolution cameras.

Canvas accepts both H.264 and H.265 video feeds from cameras and analog to H.264 encoders, accessing both live and recorded content. Canvas streams HD video from the secure internal network to remotely located control rooms where disaster mitigation teams can monitor or assist as necessary. In the near future, Canvas streams will be pushed through a Verizon private network to secure authenticated mobile clients for on-the-highway, real-time viewing of fire, accident and weather situations.

Canvas data is routed to PixelNet for display on the videowall. The DCS system continually monitors the health of all components, including InFocus/Jupiter hardware and software, Cisco equipment, Vivotek and all other pro-cess automation components, logging

data and audibly announcing any issues to the operators with simultaneous SMS notifications to technical staff.

Transmitting data through a moun-tain of rock in near real time is chal-lenging. Simultaneous broadcasting of bidirectional streams of multiple data feeds with continual monitoring of every component within this system is enormously complex. Above all, the entire system must be robust and completely failsafe. This system was de-signed to survive the remote possibility of multiple point failure from any point in the system, from field I/O to video management systems. Single switch failures could not be allowed to impact the system and, further, the system must be able to tolerate an entire bore failure (tunnel fires will melt fiber) without any data loss.

Accordingly, all critical components, such as the Cisco 3850XS switches, the PixelNet Domain Control servers, the PixelNet video nodes, PixelNet switches and Canvas servers are all redundant, so any failure will result in minimal loss of video and data streams, if any. Further, the system must be able to be maintained by field staff alone for 24/7/365 operation. Rockslides and

The operators manage the DCS in addition to radio communications with fire departments, law enforcement and maintenance news. This centralized approach ensures that seamless communication is possible for all responding agencies.

(continued on page 33)

33Spring 2016

CONNECTRIX

INFOCOMM BOOTH #C8833INFOCOMM BOOTH #C8833

Custom Panels forRack • Wall • Stage Pocket

www.acebackstage.com/sc

ACE BACKSTAGE CO., INC.STAGE POCKET SYSTEMS

systems are provided for all key compo-nents of the switching and daily opera-tion of the system.

Modern control rooms cannot run properly without the use of a high-quality KVM system. We installed a KVM system from Thinklogical in the control room, and each PC required for operation or used as source equipment is located in the rack room. We also installed KVM control of all systems delivered to each position required through the system over an IP network. The system has individual access lists and requirements for each user, based on their privileges and workflow re-quirements. In a pinch, any seat could access any system in the design.

Four Samsung 75-inch flatpanel LCDs installed on the walls in the control room replicate the content on the LED screens. Additionally, two Samsung 48-inch LCDs located at the monitoring stations provide a preview of the games.

Nearly 10 miles of data cabling con-nects the video sources, and processing and distribution equipment to the LED video-wall. Signals from the LED control-ler out to the videowall are sent over Cat5 using proprietary IP-based protocol.

There are two zones of audio for daily operation: one for general sports and one for racing. The audio systems were selected, in coordination with Westgate’s onsite staff, to match equipment already in use. At the heart of the audio system are three Harman Soundweb London BLU-806 DSPs, three Biamp Tes-

iraFORTE DSPs with AVB, two Ashly eight-channel stereo line mixers, Shure mics (including a ULXD system), VUE Audiotechnik al-4 line arrays, as well as best-of-breed components from Crest-ron, Extron, Motu, Genelec, Cisco, Bit-tree and Adaptive Technologies. Audio processing with Dante interconnectivity is located in the new rack room.

The front of house (FOH) system serves a number of zones with specific requirements, with some systems used in the daily operation and other sys-tems used only for events. Four sets of line array speakers are installed above the LED screens at selected points in the facility to treat the floor. Sound from the line arrays can be focused to the areas desired.

Connectivity to Westgate’s AV wireless network enables authorized personnel to make changes from re-mote locations through an iPad, when necessary.

There are 22 racks with audio, video and RF equipment in the room down the hall from the control room. The rack room is climate controlled and fea-tures raised flooring similar to a server or data center level room. Twenty tons of chilled air is routed through the bottom of the racks and exhausts out the top.

No one had ever placed an LED order this big before, and lead time for manufacturing was a huge concern. We had to work with Christie to build and test the LED videowall in sections to meet our deadline. This required

SUPERBOOK IS A GAME CHANGER(continued from page 27)

COLORADO DOT EXPANDS COMMAND & CONTROL SYSTEM(continued from page 22) heavy snow conditions will sometimes isolate command and control opera-tors for significant periods of time. In addition, any issue has to be reported audibly in the control room, visually in the DCS and via SMS to technical staff.

“The Novate team has in-depth, long-term knowledge of this facility,” said

DeCoursey. “Not only do they know the systems here, but they understand where technology is going and they keep us on top of our game. They consistently exceed our expectations and meet significant challenges in a practical and cost-effective way, with an overriding focus on public safety.

cess points, network devices and other IT systems are coming to touchpanels, videoconferencing cameras, digital signage displays and more. AV profes-sionals are getting good at standardized, simplified AV solutions. The biggest challenge now is making them secure. When a customer says that his organ-ization is considering prohibiting AV

HOW AV IS BECOMING MORE LIKE IT (continued from page 6)

detailed coordination with the manu-facturer, the component suppliers and the shipping company to coordinate all of those moving pieces. The main LED videowall was only part of the $13 million AV budget, so we had to be cost conscious.

The scale of the project required greater attention to detail and precision alignment than smaller videowalls. We had great support from our partners and manufacturers, especially Christie Spyder, Crestron and Thinklogical. Reflecting on the project, our teams at HB are grateful for the experience of working on such a unique installa-tion, and proud of Westgate LV’s new SuperBook.

systems based on security concerns, we all need to pay attention.

In describing his ideal AV vendors, William told researchers, “They need to have an understanding of all the ways the AV solution will communicate with, or impact, IT and security.”

If you don’t understand that yet, you’re in for an IT-like awakening.

“Glenwood Canyon will always face significant environmental and weather issues. The Hanging Lake Tunnels command and control facilities must keep pace with both what nature and drivers throw at us every day to keep this breathtaking place accessible for travel, recreation and transit.”