FCC Rules for Backup Power Application Note 4-28-14 · PDF...

www.questcontrols.com • Tel: (941) 729-‐4799, Fax: (941) 729-‐5480 • 208 9th Street Drive West, Palmetto, Florida 34221 Property of Quest Controls

FCC Rules For Backup Power – Application Note

Summary

In December 2013, the FCC issued regulation FCC 13-‐158 “Improving 911 Reliability” requiring Service Providers to meet specific performance standards for their E911/PSAP central office and headend sites. One part of the regulations requires more adequate backup power for E911/PSAP Sites. The regulation states that Service Providers must:

! Provide 24 or 72 hour backup power (depending on site) ! Follow equipment best practices in maintaining backup power equipment ! Perform and document annual certifications for backup power ! Maintain 2 years of records for certifying and maintaining backup power equipment ! Be prepared for periodic FCC audits for sites of concern

Quest offers Service Providers survey and audit services, monitoring equipment and installation services, and enterprise software monitoring and reporting solutions to meet all of the new FCC’s backup power requirements listed above for E911/PSAP sites. The diagram, Central Office/Headend Backup Power Monitoring E911/PSAP Sites, on page 2 presents all the relevant backup power alarms and information that Quest provides in its monitoring solution.

Background

As a result of the destructive derecho windstorm in June 2012 that killed 22 people, seventy-‐seven 9-‐1-‐1 call centers (also known as “Public Safety Answering Points” or “PSAPs”) serving more than 3.6 million people in six states lost some degree of connectivity, including vital information on the location of 9-‐1-‐1 calls, mostly due to service provider network problems. From isolated breakdowns in Ohio, New Jersey, Maryland, and Indiana, to systemic failures in northern Virginia and West Virginia, 9-‐1-‐1 systems and services were partially or completely down for up to several days. Seventeen PSAPs in three states lost service completely, affecting the ability of more than 2 million people to reach 9-‐1-‐1 at all.

The FCC intensively reviewed the resulting 9-‐1-‐1 outages and concluded that they were caused by inadequate physical diversity of critical 9-‐1-‐1 circuits , a lack of functional back up power in Telecom central offices, and failure of key data to be received by NOC’s. The FCC concluded that voluntary service provider actions has not been adequate. Among many FCC actions, the FCC is requiring more adequate central office backup power. This includes a requirement for 9-‐1-‐1 Service Providers to maintain two years of records for backup power deployment and maintenance records that supports annual certification of their backup power as well as follow equipment best practices for maintenance and testing of generators, batteries and other backup power sources. In addition, follow-‐up FCC site inspections could be done on a periodic, case by case basis where reliability concerns are raised.

2Property of Quest Controls, Inc.

3 Property of Quest Controls, Inc.

The Backup Power requirement calls for 24 hours of backup at full office load for those central offices that directly serve PSAPs; critical center offices that host selective routers must have 72 hours of backup power at full office load. Backup power to meet these time requirements can be achieved through fixed and/or portable generators, batteries, fuel cells and a combination of other sources. Tandem generators must be electronically separated to ensure the failure of one generator does not cause the failure of the other.

Problems with Current Backup Power at E911/PSAP Sites

From this derecho windstorm, other hurricane, snow/ice storms, tornadoes, power outages and other widespread outage events, Service Providers have learned that some of their key 911 sites exhibit some of the following critical problems with their backup power equipment:

1) Due to changes and equipment additions at their sites, backup power capacity is not adequate to meet the full load power requirements for any extended time

2) Equipment problems with generators, ATS, DC Plants, UPS systems and other backup power sources are often not identified in real-‐time and therefore not resolved prior to extended commercial power outages

3) There is lack of certainty as to whether backup power equipment alarming is working or not

4) Service Provider management lacks remote visibility of critical 911 sites for equipment status, health and recurring alarm activity

5) Regular equipment runs are not tracked, recorded and reported on a consistent basis 6) A single source for viewing critical 9-‐1-‐1 sites has been lacking. Service Providers have

not had remote access and interrogation of equipment maintenance, alarming and reporting data in real-‐time

7) Surveys were not conducted for each site to complete definition of the problems above

Customized Compliance Reports to meet FCC Requirements

OspreyFMS® Enterprise Software provides your required Compliance Reports to meet FCC Requirements. OspreyFMS reports can be customized to the specific existing power equipment and site profiles. Figures 1 – 4 below are examples of critical compliance reports.

How Can Backup Power Equipment Reliability be Improved to Meet FCC Requirements?

Backup power equipment includes Generators & ATS (Automatic Transfer Switches), DC Plants and Batteries, UPS Systems and Batteries, Fuel Cells and other backup power sources. Quest TELSEC® and/or Other Monitoring Equipment can receive discrete alarms (contact closures), analog sensor alarms and data, and/or Modbus data and communications. TELSEC’s can convert these to SNMP, TL1 and/or email alarm messaging, and transport alarms and data to Service Provider NOC’s and Local Maintenance Centers.


1. Generators & ATS

Key Generator & ATS alarms and data include:

! Generator run alarms: in conjunction with commercial power failures, generator run alarms confirm that a generator is running when it should be during an outage

! Generator run time: run time enables Service Providers to plan maintenance and document date/time of run times to meet FCC certification requirements including total run time and number of run time events

! Generator weekly or periodic test and monthly load tests: create documentation for gen run time tests both for gen run and for full load tests to meet FCC certification requirements. Control and schedule gen run and load tests remotely

! ATS Emergency and Normal position: in conjunction with commercial power failures, ATS in Emergency alarms confirm that the full power load has transferred to emergency power. Transfer Switch failures and Not in Auto alarms also can be monitored

! ATS Transfer Switch Failure or Not in Auto: switch failure indicates the ATS has not switched the power load to the generator and Not in Auto Alarm informs you that the ATS is not in automatic mode but in bypass

! Generator Low Fuel alarms and data for diesel and propane generators: gen low fuel can be monitored using fuel level analog sensors or generator pre-‐set low fuel switches to provide data and alarms for low fuel conditions; during an extended power outage, Service Providers can make intelligent decisions for dispatching fuel trucks and portable generators to maximize site uptime

! Generator Not In Auto alarm: if the generator has been transferred to manual operation and accidentally left in that position OR if the generator is under maintenance and out of automatic mode, these preventative alarms ensure corrective action can occur before a commercial power failure occurs

! Generator AC Charger Failure or Low/High Battery Voltage: these alarms inform you whenever the battery voltage is not at the needed level for generator startup OR if the charger itself has failed. If the generator battery is not operating at correct voltage, the generator will not turn ON -‐-‐-‐ these are preventative alarms to ensure corrective action can occur before a commercial power failure occurs.

! Generator Summary Trouble Alarms including Overspeed, Overcrank, High and Low Engine Temperature, Low Oil Pressure, Tank Rupture, Generator Controller Failure, EPS Supplying Load and Other Trouble Alarms: several of these trouble alarms will shut down the generator after it starts, even during an outage. These preventative trouble alarms ensure correction action can occur before a commercial power failure occurs

! Generator Breaker Alarm: if the generator breaker has tripped or turned off, the generator will not turn ON; this alarms alerts staff to a generator breaker tripping or manually turned off

! Key Generator & ATS Data: fuel level data, voltage, current and temperature trends, battery voltage, oil PSI, coolant levels and load tests, load shedding and curtailments


This Generator Run Report meets the PSAP Compliance Requirement needed to prove the backup generators (or other power equipment) were tested and verified working properly. This customized report can include site name, address, test run dates, hours active, whether the run test pass or failed, when the next scheduled run is, and more.

Figure 1

This Power Alarm Report provides a history of active and/or historical alarms from power equipment including Generators, DC Plant, Commercial Power Failures, UPS, TVSS and other power equipment. The report can be customized and can provide date/time, site location, alarm name and description, alarm severity, when the alarm cleared, how long the alarm was active, and more.


Figure 2

2. DC Plants & Batteries

Key DC plant alarms and data include:

! Major & Minor Rectifier Failures: if one rectifier fails, a Minor alarm is sent; if two or more rectifiers fail, a Major alarm is sent. Multiple rectifier failures can inhibit backup DC Plant power capability.

! High & Low Battery Voltage Alarms: high or low battery voltage will prevent the DC Plant from charging properly, and therefore limit the backup power capability for the DC plant to support any DC power load during an outage

! Circuit Breaker/Fuse Alarms: if there are any tripped circuit breakers OR fuse failures, alarms will be sent. These preventative alarms ensure correction action can occur before a commercial power failure occurs


! Battery Disconnect Alarms: if a 48VDC battery string is disconnected, the service provider will lose that battery string backup power. The battery disconnect alarm enables preventative service response to restore the battery string backup power

! Battery Discharge Events: when a 48VDC battery string goes into discharge, an alarm can be sent at pre-‐set voltage threshold to inform when a DC plant voltage is dropping. This data can also be documented to meet FCC Certification Requirements

! Battery Health for Individual Batteries & Strings by monitoring conductance, voltage & temperature alarms: since battery performance declines over time as the typical 2V or 12V batteries age, early detection of failing batteries can identify individual battery impending failure before the battery fails. This enables individual battery replacement to ensure a full 48VDC string does not fail and lose backup power capability during a power outage. Rising temperatures and high float current are early indicators of thermal runaway

! Fuse & Breaker Panel Failure Monitoring: DC Plant critical rack equipment can be monitored for individual or equipment row fuse, power, and A/B side failure alarms to prevent rack equipment failure

! DC Plant Controller Failure: when the DC plant controller fails, an alarm is sent ! Hydrogen Gas Alarms: using a hydrogen sensor mounted near the DC plant batteries,

when hydrogen gas builds up in the air, an alarm is sent warning of a potential explosion ! Other Key DC Plant Data: voltage and current trends, plant capacity

3. Inverter Systems

Inverters convert AC to DC power and visa versa.

! Inverter Failure & Status Monitoring: as more Invertors are deployed for AC-‐DC power conversion, high/low voltage alarms major and minor equip failure alarms, inverter bypass, inverter capacity exceeded, inverter fuse failure and alarms can be sent for correction action. Voltage and current trends can also be monitored

4. UPS Systems & Batteries

During a commercial power failure, until the time the site building power load is transferred to Generator Power, the power load may fall on to the UPS. If the UPS fails during this critical time, the site may lose all power and go down. Key UPS System alarms and data include:

! UPS on Bypass: if the UPS has been transferred to manual operation and accidentally left in that position OR if the UPS is under maintenance and out of automatic mode, these preventative alarms ensure corrective action can occur before a commercial power failure occurs

! UPS on Battery: during a commercial power failure, when the UPS supports the site power load, this critical alarm is sent. Most UPS systems can only support full building power loads for minutes, so response must be prompt and backup power transferred to the generator or another major power source quickly


! UPS Low Battery: when UPS battery voltages are running down, an alarm is sent warning of low battery voltages.

! UPS Summary Trouble Alarms: some trouble alarms will shut down the UPS even during an outage. These preventative trouble alarms ensure correction action can take place before a commercial power failure occurs Battery Health for Individual Batteries & Strings by monitoring conductance, voltage & temperature alarms: since battery performance declines over time as the batteries age, early detection of failing 12V or 2V batteries can identify individual battery impending failure before the battery fails. One failed battery will cause the full battery string to fail. This enables individual battery replacement to ensure a full UPS battery string does not fail and lose backup power capability during a power outage

! Key UPS Data: input and output voltage and current, UPS load and capacity

5. Fuel Cells

Fuel Cells are being used a backup power source and provide alarming as follows:

! Fuel Cell Running: alarms when the fuel cell is not active and supplying power; data point to know that the fuel cell is active and supplying power

! Low Hydrogen Fuel Level & Lower Pressure Warning: a warning indicating you are about to run out of fuel and hence shutdown. Fuel pressure is measured in PSI

! Power Available Fuel Cell: when the fuel cell is operational and supplying DC power, an alarm is sent. Typically a minor or informational alarm is sent, indicating that the fuel is operating in normal mode

! Stack Heater Fuel Cell: Stack heater is used during fuel cell startup operation and will reduce output or turn off after fuel cell reaches target temperatures. A failure alarm is transported if the stack heater fails

! General Fault: summary alarm for equipment failure of some type ! Intrusion Fuel Cell: alarm for opening of the fuel cell cabinet or the hydrogen gas

storage area ! Key Fuel Cell Data: hydrogen fuel pressure, fuel cell run hours (events and total), fuel

level, voltage and current and fuel cell load

6. TVSS

TVSS or Surge Suppressions Systems protect equipment from AC power surges and spikes.

! Surge Suppression Failure: a surge suppression failure alarm indicates that the surge suppression system has failed typically due to an AC power surge


7. Power Meter Monitoring for Main AC Power Panel

Installing a power meter at the main AC power panel provides additional key alarms and current and voltage trend data:

! Phase Loss Alarm: any loss of A, B or C Phases can be immediately alarmed ! Phase Overload Alarm & Data: when an individual phase exceed an acceptable

threshold, between 70-‐100%, then alarms and/or email alarm messages can be sent alerting staff of a phase overloading problem. As % overload increases, these alarms can be escalated

! Phase Imbalance Alarm & Data: when individual phases exceed 10-‐20% difference compared to other two phases, unhealthy harmonic power distortions can damage equipment; alarms and/or email alarm messages can be dispatched

! Over/Under Voltage: when AC power exceeds ~10% of voltage (480VAC, 240VAC or 208VAC), then alarms and/or email alarm messages can be sent

! Low Power Factor: power factor is defined as the existing power used as a % of apparent power available. When power factor declines, the power quality declines and an alarm and/or email message can be dispatched

! Low Frequency: when AC power frequency drops below 60 Hz, an alarm and/or email alarm message can be dispatched

! Energy Consumption: site power energy usage can be monitored via kwh


Figure 3

This Backup Power Equipment Status Report meets another PSAP Compliance Requirement by providing backup power equipment runtime details, fuel details, generator details, backup testing history, and alarm history by Site.


Figure 4

This Fuel Site Trend Report provides fuel trends in a graphical format. This reports verifies appropriate fuel levels to meet any compliance requirements. While this Report shows Fuel Cell Hydrogen Fuel Levels, similar reports can be generated for Diesel, Propane, and Other Fuels over time.

Quest Control’s Solution for Improving Backup Power to meet FCC Requirements

Quest’s Solution for assisting Service Providers to meet the new FCC Requirements includes survey and audit services, specific monitoring equipment and installation where real-‐time alarms are lacking or not functional, and/or enterprise software solution for monitoring and reporting backup equipment performance.

1) Survey and Audit Services

• Quest qualified application engineers and field staff can perform site surveys to determine the backup power problems needing to be resolved to meet FCC’s new requirements. Surveys will include documented survey results and pictures along with recommended actions including costs to remedy any backup power deficiencies.

• To support annual certifications or periodic FCC audits, Quest qualified field staff can assist on-‐site and remotely in certification preparation, testing and documentation to meet FCC backup power requirements.

2) Backup Power Equipment Monitoring & Turnkey Installation Services

• Quest designs and manufactures a full line of cost effective monitoring equipment for all backup power and facility critical equipment. Our TELSEC® product line of monitoring and control systems is capable of monitoring, alarming, and documenting alarms, events and key data including real-‐time power equipment problems and testing events of generators, ATS, DC Plant, UPS systems and other power source equipment.


• Quest also provides turnkey installation and/or commissioning of Quest TELSEC® or Other Monitoring equipment at Service Provider sites nationwide. Turnkey installation includes full project management, installation and commissioning including full validation of all alarms and communications to Service Provider NOC and/or Local Maintenance Centers.

• Quest also offers 24/7 backup monitoring capability within its QNOC based in Quest’s Palmetto FL HQ.

3) Enterprise Software Solution for Monitoring and Reporting Multiple Sites

Quest offers a leading edge Enterprise Software Solution for one to thousands of sites nationwide called OspreyFMS® which:

! Accepts any SNMP-‐enabled, BACnet-‐enabled and/or Modbus-‐enabled equipment including TELSEC® Systems and/or other existing monitoring systems.

! Is capable of handling all intelligent monitoring from any brand of generator, ATS, DC Plant, UPS and other critical backup power source or site facility equipment. Provides a versatile user-‐friendly Dashboard for remotely viewing multiple sites from any location as well as powerful multiple site trending, logging, alarming, reporting, and tracking power and energy management analytics in real-‐time and over time.

! Accepts data feeds from any other Internet sources such as weather, outside temps, etc. ! Capably links to any IP-‐based system for viewing any equipment embedded web servers

directly. ! Enables Service Providers to track, document and report generator, ATS, DC Plant, UPS

alarms, run times, real-‐time power loads, and test results, and then generate the necessary compliance/non-‐compliance reports to meet FCC’s new backup power requirements.

! Takes information collected from monitoring systems or any IP device and runs deeper analysis as well as calculates and presents metrics. Works to automatically provide update notifications supporting the DIRS mandate: The NORS System is a compliment to the DIRS mandate, and is utilized for recording and tracking network outages. As a centrally hosted application OspreyFMS is capable to receive and output XML formatted files directly into the NORS (Network Outage Report System). Administrators can easily bring in XML files from the NORS system into OspreyFMS. With this information, OspreyFMS provides outage histories and overlays that information on the actual monitored data from the facility. Likewise when an outage is detected by OspreyFMS, this information is automatically noted and can then be sent to the NORS software.

FCC Rules for Backup Power Application Note 4-28-14 · PDF...

Documents

Transcript of FCC Rules for Backup Power Application Note 4-28-14 · PDF...