NTOC Talking Operations SeriesProfessional Development Series Course NTOC 05:

Performance Management for Signal SystemsWebinar Transcript

January 18, 2012

Jocelyn BauerHello and welcome to today's National Transportation Operations Coalition Talking Operations webinar on Performance Management for Signal Systems. My name is Jocelyn Bauer and I will give a brief introduction to the web conferencing environment before turning this session over to our instructor.

This webinar is the fifth in a series of professional development courses offered by the National Transportation Operations Coalition on signal systems. To access the recordings of previous signal courses, you can visit the webinar archives page of the National Transportation Operations Coalition (NTOC) website. I will type that into the chat box.

Our webinar is being recorded. It will last approximately 90 minutes. The format will be different than previous Talking Operations webinars. Instead of having several presenters, we will have just one instructor, and it will be more interactive. During the webinar, if you think of a question you can type it into the chat box on the left side of your screen. Make sure you send your question to everyone rather than just the presenter. The instructor will address the questions as he goes.

A file containing the audio and visual portions of this webinar will be posted to the NTOC website within the next week. Attendees will be notified of the availability of the presentation, recording and closed captioning of this webinar by e-mail. We encourage you to direct others in your office who were not able to attend to access that recording online. The presentation is available for download in the file download box on the left-hand side of the screen. Click on the name of the file and click the button at the bottom of the download box that says “save to my computer.”

At this time, I would like to introduce our instructor for today’s webinar, Gary Thomas. Gary is a research engineer with the Texas Transportation Institute. He has taught nearly 70 workshops and seminars on a wide range of topics, including ITS standards, traffic signal design and operations, intersection safety improvements, and freeway operations. He has been a National Highway Institute (NHI) instructor for 9 years and trains future NHI instructors.Gary, you can start when you are ready.

Gary ThomasThank you very much, Jocelyn. I appreciate the opportunity to be here today. I'm going to say hello to everyone out there, and hopefully you can put a face to the name. I won't have it up here the whole time because it's more distracting, but I wanted to say good morning or good afternoon and possibly good evening if we have anyone dialing in from other parts of the world, which is not unusual. I'm glad you are with us today, and more people are logging in as I speak. We will get started right now.

Please go ahead and chat in questions as we go along. I try to give my webinars as much of a real classroom flavor as I can and address questions as we go along rather than to having to think back. If I see things come through I will try to answer them as we go along. Jocelyn will answer any questions of a technical nature, like those regarding where to download things or whre can I go to see this later. Any subject matter questions, I will answer on the fly as we go along. I think a lot of you are listening on computer speakers. If you dialed in, we will stop once or twice during the webinar to give you a chance to ask over the phone if you prefer.

This is Performance Management for Traffic Signal Systems. The target audience for this webinar includes directors of traffic engineering, traffic operators, practitioners responsible for day-to-day operations of signal systems, and decision makers responsible for budgeting decisions regarding signal systems. There is a pretty wide swath on our target audience. Hopefully you fall into one of those categories and you will get something out of today's webinar.

I am with TTI in College Station, Texas. I frequently teach for not only ITE and FHWA but the National Highway Institute as well. I’m a fellow of ITE and other organizations and active in TRB as well.

Let's talk a little bit about what I hope you get out of this course today. I have four learning outcomes. You should be able to: identify the elements of a performance management system and describe the benefits that can be gained from using one; explain how a performance management system can be applied to traffic signal operations; identify measures and supporting data sources for traffic signal management and operations; and explain ways to integrate performance measures into your agency’s process.

As we go a long, you will see in the bottom that I have identified which learning outcomes each slide applies to so that can help you keep track of where we are. You can download the slides in the file share box.

I'm going to take a few minutes to look at performance management in a generic sense and not necessarily specific to traffic signal operations. This may be a little bit of a review for those of you who are familiar with what a performance management system is and how you use one. I imagine there are a few people out there who are not familiar with performance management systems, and I want to bring everybody up to speed. Performance management includes activities that ensure that goals are consistently being met effectively and efficiently. Performance management can focus on the performance of an organization as a whole. You can focus on a department, an employee, or even the process to build a product or service as well as many other areas. I'm going to take a few moments to relate it to performance management in personal areas so we can see a direct correlation between performance management in the workplace as well. What is performance measurement? I'm using a slightly different word: what is performance measurement as opposed to management? We want to look at performance measures. I have three definitions. One is the dictionary definition: the process whereby an organization establishes the parameters within which programs, investments and acquisitions are reaching the desired results. If we look more specific to transportation, a simple way to define it might be to say a way to quantify how well a transportation system is working. FHWA has an added

definition to that, and they say the use of statistical evidence to determine progress toward specific defined organizational objectives. The operative word is the use of statistical evidence. Performance measures can be hard and fast measured parameters, such as pavement surface smoothness. Travel times, if you're in traffic operations are, can be something that can be measured. They could be measures of customer satisfaction, which, taking those same two, something that is more customer satisfaction would be the perceived ride quality of the pavement. It's not something that can be hard and fast measured with a device, but it's going to be more of a soft measurement. Where travel times can be accurately measured, it could be on-time arrivals or perceived delays by a driver. They may not be measuring it with anything, but they might say most of the time I feel like I am late, or most of the time it seems like my travel time varies, or it's pretty consistent, so that would be more of a soft measurement. Regardless of how they are measured, performance measures can be used to provide feedback about how well the system is performing, both from a user’s perspective and an operator’s perspective.

Major contributors to the inconsistency found in traffic signal operations and maintenance budgets can include some of these things: a lack of clear guidelines that describe traffic signal operations and maintenance activities and resources required to spark these activities. Also, there may be a lack of documented objectives and performance standards. If we don't have them, how do we know if we have met our goals? We need them to be documented. Another roadblock is that funding mechanisms are geared more toward project development than operations and maintenance. I think we have kind of seen a little bit of a shift in the last 5-6 years, and we are seeing more funding geared toward O&M that we didn't see 10 years ago. In smaller organizations, you may still struggle with needing that ongoing funding for measuring how well your system is performing. Part of the reason for that is it's not as exciting to cut the ribbon on a performance management system as it is to cut the ribbon on a new roadway or bridge or something of that nature. The governance structure and division of responsibilities among numerous agencies at the Federal, State and local levels can cause roadblocks along the way. The proprietary nature of the signal systems industry, both hardware and software, can cause issues. Again, I think we are seeing a little bit of improvement there, especially as we start adopting ITS standards, open source systems, open platform systems, and things like that. Tort liability: I think we are all familiar with that. Some are worried that if we show that our system is performing poorly, then we are opening ourselves up to some liability. I don't have any particular instances to give you there. We also have equipment issues. Some of us are dealing with old equipment that simply cannot provide this type of data we need: different controllers, detection systems, etc. If they have been out there awhile, detection systems may not be up to par with what we need, and maybe we are just now moving into other systems that are more reliable.

Somebody asked: wouldn't deliverables in the project funding cover O&M elements or procedures? Yes, if that is in there; if that is part of the project scope as a whole. We are seeing that, and that’s getting at that shift in attitudes. Ten years ago, you might have been hard-pressed to see a lot of the O&M stuff in the project scope of work, but now I think we are seeing that a lot more, so that is a good route we are taking.

Let's turn to more about why you should use a performance management system, and I have posted a lot of different reasons you might. All or some may apply to your particular situation.

Legislative mandates are one. Many years ago, for those of you have been around awhile, back in the days of ICE-T, the 1991 legislation that specified a number of different management systems that were required. I would be hard-pressed to come up with five or six right now, but I know pavement management systems are one of them, in addition to others. They eventually got taken out, but I think it got the ball rolling for our need as system operators to look at managing our systems and performance management. You may have some State mandates that require new projects to be monitored. That might be one reason why you would have to move into a performance management system. For the planning processes, including budget and funding allocations, you may be required to have performance management. Maybe your local jurisdiction, if you're working for a city, might have an ordinance in place that says in the planning process, you must include performance management. Maybe your organization has quality initiatives they are looking at in performance management systems, and that would be the way to monitor those. Congestion management systems and evaluation is another one; ITS operations and evaluations; and safety management systems. One that I came up with later that I don't have on the side is dealing with permit processes for commercial development. We may be looking at how a system would perform as you get more development on an arterial street, for example. That might be one additional reason. There may be others I haven't included. At any time if you think I left off something, I would encourage you to chat something in for the good of the class. There will be a couple of times that I'm going to ask you some questions. I will try to give you a heads-up before I do that.

Transportation Asset Management would be another example. If you are dealing with certain assets—I don't know if you're thinking about transit and how you would manage your assets there, buses and things like that: street furniture, bus stops, or even traffic signal poles. There was a mention about pavement management systems. As someone from ODOT chatted in, whatever gets measured gets improved, and that's a good point to make. When you do measure something, you can show to the decision-makers, a city council or city manager or a State transportation commission—whoever makes the decision about budgets—when you show we are measuring this, they will say okay, we will steer more funding toward that area and measure it, and hopefully by next year it will be better. Somebody else mentioned about learning from your actions and expanding your knowledgebase. If you implement a new traffic signal timing plan, if it's not measured, you don't know if you have done any good, and you can learn from that. There are a lot of benefits as well. Just because you are forced to use a performance management system through a mandate, doesn't mean you're not going to get numerous benefits. Good leaders keep their organizations focused on the highest business priorities. Agency leaders setting an agenda help to motivate staff. If you are in charge of some type of system and you don't give specific direction to your staff, they will lack that motivation. How do they know if they're doing a good job or operating the system that best it can be? Objective performance data helps them understand challenges and set appropriate policy priorities. I often find that strong performance emerges when day-to-day business processes are aligned with well thought out agency-wide strategic priorities. When things come together that line up with what you are doing, with the strategic agenda, the evidence is there that we see a stronger performance. Accountability is a fact of life, certainly for public agencies. Transportation agencies that ignore the expectations of their elected officials and citizens run the risk of stimulating adversarial relationships that drive up the risk of negative policy mandates and reductions in funding. If we are not being held accountable, we have lots of people looking over our shoulders these days.

People are very concerned about use of public funds, tax dollars, and where they're going and that not being squandered away. Having a system in place and will improve your accountability to those people. It can provide better customer service as well. You can develop tools to predict future performance. For pavement management, a lot of work has been done in that area since the 80s and 90s, and we are able to start predicting our performance of pavements and understanding the opportune time to come in and replace or rehabilitate pavements. The same can be done for systems. If we have an idea of how often we need to look at retiming signals and if we re-time our signals and start measuring things like travel times and delay, we will probably see delays increase over time. We will see travel times increase over time, especially in areas that are growing. By looking at that and measuring things, either on a daily or monthly basis or an annual basis, we can look at when we need to do this again and predict future performance. Hopefully it can also help transform public images and reduce risk to using public funds, and that gets back to accountability. Also, data archiving is not mentioned up there, but if you properly archived data, transportation agencies can use that historical data. They can make better use of those limited resources.

Let's talk a little bit about the structure of a performance management system. The thing to remember about a performance management system is that it's only a part of a larger process that includes a strategic planning component, which are the green boxes and circles in this particular diagram. We’re not going to talk about them today. We are more in the middle and talking to the purple area, which is the performance management system with things around it and below. The areas of the orange are not going to be addressed today in this particular webinar. In doing some research and surveys, we found there are few traffic signal programs that have a documented strategic plan. There are some out there, but it is not widely used. A performance management systems is a continuous cycle. It consists of four components: selecting measures, setting targets, making decisions, and evaluating the system. The third of these steps, using performance measures to make decisions, is the primary focus of this webinar, but we will talk about the other things as well. The two basic forms of performance-based decision-making that are influenced by a system are those that address resource allocation, which could be rephrased “doing the right thing,” and resource efficiency, which we could call “doing things right.” This is where the measures get fed back to the audit process that may exist, doing the evaluation of the system. Performance management is the application of data analysis and innovation to support these decisions we are making. This is taken from the NCHRP 660 report about performance management structures; if you Google “NCHRP 660”, usually it will come up.

We’re going to look at selecting measures, start there and go around the performance management system. The strategic plan should serve as the guiding document in selecting performance measures. Measures should reflect agency goals and objectives, providing the data needed to answer the question: how we are doing? Your measures should be outcome-oriented, much like the learning outcomes of this webinar; you should be able to explain things and to find things when you are done today. Your measures should be outcome oriented, meaning they examine the impact of decisions made rather than simply the amount of resources being devoted to a particular practice.

One question I want to ask is if you are with a public sector agency, does your organization have a strategic plan for its traffic signal system? Jocelyn, can you do a poll on the fly? Public-sector agencies, does your organization have a strategic plan for your signal system?

The measures an agency chooses to collect and track data should be specific enough to directly address the objectives set out in the strategic plan. You will want to identify measures that are relevant, and that is sometimes a tough one to come up with. What are relevant objectives? What may be relevant to the traffic signal engineer may not be relevant to the mayor or city council. We want to identify those measures that are relevant to the achievement of agency objectives. Data for the measures should be reasonably easy to collect and work within the agency’s operations and maintenance constraints, complementing them to the greatest extent possible. It's important to employ measures that can be tracked incrementally and compared against performance targets. It should be easy to collect and easy to collect fairly often or at certain increments. Performance targets are often very ambitious, and measurable incremental progress may be an indication the agency is doing the right things but needs to step up efforts. Linking that more toward life in general, if you've got certain performance measures you want to meet, you can have a very ambitious goal—perhaps it's the brand-new year, people say they're going to lose weight—you might set ambitious goals, but you need to do something that you can track incrementally and know you're making progress instead of waiting to the end. We will talk a little bit more about doing the targets in just a minute.

As data collection and analysis improves, new measures will become possible that better reflect a particular outcome than whatever measure is currently in use. Agencies should be careful to update measures only when the new measures clearly improve the decision-making process. You don't want to jump around from measure to measure; you want to be consistent and update them only when you have a clear reason for doing so. Maintaining existing measures provides continuity throughout the overall process and gives you the historical information that you need to understand trends.

A bunch of you have been answering the poll question (For public agencies, does your agency have a strategic plan for your signal system?). Most of you (about 73%) said no; 16% of you said yes; and 11% of you were not certain. That was for and those of you in a public agency and we probably have a lot that are not public agency members in the room today. A lot of our agencies out there that are participating don't know if they have a strategic plan for its traffic signal system. It's something you might want to think about doing. If that's not your particular job, you may want to talk about it to those that are responsible for doing something like that.

One of the first steps in designing a performance management system is to bring together stakeholders to discuss and select performance measures. The measures should be meaningful to the typical driver. For example, agencies oftentimes focus on traffic performance measures such as delays and stops, and we base those on aggregate measures—overall delay, average number of stops per day, etc.—which is not perceivable by an individual motorist. Things like travel time and reliability hold more meaning to them because they know what their travel time is or they know if it fluctuates wildly, that on some days it takes an extra 50% of their time to get to their destination; that they perceive quite readily. The stakeholders should review any strategic plan documents that are relevant to the operation of the signal system. You would want to identify

data elements that are already collected by the agency. You may or may not use them, depending on how accurate and reliable the data is, but you want to start from somewhere. There may be data elements not currently collected that you have the capability of collecting. As our controllers get more advanced and detection gets more advanced, we can start collecting other things. We are seeing an increase in Bluetooth technology and using Bluetooth to collect data. I can talk more about that with you separately if you want, but a number of jurisdictions are starting to use Bluetooth capabilities to collect that information. Once you have looked at that data, what are and are not collecting, you want an assessment of equipment needs to complete anything. Maybe you have data you want that you don't have the ability to collect right now, and you want to identify that.

Let's move on to the second rectangle around the performance management system: setting targets. There are a number of acronyms you can find for setting targets and different criteria to set a target. This is one I use a lot when describing setting targets; they all vary just a little bit. The acronym is SMART. There is a typo on the slide: the “s” stands for “specific.” Targets need to be specific in nature. An example of a specific one for traffic operations might be “improve travel times in the peak hour by 10%.” It is very specific, not just “improve travel times.” The “m” is for measurable; they need to be something that can be measured on a regular interval. As I pointed out, you may already collect some of this data, maybe you have the ability to collect it but do not presently do so, or maybe you need new equipment to capture that particular data, but a target should be measurable. The “a” is “ambitious.” I think it's important to set ambitious goals, whether it is a traffic signal system or your personal life goals. If you're looking at getting out of debt, sec your gold to be ambitious, but not outrageous, which get’s to the “r,” “realistic.” Be ambitious in setting your targets, but be realistic in finding some middle ground. They also need to be time-dependent. If you say that you're going to do something, you need to set a time to do that, otherwise it's not going to get done. It’s the same with performance management systems. You want to set time dependency on all of your targets, otherwise you can say that is our target, but I don't know how long it will take to get there, and we will get there when we get there, in which case you will probably never get there. Be short-term enough that progress can be measured—yearly, monthly—and targets that support strategic planning objectives should probably look no more than none 2-3 years ahead. It is tough to set a target for something 10 years out because something will change a lot with technology. Even in 2-3 years, technology changes pretty fast, but we can probably do that and have a good target.

We brought together stakeholders, and now, how do we use those measures in decision-making? We've got strategic decisions that have been made, and now we look at how we are going to allocate resources based on those targets in the performance management system? You also have all kinds of decisions you will be making based on the performance of your system: programming decisions, operational decisions such as making changes to peak hour timing. As we get the information back, that may trigger decisions about “our peak ours spreading out a lot more; it used to be mostly from 5:00 to 6:30, but with new development, the hours are spreading out” That is an operational decision, as is when you simply change cycles or something like that. Human resource decisions can come from measures such as how you are staffed up and how many technicians you have per traffic signal or per 50 traffic signals, something like that. If you are a growing area building a lot of traffic signals each year, you may use that to trigger human resource decisions, like hiring more staff or re-tasking certain people to do other things.

Programming decisions I think of more as “when and how often do we add signals? Are we going to start doing things differently as the system expands and as travel times and delays go up?? That is a programming decision. When you use measurable things, they can trip certain decision-making things you need to do at specific times.

The performance management system itself should be regularly updated and evaluated. You don't want to do it every month, but you probably want to at a minimum come back on an annual basis and look at how your system is working—at most a year, 6 months might be alright if you’re just getting started. The interaction between setting strategic agency priorities and selecting performance measures and targets will lead to changes in both priorities as your data demonstrate more efficient courses of action and measures and targets, as changing priorities required new measures and targets. Another thing we see is that technological advances and feedback from employees and external sources may lead to a development of an improved measure or more reliable collection measurements, an upgraded data system, data archiving. Going back to thinking about this Bluetooth data collection that some jurisdictions are doing, 5 years ago we were not even talking about it. Three years ago there was maybe initial discussion about it being used, and now agencies are doing it. The stakeholders need to be educated that performance measures may vary period to period. Regular operations and maintenance of the system are also valuable in evaluating the system.

I haven't seen any questions come in so I will just take a minute and turn to the operator to see if anybody wants to answer a question over the phone. Operator, please tell them what they need to do.

What we have been looking at up until now was kind of the overview of what a performance management system might look like, and in a moment we’re going to look at specific traffic signal management plans.

A question came in about the most common system measurements. We will talk about those coming up, so we will see if your question gets answered as we go along.

Let's talk more specifically about traffic signal systems. It has been estimated in previous research that improper traffic signal timing can account for 5% to 10% of all traffic delay, or 295 also estimated that 295 million vehicle-hours of delay on major roadways alone. That’s from a DOT report I listed below called “Temporary Losses of Highway Capacity and Impacts on Performance.” Another study found that congestion causes the average peak period traveler an extra 38 hours of travel time and an additional 26 gallons of fuel, amounting to about $710 per traveler per year. That is from the 2007 Urban Mobility Report from TTI. We just came out with the latest Urban Mobility Report. I will check those numbers and maybe get these updated, but it has probably gone up. We need to realize that traffic signal system performance is critical to reducing delays and reducing congestion.

This was taken from the outcome-oriented performance measures for the management of signalized arterial capacity from the repot Transportation Research Record (TRR) 2192, and that is completely dedicated to performance management systems and traffic signal operations. If you don't have a copy of that and it's something you want to get into, I would encourage you to

seek out that document. While it is research-oriented, I find some of this stuff is very practical; you can put it to use in real-life situations. This diagram illustrates what would be a typical organization’s approach to signal timing and maintenance. Based on objectives, data are collected for specific times of the day. There are various software packages that will design cycle split and offsets geared toward those particular objectives. Again, we set priorities, look at getting data, and then we might model it using some particular software. Then you would do the design and documentation. You can input the data, get the data out, and deploy it. You might do a little bit of feedback here. You look at the timing and documentation, and any good traffic engineers would not simply take those timings and put them in verbatim out in the field; a good traffic engineer would look at them and see if they makes sense and probably go back to the software and make some adjustments, then come back and get some timing designs. Then, once you're satisfied, you move over to the deployment stage, and then the last step is evaluation. For the evaluation, make sure that what you put in the controller is what you expected and that things are improving. Then hopefully we have a feedback cycle. Sometimes we take too long to go back and re-evaluate and look at doing a different timing design and documentation. We have a significant amount of quantitative feedback in the timing design, but very little in the deployment area. Certain inherent weakness these can lead to operational deficiencies, such as the data collected often doesn’t reflect weekend and off-peak times. We are typically very concerned about peak hour travel, so we forget to go out on Saturday afternoon or we assume that if it works okay during peak hours then it’s working okay during non-peak hours.

Timing plans are oftentimes static, or at least most agencies still deal with static timing plans, meaning they don't change on-the-fly. They may change due to a new traffic plan at a particular time of day, but they are not dynamically responsive to traffic. The data is collected sometimes on the order of years or on customer complaints. We might take 2 or 5 years to evaluate something again because it is so labor-intensive to make travel time runs. Different ways of collecting data are making it a little easier, but back in the day it was a lot of hassle to go out and collect this data. Even new data can yield the same deficient results from software programs, so it's not always going to solve our problems or make things better, but you never know. We need to look at more of this evaluation and having a performance management system where this is collected automatically can help out quite a bit.

The question becomes, is there a better way? The last National Report Car (this one is 2007) just closed a month or two ago. Hopefully some of you participated in that in doing self-reporting for traffic signals. Self-reporting in the last one said traffic monitoring and data collection got an "F." The report said it is the greatest potential for improvement. Forty-three percent reported little to no regular ongoing program for collecting and analyzing traffic data, and half of the agencies do not assess the quality of the data collected. This is self-reported information, so it's agencies regularly admitting their shortcomings when it comes to operating and maintaining traffic signal systems.

Before we explore the answer of can we do better, let's look at the principles of performance management as it relates to traffic signal systems. The following principles provide the basis for agencies to develop an objectives-based traffic signal management program. First of all, we want clarity of objectives. The attainable performance evaluation should be linked to those objectives. Standards of performance, for example scoring well on the signal timing report card, should be

based on objectives rather than agency activity, especially the arbitrary frequency of activities. Resource requirements should be based on objectives rather than industry norms. The program, should be based on clear and consistent communication with policymakers and elected officials, and also based on a systems engineering process in thought rather than merely in deed. If you don't know much about systems engineering, I encourage you to find more information on that. They really go together; the two are directly linked.

Drivers have simple overall expectations. They want to drive to their destination at their desired speed with minimum of attention, or at least they want to be treated fairly and predictably so they can plan their day with a minimum of uncertainty. That is a customer expectation, and how you might drive your objectives might come from that. That is not going to vary from location to location. That is your customers and what they are thinking about.

The agency objective is slightly different. It can be very simple. This is an example: we will do our best to avoid making drivers stop, and when we must make them stop, we will delay them as little as possible within the context of safe operations. That is an agency of objective, which should correlate well with customer expectations or customer objectives. From there, we can develop a good strategic plan.

What are some of your objectives? You can just type them in if you want to chime in on any of your objectives for a traffic signal system, whether it’s as an agency or a public agency. Someone said 100% operation. No downtime, I guess: your system operates all the time and you’re not having any malfunctions. Someone else wrote “a huge issue for signal systems is addressing performance for pedestrians as well.” I've been more vehicle-centric so far, but pedestrians: what are their objectives or their expectations? They expect a safe environment that is reasonably direct; that is obvious and gives them time to cross streets and things like that. “Minimize total delay at intersections.” That is an agency of objective. It doesn't mean much to the driver. All I know is if I am being delayed. You can have those other objectives as well. “Able to respond to complaint:” you might have an objective to respond to all complaints within a certain timeframe, at least begin to address or analyze them. “Monitor the health of your detection and preemption priority systems.” That’s a good one, knowing whether your detectors are failing. If you have a lot of loop detectors and you are in an area of the country that experiences a lot of freestyle cycles, those detectors are very susceptible to damage, or in areas of extreme heat where you have a lot of asphalt, it will break loop detectors. Video systems are not perfect either. You want to make sure those are operating properly. Cameras can move over time. You want to make sure that your video detection is healthy as well. “Reduce number of complaints.” “System approach for the network trip times, not just link speeds or delays,” so more aggregate measures, going from point A to point B, not just the link speed between two streets. “Minimize minimum green time when using detection,” so there are not many unnecessary delays. Some more are minimize delays and complaints, minimize response statements by improving preventive maintenance, able to update and optimize traffic timing quickly and efficiently. Those are your objectives, which you do need to flush out a little more to be more specific on how you measure them.

I mentioned this relationship between performance measures and systems engineering. Once you have agreed on the agency objectives, it is time to develop performance measures. Each

performance measure should be linked back to at least one objective, and similarly, each objective should have at least one performance measure. It can have more than one, so the objective can be one-to-many, and the performance measure should link back to a one-to-one agency objective. If you're familiar with systems engineering processes, this probably sounds familiar. In the case of systems engineering, we develop user needs first and then functional requirements. Every user need should have at least one functional requirement, and every functional requirement should trace back to at least one user need. Functional requirements must be specific, measurable, and subsequently testable. The systems engineering process can be applied to a traffic signal performance management process. If you are not familiar with systems engineering, I would encourage you to look that up. There a lot of good material out there on FHWA’s website on systems engineering as it applies to ITS projects.

Let's look at an example performance measure. You see I have my SMART topics at the top: specific, measurable, ambitious, reasonable and time-dependent. To reduce travel time, if we took that as our performance measure, it probably doesn't really meet all of them. I would argue it is not specific and it isn't measurable, and there’s no ambitiousness, reasonableness, or time-dependency to it. Reduce travel time for normal traffic conditions: that might be a little more specific, but it probably doesn't meet the other ones. Reduce travel time by 25% for normal traffic conditions: we are getting closer. I would argue that is very specific, measurable and ambitious, but it probably doesn't meet the criteria of something that is reasonable or reachable. Reducing travel time by 1% for normal traffic conditions: now we have just switched it around. It's not terribly ambitious, but certainly is achievable without too much trouble. Reduce travel time by 3% for normal traffic conditions when signals are retimed: now we are getting something that one could argue does meet all five criteria. It is specific, we can measure it. Maybe it is ambitious, and hopefully it is reasonable. It gives a little time dependency, when signals are retimed. One might argue in the next six months would make it even more time-dependent, so it could still be a little better.

If we continue with this example, we ask ourselves what we need to measure/test in order to determine if our objective has been met? The obvious answer is travel time. The next question becomes, how are we going to measure travel time? We can manually collect the data, and this is probably the most common we use. Historically it has been the most common method of collecting data, but it has always been the most intensive: labor-intensive and time intensive. Analyzing the data can be very intensive. If it's manually collected, it needs to be put into something, a spreadsheet or whatever. There are lots of downsides to manually collecting data. You can automatically collect the data. New technologies at our disposal may allow us to do this more easily, and it is working. I keep mentioning the Bluetooth example for collecting travel time data that’s being used around the country in several different cities. You can use simulated data using simulation software. I'm not going to go into any details on particular software. They can give us estimates of travel times on corridors, but the results are only as good as the data that is input and the assumptions of the model algorithm. You can certainly do simulated data, but it may not be the most reliable for actual data. Finally, direct observation is a good way to get a quick feel of the data that represents reality.

Brian asked what travel time we’re talking about: through vehicles? Side streets? Pedestrians? This is the complexity of the real world. I was again being driver-centric and thinking of the

travel time from point A to point B, not necessarily link specific. If we do collect it automatically, people will drop off the system and vehicles will drop off the system. You have to be very specific: who's travel time are we talking about, you are right. We can be even more specific. In the case of more freeway operations, you have got more specific locations where people can enter and exit the system. TxDOT happens to be experimenting with this on I-35 as they do reconstruction there and are looking at collecting travel time data using Bluetooth that will feed back into a system that will tell people “it’s so many minutes of delay between here and there,” and that is displayed on DMS and mobile devices and things like that. I am being very open about travel time; it is not very specific.

We need to identify resources. Once we have identified the targets and data we are going to collect, are we using manual collection of license plates and time stamps? That is certainly the most time- and labor-intensive. It is very difficult to do. Employees driving routes, having them keep a log or something like that. It is not real-time, but it can help you identify historical trends. I think you can see the downside of that as well.

What about equipment? The collection of license plates and time stamps you can do automatically. You can have probe vehicles instrumented with GPS and Bluetooth readers at key intersections. You need to identify resources you have or you need. I'm going to skip over this slide because I think we already talked about it, and that is what are the advantages and disadvantages to that. We will come back to it if we need to, but I think we covered that already.

Test runs: these are manual data sources you can look at. Employees at the agency can perform test runs, and I have certainly done that. I had a GPS unit in my vehicle and made travel time runs to see what type of improvement I could make. The data is slightly skewed because I am not the average driver. I have a vested interest in the data improving and that might skew the way I drive. I might deliberately slow down if I know the signal is supposed to be turning green rather than go a couple of miles an hour over the speed limit. Test runs can be very helpful as long as you do it correctly and without bias. Citizen complaints were mentioned as an objective, and that is measurable. We can measure the number of complaints we get per signal, or the failure of complaints to keep pace with growth would indicate progress. If you are growing and you are getting more signal systems but your complaint levels are not going up, that is a good sign. The prevention of a sudden and sustained increase in complaints would indicate that the agency has not slipped in this measure. Employees or consultants can perform various traffic studies in locations. Make sure you are collecting it without bias. Trouble calls: several of you mentioned making sure detections are working, or 100% operation. Whether your trouble calls are triggered automatically by your system, we want to make sure we can track those and eventually that they are rare occurrences. Accidents are another good source of data, certainly for helping long-term trends. Historically, the accident time lag is what I'm looking for. We sometimes don't get the information for a year past when the data was collected. As we see more and more automated systems coming online, we are seeing the time lag reduced, and that is a good thing. Accident trends are very important to monitor the health of your traffic signal control system. Hopefully you get that data fairly often.

I already mentioned most of these for automated data sources: loop detectors and video detection. Video detectors are being used more because the costs have come down; they are finally comparable to loop detectors. Traffic signal controllers are becoming more able to collect different types of data. Probe vehicles are still used occasionally, and Bluetooth technology. In the future, connected vehicles could be used. As vehicles start talking to each other and the roadside, we're going to see more data coming in; sudden stops I see as s potential data collection. If you're getting the data back from connected vehicles that are saying 20% of the vehicles on this approach are decelerating very quickly, that might tell us we have an offset problem. They are expecting it to turn green and it's not, or something like that. Examples of V/C ratios, again, that doesn't mean much to an individual driver, but it can mean something from a system-wide view. There was some work in Arizona where they retro-fitted a signalized intersection with the goal of developing event-based performance measures while leveraging existing infrastructure to its fullest extent. In addition to video from the cameras, they put it interface cards to provide contact closure outputs of vehicle flow rates. These outputs were conveyed to an on-board event -based data locker within the traffic controller. They were doing fancy stuff inside the controller with this information, but they were able to get performance measures generated: volume to capacity ratio (V/C) and cumulative counts from the video-generated data. That was compared with manually counted data. That is also from the same Transportation Research Record 2192, so I would encourage you to get your hands on this, and it can have very practical advice.

Someone mentioned “in a broader sense, all ‘control’ data, independent variables that affect performance, so that effect can be related to cause, do you want to comment on that?” So as your independent variable changes, that can affect your performance more in a cycle, I guess is what you are saying. As one reacts to the other, it kind of feeds back and could possibly make things worse. As incidents go up, that is one measure, but as incidents go up, it can also cause problems on your system, possibly causing more incidents, or something like that. They are all related, and you need to be careful about causation-correlation. Just because something is correlated, doesn't mean there is causation. We need to be cognizant of that and not make the wrong conclusions.

Here is one on calculating queue lengths. This is a concept of operations out of the Denver Regional Council of Governments. This is a concept they used of detectors to calculate queue lengths. You've got advanced detector actuations. They measured arrival flow and profile of the current cycle length. You have stop bar detection down here for actuations with the signal phase change data to develop a discharge flow profile. The two profiles are combined to determine if there is an accumulation at the intersection. With the maximum queue length, this has been estimated once for each cycle. They are using video detection here and here, and they have got the advanced detection and the output detection, and again going into the controller and back to the TMC, they can make measurements about queue lengths with the objective to minimize queue lengths during peak periods.

Another example is collecting travel time using Bluetooth. For those of you that don't know much about Bluetooth, it is a short-range communication that is very prevalent these days, mostly in mobile devices. A lot of cars these days have Bluetooth communications in the car that can talk to your mobile device so technically you can do hands-free talking on your cell phone,

however I will say there's no research that shows hands-free cell phone talking is any safer than talking on a device you hold up to your ear. By having this Bluetooth technology in there, they give off an MAC address that is unique to that advice so these two can talk to each other. If we start back here, this cab can say that the device with this MAC address has just passed me, and it sends that information to the TMC. Later on, this yellow car goes by this signal, and the device in here sees that same MAC address and sends it to the traffic management center, and then they can estimate the travel time for this particular intersection to this one. That is how we are seeing some of this being done. It is pretty much cutting-edge right now; some jurisdictions are doing it, but not a lot.

We’re going to talk about some of the factors for success. You want to begin by focusing on a clear and present challenge faced by your agency. Use performance measures to describe the problem and provide evidence for the most appropriate solution. You want to focus on your challenges. Bring managers and employees along, building their capability to use and manage the data. Make sure you have everybody involved. If you want to expand, start small expand over time into the day-to-day processes. If you start with everything in the kitchen sink, you are probably destined to fail. Start small so that there’s an expectation that quality data will be used to make decisions and that agency staff will take ownership of their work. Train your agency managers and employees to focus on the needs of agency customers and to balance standard engineering and programmatic considerations against these needs so that the agency appears credible and capable to the public and legislative bodies Sustained these efforts over time by ensuring the program is not connected to a single individual or office within the agency. You want to keep it broad-based. Ensure broad distribution of performance data to your legislators or stakeholders, whoever that may be, and the public as well, building constituencies for the continued use of the performance management system at the agency.

One of the overarching goals of performance management is to increase transparency and accountability. This is done through reporting; collect and analyze all the data you want, but if you keep it to yourself, they’ll say you are not being transparent. Frequent public reporting can produce numerous positive results: building accountability, credibility and trust between your organization and its constituencies. It helps strengthen support for budget and program proposals. It promotes friendly competition and information sharing between offices that have experience different results. You may want to look at how you do that; not pitting employees against each other, you don't want that, but you want to have them spurred on. “They are doing that? We can do better than that.” Creating an expectation of continued reporting and incremental improvements can solidify the program you’ve implemented.

Here is an example of reporting done by VDOT. This is more related to pavement, but this is a dashboard that is available to the public (dashboard.virginiadot.org). It has performance, safety, condition, finance, projects, management, and citizen survey results. I encourage you to take a look at that.

Start thinking about questions you want to ask and we will take them. Here is a good example you might want to go to on your own time. This was done in 2008 by Denver Regional Council of Governments (DRCOG). This report was very collaborative: DRCOG had a steering committee, seven local agency representatives, FHWA, consultants. These are the key questions

they felt anybody should be asking if they’re going to undertake this endeavor. What types of data should be collected? How are you going to store this? What frequency? How will the information be disseminated? What information will be provided to the public? Some of the issues and challenges they came across included the capability to store recorded data for longer periods of time. If you are collecting video data, it is hugely intensive storage space. Simple volume or speed data is just numbers and won't be much of an issue. You want to maintain good communication to your field devices. Educate the public. All of these things they identified as particular challenges they had to go through. Here is their concept of operations, which is part of the engineering systems process. They developed a concept of operations (ConOps) from the uses and users to data processing to field devices, and then detection as well. That was their generic concept of operations.

I think we have covered all of the learning outcomes. You can look at them again. Feel free to chat in some questions. For more information, look for Transportation Research Record 2192. I'm trying to remember if it is available for free or if you have to buy it, but you can go to the TRB website, TRB.org, and look for that. A PDF may be available online.

Any questions as we finish up? I'm not seeing anything coming in the chat box. Jocelyn, is there anything you wanted to cover before we end the call?

Jocelyn BauerYes, I wanted to go over the NTOC organization.

On the first side, you see the member organizations of NTOC, or the National Transportation Operations Coalition. We encourage you to go to the NTOC website that will be listed on the following slide to find out more about these organizations. The NTOC website contains information about upcoming webcasts and contains a webcast archive webpage with the slides and recordings of previous Talking Operations webcasts, all the way back to about 2004. We will have the recording and slides from today's webinar up within a week. For those of you who registered, you will see that e-mail giving you notice of the availability of that recording and the slides. NTOC has two discussion forums, one focusing on high-level or strategic issues and the other focusing on ITS deployment and lessons learned. You can sign up on the website for the NTOC newsletter, which is mailed out by e-mail twice monthly.

That concludes all the information I have to share on NTOC.

Gary ThomasThank you all for your time and attention, and hope you have a great rest of the day.