A Guide to Clinical Interpretation of the Test of Variables of Attention (T.O.V.A.TM)

Steven J. Hughes, PhD, LP, ABPdN Director of Education and Research

The TOVA Company © 2008 The TOVA Company

Purpose: The purpose of this Guide to Clinical Interpretation is to help new T.O.V.A. users quickly come “up to speed” in interpreting the T.O.V.A. when used in evaluating attention problems, and to help veteran users get maximum value from the test by introducing the T.O.V.A. “Process Interpretation.” The T.O.V.A. Process Interpretation provides a roadmap through the T.O.V.A. report, and helps you get the maximum amount of information from the T.O.V.A. Background: The Test of Variables of Attention (T.O.V.A.) is the state-of-the-art continuous performance test (CPT) that provides highly accurate, reliable information about an individual’s sustained attention, speed and consistency of responding, and behavioral self-regulation – critical aspects of attention and executive functioning known to be compromised in persons diagnosed with Attention Deficit Hyperactivity Disorder (ADHD) and other conditions affecting the central nervous system. As a tool for routine clinical use, the T.O.V.A. helps to screen for and confirm the clinical diagnosis of ADHD and other attention disorders, quantify treatment effects, and monitor progress over time. It is also an outstanding psychophysiological tool for use in research on the CNS effects of disease, brain injury, medication, or other factors that impact attention and executive functions. A wealth of information is present in the T.O.V.A. report. For many users, more information is available than might readily be integrated into case management. However, while it is possible to base your interpretation on only one or two elements of the full T.O.V.A. report, the time taken to make a thorough review of results is well spent. The Process Interpretation consists of the following steps: 1) Review of the T.O.V.A. narrative interpretation; 2) Review of the ADHD Score; 3) Review of overall performance; 4) Comparison of performance in the infrequent and frequent target conditions (first half and second half, respectively); and 5) Review of performance changes across the four quarters of the task. You will also find some information that may, utilizing subtle differences in reaction time, tell you about the motivation of your subject to perform well. This is the subject of current research. Additional information about this work will be made available on The TOVA Company website as it becomes available (www.tovacompany.com). The T.O.V.A. Report Forms There are seven forms that comprise the full T.O.V.A. Clinical Report. While you don’t need to review all seven forms, you should be aware that if you are only reviewing one or two, there is additional information that you might want to consider in other parts of the report.

Each of the T.O.V.A. Clinical Report forms is described in detail in Chapter 2 of the T.O.V.A. Clinical Manual. You can select which forms to review or printout, and you may return to a previously scored T.O.V.A. report and print out other forms if you choose. To complete a T.O.V.A. “Process Interpretation” you should have the following forms at hand:

• T.O.V.A. Interpretation (Form 1) • T.O.V.A. Analyzed Data (Form 3) • T.O.V.A. Signal Detection Data (Form 4–we’ll just use the bottom information on the ADHD

Score) • T.O.V.A. Information and Results (Form 5)

To help you understand this process, we’ve included a case to help illustrate. The sample material is from a T.O.V.A. report for a 34 year old man with ADHD. Step 1: Review of The T.O.V.A. Interpretation Form 1 of the T.O.V.A. report provides demographic information about the current subject, a simple statement about whether the subject’s performance is within or outside of normal limits, and additional information about the validity of the T.O.V.A. performance. It is a simple, quick and clear interpretation of test results. If you only look at one portion of the T.O.V.A. Clinical Report, this is the form to examine. A detail of Form 1 can be seen in Figure 1. Under the heading “T.O.V.A. Interpretation” you will find two statements. The first statement will be whether the overall results are “suggestive of an attention problem, including ADHD/ADD,” or “within normal limits.” This interpretation is based on an empirically derived algorithm (described in greater detail in Chapter 2 of the T.O.V.A. Clinical Manual). The algorithm reviews general validity indicators (excessive omission or commission errors, excessive anticipatory responses, or user interrupts) and produces interpretations based on the profiles of standard scores for response time variability, response time, commissions and omissions. The second statement is whether the ADHD Score (detailed in Form 4) is “not within normal limits” or is “inconclusive.” Both methods (review of profile standard scores and the ADHD Score) of interpreting the T.O.V.A. are considered independently in making the general interpretation. The ADHD Score is based on the three profile standard scores that have been found to be the most discriminating between independently diagnosed ADHD cases and normal control subjects. The ADHD Score tells you whether the performance was like that obtained from most (but not all) persons diagnosed with ADHD. Remember that not everyone with the diagnosis of ADHD performs poorly on objective measures, so a normal-range ADHD Score does not rule out the diagnosis; hence, we call it “Inconclusive.” An ADHD Score that is “not within normal limits” helps to confirm the diagnosis of ADHD while an “Inconclusive” ADHD Score does not rule out the diagnosis. The person’s profile standard scores and the ADHD Score can be discrepant, and you should remember that the finding of normal ADHD Scores and/or normal profile standard scores does not necessarily rule out a diagnosis

of ADHD. Bright or unusually well motivated persons may be able to “keep it together” for the time necessary to complete a T.O.V.A. while demonstrating impairment in other settings. In addition, since some persons with ADHD have primarily auditory rather than visual information processing problems, the visual T.O.V.A. may be entirely within normal limits while the auditory T.O.V.A.-A. documents the disorder. Thus, a number of clinicians use both the T.O.V.A. and the T.O.V.A.-A. to assess their patients. Additional notes on Form 1 provide information about the validity of the test administration, and flag the presence of quarters with excessive commission, omission, or anticipatory errors. When one or more quarters are flagged as “invalid,” and the remaining valid quarters are within normal limits, the protocol will be interpreted as “within normal limits” based on the valid quarters. When this occurs, a statement is included indicating that the invalid quarter(s) may have been the result of an attention disorder, and that the interpretation should be viewed cautiously.

Figure 1. Detail of the T.O.V.A. report, Form 1: T.O.V.A. Interpretation, Additional Interpretation Notes, and

Validation Notes. Note that the presence of one or more invalid quarters does not mean that the T.O.V.A. is uninterpretable. However, the ADHD Score cannot be calculated, and the Form 1 algorithm may not be able to formulate an opinion regarding the presence of an attention problem. In such a case, you can review the

protocol quarter by quarter (see below). The diagnosis of any disorder must be made by a clinician based on history, symptom review, collateral information, and the results of objective testing such as the T.O.V.A., with appropriate consideration of potential comorbid or “masking” disorders (e.g., a mood disorder, a primary behavioral disorder, or a learning disability). A T.O.V.A. result that is “within normal limits” does not rule out a diagnosis of ADHD, particularly in the case of high ability, highly motivated individuals, or persons with auditory processing problems. Similarly, an interpretation that findings are "suggestive of an attention problem" should not be taken as certain proof that the person has ADHD. This statement indicates that the ADHD Score and/or profile standard scores are outside of normal limits and characteristic of the results seen in individuals independently diagnosed with ADHD. As noted above, the final diagnosis is to be made by a clinician after taking additional factors into consideration. After reviewing Form 1, you know whether the T.O.V.A. performance is valid, and whether you have evidence of the presence of an attention problem. If you are simply seeking confirmation of your clinical diagnosis of ADHD, or are utilizing the T.O.V.A. as part of a multi-method diagnostic procedure, this information may be sufficient for your purposes. However, as you’ll see in the rest of this guide, there is additional information in the T.O.V.A. Clinical Report that may be of clinical interest. Step 2: Review of the ADHD Score When Form 1 indicates the possible presence of an attention disorder and indicates no issues that compromise the validity of the T.O.V.A. performance, you can obtain additional information from other parts of the T.O.V.A. Clinical Report. An example of the relevant portion of Form 4 is shown in Figure 2.

Figure 2. Detail of T.O.V.A. Report, Form 4: The ADHD Score.

Your next step in the T.O.V.A. Process Interpretation is to find out why the subject’s performance was outside of normal limits. To do so, begin by turning to Form 4 to examine the ADHD Score. You already know if the ADHD Score is outside of normal limits (from reading Form 1), but now look at this index more closely. The ADHD Score is reported at the bottom of the page. It is derived from the sum of z-scores for Response

Time in Half 1, d’ in Half 2, and Response Time Variability (Total). Note that the ADHD Score is a SUM of three z-scores1 and is not itself a z-score. The three scores that comprise the ADHD Score were identified through an analysis comparing carefully diagnosed ADHD cases with normal cases. These three scores provided the best discrimination between the two groups, and the ADHD Score was constructed as the sum of these scores to combine their discriminating ability into a single index. This analysis showed that if the sum of these three scores was equal to or more negative than -1.80 (e.g.: -1.81), the protocol more closely matched the known-ADHD group than the “normal” group. The ADHD Score was developed as a tool to differentiate performance between ADHD cases and controls. It is a quick way of gauging the degree to which a T.O.V.A. report is similar to that of known-ADHD cases. An ADHD Score within normal range does not mean that the person does not have ADHD. An ADHD Score within normal range is “Inconclusive” and is not clinically relevant. As a single index, the ADHD Score is helpful, but there is much additional information in the profile standard scores, and it is best to consider both the ADHD Score and the profile standard scores. Consideration of both the ADHD Score and profile standard scores allows a more nuanced interpretation of the general results from the T.O.V.A. Note that the ADHD Score and profile standard scores may not be in agreement. You can, however, integrate both sources of information. Table 1. Integrating analysis of profile standard scores with the ADHD Score.

Profile Standard Score ADHD Score Interpretation Significant Standard Scores ≤ 80

Significant ADHD Score ≤ -1.80

The overall performance was significantly deviant from the norm and suggestive of an attention problem/disorder, and the ADHD Score was characteristic of that seen in individuals diagnosed with Attention Deficit, Hyperactivity Disorder.

Not significant Standard Scores > 80

Significant ADHD Score ≤ -1.80

Although the overall performance was not deviant from the norm, the ADHD Score was characteristic of that seen in individuals diagnosed with Attention Deficit, Hyperactivity Disorder.

Significant Standard Scores ≤ 80

Not Significant ADHD Score > -1.80

The overall performance was deviant from the norm and suggestive of an attention problem/disorder although the ADHD Score was not characteristic of that seen in individuals diagnosed with Attention Deficit, Hyperactivity Disorder.

Not significant Standard Scores > 80

Not Significant ADHD Score > -1.80

Neither the ADHD Score nor any profile standard scores were deviant from the norm. This performance does not suggest the presence of an attention disorder.

This can be done with the aid of the guidelines provided in Table 1. You can use the appropriate summary statement in your report. The first interpretation statement is appropriate in the example case since the profile standard scores and the ADHD Score were both significantly deviant from the norm. 1 The z-score statistic represents an individual score in units of standard deviation; [average score - observed score]/standard deviation. A z-score of 0 indicates mid-average performance--no deviation from average--while a z-score of say, -1.5 indicates a performance falling one-and-a-half standard deviations below average.

The overall performance was significantly deviant from the norm and suggestive of an attention problem/disorder, and the ADHD Score was characteristic of that seen in individuals diagnosed with Attention Deficit, Hyperactivity Disorder.

Having performed these steps, you now know the degree to which the subject’s performance is similar to that of known-ADHD cases, and have sense of his or her performance relative to the normative sample. Even more information is available if you review T.O.V.A. performance by condition, as shown in step 4. Step 3: Review of Profile Standard Scores For step three, examine the “Total” column at the right side of “T.O.V.A. Analyzed Data (Form 3)” (Figure 3.). Here you see a summary of the subject’s performance across the entire test, integrating results from the target infrequent (quarters 1 and 2) and frequent (quarters 3 and 4) conditions. From the ADHD Score, you know whether the performance is characteristic of that seen in persons diagnosed with ADHD. Here, you can see the aspects of their performance that were most discrepant from the normative sample.

Figure 3. Detail of T.O.V.A. Report Form 3: Analyzed Data.

The profile standard scores provide information about the subject’s response style, such as whether they tended to make impulsive errors (commission errors) or errors due to inattention or distraction (omission errors), and about the pace and consistency of their reaction time (response time and response time variability). In the example shown, you can see that the subject demonstrated his poorest performance on the RT Variability score (Total standard score = 65), and showed borderline performance on the Commission Errors score (Total standard score = 83). In an interpretive summary, you could write:

Performance was notable for significant inconsistency in his overall speed of responding, and

some tendency to make impulsive errors. Now you know if the T.O.V.A. results are characteristic of that seen in patients with ADHD, and have information about the patient’s performance and response style. By the way, you may have noticed that the standard scores for Half 1, Half 2, and Total are not the arithmetic mean of the relevant quarters. For example, look at commission errors in the example case. The standard scores in Quarters 1 and 2 are 85 and 88 (both in the normal range), but the standard score for Half 1 is 73 (significantly deviant from the norm). Why? Because the calculation of the standard scores for each half and the total are based on all the data for trials in that interval, and are not themselves calculated as a “mean of the means” by quarter. This is similar to the method used to calculate IQ scores.2 Step 4: Analysis by Condition The T.O.V.A. is comprised of two subtests, one presenting the target stimulus infrequently, the other, presenting frequent target stimuli. During the first half of the test (the infrequent target condition), stimuli appear at a ratio of 1 target to 3.5 non-targets. This ratio changes during the second half (frequent target condition), where the ratio is 3.5 targets per non-target. Subjects experience the two conditions quite differently. The first half is frankly quite boring while the second half can (for some people) be overwhelming. These two conditions were designed to provoke errors of omission (during the infrequent target condition) and errors of commission (during the frequent target condition). Review of differences in performance across the two conditions will tell you how the patient responded to these understimulating and overstimulating conditions. Review Half 1 and Half 2 data from the example report in Figure 3, “Detail of T.O.V.A. Report Form 3, Analyzed Data”. Beginning with the RT Variability scores in our example case, you can see that variability worsened from the significantly below average range in Half 1 (standard score = 77) to Half 2 (standard score = 58). Response Time slowed from Half 1 to Half 2 although both scores are in the normal range) while Commission Errors improved from significantly deviant from the norm to within the normal range. Thus, his response times were excessively variable (i.e.: some responses were too fast and some were too slow) throughout the test and worse in Half 2. However, his average time of responses slowed in Half 2 and his impulsive errors (commissions) significantly improved.

2 Index Scores are calculated from all the items that comprise the respective subtests, not from the subtest scores themselves–and like the Quarter, Half, and Total scores on the T.O.V.A., overall IQ scores generally do not equal the mean of subtest scaled scores.

This pattern of performance can be interpreted as follows:

Analysis of the subject’s responses by stimulus condition (infrequent versus frequent targets) shows that his ability to respond in a consistent manner was significantly deviant from the norm and worsened in the more stimulating frequent-target stimulus condition. However, he was able to slow down in the frequent-target stimulus condition, and his commission errors decreased and normalized. This decrease in response speed may be seen as an adaptive response that helped him improve the accuracy of his performance even though his consistency of response speed worsened.

You can provide an even more detailed analysis of infrequent versus frequent condition performance by reviewing change in performance over time in Step 5. Step 5: Review of Quarter by Quarter Performance A detailed view of quarter-by-quarter data from Form 3 is provided in Figure 4. By reviewing test performance at this level, you can obtain information about the subject’s response to novelty and stimulation as well as duration of attention and ability to adapt to changing task demands. For example, it is not uncommon for subjects to show normal or near normal performance in quarter 1 only to have their performance decline as the initial novelty of the task wears off. The unannounced onset of the frequent stimulus condition in quarter 3 can reintroduce novelty as well as stimulation, and performance may improve at least temporarily. Subsequent decline in performance in quarter 4 represents a fairly “classic” short-attention span, need-for-novelty profile of T.O.V.A. that commonly occurs in ADHD cases. Performance that improves in quarter 3 and remains good in quarter 4 may suggest a subject who performs best with continued higher levels of stimulation. There is some evidence that improvement from quarter 1 to quarter 2, followed by decline in quarter 3 and improved performance again in quarter 4 may be characteristic of persons who are anxious. Now review Figure 4. Some stability is seen in RT Variability from quarter 1 to quarter 2, followed by a significant decline in performance in quarter 3 and further decline in quarter 4. The subject’s ability to attend consistently declined once the frequent condition began, and never fully recovered. Examination of commission errors shows steady low average performance in quarters 1 and 2, with an abrupt worsening in the frequent target condition beginning in quarter 3 followed by recovery of performance in quarter 4. The improvement could be due to the increased stimulation of the frequent task, or that the subject was able to reduce his commission errors in quarter 4 by slowing down his response speed.

Figure 4. Quarter-by-quarter detail from T.O.V.A. Form 3: Analyzed Data. An interpretation based on this detailed analysis could look something like:

While stable during the infrequent target portion of the task, response time variability and commission errors increased after the onset of the frequent target condition, while response speed remained in the average range. As the frequent target condition continued into the fourth quarter of the test, the subject’s variability worsened. However, by slowing down in the fourth quarter, the commission errors normalized.

You may also be able to make some useful recommendations based on T.O.V.A. data. As examples, some patients perform better with task stimulation, and some do better with less. Some patients have difficulty sustaining attention after 5 or more minutes and might respond to refocusing interventions. Some have difficulty with abrupt transitions. As noted earlier in the example case, we find that the subject made commission errors across the first three quarters of the task, and didn’t manage to bring his impulsive responding under control (by slowing down) until the final quarter of the task. How might you use this information to underscore the observations you’d make regarding his approach to work-related tasks? Do you think showing the T.O.V.A. results to him might illustrate your point? At this point, you’ve made a thorough review of data from the T.O.V.A. report. You have some information about the likelihood of an ADHD diagnosis, you have a sense of your subject’s performance relative to the normative sample, and across condition and time. You also know about your subject’s adaptation to change and duration of attention. Some new work with the T.O.V.A. suggests that there is even more that you can learn by examining additional timing data on Form 5.

Step 6: Symptom Validity and Organization Recent analysis of T.O.V.A. reaction time data suggests that you can learn about other important aspects of a subject’s performance. Because of the use of advanced software and hardware technology, the T.O.V.A. is the only commercially available objective measure of attention that records reaction times with millisecond accuracy. This degree of precision permits the evaluation of subtle reaction time differences that have led to new (and unique) methods of assessing response validity. This work provides answers to basic questions such as: “Was the subject providing their best effort?” “Did they understand the nature of the task?” and “Were they able to meet the basic demands of the task?” This information is particularly helpful in making sense of T.O.V.A. results that are unusually deviant. As an obvious test of attention, the T.O.V.A. has strong face validity. For this reason, it is an easy target for “faking bad” in subjects who, for their own reasons, wish to represent themselves as more impaired than they may actually be. Should you trust that shockingly poor performance in the college student who has never been diagnosed with ADHD, but presents with the complaint that they “need something” to help them get through finals? What about a subject with significant affect regulation problems? Is their poor performance on the T.O.V.A. due to inability to follow directions, or manage the basic requirements of the task? Information to help you address these questions can be found on Form 5: Information and Results. A detail view from Form 5 is presented below in Figure 5. While a range of information is presented here, our interest now is in the “Response Time” values shown in the second line of the table, and “Post-Commissions” values presented in the ninth row (indicated by arrows). The Response Times are averaged reaction times (± 1 millisecond) for correct responses for each quarter, half, and for the task overall. The data in the row labeled “Post-Commissions” represent the averaged response times for the first correct response following a commission error. The average Post-Commission Error (PCE) response time is shown for each quarter, half, and for the task overall. In most T.O.V.A. reports, you will find that the average response time is consistently shorter than the average PCE response time. Most subjects are trying to do their best on the T.O.V.A., and feel a “cringe” when they make a commission error (“oops!”). When commission errors are made, most people seem to resolve to “be a little more careful” on subsequent trials, and this increased care is reflected in a slightly longer latency for the next response. This has been observed in both normal and ADHD samples. Further work will determine whether the increased PCE response latency is present in other groups. It appears that Response Times that are consistently faster than PCE response times is one marker of a “good” T.O.V.A.; the subject is responding in a manner that is consistent with good effort, and we have reason to believe that they are trying their best. T.O.V.A. reports from some groups do not show this difference. For example, early in the development of the T.O.V.A., it was observed that adolescents diagnosed with Conduct Disorder did not show this difference. Their

PCE response time was not different from their average response time or was faster (maybe they don’t cringe!). Another group that does not consistently show the increased PCE response latency is those who may be motivated to perform below their actually ability level–individuals seeking to “fake bad.” While the T.O.V.A. is no lie detector, failure to observe the expected difference in response times may be a sign that the subject is not feeling that “cringe” when making commission errors: those errors may be deliberate. Finally, there is some evidence that failure to observe consistent response time differences may indicate a subject who is having difficulty meeting the basic requirements of the T.O.V.A. PCE response times vary widely in individuals who have psychiatric issues that may impact their ability to comply with task requirements.

Figure 5. Detail of T.O.V.A. Report Form 5: Information and Results. Note “Response Time” and “Post-

Commissions.”

For example, it has been observed clinically that some children with Reactive Attachment Disorder produce mixed PCE response times that are both faster than the average reaction time, and dramatically slower. These wide-ranging values suggest unusual variability in compliance and/or ability to perform. Overall performance tends to be extremely poor when such a pattern is observed, which may be a marker of the subject’s general disturbance, due, at least in part, to some general decompensation during the task (and not necessarily a specific attention deficit).

You can see in the example case (Figure 5) that this subject made a total of 9 commission errors. In every quarter, the PCE response times are slower than his average response time for that quarter. In some cases, the difference is quite small (e.g.: 23 milliseconds in quarter 1), but it is always in the expected direction. We think this consistent, increased latency of PCE response times reflects the recurrent “cringe” and redoubling of effort following commission errors, and that this pattern suggests good effort and an attempt to produce his best performance. Reanalysis of research examining “faking bad” on the T.O.V.A. supports this interpretation; however, bear in mind that this is “probabilistic” information. Failure to see a consistent RT < PCE response time pattern does not necessarily indicate faking bad, conduct disorder, or affective decompensation; however, such a pattern appears to be more common in such cases. It is possible that examination of PCE response times may provide additional information about validity of responding during individual quarters of the T.O.V.A. However, at present we feel that if you do examine this response time difference that you limit interpretation to the values in the Total column. Also, keep in mind that there is a real limit to how much we can make of PCE response times when the subject makes only one or two commission errors. Finally, we want to emphasize that PCE response time analysis is still investigational and we do not recommend that you make clinical judgments about test validity based solely on this parameter. Further work examining PCE response time latencies will continue, and this work will be presented at professional conferences and in our workshops during 2008 and 2009. Future Directions in T.O.V.A. Interpretation We hope you find the material in this guide to be useful to you in your clinical work. We continue to support research on attention using the T.O.V.A. through grants from the T.O.VA. Education and Research Foundation, and some of that work will help us refine T.O.V.A. interpretation procedures in the future. We are doing some very exciting work for the next major release of the T.O.V.A. software. This will involve reformulation of the calcuation of the ADHD Score, simplification and visual enhancement of all aspects of the report, and additional changes to improve the clinical utility of the T.O.V.A. Don’t forget, in additional to free software updates, we are always ready to provide free clinical support and free technical support for T.O.V.A. users. In addition, the T.O.V.A. is available through an annual site license program to schools, and at a significantly reduced rate for approved training programs. Research support is also available through grants from the TOVA Research Foundation. Give us a call at 1-800-PAY-ATTN or visit wwwtovatest.com for more details. We deeply appreciate your use of the T.O.V.A., and are committed to helping you provide the very highest standard of clinical care. Please call us any time if you have questions, or if we can help you in any way.