Feedback on Feedback: CAMPO’s Findings from Testing Various Feedback ApproachesTRB Applications ConferenceMay 11, 2011Session 18B

Feedback on Feedback: CAMPO’s Findings from Testing Various Feedback Approaches

Kevin LancasterCapital Area Metropolitan Planning Organization

Jonathan AvnerWilbur Smith Associates

Karen LorenziniTexas Transportation Institute

Feedback on Feedback: CAMPO’s Findings from Testing Various Feedback Approaches• Why Feedback?• What Did We Test?• What Did We Find?• Where To Next?

The CAMPO Model• Capital Area Metropolitan Planning Organization• Five Counties Encompassing the Austin – Round

Rock, Texas Metropolitan Area• Auto, Truck, Fixed Route and Bus Transit,

Bicycles, and Pedestrians• Generalized Cost Assignment Including Tolled

Facilities• 1413 Internal, 49 External Traffic Analysis Zones• 16628 (2035), 14480 (2005) Links• 11575 (2035), 10443 (2005) Nodes

Why Feedback?

• Recommended by previous peer reviews

• Intuitively justified because inputs into earlier steps of the model could be inconsistent with the model outputs at later stages

Original CAMPO ProcessTraditional Four-Step Sequential Process

Trip Generation (TRIPCAL5)

Trip Distribution (ATOM2)

Highway Assignment

Assigned Volumes/Congested Speeds

Speeds are not an input to Trip Generation

For Trip Distribution, speeds representative of 24-hour free-flow are used as input.

Speeds representative of 24-hour free-flow are used as input for the first Traffic Assignment iteration.

Mode Choice

Transit Assignment

Ridership/Boardings

How Did We Approach Feedback?• We Need to Decide:

What gets fed back? What convergence criteria to use?

• How We Decided: Research literature Research State of Practice

(TMIP and other Texas MPOs)

Various Common ApproachesDifferent Possible

ApproachesOptions for What

Gets Fed BackTypical Convergence

Measures• Naïve (Direct)• Fictive Costs• Methods of

Successive Averages (MSA)

• Constant Weight Methods

•Link Time•Link Volumes

(converted to time)

•Trips•Skims

•Absolute or Percentage Differences

– Typically system-wide measures

•Total Misplaced Flows– Typically trip matrices or

link volumes•Root Mean Square Error

(RMSE)–Typically skims or trip

tables•GEH Statistic– Empirical formula typically

applied to link volumes

What CAMPO Tested – Feedback Approaches

Approach What Gets Fed Back

• Method of Successive Averages (Caliper’s MSA Implementation)

•Link Volumes Processed into Time Values

• Constant Weight Method - 50 – 50 - 70 – 30 - 80 – 20

•Trip Tables Processed Prior to Assignment

What CAMPOTested

1ST MODEL RUN "LOOP 1"

TRIP GENERATIONTRIP DISTRIBUTION

MODE CHOICEASSIGNMENT (24-Hr and AM Pk Per Highway)

if MSA Method:Apply MSA formula to link volumes, derive

"MSA-times" to feedback

Convergence or Maximum Loops?

No

Yes Finalize Model Reports

ASSIGNMENT (24-Hr and AM Pk Per Highway)

SUBSEQUENT LOOPS

TRIP DISTRIBUTION (all steps)MODE CHOICE (all steps except Mkt Seg)

TRIP TABLES (24-Hr and AM Pk Per Highway)

Update Time Fields for SkimmingFeedback

if Constant Weights Method:For each of the 24-hr and AM Pk TRIP TABLES,

create a new trip table based on the Constant Weight factor: CW * (current loop trips) + (1-CW) * (previous

loop trips)

Feedback

if Constant Weights Method:

Feedback times resulting from assignment

24-Hour / Non-Work Trip

Purposesif MSA Method:

Apply MSA formula to link volumes, derive

"MSA-times" to feedback

if Constant Weights Method:

Feedback times resulting from assignment

2-Hour Peak / Work Trip Purposes

MSA Method Formula

What CAMPO Tested – Convergence Criteria• Aggregate

Total number of trips• Matrix Level

Trip and skim table changes• Link Level

GEH statistic Maximum link flow change

Feedback Report

Measures for Convergence Criteria• Total Number of Trips

Absolute value, percent change• Trip and Skim Table Changes

Percent RMSE, Percent Total Misplaced Flow

• Link Level Total link flow change,

maximum link flow change,GEH statistic

GEH Statistic

• What is it? Empirically-based, not true statistic test

Typically applied to link volumes

Invented in the 1970s

What Did We Find?

• For All Approaches, the Measures of Convergence We Tested Tended toward Stability

• Some Converged Fasterthan Others

Daily / 24-Hour MetricsPercent Change

Total Trips

Trip Table Change - % RMSE

0

50

100

150

200

250

1 2 3 4 5 6 7 8 9 10

Perc

ent R

MSE

Iteration

MSACW 50%CW 75%CW 80%

Skim Table Change - % RMSE

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10

Perc

ent R

MSE

Iteration

MSACW 50%CW 75%CW 80%

Maximum Link Flow Difference

-0.014

-0.012

-0.01

-0.008

-0.006

-0.004

-0.002

0

0.002

0.004

1 2 3 4 5 6 7 8 9 10

Perc

ent

Iteration

MSACW 50%CW 75%CW 80%

0

5000

10000

15000

20000

25000

1 2 3 4 5 6 7 8 9 10

Max

Flo

w D

iffer

ence

Iteration

MSACW50CW75CW80

Daily / 24-Hour Metrics - GEH

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5 6 7 8 9 10

% o

f Lin

ks G

EH >

5

Iteration

MSACW50CW75CW80

2-Hour / Peak Period MetricsPercent Change

Total Trips

Trip Table Change - % RMSE

Skim Table Change - % RMSE

Maximum Link Flow Difference

-0.014

-0.012

-0.01

-0.008

-0.006

-0.004

-0.002

0

0.002

0.004

0.006

1 2 3 4 5 6 7 8 9 10

Perc

ent

Iteration

MSACW 50%CW 75%CW 80%

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10

Perc

ent R

MSE

Iteration

MSACW 50%CW 75%CW 80%

0

50

100

150

200

250

300

350

1 2 3 4 5 6 7 8 9 10

Perc

ent R

MSE

Iteration

MSACW 50%CW 75%CW 80% Not

evaluated for peak period

Skim Change – % RMSE24-Hour Versus Peak Period

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10

Perc

ent R

MSE

Iteration

MSA 24-Hour

MSA Pk-Hour

Decision MatrixConsideration MSA Constant Weights

Performance • No significant differenceMathematical Rationale

• Mathematically proven to converge

•Empirically-demonstrated performance

Implementation and Maintenance

• Supported in TransCAD GISDK

• Coded using GISDK, not explicitly supported

State of Practice

• Seems that MSA might have a slight edge in the modeling community discussions

CAMPO’sChosen FeedbackMethod

1ST MODEL RUN "LOOP 1"

TRIP GENERATION

TRIP DISTRIBUTION

MODE CHOICE

ASSIGNMENT (24-Hr and AM Pk Hr Highway)

Feedback MSA-derived

24-Hour Timesfor

Non-Work Trip Purposes

Convergence or Maximum Loops?

No

Yes Finalize Model ReportsFeedback

ASSIGNMENT (24-Hr and AM Pk Hr Highway)

SUBSEQUENT LOOPS

TRIP DISTRIBUTION (all steps)

MODE CHOICE (all steps except Mkt Seg)

TRIP TABLES (24-Hr and AM Pk Hr Highway)

Feed

back

Update Time Fields for Skimming

Feedback MSA-derived

2-Hour Peak Timesfor

Work Trip Purposes

Feedback

Feed

back

ConvergenceCriteria:% RMSE ofSkim< .1

Lessons Learned

• Opportunity to address other inconsistencies

• For testing, run many, many iterations• Be cognizant of assignment convergence

issues that affect feedback• Running mode choice for each iteration

was appropriate (and defensible)• Run time was a factor in our

decisions

Where To Next?

• For the 2005 Model, CAMPO Continues to Investigate Project- and Link-Level Implications of Modeling with Feedback

• CAMPO is Working Toward a Time Period Modeling Approach for its 2010 Model

• Long-term, Investigating Incorporating Accessibility into Trip Generation, andLooping Feedback to Trip Generation

Feedback on Feedback: CAMPO’s Findings from Testing Various Feedback Approaches

For further information, please contact:Kevin Lancaster, Capital Area Metropolitan Planning Organization512/974-2251kevin.lancaster@campotexas.org

Jonathan Avner, Wilbur Smith Associates512/592-3842javner@wilbursmith.com

Karen Lorenzini, Texas Transportation Institute512/467-0952k-lorenzini@ttimail.tamu.edu