PRIME PROVIDER EVALUATION · Frequently Asked Questions (continued) 29 Shouldn’t TxDOT use a...

2017 PEPS Conference

PRIME PROVIDER EVALUATION2017 PEPS Conference

Thuy Luong, P.E.

November 3, 2017


Table of Contents

2

Purpose of Prime Provider Evaluations

Converting ESA score to Past Performance Score

ESA Score for Providers with No Past Performance Scores

Upcoming Changes to Prime Provider Evaluation

Frequently Asked Questions

11

33

44

55

66

Scoring System22


Purpose of Prime Provider Evaluations

3

• Have to –required by rule (43 TAC §9.41 (d))

• Want to – used as a multi-purpose tool

Why do provider evaluations?

• Management tool - use it to communicate our level of satisfaction

• Selection tool - past scores are used to compute the ESA, which is used during selection

How are they used?


Evaluations - All About Communication

4

At the beginning to identify what is expected

At the beginning to identify what is expected

At logical points to identify and

correct problems

At logical points to identify and

correct problems

At the end - the final evaluationAt the end - the final evaluation

TxDOT communicates on two levels:

TxDOT communicates:

1 - Is the job getting done? 2 - How well is it being done?


Evaluations - A Management Tool

The Prime Provider Evaluation Form:

– Provides performance feedback

– Improves contract management

– Improves contract product

It contains a list of criteria and outcome standards

– Evaluates the consultant PM’s skills and abilities as a project manager

– Does not evaluate consultant PM’s skills and abilities as a professional engineer, surveyor, or architect

5


Evaluations - A Selection Tool

TxDOT PM enters the evaluation into CCIS

Evaluation Score Average (ESA) is computed in CCIS

The ESA includes a component for:

– the consultant project manager

– the firm

The ESA is used during the selection process at either

– the SOQ evaluation stage, or

– the proposal/interview evaluation stage

Typically, past performance is weighted at 5-10% of the SOQ or interview score or proposal/interview scores

6


Scoring System

7

Project manager starts with 100 points

Firm starts with 50 points

For satisfactory performance on a criterion, the base score is given

For less than satisfactory performance on a criterion, points are deducted (e.g. -3, -6)


Scoring System (continued)

8

The PM/firm evaluations are divided into three groups:

The first group consists of the evaluations entered into the CCIS at any time from/including the current day to five years prior to the current day.The second group consists

of the evaluations entered into the CCIS at any time from/including five years prior to the current day to

ten years prior to the current day.

The third group consists of the evaluations entered into the CCIS at any time from/including

ten years prior to the current day and earlier.



9

The evaluation scores for Group 3 are ignored entirely. TxDOT is not interested in evaluations 10 years old and

older.

The evaluation scores for Group 2 are averaged. This number is then be averaged with 100.00

The evaluation scores for Group 1 are averaged. (Note: This number will not exceed 100.00)



10

The PM score is calculated as follows. •If there is an average evaluation score available

for Group 1, that number is divided by 12.50 to yield a PM score equal to or less than 8.00. Thus, an average score of 97.00 would yield a PM score of 7.76.

•If there is no average evaluation score available for Group 1, the effective average evaluation score for Group 2 is used. As above, this number is divided by 12.50 to yield the PM score.



11

The firm score is calculated as follows.•If there is an average evaluation score available

for Group 1, that number is divided by 25.00 to yield a firm score equal to or less than 2.00. Thus, an average score of 49.29 would yield a firm score of 1.97.

•If there is no average evaluation score available for Group 1, the effective average evaluation score for Group 2 is used. As above, this number is divided by 25.00.


ESA Procedure

12

For a given PM and prime

provider firm, PEPS obtains

the ESA in CCIS

The ESA for each provider is

entered into the scoring

spreadsheet during the selection process

Documentation -An evaluation

report, listing all of the

evaluations used to compute the

ESA is maintained in selection file


CCIS Software Application

13


PDF lists evaluations used to compute Firm and PM scores

PEPS files the PDF to Selection File

14


Converting ESA score to Past Performance Score

ESA Score Past Performance Score

= 10 5

9.52 < ESA < 10 4

9.04 < ESA <9.52 3

8.56 < ESA <9.04 2

8.56 < ESA 1

15


Example of Scoring Spreadsheet

16

Past performance is included as a criterion at the SOQ stage, Interview, or Proposal/Interview stage


Example of Scoring Spreadsheet (continued)

17

Example of Past performance used at the Proposal/Interview stage:


ESA Score for Providers with No Past Performance Scores

For Project Managers (PM) and Firms with no previous TxDOT experience or no evaluations on files, TxDOT uses an average score. This average is calculated using data from the past 5 years. Summary of the data points used:

18

Number of Data Points Used

Score ESA

PM 5 YR Average 752 98.09 7.84

Firm 5 YR Average 770 49.39 1.97

Total (perfect score 150) 147.48 9.81


How is the 5-year Average Score Used?

19

Evaluations available

ESA Evaluations available



ESA

PM 8* 8* X 7.84 X 7.84

Firm 2* X 1.97 2* X 1.97

10* 9.97 9.84 9.81

P.P. Score 5 4 4 4

* Assuming perfect score for Past TxDOT Experience


Upcoming Changes to Prime Provider Evaluation

20

•TxDOT PM completes evaluation in PS-CAMS•TxDOT PM’s supervisor reviews & approves

evaluation •Prime provider PM reviews, comments, and dates

evaluation using PS-CAMS Consultant Portal

New system will be in PS-CAMS

New system will be ready for use in Spring 2018

Scores for each criteria will be from 1 to 5


Upcoming Changes to Prime Provider Evaluation (continued)

21

PM Evaluation Criteria

1.a Quality of Deliverables: Accuracy

1.b Quality of Deliverables: Completeness

2 Timeliness of submittals

3.a Contract administration - Budget

3.b Contract administration - Schedule

3.c Contract administration - Invoices

4 Responsiveness and availability of the PM

5 Resolution of issues

6 Communication and coordination

7 Management of subproviders

8 Adequate use and prompt payment of HUB/DBE firms

9 PM performance

Firm Evaluation Criteria

1 Firm Responsiveness

2 Firm Invoicing

3 Firm Resource Management


Frequently Asked Questions

What should I do if the TxDOT PM does not evaluate me?

Prime provider evaluation is required:

o Per TAC (43 TAC §9.41 (d))

o Per TxDOT Chief Engineer’s Memo dated April 20, 2017: “Each TxDOT project manager shall complete an end of project evaluation on each consultant contract that they were managing currently. Each TxDOT project manager shall also complete an interim evaluation anytime that a consultant is not performing to expected standards for the Department.”

o PEPS Division assists in this effort by providing reminder email notices to District and Division project managers when contracts are due for an annual evaluations and when contracts are due for end of project evaluation.

22


Frequently Asked Questions (continued)

23

Can the TxDOT PM share the evaluation results with the provider PM? That way it

could be a lessons learned or opportunity to do better.

Yes. Once the TxDOT PM has completed the provider

evaluation, he/she must share results with the

provider PM and give the provider PM an opportunity to review and comment on

the results.



24

If I don’t agree with the TxDOT PM on the results,

can I not sign the evaluation?

If you don’t agree with the results, you should

discuss with the TxDOT PM and if the TxDOT PM

doesn’t change the scores, you don’t have to

sign the evaluation.



25

If I don’t sign the evaluation, will the

scores be included in the TxDOT database?

Yes. The evaluation will be input in the database and the

scores will be included in the ESA

calculation.



26

As a provider PM, if I disagree with the TxDOT

PM’s evaluation, how do I protest/dispute the

scores?

First, you should discuss with the TxDOT PM to get the issue resolved. If it

doesn’t get resolved, you may request a meeting

with the TxDOT PM’s supervisor to discuss.



27

Why doesn’t TxDOT use the extra points (i.e. +3, +6) when calculating the

ESA?

Extra points are not included because the purpose of the ESA is to identify less-than-

satisfactory performance.



28

When TxDOT makes a lot of changes to a project, what must consultants do to get above base

scores on meeting deadlines and staying

within budget?

The consultant should put forth extra effort

to submit quality deliverables and still

meet project deadlines.



29

Shouldn’t TxDOT use a small percentage of the

prime provider evaluation score in the selection process since firms and provider PMs

change?

Past performance is weighted at 5-10% during the selection process.

The PM has his/her own evaluation scores and the firm has its own scores. When the PM goes to a

different firm, the PM’s evaluation score goes with him/her.

2017 PEPS Conference 30

Questions and Discussion

2017 PEPS Conference 31

Thuy LuongTxDOT| PEPS Center of Excellence(713) [email protected]

Thuy LuongTxDOT| PEPS Center of Excellence(713) [email protected]

PRIME PROVIDER EVALUATION · Frequently Asked Questions (continued) 29 Shouldn’t TxDOT use a...

Documents

Transcript of PRIME PROVIDER EVALUATION · Frequently Asked Questions (continued) 29 Shouldn’t TxDOT use a...