CONSULTANT DEBRIEFS - Texas A&M University · 2017 PEPS Conference CONSULTANT DEBRIEFS 2017 PEPS...
Transcript of CONSULTANT DEBRIEFS - Texas A&M University · 2017 PEPS Conference CONSULTANT DEBRIEFS 2017 PEPS...
2017 PEPS Conference
CONSULTANT DEBRIEFS2017 PEPS Conference
Gail Eileen Morea, P.E.│ Norma Glasscock, P.E.
November 3, 2017
2017 PEPS Conference
Table of Contents
Objectives
Consultant Debriefs and What They Should Cover
Types of Debriefs: Individual and Group
The Rules of Engagement for Consultant Debriefs
Tips for Successful SOQ
Tips for Successful Interview
Questions and Discussion
11
22
33
44
55
66
77
2017 PEPS Conference
Objectives
Define consultant
debriefs
Discuss when
debriefs should be scheduled
Discuss debrief
outcomes
2017 PEPS Conference
Objectives
Definition of a consultant debrief
– The opportunity to share information about a specific procurement in a planned and controlled, formal environment.
– Essential to the continuous learning process
When should a debrief be scheduled?
– Disqualified firms are notified at the screening stage via an e-mail identifying the missing or incomplete items that led to their elimination from further evaluation
– Processes without interviews, once selections are made
– Processes with interviews, after the shortlist is announced
– May be held up to four months after contract execution but the goal is the earlier the better
2017 PEPS Conference
Objectives (continued)
Outcome of a debrief
– Factual information about the performance of the provider
– Demonstrates that a thorough, fair evaluation based on sound decisions took place in accordance with the selection methodology developed prior to advertisement.
– Open and honest feedback to assist the provider in improving their future submissions in response to new procurements.
– Identifies strengths and how to learn from mistakes going forward.
2017 PEPS Conference
Consultant Debriefs - What Should They Cover?
Reference Solicitation Number, Discipline and the Selection Process followed
Overall high score, low score, the provider’s overall score and their ranking within the solicitation pool itself
The average score for each question in the SOQ and/or Interview
The firm’s individual score by question
Stress the importance of the weighting factor for each question
Identify questions that warranted more information
2017 PEPS Conference
Consultant Debriefs - What Should They Cover?
Was the question answered thoroughly or were the responses too broad brush?
Lessons learned by handling similar projects
Implemented innovative techniques
How was the role of the Project Manager described in the response to a question? Was it one of managing the project, budget, and schedule along with demonstrating their problem solving abilities or was it more of a design role?
2017 PEPS Conference
Consultant Debriefs - What Should They Cover?
o Reference that interview time should be used wisely.
o The interview gives the CST the opportunity to consider the team dynamics and interaction.
o Avoid disclosing what was looked for on a point by point basis when addressing responses.
2017 PEPS Conference
Consultant Debriefs
How is past performance evaluated?
Adjusted Evaluation Score Average (ESA) based on past performance on TxDOT contracts (5-year window) using a 10-point system
Full credit is no longer given to Project Managers without past performance scores in the TxDOT database
The CCIS Software application provides the ESA and it is then adjusted to a 5-point score based on the 5-year average of all evaluations in the database.
Minimizes potential bias towards firms with TxDOT experience but does not give the PMs lacking TxDOT references an unfair advantage by giving them a perfect score.
2017 PEPS Conference
Types of Debriefs
Individual (face to face)
Group
Conference call Webex
Debrief can be held as:
Preferred methods: Individual or Group DebriefDuration: 30 minutes to 1 hour
2017 PEPS Conference
Consultant Debriefs - Individual Debrief Preparation
Review the team’s SOQ and CST members score sheets
Request input from the CST if you do not have their notes
Review their screening checklist and identify potential issues that did not result in a disqualification but could provide relevant information
Review the top team or teams’ information and take notes to serve as a comparison.
Comparison is generic with no specific details about another team’sresponses given
Information should be detailed enough to be meaningful and helpful Focus on highs, lows, and patterns General expectations of each question Areas for improvement Trends exhibited by the CST that impacted selection
2017 PEPS Conference
Consultant Debriefs - Individual Debrief Meeting
Provider should have a copy of their SOQ available at the meeting
Delivered by either the Procurement Engineer or the Service Center Manager
For in-person individual debriefs, only notes taken to prepare for the debrief are allowed in the meeting. No actual selection documents or files are to be brought to the meeting.
Remember that every CST is different
Maintain professional conduct during the debrief
It is not a time to debate the score!
Purpose is to help improve future responses
2017 PEPS Conference
Consultant Debriefs - Individual Debrief - Example
Yellow Rose Consulting was not selected for a contract and requested a debrief. They ranked 36 out of 72 firms.
To prepare for the debrief the Procurement Engineer or Service Center Manager read and evaluated the top 2 or 3 ranked submittals for comparison.
Findings:
All 3 were well organized and had clear responses
All had headings that followed the main points of the questions
Contained thoughtful use of bullets and dialogue that provided clarity to the response
Provided depth to responses that related to the added value scoring
De-briefer scoring correlated with outcome
2017 PEPS Conference
Consultant Debriefs - Individual Debrief - Example
De-briefer then read and evaluated the response of the requesting consultant
Found blocks of dialogue without breaks between paragraphs –difficult to follow direction of response
Each question started with a large paragraph of marketing information
Did introduce members of the team through the response, explaining the team expertise related to the question
2017 PEPS Conference
Consultant Debriefs - Individual Debrief - Example
De-briefer will provide information on average scores per question and the scores of the requesting consultant
Question Q1 Q2 Q3 Q4 Q5
Percentage Wt. 10% 17% 13% 20% 25%
Average score 3.05 3.09 3.26 3.13 3.43
Yellow Rose Consulting 3.33 3.00 3.67 3.00 3.67
De-briefer will:
– Address any items in the screening that may have been noted
– Provide examples of what constitutes “meets” and “exceeds”
– Provide time to address questions from the consultant
2017 PEPS Conference
Consultant Debriefs - Individual Debrief - Example
Trends that lead to success for the top 3 firms
Immediately started answering the question
Did not have introductory marketing paragraphs (Our firm has 30 years of experience….), though there may be a sentence or two provided in information on expertise related to the topic of the question
All responses above average
Goal: Help the consultant prepare for afuture with successful SOQ responses.
2017 PEPS Conference
Consultant Debriefs - Group Debrief
– Triggered by the number of SOQ respondents
– The presenter is the Service Center Manager
– The Group Debrief focuses on the responses of the top scoring firms
– Procurement Engineer and/or Service Center Manager should prepare their own notes. If the Consultant Selection Team is not participating in the debrief, request their input
– Information that may be covered by a group debrief
Focus on the highs, lows, and patternsGeneral expectations of the questionsAreas for improvementsTrends that impacted the selection
2017 PEPS Conference
Consultant Debriefs - Group Debrief
Additional Information to be shared in a Group Debrief:
Number of SOQs received
Number of disqualified firms (if any), and general information about why these firms were disqualified
Range of total scores
Rounded average score by question
Summary of each question, the applicable weighting and examples of what constitutes “meets and exceeds” requirements
The end result……High performing teams lead to more competition!
2017 PEPS Conference
Consultant Debriefs - Group Debrief
Can I obtain our team’s scores by question and overall total
score?
Yes, simply e-mail the presenter of the Group Debrief
and request the information
Does a group debrief mean that I am unable to request an
individual debrief?
No, an individual debrief may be requested; however, it will
be scheduled after thegroup debrief
The goal of a group debrief is to provide as much information to all respondents regarding the procurement expectations in a timely manner.
For procurements where there are 50, 60 or even 90 respondents it is not feasible to schedule an individual debrief in a timely manner due to the focus on negotiations for the current procurement. A group debrief enables the timely sharing of information
2017 PEPS Conference
Rules of Engagement for Consultant Debriefs
The provider may state that their competitor used the same sub who responded to the question with the same response. Though the same sub may be used by both providers this does not mean the same response is in actuality used without modifications.
If a provider asks how a competitor did, the de-briefer should respond that another firm’s information to include SOQs, proposals, interviews, scores or rankings is confidential and is not shared with others.
2017 PEPS Conference
Rules of Engagement for Consultant Debriefs
• Come prepared to the meeting with your SOQ • Be courteous and professional• Be respectful of others speaking• Be respectful of the time
2017 PEPS Conference
Tips for a Successful SOQ
Read the RFQ thoroughly!
Prior to the SOQ submittal, take advantage of the question period
Watch for the addenda
Schedule the SOQ development plan to allow time to QA/QC your SOQ before submittal
Verify:
•Your administrative qualification status, and those of the sub-providers for the Federal process
•Precertification status of each task leader on a standard work category•If the PTC form cell turns red, investigate•Person’s name matches the name shown in CCIS•Firm names are the actual legal firm names shown in PS-CAMS
2017 PEPS Conference
Tips for a Successful SOQ (continued)
Address the NLCs completely
•Credentials•Beginning dates, month and year, as well as ending dates, month and year•Direct response for the exact items requested in the minimum requirements
Answer the questions being asked
Be clear, concise, specific, and technical
Don’t waste space marketing
Consider innovative techniques, challenges, and lessons learned
2017 PEPS Conference
Tips for a Successful Interview and RFP/Interview
Read the ICG or RFP/ICG (Federal process) thoroughly
Focus on
The proposal format and requirements (Federal process)
The interview (and proposal, if Federal) topics as well as the evaluation criteria and weighting
Technical expertise Only! – marketing material will not be evaluated
Anticipate follow-up questions from the CST. These questions clarify and expand on the topics
Expect tough competition
Show team’s leadership between the PM and the task leads
2017 PEPS Conference
Contact Information
Gail Eileen Morea, P.E.TxDOT | PEPS Houston Service Center Director(713) [email protected]
Norma Glasscock, P.E.TxDOT| PEPS Fort Worth Service Center Manager(817 )[email protected]