MAT-PDOAProgram Evaluation

Diana Seybolt, Ph.D.Karen McNamara, Ph.D.

Systems Evaluation Center (SEC)

Program Evaluation

A systematic method for collecting, analyzing, and using information to answer questions about projects, policies and

programs, particularly about their effectiveness and efficiency.

Purpose of MAT-PDOA Program Evaluation

• Track progress on MAT PDOA goals and objectives• Identify potential disparities in sub-population

access, utilization, and outcomes • Inform ongoing implementation efforts at the state

and local levels and identify areas requiring adjustments

• Provide information for required reporting to SAMHSA regarding program implementation efforts, challenges encountered, and strategies implemented

Baltimore City MAT-PDOA ObjectivesObjectives

1. Increase the number of individuals evaluated for MAT by recruiting opioid overdose survivors into SRD treatment

2. Increase the number of opioid overdose survivors receiving MAT

3. Increase length of stay in MAT by 40%; Reduce patients leaving MAT against medical advice by 20%

4. Decrease illicit drug use by those in MAT treatment

5. Increase number of residents in 3.7 treatment undergoing buprenorphine induction treatment

6. Increase referrals from 3.7 treatment to MAT programs

7. Prevent opioid-related deaths

Baltimore City MAT-PDOA ObjectivesObjectives Evaluation Activities

1. Increase the number of individuals evaluated for MAT by recruiting opioid overdose survivors into SRD treatment

Survivor Program Contact and Referral Data, Stakeholder Interviews

2. Increase the number of opioid overdose survivors receiving MAT

Claims/Service Utilization Data, Contact and Referral Data, Stakeholder Interviews, Survivor Program Contact and Referral Data

3. Increase length of stay in MAT by 40%; Reduce patients leaving MAT against medical advice by 20%

Claims/Service Utilization Data, Stakeholder Interviews

4. Decrease illicit drug use by those in MAT treatment

GPRA Questionnaire (Section B, alcohol and drug use), Stakeholder Interviews, Treatment Progress Data

5. Increase number of residents in 3.7 treatment undergoing buprenorphine induction treatment

Contact and Referral Data, Claims/Service Utilization Data, Stakeholder Interviews

6. Increase referrals from 3.7 treatment to MAT programs

Contact and Referral Data

7. Reduce opioid-related deaths

Office of Chief Medical Examiner Data

Additional Outcome and Process EvaluationObjectives Evaluation ActivitiesProgress, Barriers, Efforts to Overcome Barriers

All of the above

Outcome Questions: intervention effects, factors associated with outcomes, effect durability, 6-month follow up.

GPRA Questionnaire (Section B, Alcohol and Drug Use), Stakeholder Interviews, Treatment Progress Data

Process Questions: implementation, changes made, changes to address disparity, service provision, # reached through project

Stakeholder Interviews, Contact and Referral Data, Claims/Service Utilization Data

Evaluation Guidelines

• Perform a thorough evaluation that provides useful information to our grantor as well as to program decision-makers

• Minimize intrusion on staff time• Build upon your existing procedures• Develop methods of data collection that

provide useful information to individuals “on the ground”

Evaluation Activities

• Contact and Referral Data• Treatment Progress Data• Stakeholder Interviews• Claims/Service Utilization/OCME data• Government Performance and Results Act

(GPRA) Questionnaire

Contact and Referral DataWho• State Care Coordinators and Care Coordinators at MAT programs• OSOP/Other Outreach Peers • Administrative Data

WhatCounts of the following, including demographic information, to understand client flow and assess for potential disparities• clients inducted for buprenorphine• clients referred from induction to MAT services• clients with opioid overdoses receiving peer outreach• clients referred to Care Coordinator for MAT Where• Total Health Care• Reflective Treatment Center• Daybreak Rehabilitation Center• Tuerk House• Other Locations (for peers and outreach)

When• Collected monthly, reported quarterly

How• Contact Logs, Clinic Data Collection Procedures

Treatment Progress Data• This evaluation piece is under development.• General idea is to capture basic information about client

progress, such as number of clients making progress on treatment goals per month.

• Will be developed in consultation with MAT-PDOA community partners.

• We will also be able to capture more traditional outcomes data through the GPRA questionnaires, and perhaps eventually the BHA Outcomes Measurement System (OMS).

Stakeholder InterviewsWho• Clients• Peers (MARS, outreach)• Care Coordinators• MAT program staff• County and State Administrators (grant leadership)

What:Group interviews with each stakeholder group using a semi-structured interview (currently in development)

Where• Locations convenient to stakeholder groups (to be determined)

When• Approximately every 12 months on a rolling basis

How• Interviews conducted and summarized by the UMB Systems Evaluation Center

Claims/Service Utilization/OCME dataWho• Treatment Providers will enter claims and authorization data into the ASO system (ValueOptions®)

as usual business practiceWhatSpecific variables to analyze currently being identified.

Where• Data files provided by ValueOptions® to BHA• Office of the Chief Medical Examiner (OCME) currently provides monthly data to BHA regarding

Maryland deaths.

When• Extracted and analyzed monthly, once available

How• Data to be analyzed by UMB Systems Evaluation Center

Government Performance and Results Act (GPRA) Questionnaire

Who• State Care Coordinators conduct interviews and enter data

What:a client interview/performance measure mandated by SAMHSA

Where• Healthcare Access Maryland enters information into SAMSHA database

When• Enrollment, 6 months, and Discharge

How• Data made available by SAMSHA will be analyzed by UMB Systems Evaluation Center staff

What we’ll need from you

Everyone• Meet with evaluators so that we can understand

your workflow, data availability, etc.• Provide feedback regarding program evaluation

workflow and evaluation tools as developed• Assist in arranging annual group interviews for

staff and/or clients (inviting participants, possibly arranging space, etc.)

• Participate in annual group interviews

What we’ll need from youMAT Programs• Collect Contact and Referral Data and submit to SEC monthly • Collect Treatment Progress Data and submit to SEC monthlyTuerk House• Collect Contact and Referral Data and submit to SEC monthly • Collect Treatment Progress Data (possibly) and submit to SEC monthlyState Care Coordinators• Conduct GPRA Interviews and enter dataProgram Care Coordinators and Other Peer Outreach• Collect Contact and Referral Data and submit to SEC monthly MARS • Collect evaluation information to be determined (e.g. Total meetings

per month, total number of attendees)

What we will produce• Quarterly Summary Reports - The SEC will create

summary reports integrating these various sources of data and provide them to the Project Director and local administrators at least quarterly for grant management.

• Group Interview Summaries - As stakeholder interviews are completed, the participating group will also receive a brief report summarizing the results.

Questions?

Karen McNamarakmcnamar@psych.umaryland.edu410-646-0049

Di Seyboltdseybolt@psych.umaryland.edu410-646-1756