John Gagnon | SearchLove San Diego, 'Trends and Techniques for Marketers and Data Geeks'
For Data Geeks…
description
Transcript of For Data Geeks…
![Page 1: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/1.jpg)
For Data Geeks…
![Page 2: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/2.jpg)
Connecting Assessment with Practice: Moving Information fromInteresting to Valuable
Darlena Jones, Ph.D.Director of Research and Development
Educational Benchmarking Inc
![Page 3: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/3.jpg)
Assessment to Practice: A Strategy
• Assessment Instruments– Defining the goals of an assessment project– Designing assessment instruments– Identifying the various key stakeholders
• Reporting Strategies– Comprehending the range of reporting methods– Linking effective reporting methods and key institutional stakeholders– Discussing how effective reporting can promote changes in
institutional practice
• Examples in Action– MAP-Works Student Reporting– MAP-Works Faculty/Staff Reporting
![Page 4: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/4.jpg)
Assessment Instruments
![Page 5: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/5.jpg)
Assessment Instruments
Step 1: Understanding the Outcomes of the Assessment
Step 3: Use of Assessment Information
Step 2: Breadth or Depth of Assessment
Step 4: Appropriate
Survey Items
Keys to Successful
Assessment Instruments
![Page 6: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/6.jpg)
Keys to a Successful Assessment Instrument
• Step 1: What are the outcomes of the assessment?– Program improvement?– Measuring the climate?– Measuring student learning?– Support accreditation / program review?
![Page 7: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/7.jpg)
Keys to a Successful Assessment Instrument
• Step 2: What is the breadth or depth of your assessment?– To thoroughly educate your audience on a range of items
(diagnostic assessment)? • Strengths and weaknesses• Participant needs• Areas of improvement• Longitudinal trends• Implications and suggestions
– To quickly provide “just the facts” (targeted to a specific topic)?• Summaries• Focus on Outcomes
![Page 8: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/8.jpg)
Keys to a Successful Assessment Instrument
• Step 3: What will your assessment inform?– Internal to Institution
• Decision making• Program evaluation and improvement• Budget allocations• Marketing and education• Interventions• Program development
– External to Institution• Accreditation reports• Grant reports• Benchmarking• Professional development• Publicity for alumni news, local media, etc.
![Page 9: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/9.jpg)
Keys to a Successful Assessment Instrument
• Step 4: What questions are on your instrument?– Potential Problem: Survey is long and confusing
• Solution: Create survey that is focused on:– Improvement, not marketing– Performance, not activity (measure outputs, not activity)– But, most important, MISSION
– Potential Problem: Survey design is bad; factors have low reliability; results display poor validity
• Solution: Have “survey experts” design survey or involve people closely connected to issue help design survey
– Potential Problem: Don’t know where to begin improvement once analysis is finished
• Solution: Must have both analyses/systems that help guide action planning
![Page 10: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/10.jpg)
Reporting Strategies
![Page 11: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/11.jpg)
Reporting Strategies
Step 1: Understanding the Audience
Step 3: Using Valuable Information
Step 2: Appropriate Form for Information
Step 4: Choosing the Best Delivery
Method
Keys to Successful Reporting
![Page 12: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/12.jpg)
Understanding the Audience• Step 1: Who will be reading your assessment report?
– Internal Stakeholders• Administrative decision-makers• Boards of trustees• Budget administrators• Faculty or staff• Internal governing bodies (Faculty/Staff Senate, Unions, etc.)• Students
– External Stakeholders• Accreditation bodies & reviewers• Alumni• Community members • Donors• Grant reviewers• Prospective students & parents• State and federal governments
![Page 13: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/13.jpg)
Understanding the Audience
• Step 1 (cnt): What is your audience’s experience with information?– Quantitative vs. Qualitative
• How comfortable are they with statistics?• Do they prefer narratives or numbers?
– Interest and experience - How much explanation regarding the…• Topic?• Assessment methods?• Results?• Implications?
– Time available - How much time can or will they spend reading the results?
– Level - Will they use university level data? College level? Department level? Individual level?
![Page 14: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/14.jpg)
Appropriate Form for Information
• Step 2: What is the best form for the information?– Types of reports
• Executive summaries• Comprehensive reports• Assessment summaries• Notes, brochures, flyers, and memos• Institutional snapshots• Interactive data
– Easily read?• What does the report look like (size of font, appearance, visuals,
etc.)?• Do they want to read this? Does it draw them in? Does it intimidate
them or overwhelm them?
![Page 15: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/15.jpg)
Using Valuable Information
• Step 3: What is the content of the report?– Importance: Does this report…
• Include important issues?• Highlight and emphasize the important results?• Differentiate between important and non-important results?
– Usefulness: Does this report…• Discuss the implications of the results?• Clearly link the results to practice?• Help practitioners determine what should be done?• Differentiate between useful results and interesting results?
![Page 16: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/16.jpg)
Choosing the Best Delivery Method
• Step 4: How will you deliver the assessment results?– Media formats
• Paper• Electronic (websites, downloadable files, CDs, emails, etc.)• Oral presentations• Combinations
– Easily accessible?• How hard is the information for Stakeholders to access or find?• Can they find what they need quickly?
![Page 17: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/17.jpg)
Examples in Action: MAP-Works Student Report
EBI’s “best practices” model of reporting
![Page 18: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/18.jpg)
Who is Responsible?• Who is responsible for student
success on your campus?– Enrollment Management/
Retention?– Student Affairs?– Academic Affairs?
• What information do you know about this first-year student?
Student ID: YD252952HS GPA: 3.93SAT Verbal: 29Location: In stateGender: FemaleRace: African AmericanAge: 18Major: Undecided
Do you really know them?
Enrollment Management/R
etentionStudent Affairs
Academic Affairs
![Page 19: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/19.jpg)
Paradigm Shift• What would happen if…
– ALL faculty/staff were responsible for student success?
– YOU knew student was struggling?
• Could you do something about it before it was too late?
Enrollment Management /
Retention
Residence Hall Staff
Academic Advisor
First-Year Seminar Instructor
Academic Department
Heads
Financial Aid
Minority Student Affairs
I’m really homesick
I don’t think I can afford college My roommate
and I argue all the time
Student AffairsAcademic
Affairs
I’m thinking about
transferring
I’m struggling in my math
class
![Page 20: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/20.jpg)
MAP-Works Mission – 4 WayAcademic Success: Improve students' ability to succeed academically by realigning behavior with grade expectations and focusing on elements of academic success
Retention: Minimize percentage of capable students who drop out due to issues that could have been addressed by self-awareness or timely intervention by staff/faculty
Student Development:
Facilitate the establishment of
relationships, address
homesickness, identify residence
hall living issuesStudent Involvement: Connect students with campus resources to facilitate involvement with student organizations and campus programming
![Page 21: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/21.jpg)
MAP-Works History
1988, Ball State developed concept
2005, Ball State partnered with EBI to create MAP-Works
1989 to 2004, Ball State used MAP in-house
Fall 2008 -approximatel
y 40 campuses
will use MAP-Works!
• In 1988, Ball State had a number of concerns…
– First-year students arrived with unrealistic expectations (academics, grades, housing, etc.)
– Retention rates were not as high as they wanted them to be
– Faculty and staff were concerned that identifying student issues at mid-term was too late
– Faculty and staff wanted better data about incoming students
![Page 22: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/22.jpg)
MAP-Works Process
•Expectations
•Behaviors
•Social Norming
•Expectations
•CampusResources
•Student Summary
•Scan Students
•Student Profile•Institution Profile•Campus
Resources
![Page 23: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/23.jpg)
Understanding the Student’s Experience
Survey and Profile Items
• Academic Integration– Academic Self-Efficacy– Core Academic Behaviors– Advanced Academic Behaviors– Commitment to Higher Education
• Self-Assessment– Communication Skills– Analytical Skills– Personal Management– Time Management– Health and Wellness– Potential Issues (stress, financial,
etc.)
• Profile Information– Student information like gender and
race/ethnicity– Entrance exam scores– # credit hours enrolled
• Social Integration– Campus Relationships– Living Environment (on/off campus)– Roommate Relationships– Homesickness
![Page 24: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/24.jpg)
• Who benefits from MAP-Works?– First-Year Students– Departments like…
• Housing & Residence Life• Academic Advising• Enrollment Management / Retention• First-Year Seminar Instructors• Minority Student Affairs• Athletic Department• Student Activities
– Academic Assessment & Institutional Research– Upper Administration
Who Benefits from MAP-Works?
![Page 25: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/25.jpg)
MAP-Works Student Reports
• Student Reporting– Individualized on-line report provided directly to students within days of
assessment
• 3 Main Reporting Purposes– Purpose 1: Realign expectations– Purpose 2: Information to help them plan for their success– Purpose 3: Connect with appropriate campus resources
![Page 26: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/26.jpg)
Purpose 1: Realign Expectations (Example 1)
Report provides benchmark information to help students realign
expectations
REPORTING TIP: Only provide the most valuable information in an summary report. Keep the “interesting” information for the larger report.
![Page 27: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/27.jpg)
Purpose 1: Realign Expectations (Example 2)
Report provides benchmark information to help students realign
expectations
REPORTING TIP: Know your audience! Consider using very simple charts to relay complex information (most students don’t have experience reading complex charts/tables).
![Page 28: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/28.jpg)
Purpose 2: Plan for Success
Report provides feedback to help student understand need to plan for their future success
REPORTING TIP: Consider providing written explanations to help the reader draw conclusions.
![Page 29: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/29.jpg)
Purpose 3: Connect with Resources
School lists campus resources/offices
that link with reporting area
REPORTING TIP: Consider providing additional information the reader may want to access.
![Page 30: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/30.jpg)
Evaluating MAP-Works Student Reporting
Step 1: Understanding the Audience – Students have less experience with information; reporting is easy to read
Step 3: Using Valuable Information - Targeted to helping them be successful in school
Step 2: Appropriate Form for Information –Extremely visual and colorful. Minimal information per slide.
Step 4: Choosing the Best Delivery
Method – Web based is a delivery
method most expected by this
generation
Keys to Successful Reporting
![Page 31: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/31.jpg)
Examples in Action: MAP-Works Faculty/Staff Report
![Page 32: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/32.jpg)
MAP-Works Faculty/Staff Reporting
• Interactive system for faculty and staff– Residence hall staff – Academic advisors– First-Year Seminar Instructors– Other staff positions?
• 3 Main Reporting Purposes– Purpose 1: Identify students who may benefit from personalized
attention – Purpose 2: Provide information for one-on-one meetings with students– Purpose 3: Provide input regarding programming and training needs
![Page 33: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/33.jpg)
Purpose 1: Identifying Students (Example 1)
These students are not adjusting academically or socially and are not committed to their institution.
REPORTING TIP: Consider creating high level “dashboards” that provides a lot of information in an easy-to-read format.
![Page 34: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/34.jpg)
Purpose 1: Identifying Students (Example 2)
These students very interested in leadership positions – Recruit them!
These students are moderately interested in a leadership position – Convince them?
These students definitely not interested in a leadership position – Why? Important on résumé
REPORTING TIP: Consider using simple color codes instead of statistical information to relay information.
![Page 35: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/35.jpg)
Purpose 2: Individual Meetings (Example 1)
Kimberly is having issues adjusting to college (both academically and socially) and early warning indicators are not good.
REPORTING TIP: Consider creating “dashboard” reports that give a 30,000 foot look at the data.
![Page 36: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/36.jpg)
Purpose 2: Individual Meetings (Example 2)
REPORTING TIP: Consider providing easy access to in-depth information if the reader needs more clarification.
![Page 37: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/37.jpg)
Purpose 3: Programming
These students have self rated themselves low in Public Speaking skills.
How could this be addressed through programming?
REPORTING TIP: Consider that the information you provide can be used in multiple ways and how that information can be presented to best tell the story.
![Page 38: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/38.jpg)
Evaluating MAP-Works Faculty/Staff Reporting
Step 1: Understanding the Audience – Reporting is color-coded for readability; No special training required
Step 3: Using Valuable Information – Highlights students with more critical issues; dashboards are focused on critical transition
issues
Step 2: Appropriate Form for Information –Extremely visual and colorful. Easy navigation.
Step 4: Choosing the Best Delivery
Method – Web based is a delivery
method that provides most
flexibility in searching through
information
Keys to Successful Reporting
![Page 39: For Data Geeks…](https://reader035.fdocuments.net/reader035/viewer/2022062222/56815d33550346895dcb2f64/html5/thumbnails/39.jpg)
Additional Questions and Discussion…
Darlena Jones, Ph.D.Director of Research and Development
Educational Benchmarking [email protected]
For more information about MAP-Works, contactTodd Pica, [email protected], visit www.MAP-Works.com