ASSESSEMENT METHODS AND ACTIONABLE DATA

Richard Mahon, Riverside City College, facilitator

Lesley Kawaguchi, Santa Monica College Maggie Taylor, Fresno City College Linda Umbdenstock, member, ASCCC-RP SLO

Collaborative

AUTHENTIC ASSESSMENT

“. . . [A]uthentic assessment simulates a real world experience by evaluating the student’s ability to apply critical thinking and knowledge or to perform tasks that may approximate those found in the work place or other venues outside of the classroom setting.” SLO Terminology Glossary: A Resource for Local Senates, pp. 2-3

SAMPLE STUDENT LEARNING OUTCOME:

History: “Demonstrate the ability to interpret historical information by applying analytical skills used by historians – such as synthesizing evidence from both primary and secondary sources, comparing and contrasting multiple perspectives, contextualizing information, and/or identifying causes and effects of change and continuity – to the course content.”

POSSIBLE WAYS TO ASSESS:

Essays Research Papers Midterm or Final Exams Oral Discussion and/or Presentation

RUBRIC DEVELOPED BY THE FACULTY

REGISTERED NURSING SLOS

Fundamentals of Nursing Skills Lab SLO: “Provide safe, comprehensive and proficient

care for the adult client in a simulated environment.”

Each procedure has a rubric. At end of course, a patient scenario is given which

involves a set of procedures. Performance of procedure, including rationale Documentation

Core RN faculty discussed the rubrics, the scenarios, and use the results to make any changes.

ENGL 252: WRITING IMPROVEMENT

Assigned English faculty met several times, developed rubric, and exam.

Held several norming sessions to assist in evaluating papers.

When finished, had an overall critique of the exam.

Results will be used in the next semester.

IDENTIFYING ACCESSIBLE EXTERNAL SOURCES OF DATA

• Chancellor’s Datamarthttp://www.cccco.edu/ChancellorsOffice/Divisions/TechResearchInfo/MIS/DataMartandReports/tabid/282/Default.aspx• NCES National Center for Education

Statisticshttp://nces.ed.gov/• CPEC California Postsecondary

Commissionhttp://www.cpec.ca.gov/• CalPASS - http://www.cal-pass.org/

IDENTIFYING INTERNAL DATA SOURCES

STUDENTS PROGRESS IN A NON-LINEAR FASHION

DEFINING ACTIONABLE DATA – USING DATA INTERACTIVELY

Data collection does not equate to action or improvement.

Even the most valid and reliable data are not a substitute for action and will not by themselves motivate action.

Actionable data provide information that leads to improved practice.

Actionable data result from “action-oriented” research.

ACTIONABLE DATA? ARCC DATA Statewide Rates

Indicator 2008 2009 2010 Student Progress & Achievement 51.2% 51.8% 52.3% Completed 30 or more units 70.4% 71.2% 72.4% Fall to Fall Persistence 68.3% 69.2% 68.7% Voc Ed Course Completion 78.2% 77.7% 77.6% Basic Skills Course Completion 60.5% 60.5% 61.5% Basic Skills Course Improvement 50.0% 51.2% 53.8% ESL Course Improvement 44.7% 50.1% 50.2%

Is this data actionable?

ACTIONABLE DATA?Basic Skills Success Rates

African-American 31.3% Asian 75.0%

Decline to state Filipino 68.4%

Hispanic 62.5% Native American Pacific Islander

Unknown White 56.1% Total 59.7%

Is this data actionable?

ACTIONABLE DATA?

Is this data actionable?

ACTIONABLE DATA?

Basic Skills Success at Levels prior to TransferEnglish English English ESL ESL ESL Math Math Math

3 levels prior Pre-Intermediate 68% Fundamental 65%2 levels prior 67% Intermendiate 35% 69% Algebra 32% 67%1 levels prior 40% 74% High Intermediate 22% 37% 72% Intermediate 11% 27% 72%transfer 25% 46% 79% Transfer 11% 13% 31% 5% 13% 31%

16% 29% 42%

Is this data actionable?

STUDENT LEARNING OUTCOMES DATAESL Level 1 Outcomes Assessments % passed Listening/Speaking

SLO: Demonstrate understanding of frequently used words, phrases and questions in familiar contexts. Engage in limited social conversations to communicate basic survival needs.

Assessment = Interview Outcomes = 84% passed successfully

Reading SLO: Construct meaning from simplified print materials on familiar topics. Assessment = CASAS Level A (A comprehensive standardized test) Outcomes = CASAS Success rate 73%   Writing SLO: Produce simple sentences in paragraph format and complete simple

forms. Assessment = Written Paragraph and Rubric for scoring Outcomes = 74% pass rate

Principle 1 – Use longitudinal data when possible

Principle 2 – Use data in context

Principle 3 – Look for both direct and indirect data

Principle 4 – Do not oversimplify cause and effect of data

Principle 5 – Use appropriate levels of data for appropriate levels of decisions

Principle 6 – Perception is the reality within which people operate

Principle 7 – Use of data should be transparent

Principle 8 – Consider carefully when to aggregate or disaggregate data

Principle 9 – Focus on data that is actionable

Principle 10 – Consider implications and the “What if?

EXPLORING DATA - DATA 101

CREATING  INSTITUTIONAL PROCESSES FOR CONDUCTING RESEARCH AND USING DATA TO INFORM PRACTICE

-Information Capacity Challenges

-Research Processes and Procedures

-Research Agendas

-Action Research Guided Questions

Building an Evidence-based Infrastructure Managing and responding to myriad requests Maintaining quality and integrity of data

process Making data and information widely accessible

Keeping Up with the Demand Responding to heightened accountability

mandates Linking research to (resource) planning Supporting data-driven decision-making

Turning Data into Action Making data available and applicable at all

levels Making sense of and taking action on the data Building a Culture of Inquiry

INFORMATION CAPACITY CHALLENGES

Guidelines for use of data and information Protection of human subjects policy Review panels and committees Request and fulfillment procedures Criteria for prioritizing ad hoc requests Linking requests to broader goals &

initiatives Creating and using Research Agendas

PROCESSES & PROCEDURES

College-wide Research Agenda Supports major college-wide initiatives &

activities Tied to college-wide plan (goals & priorities) Includes recurring requests

Topical Research Agenda Focused on a single topic or group of interest Tied to a specific initiative or activity Fewer research activities than college-wide

RESEARCH AGENDAS

Developing the Research Agenda

1. What and who will be researched?

2. How is research tied to college plans, goals, initiatives and/or activities?

3. How will the information be used, by whom and how often?

4. Which methodology or approach will be used?

Turning Data into Information

1. What do the data tell us?

2. Which questions were fully answered by the research and which need more exploration?

3. What are reasonable benchmarks based on the research?

Taking Action on the Information

1. What interventions or strategies do we need to deploy in order to move the needle?

2. How should this information be shared and applied across the college?

ACTION RESEARCH GUIDED QUESTIONS

DEVELOPING A CULTURE OF COLLABORATIVE INQUIRY -Operational Definition -Building a Culture of Inquiry -Data Integration & Inquiry Strategies

Good evidence used in evaluations has the following characteristics:

1. It is intentional, and a dialogue about its meaning and relevance has taken place.

2. It is purposeful, designed to answer questions the institution has raised.

3. It has been interpreted and reflected upon, not just offered up in its raw or unanalyzed form.

4. It is integrated and presented in a context of other information about the institution that creates a holistic view of the institution or program.

5. It is cumulative and is corroborated by multiple sources of data.

6. It is coherent and sound enough to provide guidance for improvement.

CHARACTERISTICS OF EVIDENCE

Source: ACCJC/WASC Guide to Evaluating Institutions, 2009

EVIDENCE AND INQUIRY

Data are provided for multiple purposes.

Data demonstrate performance & outcomes.

Data are accessible to all constituency groups.

Data are continuously distributed.

Dialogue centers around data/evidence.

Dialogue focuses on taking action.

Dialogue is open and collaborative.

Dialogue is continuous and widespread.

Dialogue is reflective and dynamic.

Culture of Evidence Culture of Inquiry

1. Make evidence and inquiry the paradigm.2. Communicate often, widely and clearly.3. Embed the cycle of collectively assessing,

planning, implementing, re-assessing and re-planning…..

4. Create opportunities for continuous collaboration with multiple and mixed constituency groups.

BUILDING A CULTURE OF BUILDING A CULTURE OF INQUIRYINQUIRY

WHAT MIGHT THIS LOOK LIKE?

ORGANIZING DATA: WEAVE, TRAKDAT, TASKSTREAM, ELUMEN

“The New Guys in Town”

QUESTIONS? Thank You

Please fill out the evaluations