AssessingAssessingStudent Learning OutcomesStudent Learning Outcomes
ininWriting CoursesWriting Courses
Lisa Marchand
Cosumnes River College
Sacramento, California
ACCJC Accreditation ACCJC Accreditation StandardsStandards
California Community Colleges are required to establish “well-documented, ongoing processes for evaluating” effectiveness at all levels of the institution.
The foundation of this evidence must be data collected and analyzed in a consistent way, over time.
By the Year 2012By the Year 2012Engage in an ongoing dialog
about student learning Have a well documented process
for evaluating Student Learning Outcomes
Show tangible evidence that your program is continuously moving toward improved student learning.
Think about your own Think about your own classes or program…classes or program…
How do you assess students writing skills?
Are Student Learning Outcomes in place?
Are they measurable?How are they measured?
Take five minutes to talk it over!
Cosumnes River CollegeCosumnes River College
Questions:How well can students demonstrate
that they have met the Student Learning Outcomes for writing courses that they have passed?
How well can students who have placed into a level through the assessment process demonstrate the skills specified in the prerequisite course’s SLO’s?
Existing DiagnosticsExisting Diagnostics
Not specifically aligned to course SLO’s (Either too general or too specific)
Mining data from these diagnostics would be too difficult
ConcernsConcerns
We wanted to assess SLO skill sets but we didn’t want to have these assessments affect the students’ grades.
We didn’t want to have the SLO assessments be used to evaluate the teachers in any way.
Assessing SLO’s at the end of a course wouldn’t provide the best information on students who entered through the placement process.
Our Goal: Our Goal: Sustainable Continuous Quality Sustainable Continuous Quality
ImprovementImprovement
Analyze Data and Discuss Observations
Measure Student Learning Outcome
s
Make Adjustments
Ramping UpRamping Up
Develop student learning outcomes
Define metrics to measure SLO’s
Experiment with various measures and discuss observations
Creating Departmental Creating Departmental Diagnostic Writing Diagnostic Writing
AssessmentsAssessments
Measure expressive writing skillsA holistically scored writing sample
Measure meta-linguistic abilitiesAn objective multiple choice
assessment
Analyze SLO’s to Identify Analyze SLO’s to Identify MetricsMetricsMeasurable elements of writing
include:◦Syntactic structures and variety◦Accuracy of verb tense form and use◦Appropriate word forms◦Range of vocabulary◦Rhetorical patterns◦Development◦Use of cohesive devices◦Other
……include measures of meta include measures of meta skillsskillsMeta-linguistic skills are included in
SLO’sExamples:
◦Identify X◦Label X◦Analyze X◦Recognize X◦Differentiate between X and Y
An objective instrument can provide discrete measures of each of these
Build a Scoring GuideBuild a Scoring GuideCRC chose an ETS style holistic
scoring guide model with 5 levels:
◦Insufficiently prepared = 1◦Under-prepared = 2◦Fairly prepared = 3◦Well prepared for the level = 4◦Demonstrates skill above level = 5
……write descriptors for each write descriptors for each levellevelStart with the SLO’s – write a
descriptive narrative to match expressive skills they describe
This will be the score 4 – “well prepared for the course”
To write descriptors for scores 1 – 3 and for score 5, examine the sample papers already marked by faculty
Select A Writing PromptSelect A Writing PromptFor the lowest levels, we chose a
picture and guiding questions:“Look at the picture. Who is the man? Where is he? What happened? How does he feel? Why does he feel that way? Where is he going? What will happen next? Write the story!”
For intermediate SLO’s and beyond, we selected this prompt:
“Some people like to live in big cities, others do not. Tell where you prefer to live and explain why.”
Create an Objective Create an Objective InstrumentInstrumentAlign assessment items directly to SLOMake more than one item to measure
each skill…vary the contextsUse multiple choice format & SCANTRON©
Check to see that the terms used are consistent with what students have encountered in classes and texts
Collaborate to check for validity, bias, and clarity
Validity CheckValidity CheckAccording to Test Better, Teach Better
by W. James Popham, content-related evidence of validity can be determined by relying on “human judgement”
A team of expert colleagues agrees that each test item measures the skill it was designed to reflect.
The team must also agree that the test items are accessible and free of bias.
Check for ClarityCheck for ClarityConduct item-by-item study of
the testBe sure that the language level
matches the textsBe sure that vocabulary is
appropriate and standardBe sure the content doesn’t
unfairly affect a student’s ability to respond—it is not biased.
Check for BiasCheck for BiasContent Bias
◦Gender◦Culture◦Experiential
Linguistic Ambiguity◦Vague wording◦Confusing grammar◦Low-frequency vocabulary
Difficulty
Create a Score Tally in Create a Score Tally in ExcelExcel
student id
SLO's 1 - 4
Holistic Score
SLO #1 Evidenc
e of Plannin
g
SLO #2 Logic and
Development
SLO #4 Syntax
SLO #3 Mechan
ics
Identifying
Sentence
Parts SLO #5 Items 1-7
(7Q's)
Identifying
Parts of Speech SLO #6 Items
8-14 (7 Q's)
Verb Tense Forms SLO #7 Items 15-20 (6
Q's)
Verb Tenses
in Questions SLO
#2 Items
21-24 (4 Q's)
475549 4 2 3 2 2 5 4 1 2485646 2 2 2 1 2 4 4 4 3922766 3 2 2 2 2 4 4 6 4987991 3 2 2 2 1 5 6 5 31020104 3 2 3 3 2 2 3 3 21036656 2 2 1 1 1 4 3 4 2
……includeincludeColumns for
◦Student ID #’s◦Holistic Writing Score◦Scores for component skills
evaluation◦Scores from each section of the
objective assessment
Analyze DataAnalyze DataCollect descriptive statisticsMake correlationsFocus on evidence of student learningProvide data sets to all teachers for
use in self-studies and program reviewUse the assessments over several
semesters to enable longitudinal study of a significant sample of student work
Continue the DialogContinue the DialogUse collective observations to suggest
and implement changes to ◦Curriculum◦Methods◦Materials
Select appropriate data to support resource requests like facilities and FTE
Have faculty experienced in the process mentor and norm their colleagues
Other than ACCJC Standards, Other than ACCJC Standards,
Why Assess SLO’s?Why Assess SLO’s?
Find out what you’re “teaching” that the students aren’t learning.
Move faculty dialogue away from anecdotal discussion
Confirm intuition (or not)Gather substantive support for
resource requests
Example SLO’sExample SLO’sSLO 1: Identify and apply basic essay components
SLO 2: Employ the writing process
SLO 3: Interpret and respond to writing prompts
SLO 4: Develop paragraphs appropriate to rhetorical mode
SLO 5: Identify and correct grammatical errors through editing
SLO 6: Write simple, compound, and complex sentences with correct punctuation
SLO 7: Identify and correctly use grammar structures and word forms
Example of an SLO with Example of an SLO with MetricsMetrics
SLO 7: IDENTIFY AND USE GRAMMAR STRUCTURES
AND WORD FORMS
identify and use all basic verb tenses with very rare errors in form (simple and progressive aspect of present, past and future)
identify and use some perfect verb tenses with occasional error in form and usage (present perfect, present perfect progressive, past perfect)
identify and use singular, plural and non-count forms and appropriate quantifiers and articles without errors in form
identify parts of speech and use correct word forms in writing
W 50 SLO#7 W 50 SLO#7 IDENTIFY AND USE GRAMMAR IDENTIFY AND USE GRAMMAR
STRUCTURES AND WORD FORMS STRUCTURES AND WORD FORMS
0
1
2
3
4
5
6
7
8
9
10
Holistic Score Grammar Correcting Errors
Total Possible
Average
Best Score
Lowest Score
Convert to a Comparable Convert to a Comparable FormFormDo some simple arithmetic:
◦Divide the total possible into 100 100÷5 = 20
◦Multiply the answer by the average score 20 x 2.18 = 43.6%
W 50 SLO#7 W 50 SLO#7 IDENTIFY AND USE GRAMMAR IDENTIFY AND USE GRAMMAR
STRUCTURES AND WORD FORMS STRUCTURES AND WORD FORMS
0
10
20
30
40
50
60
70
80
Holistic Score Grammar in Writing
Correcting Errors on objective test
Student Success
Student Success
Observations from the Observations from the DataDataNeither the students’ ability to
correct grammar errors on the objective assessment nor the teachers’ evaluation of students’ grammar in the writing sample were significantly correlated with their success on the holistically graded writing sample.
There was no significant correlation between students writing sample and the objective assessment.
Significant CorrelationsSignificant CorrelationsThe students holistic writing
scores were directly correlated with the score the rater gave them on their essay organization.
The students holistic writing scores were directly correlated with the score the rater gave them on their essay development.
Top Related