Spring2010CEAreport

7/31/2019 Spring2010CEAreport

1/8

First-Year Writing ProgramSpring 2010 Course-Embedded AssessmentReport of Results

Dr. Clancy Ratliff, Director of First-Year WritingThanks to Amanda Anderson, Joe Baumann, Garnet Branch, Joanna Davis-McElligatt, Jack Ferstel, John Guidry, Marie Hendry,Amanda LaRoche, Katie Manning, Jennifer Page, Christal Seahorn, and Elizabeth Sonnier, who served as readers.


2/8

Assessment Spring 2010

The readers scored papers in each of the three categories from 1-6, where 1-2 were poor, 3-4 were satisfactory, and 5-6 were outstanding. The First-Year Writing Program rubric is on page 7 of this report. After a thorough norming exercise, two readers scored each paper. We met our benchmark of70% of student papers scoring satisfactory or higher.

Overall results (n=100) Poor Satisfactory | Outstanding

Content 24% 76%

Organization 25% 75%

Language Issues 29% 71%

Overall 28% 72%

First-Year Writing Program CEA HistoryThe First-Year Writing Program conducted rounds of course-embedded assessment (CEA) in Fall 2008, Spring 2009, and Spring 2010. In Fall 2009, in lieu of CEA,students in English 101 took the Collegiate Learning Assessment exam.

Students in the Fall 2008 CEA performed below expectations (our benchmark was 70% or more students with an overall score of satisfactory or higher); thatsemester, only 49% of student papers were scored satisfactory or higher. We attributed these results to unusually high attrition and student distraction in the wakeof Hurricane Gustav and Hurricane Ike. Also, we revised the rubric.

Students in the Spring 2009 CEA performed below expectations, but that semester, 59% of student papers were scored satisfactory or higher.

For this round of CEA, I made three major changes in the process:

1.Skip Fox, a faculty member in English, suggested including papers from English 115 in the assessment. We decided to do this when we collected our Spring2010 samples; using samples from one section of English 115, which is an honors course, gives a more accurate measurement of the range of achievement

2


3/8

across the freshman class. We had heretofore been scoring only papers from English 102 classes.

Of the 100 papers we scored, 87 came from English 102 classes, and 13 came from an English 115 class. I do not know exactly what percentage of thefreshman population typically places into honors courses from year to year, so it's unclear whether English 115 students are overrepresented orunderrepresented in this sample.

2.Another change I made was to change the scoring range from 1-3 to 1-6. This change follows a method used by Dr. Asao Inoue, professor and Assessment

Coordinator for the College of Arts and Humanities at California State University, Fresno, which directs assessment readers to put student work into"conceptual buckets": a score of 1-2 would put student work into the "poor" bucket, 3-4 in the "satisfactory" bucket, and 5-6 in the "outstanding" bucket.Readers now had the discretion to give a paper a "low satisfactory" or "high satisfactory" if there was some overlap between the categories, as therealways is.

3.Finally, one key difference between the Spring 2010 CEA and previous rounds was that the First-Year Writing Program rubric had become widely availableto students. It is printed in their required textbook,The Freshman Guide to Writing, beginning with the academic year 2009-2010 edition of the book, and itis on the First-Year Writing Program web site. All teachers of First-Year Writing courses are asked to use that rubric when grading student writing, and it isthe rubric we use to settle grade disputes.

I believe that these changes provided enhanced construct validity.

We plan to conduct another round of CEA in Fall 2011. In the Spring 2011 semester, we will be doing a large-scale indirect survey assessment of the First-YearWriting Program, a project that is being supported by the Office of Academic Planning and Faculty Development in the form of an Instructional Improvement Mini-Grant. For the next round of CEA, I will attempt to gather data about the percentage of freshmen who qualify for the Honors Program and use exactly that numberof English 115 papers in the 100-sample set.

Starting in Fall 2010, we made some changes to the First-Year Writing curriculum. We are now using new books that are, I believe, more closely aligned with theoutcomes and the rubric we use. Perhaps these changes will raise or lower the scores in the next round of CEA, though I know that a cause-and-effect relationshipcannot be proven.

An ongoing concern I have is the increasing population of freshmen who take their first-semester writing course in high school dual enrollment programs. I am

skeptical that any dual-enrollment program, no matter how rigorous, is equivalent to taking separate high school and college English courses. Students who gostraight into English 102 simply have less time to practice writing; I advocate a curriculum that provides students with at least five full years of English courses: fouryears in high school and one full year in college.

Most relevant to the CEA, though, is the question of what exactly is being assessed. Our writing program consists of a two-semester sequence: English 101 andEnglish 102. Admittedly, there will always be transfer students who have taken their first-semester writing course at another institution, but the dual enrollmentpopulation is sizable, and some of Louisiana's dual enrollment programs have come under criticism for their lack of rigor above and beyond the regularrequirements of high school English 4. One possible method for addressing this issue is to use the special-roster data the English department receives eachsemester and check the student writing samples against that data set; only students whose records show that they took English 101 at ULL would be included in theCEA, or their papers could be included but given designated numbers (unbeknownst to the readers) in the CEA sample set.

3


4/8

Raw Data (R1 and R2 refer to "Reader 1" and "Reader 2")

4

Paper Number Content Score R1 Content Score R2

1 4 2 3 3 3 3 3 2 2.5 2.88

2 3 3 3 3 3 3 3 3 3 3

3 3 2 2.5 3 3 3 2 3 2.5 2.72

4 3 4 3.5 4 4 4 5 4 4.5 3.94

5 3 3 3 4 5 4.5 5 5 5 4.06

6 1 1 1 1 1 1 1 1 1 1

7 4 4 4 3 3 3 5 4 4.5 3.75

8 5 3 4 5 3 4 5 3 4 4

9 5 2 3.5 2 2 2 6 3 4.5 3.19

10 4 3 3.5 4 2 3 4 3 3.5 3.31

11 2 2 2 2 2 2 4 3 3.5 2.38

12 3 3 3 2 4 3 3 3 3 3

13 3 3 3 2 2 2 2 2 2 2.38

14 2 4 3 2 3 2.5 3 3 3 2.81

15 3 4 3.5 4 5 4.5 4 5 4.5 4.13

16 1 2 1.5 2 3 2.5 2 2 2 2

17 2 3 2.5 2 3 2.5 1 2 1.5 2.25

18 4 3 3.5 4 3 3.5 5 3 4 3.63

19 6 4 5 5 4 4.5 5 3 4 4.5620 4 5 4.5 3 4 3.5 4 4 4 4

21 5 4 4.5 5 4 4.5 4 3 3.5 4.25

22 2 3 2.5 4 2 3 5 2 3.5 2.94

23 5 3 4 4 3 3.5 5 3 4 3.81

24 3 2 2.5 3 5 4 3 5 4 3.44

25 5 3 4 4 3 3.5 5 2 3.5 3.69

ContentAverage

Organization ScoreR1


OrganizationAverage

Language ScoreR1

Language ScoreR2

LanguageAverage

OverallAverage


5/8

5

Paper Number Content Score R1 Content Score R2ContentAverage



OrganizationAverage

Language ScoreR1

Language ScoreR2

LanguageAverage

OverallAverage

26 3 2 2.5 2 3 2.5 3 1 2 2.38

27 3 3 3 3 3 3 4 2 3 3

28 2 3 2.5 1 3 2 3 2 2.5 2.31

29 5 3 4 5 2 3.5 3 3 3 3.56

30 5 3 4 5 3 4 4 3 3.5 3.88

31 2 2 2 2 2 2 2 3 2.5 2.13

32 3 3 3 3 3 3 3 3 3 3

33 5 2 3.5 4 2 3 5 5 5 3.69

34 5 5 5 4 4 4 3 3 3 4.13

35 5 5 5 4 4 4 5 4 4.5 4.5

36 6 4 5 6 5 5.5 6 4 5 5.19

37 4 5 4.5 4 5 4.5 4 5 4.5 4.5

38 5 2 3.5 3 2 2.5 3 3 3 3

39 3 2 2.5 4 2 3 3 2 2.5 2.69

40 2 5 3.5 4 3 3.5 2 4 3 3.38

41 4 3 3.5 3 3 3 5 4 4.5 3.56

42 5 4 4.5 5 5 5 5 4 4.5 4.69

43 2 5 3.5 2 5 3.5 2 5 3.5 3.5

44 5 4 4.5 4 5 4.5 4 5 4.5 4.545 3 3 3 3 2 2.5 4 3 3.5 2.94

46 3 4 3.5 4 4 4 3 3 3 3.56

47 3 5 4 2 5 3.5 3 5 4 3.81

48 4 4 4 4 3 3.5 4 3 3.5 3.69

49 2 3 2.5 1 1 1 3 3 3 2.06

50 5 2 3.5 4 2 3 4 2 3 3.19


6/8

6




OrganizationAverage

Language ScoreR1

Language ScoreR2

LanguageAverage

OverallAverage

51 3 3 3 3 2 2.5 2 2 2 2.56

52 4 3 3.5 5 4 4.5 4 4 4 4

53 1 2 1.5 2 2 2 3 2 2.5 1.94

54 5 4 4.5 3 5 4 3 4 3.5 4.06

55 4 4 4 3 4 3.5 3 2 2.5 3.44

56 3 4 3.5 4 5 4.5 4 3 3.5 3.88

57 3 4 3.5 3 4 3.5 1 4 2.5 3.25

58 3 3 3 2 4 3 1 4 2.5 2.88

59 4 3 3.5 3 2 2.5 2 2 2 2.75

60 2 1 1.5 2 1 1.5 2 1 1.5 1.5

61 3 3 3 2 3 2.5 2 3 2.5 2.6962 1 2 1.5 2 2 2 2 2 2 1.81

63 3 1 2 2 1 1.5 3 1 2 1.81

64 2 2 2 4 3 3.5 4 2 3 2.81

65 2 2 2 3 3 3 2 3 2.5 2.5

66 3 3 3 3 3 3 4 3 3.5 3.13

67 4 4 4 3 4 3.5 4 4 4 3.81

68 2 1 1.5 3 3 3 2 1 1.5 2.06

69 5 2 3.5 4 2 3 5 2 3.5 3.31

70 3 5 4 3 4 3.5 4 1 2.5 3.44

71 3 2 2.5 3 3 3 4 1 2.5 2.69

72 5 6 5.5 5 6 5.5 5 6 5.5 5.5

73 1 1 1 2 3 2.5 4 3 3.5 2.19

74 4 3 3.5 4 2 3 4 3 3.5 3.31

75 2 3 2.5 2 5 3.5 3 4 3.5 3.13


7/8

7




OrganizationAverage

Language ScoreR1

Language ScoreR2

LanguageAverage

OverallAverage

76 3 5 4 2 4 3 2 3 2.5 3.25

77 3 4 3.5 4 4 4 4 4 4 3.81

78 4 4 4 4 5 4.5 4 4 4 4.19

79 5 6 5.5 6 5 5.5 6 6 6 5.63

80 5 5 5 5 5 5 6 5 5.5 5.13

81 6 6 6 5 6 5.5 6 5 5.5 5.69

82 4 5 4.5 3 4 3.5 3 2 2.5 3.63

83 5 2 3.5 4 2 3 3 4 3.5 3.31

84 4 3 3.5 2 3 2.5 4 2 3 3

85 3 3 3 4 3 3.5 5 3 4 3.44

86 3 2 2.5 4 2 3 3 4 3.5 2.94

87 2 5 3.5 3 4 3.5 2 2 2 3.13

88 5 2 3.5 4 3 3.5 6 4 5 3.88

89 5 3 4 4 2 3 5 4 4.5 3.75

90 3 2 2.5 3 2 2.5 3 2 2.5 2.5

91 2 4 3 3 3 3 2 2 2 2.75

92 4 4 4 5 4 4.5 5 5 5 4.44

93 3 3 3 3 4 3.5 4 2 3 3.19

94 4 4 4 4 4 4 4 4 4 4

95 5 6 5.5 4 5 4.5 4 6 5 596 3 3 3 3 2 2.5 3 4 3.5 2.94

97 5 4 4.5 2 2 2 4 3 3.5 3.31

98 3 5 4 2 4 3 3 5 4 3.63

99 4 3 3.5 4 3 3.5 4 3 3.5 3.5

100 3 5 4 4 5 4.5 2 5 3.5 4.06

3.4 3.31 3.39 3.36


8/8

Topic is too broad and general for a

paper of its length

No clear main idea or sustainedposition in the paper (or argument isincredible to an academic audience)

Argument, if present, is unsupported,

or evidence is insufficient

Paper is overly reliant on cliches or

culturally conditioned/ ethnocentricassumptions and bias

Sources, if used, are used

inappropriately (data dump, awkwardintegration, unintentional plagiarism,etc.)

Topic is manageable for a paper of its

length

Position/argument is comprehensibleeven if not clearly stated

Amount of evidence is sufficient

Demonstrates critical thinking (ability

to recognizecomplexity, biases, andstereotypical representations;distinguishing fact from opinion)

Use of sources is appropriate; no

serious problems with integration ofother materials

Topic is narrow enough to allow for a

rigorous, nuanced treatment of the subject

Thesis is stated clearly

Evidence is ample to support position taken

Contains some acknowledgment of

opposing/divergent views

Sources are used purposefully and

strategically, integrated seamlessly

Introduction does not orient the reader

to the concerns of the paper orcontextualize the subject of the paper

Arrangement of the paper is haphazard

and random

Paragraphs do not have transitions that

guide the reader from one idea to thenext

Conclusion is absent or abrupt

Introduction isrecognizable even if it is not alwaysreader-based

Paragraphs generally treat one idea

at a time

Attempts at transitions between

paragraphs are made, even if theyare awkward

Conclusion provides some closure to

the argument, even if only asummaryof the main points

Paper contains a

clear, reader-based introduction,development, conclusion

Logical, smooth transitions between sections

Plan of development stated (forecasting

statement, self-announcing structure toargument)

Conclusion does more than just summarize

the paper; restates the thesis in a fresh wayor includes a gesture (call for action,unresolved questions, etc.)

Frequency of error (of any type)

seriously detracts from the content ofthe paper

Grammar, punctuation, and spelling

are mostly correct

Student shows command of

language (word choice/ vocabulary)

Varied sentence structure

Paper is virtually free of error

Writer shows an unusual felicity with regard

to word choice, turns of phrase (ex. usesobscure words, bon mots)

Sentence structure is complex but not

cumbersome

8

Spring2010CEAreport

Documents

Transcript of Spring2010CEAreport