Indirect Assessment Comparison Report

7/31/2019 Indirect Assessment Comparison Report

1/13

Indirect Assessment of the First-Year WritingProgram:

Surveying UL First-Year Writing Students with the

Consortium for the Study of Writing in College(CSWC) Survey Instrument

Dr. Clancy RatliffDirector of First-Year WritingOctober 2011

Acknowledgements: This project was funded by the Office ofAcademic Planning and Faculty Development through anInstructional Improvement Mini-Grant.

-


2/13

Introduction

The field of rhetoric and composition studies has, by way of over four decades of

research, come to agree on several best practices in the teaching of writing: having

students write essays in a series of drafts, assigning brainstorming and prewriting

activities, assigning a variety of genres, assigning papers written to a specific audience,holding one-on-one conferences with students about their writing, and more. For the first-

year writing curriculum at the University of Louisiana at Lafayette, I have designedcourse outlines, outcomes, and assignments with these practices in mind. These

pedagogical techniques are the focus of a survey I distributed to students in English 101

and 102 at the end of the spring 2011 semester.

The survey is based largely on questions designed in 2007 by Charles Paine, Robert

Gonyea, Paul Anderson, and Chris Anson for the Consortium for the Study of Writing in

College, a partnership between the National Survey of Student Engagement and theCouncil of Writing Program Administrators. The questions are intended to serve as a

supplement to the NSSE to measure the degree to which students engage in writingthroughout all their college coursework in both general education and their majors butthe authors of the survey permit professors and administrators to use the instrument for

smaller studies at their own universities. I believe that the results of this survey help to

contextualize the results of our previous assessment efforts.

In the fall of 2008 and the spring of 2009, the first-year writing program conducted

course-embedded assessment with the help of Paula Phillips Carson and Carolyn Bruder.

During these semesters, we collected samples of student writing from English 102 classesand evaluated them according to a rubric I designed. The results both semesters were

disappointing; in neither round of assessment did we meet our benchmark of 70% or

more students getting a rating of satisfactory or higher. We conducted another round ofcourse-embedded assessment this semester with writing samples from spring 2010; this

time, students from English 115 were also included in the data set, which revealed a more

accurate representation of the range of achievement across the freshman class and, Ibelieve, resulted in our meeting the benchmark: 72% of student papers were scored as

satisfactory or better.

My research assistant, Christal Seahorn, and I visited six sections of English 101 and 23sections of English 102. Participation was voluntary; we visited classes with the

instructors consent, and we offered them the opportunity to read the survey questions

before deciding whether or not to let us distribute the survey to their students. One

English 101 teacher took us up on this opportunity. One important caveat for thesefindings, then, is that the teachers self-selected to have their students participate in the

assessment. Random selection of teachers and unannounced classroom visits wouldprobably have yielded different results, but for the sake of morale, I decided to let

teachers opt into the assessment. A total of 546 students took the survey, 426 of whom

were in English 102 classes. In all, we visited six sections of English 101 and 23 sections

of English 102.

-


3/13

This report contains results from both the 101 students and the 102 students, along with

my commentary on the differences in the findings. It's important to note that students in

English 102 taking the survey were asked to reflect on their experiences in both 101 and

102 as they answered the questions.

-


4/13

Which of the following most accurately describes your situation (check one):

English 101 English 102

I have taken/plan to take allmy first-year writing courses

(90, 101, 102, or 115 as

applicable) at ULL

108 (90%) 350 (86%)

I have taken one or more

first-year writing courses atanother university and am

completing the first-year

writing requirement at ULL

10 (8%) 36 (9%)

I have taken one or more

first-year writing courses inhigh school through dual

enrollment

2 (1%) 6 (2%)

I have received automatic

credit for one or more first-year writing courses from

testing (AP exam, ACT/SAT

score, CLEP) and amcompleting the first-year

writing requirement at ULL

0 14 (3%)

Overall: I asked this question to get a sense of how many students take both courses of

the writing sequence here at ULL and to differentiate between transfer students and dual-

enrollment students. I know from experience that had we done this survey in a fallsemester, these numbers would have been different; many of the first-time freshmen who

take English 102 in fall semesters have completed English 101 through dual enrollment.

1. During the current school year, for how many of your writing assignments have

you done each of the following?Brainstormed (listed ideas, mapped concepts, prepared an outline, etc.) to develop your ideasbefore you started drafting your assignment:

English 101 English 102 Difference

All assignments 29% 29% 0%Most assignments 31% 33% 2% increase in 102

Few assignments 22% 28% 6% increase in 102

One assignment 13% 4% 9% decrease in 102

No assignments 3% 6% 3% increase in 102

Overall: Brainstorming on the whole increased. More students in 102 brainstormed on at

least a few assignments, and fewer brainstormed on 0-1 assignment.

Talked with your instructor to develop your ideas before you started drafting your

assignment

-


5/13


All assignments 17% 15% 2% decrease in 102

Most assignments 29% 23% 6% decrease in 102

Few assignments 25% 37% 12% increase in 102One assignment 17% 16% 1% decrease in 102

No assignments 12% 9% 3% decrease in 102

Overall: Talking with instructors to develop ideas increased in 102, but only slightly.

Talked with a classmate, friend, or family member to develop your ideas before you

started drafting your assignment

English 101 English 102 DifferenceAll assignments 18% 26% 8% increase in 102

Most assignments 22% 23% 1% increase in 102

Few assignments 27% 27% no change

One assignment 16% 12% 4% decrease in 102No assignments 16% 12% 4% decrease in 102

Overall: Brainstorming with peers increased slightly in 102.

Received feedback from your instructor about a draft before turning in your final

assignment


All assignments 35% 29% 6% decrease in 102Most assignments 20% 26% 6% increase in 102

Few assignments 15% 23% 8% increase in 102One assignment 13% 9% 4% decrease in 102No assignments 12% 12% no change

Overall: By the end of English 102, more students reported having received feedback on

drafts. Instructor comments on rough drafts are pedagogically powerful, and in the future, Iwould like to see all students report having received feedback on drafts for at least a few

assignments (and instructors requiring submission of rough drafts for feedback and revision).However, that is difficult for teachers given the demands on their time and our class size in

101 and 102.

Received feedback from a classmate, friend, or family member about a draft beforeturning in your final assignment (this includes class peer review activities)



Most assignments 23% 28% 5% increase in 102Few assignments 15% 26% 11% increase in 102


-


6/13


7/13

Proofread your final draft for errors before turning it in




Few assignments 11% 11% no changeOne assignment 3% 2% 1% decrease in 102

No assignments 1% 1% no change

Overall: Proofreading frequency increased slightly in 102.

2. During the current school year, in how many of your writing assignments did you:Narrate or describe one of your own experiences



Most assignments 15% 13% 2% decrease in 102Few assignments 33% 38% 5% increase in 102



Overall: Narrative writing based on personal experience decreased in 102, or rather, itremained at English 101 levels. That is to be expected; while personal narratives and personal

essays are sometimes assigned in 101, 102 teachers are asked to assign research-basedwriting.

Summarize something you read, such as articles, books, or online publications

English 101 English 102 DifferenceAll assignments 8% 23% 15% increase in 102Most assignments 21% 41% 20% increase in 102

Few assignments 41% 30% 11% decrease in 102


Overall: Summary of outside sources increased significantly in 102. That is in keeping with

the 102 curriculum in which students are required to engage with/in academic source-basedresearch writing.

Analyze or evaluate something you read, researched, or observed


All assignments 23% 26% 3% increase in 102Most assignments 27% 43% 16% increase in 102



-


8/13

Overall: Analysis increased in 102, which is to be expected given the greater difficulty of

English 102 and, perhaps, students' greater awareness of what it is they are actually doing

in their writing.

Do field research (interviews, informal or formal surveys) for research-based writing


All assignments 0% 4% 4% increase in 102


One assignment 22% 23% 1% increase in 102


Overall: Field research increased in 102. It still remains at low levels, which is fine;

teachers are under no instruction to require this kind of research. Teachers who are

themselves experienced field researchers seem to be more confident with teaching (andmore likely to teach) field research.

Argue a position using evidence and reasoning






Overall: We see a considerable increase in argument-based writing in 102. Again, I

believe this can be attributed to both the 101/102 curricular distinctions and the students'increased sophistication and self-awareness.

Summarize and refute opposing views (counter-arguments) in argument-based writing



Most assignments 16% 35% 19% increase in 102Few assignments 28% 35% 7% increase in 102One assignment 28% 14% 14% decrease in 102


Overall: Engaging with counterarguments increases in 102, I believe for reasons I've

already stated.

-


9/13

Use ULL library resources (databases, books, journals) to find sources for research-

based writing


All assignments 7% 30% 23% increase in 102Most assignments 7% 28% 21% increase in 102Few assignments 9% 26% 17% increase in 102



Overall: We see a somewhat dramatic increase in library use in 102; again, this difference

has its basis in the First-Year Writing curriculum.

Create the project with multimedia (web page, poster, slide presentation such as

PowerPoint, etc.)






Overall: There's a slight increase in multimedia use in 102. Teachers are not required to

assign work that uses some of the examples of multimedia, and our technology resourcesdo not give much support to such instruction. I suspect the increase in 102 is due to

teachers' assigning research presentations in conjunction with research-based essays. I

included this question because it was in the original CSWC survey instrument.

Include drawings, tables, photos, screen shots, or other visual content into your written

assignment



Most assignments 3% 1% 2% increase in 102Few assignments 11% 7% 4% decrease in 102



Overall: Again, the inclusion of visual content in writing assignments is not something

required of teachers or students. I suspect the overall decrease of visual content in 102has to do with the fact that some English 101 teachers have students do a rhetorical

analysis of a visual argument as one of the assignments, but that assignment is not

typically given in 102.

3. During the current school year, for how many of your writing assignments have

-


10/13

your instructors in ULL First-Year Writing courses done each of the following?

Provided instructions describing what he or she wanted you to do



Few assignments 5% 3% 2% decrease in 102One assignment 3% 1% 2% decrease in 102


Overall: Student perception of having received instructions on assignments increased

slightly in 102.

Assigned readings from the course textbooks in conjunction with the assignment

English 101 English 102 DifferenceAll assignments 40% 31% 9% decrease in 102


Few assignments 18% 22% 4% increase in 102One assignment 2% 4% 2% increase in 102


Overall: Student perception of having been assigned reading from the course textbooks

increased somewhat. I asked these questions about textbook use because occasionally

(though less so in the last year and a half), teachers complain that they don't like therequired books, but I wanted to get a clear sense of how much the students and teachers

actually use the books.

Explained in advance what he or she wanted you to learn





One assignment 2% 2% no change

No assignments 3% 3% no change

Overall: There was no significant change in this number between 101 and 102. I willencourage teachers to explain learning outcomes for particular writing assignments by

continuing to give them language to express these learning goals -- referencing the course

outcomes and more specific learning goals.

-


11/13

Explained in advance the criteria he or she would use to grade your assignment




Few assignments 11% 16% 5% increase in 102One assignment 3% 2% 1% decrease in 102


Overall: Student perception of instructors' clarity in presenting grade criteria declinedslightly in 102. I suspect that this is due to the increased emphasis on research paper

writing in 102 and students' confusion about how MLA documentation format is

weighted in the grades (based on my interactions with students in grade appeal cases) --as well as their generalized confusion about the genres of academic research writing.

Referenced the First-Year Writing Program grading rubric in the Freshman Guide







Overall: Students perceived that their instructors referenced the writing program rubric

slightly more frequently in 102, with a greater percentage of students reporting that the

instructors referenced the rubric for at least a few assignments. I will continue to remindinstructors that the rubric is a useful tool for them (and certainly for students).

Provided a sample of a completed assignment written by the instructor or a student

(this includes sample ULL student essays in the Freshman Guide)







Overall: No significant change from 101 to 102. I think it's more important in English

102 for students to see examples of the genres of papers they are expected to write,

written by student authors like themselves; while they have ample opportunity to readarguments written by journalists and other professional writers, they rarely have the

opportunity to read student research papers. Journalists and professional writers in

-


12/13

popular magazines are not held to the same standards of citing their sources (certainly not

in the format students are expected ; it is assumed that the publication has fact-checkers

who

Asked you to do short, ungraded, informal writing exercises in or out of class







Overall: There was a slight increase in 102. It's unclear how to interpret these answers;

my sense is that teachers do assign a good bit of informal journal writing and in-class

writing, but perhaps students do not see it as directly connected to their major

assignments. Sometimes it's not, and the teachers are just having the students write forthe sake of practicing writing.

Asked you to give feedback to a classmate about a draft or outline the classmate had

written





One assignment 16% 5% 11% decrease in 102No assignments 7% 7% no change

Overall: There was a slight increase in peer response exercises in 102. While peer

response exercises aren't mandatory in first-year writing, they do help students meet ouroutcome of "engage in writing as a recursive process," and they help students in basic

project management -- not putting their essay-writing off until immediately before the

deadline. I'm pleased to see that 88% of students reported having done peer response

exercises for at least a few of their assignments.

Conclusion

Overall, I am satisfied with these results. While we have conducted direct assessment of

student work, I was interested in what a combination of direct and indirect assessmentwould show. I wanted to see what students perceptions were of what their instructors

required them to do. I believe its possible that teachers may require many of these

activities but that students may not recall having engaged in them, and this survey showswhich activities the students have retained. Now that I have the results of the survey, I am

planning closing-the-loop measures: first, teaching workshops devoted to the areas Id

-


13/13

like to improve, which for me include students getting feedback from peers, tutors, and

teachers prior to turning in their papers, as well as doing more reading, particularly the

course textbooks, and more analysis and evaluation in their work (though I understand

that students may actually be doing analysis and evaluation but not realizing it. Thatsanother area Id like to address: metacognitive skills in writing). I am also designing

curriculum materials targeted toward these areas -- assignment handouts, classroomactivities, etc. -- to upload to my Moodle repository of instructional support materials forfirst-year writing instructors.

-

Indirect Assessment Comparison Report

Documents

Transcript of Indirect Assessment Comparison Report