2015-2016/Fe… · Web viewFort Lewis College. Assessment Committee. Office of Assessment ....

Running Head: Liberal Arts Core Assessment 2014-2015 Report

Fort Lewis CollegeAssessment Committee

Office of Assessment

Liberal Arts Core Assessment 2014-2015

Report prepared by:Heidi Steltzer and Lisa Snyder

Assessment Committee Members (2014-2015)Heidi Steltzer (Sciences, chair)

Erich McAlister (Sciences)Shawn Meek (Arts and Humanities)

Michael Martin (Arts and Humanities)Mika Kusar (Business)

Rebecca Austin (Social Sciences)Kim Hannula (associate dean, ex-officio)

Lisa Snyder (Director of Assessment, ex-officio)

Liberal Arts Core Assessment 2014-2015 Report

I. Background and Overview

In May, 2014, The FLC Faculty Senate passed new Liberal Arts Core Learning Outcomes. These new learning outcomes, which were adapted from the Essential Learning Outcomes of the Association of American Colleges and Universities (AAC&U), paved the way for formal assessment of Fort Lewis College’s current general education program as well as the ongoing assessment of the new Liberal Arts Core, which remains in development.

In an effort to obtain guidance from experts in the fields of general education and assessment, a trans-disciplinary group attended AAC&U’s Institute on General Education Reform & Assessment (IGEA) in Burlington, Vermont. The results of the IGEA were an action plan for the reform of FLC’s General Education Program (now called the Liberal Arts Core), and an assessment plan with accompanying procedures. The structure of the 2014-2015 Liberal Arts Core assessment plan is based on consultations provided at AAC&U’s IGEA. This plan called for scoring to take place at the end of both the Fall, 2014, and Spring, 2015, giving us two rounds of assessment data prior to our HLC accreditation visit in October, 2015. Fall 2014 assessment was completed April 2015 and Spring 2015 assignments for assessment have been collected. The second round of assessment will be completed in Fall 2015.

For both rounds of assessment, a scoring committee made up of faculty members of the LAC Council, the Assessment Committee, and faculty volunteers collected cross-disciplinary samples of students’ written work in their senior seminar / capstone courses. Scoring is based on the AAC&U’s critical thinking and problem solving rubrics. This approach was suggested for several reasons. First, there is no cross-disciplinary course from which to take work samples for student’s all in the same place in their academic career. Second, critical thinking is included as an outcome for every program on campus; therefore seniors should demonstrate this skill in their senior-level work. Finally, this would allow the Assessment Committee to work on the assessment process while the Liberal Arts Core Council focused on the reform of the Liberal Arts Core program.

The experience gained from assessment of Fall 2014 work samples has greatly improved the process for Liberal Arts Core assessment at Fort Lewis College, including the approaches used to collect samples, and faculty familiarity with the process of submitting and scoring student work. The following report identifies protocols created for the collection and de-identification of student work samples, findings of what was learned through scoring Fall 2014 work samples, as well as modifications to the assessment process.

2

2


II. Process and Protocols for the Collection and De-identification of Student Work Samples

Objective for 2014-2015 Academic Year: Assess critical thinking and problem solving for liberal arts core

Summary of method for liberal arts core assessment at Fort Lewis College• Faculty teaching capstone senior courses submit writing samples for each student/group

within their course at the end of the term.• Writing samples are scored for critical thinking and problem solving, using the AACU

LEAP rubrics.• Writing samples are selected through a stratified random sampling approach

• Stratification is by area (Science, Social Science, Arts and Humanities, Business, Teacher Education) and within areas by faculty to increase chances that selected writing is not all from one faculty, as is possible.

• Approximately 20% of the work samples are scored by faculty volunteers, including individuals from the assessment committee

• Data will only be analyzed across areas and faculty

Detailed method for liberal arts core assessment at Fort Lewis College: Score writing assignments from the final capstone course (senior thesis) across all disciplines using the AACU LEAP rubrics for critical thinking and problem solving. Tasks are listed below by number and the corresponding number is included in the table on the next page to identify by whom and when tasks need to be completed.

1) Instructors were asked to select a minimum of two criteria for critical thinking and one criterion for problem solving by which their students’ assignments will be scored. All criteria could be selected if appropriate. Written assignments and cover sheets are due at the end of the term.

2) A protocol for de-identification was developed and will be used to ensure consistency and thoroughness across written assignments (see page 6).

o Assignments and corresponding cover sheets will be coded to ensure cover sheet information (one per instructor) can be matched to assignments (more than one per instructor).

o Two spreadsheets have been developed, so that the process is double-blind. One spreadsheet is for Institutional Research, the Director of Assessment, and the Director of e-learning. This spreadsheet contains identifying information for students, departments, and faculty. The second spreadsheet for the AC chair does not contain this information and will be used to select writing assignments for scoring.

3) Data on submissions will be compiled and provided to the Chair of the Assessment Committee using MS Excel spreadsheet for this purpose (template provided by AC chair). This spreadsheet will be used to select writing assignments for scoring.

4) Data on submissions will be compiled and provided to the Director of Assessment and Director of e-learning using MS Excel spreadsheet for this purpose (template provided

3

3


by AC chair). This spreadsheet will be used, if needed, to provide departments with data or resources.

5) A stratified, random sampling design will be used to select theses for scoring. o Stratification will be by area (i.e. Science, Arts and Humanities, Social Science,

Business, Teacher Education) and if needed by whether written assignments were completed as a group or by individuals. Stratification is needed, because submissions are unbalanced across areas, yet all areas should be represented. Area will not be used as a variable by which data is analyzed.

6) Assignment instructions or grading rubric will be added to each written assignment selected for scoring by copying the text into the written assignment. The instructions or grading rubric will precede the written assignment.

7) Written assignments selected for scoring will be de-identified using the protocol. 8) Notes will be taken on additional steps needed and steps not needed to improve the

protocol. The notes will be used to revise the protocol. Revised protocol and de-identified assignments will be made available to Director of Assessment and Director of e-learning director.

9) Volunteer scorers will be recruited from across areas. A system to track service has been developed by the AC.

10) Assignments will be assigned to scorers by area to the extent this is possible and uploaded to Canvas.

11) Scorers will be trained through a rubric norming session.12) Scorers will have one month to complete scoring through Canvas13) Scores will be analyzed to assess critical thinking and problem solving for the liberal arts

core across all areas. Complete a report, discussing results and process, ex. what worked, what didn’t work and steps for improving process.

Phase 1: Fall senior seminar / capstone coursesBegin Oct 1, 2014 complete by Mar 20, 2015 (actual date of completion Apr 2015) Total number of theses to select and score will be 50, each scored twice.10 volunteer scorers were recruited of which 9 completed their scoring.Service for assessment of liberal arts core was recorded for those who completed scoring.

Who was responsible and when tasks were completed:Task Whose task Needed

bySend to:

1) submission of assignments and cover sheets

Coordinated by AC chair

Dec 23, 2014


2) develop de-identification protocol Director of Assessment

Jan 27, 2015

AC chair

3) Data on submissions compiled for AC Office of Assessment

Jan 29, 2015*

AC chair

4) Data on submissions compiled for Director of Assessment and Director of e-


Jan 29, 2015

Director of Assessment and

4

4


learning Director of e-learning

5) selection of assignments for scoring AC chair Jan 30, 2015*


6) add instructions to assignments selected for scoring

Director of e-learning

Feb 6, 2015

Post on canvas

7) use de-identification protocol to de-identify written assignments


Feb 6, 2015

8) record notes and revise protocol for de-identification; provide revised protocol and assignments to Director of Assessment and Director of e-learning

Office of Assessent

Feb 6, 2015

Director of Assessment and Director of e-learning

9) recruit scorers Coordinated by AC chair

Feb 6, 2015

Director of Assessment and Director of e-learning

10) assign assignments to scorers and upload to Canvas

Director of e-learning

Feb 13, 2015

Director of Assessment and AC chair

11) train scorers (rubric norming session) Director of Assessment and AC chair

Feb 13, 2015

AC chair and Director of e-learning

12) scoring Faculty scorers Mar 13, 2015

Enter on canvas

13) analyze data and complete report AC chair and Director of Assessment

Mar 20, 2015

Director of Assessment

*task 5 requires that task 3 is completed on time. Also note that there is a different spreadsheet for the chair of AC that does not include any identifying information for students or faculty.

Phase 2: Spring senior seminar / capstone coursesBegin Apr 1, 2015 complete September, 2015 Phase 2 will be designed to improve on phase 1 including approach for assignment submission and tracking, component selection for scoring from rubrics, and approach for scoring.

Faculty senate approved a procedure for involving faculty in scoring, which will be implemented in August for fall scoring (see procedure for faculty service for liberal arts core assessment).

Due to the complexities involved with soliciting, collecting, organizing, coding and uploading work samples to Canvas, it was felt by all involved that it would be more efficient to centralize the process within the Office of Assessment. The methods listed in this report include this revision to the process for assessment of the liberal arts core.

5

5


Protocol for collecting and de-identifying work samples1) Faculty whose courses are selected to provide student work samples will be notified at the beginning of each semester. Notification will include corresponding rubrics and instructions.

2) At the end of the term, faculty will submit student work samples, the assignment instructions, the cover sheet, and the rubric components electronically, in the form of a zipped file, to the Assessment Coordinator, Office of Assessment.

3) The Assessment Coordinator will pool assignments across all courses/instructors; summary data on assignments will be compiled and provided to the Chair of the Assessment Committee using MS Excel spreadsheet (template provided by AC chair).

4) Selecting from the spreadsheet identified in #3, the Chair of the Assessment Committee will select a stratified, random sample to be used for scoring. Stratification will be by area (i.e. Science, Arts and Humanities, Social Science, Business, Teacher Education) and if needed by whether work samples were completed as a group or by individuals. Stratification is needed, because submissions are unbalanced across disciplines, yet all disciplines should be represented. Discipline will not be used as a variable by which data is analyzed. The AC chair will give sample list will be given back to the Assessment Coordinator for de-identification of samples.

5) The Assessment Coordinator will de-identify all work samples. This will include deleting names of both students and faculty from the work samples, deleting acknowledgement sections, etc. Citations will not be changed, although this could provide identification of faculty who mentored students as this will be less of an issue when work samples are not from senior seminar /capstone course. Each work sample with its corresponding cover sheet and assignment will be coded with corresponding and non-identifying codes (i.e. no faculty initials will be used) and saved as a file named with the code. For example, file F1401.001 will include the work sample, the cover sheet, and the assignment for a fall 2014 work sample for faculty 01 and student 001. A second spreadsheet will be created that includes student names, discipline, and corresponding codes and will be provided to the Director of eLearning so that she may set up the files in Canvas. The addition of this spreadsheet also serves to double blind the student work samples.

6) When scoring is complete, the director of e-learning will provide results from Canvas (all scoring will be done on Canvas) for each rubric component to the Director of Assessment and the AC Chair.

7) The Chair of the Assessment Committee will provide a report of findings to the Assessment Committee and the Office of Assessment. The Assessment Committee will make recommendations for disseminating findings and acting on assessment results.

8) The Chair of the Assessment Committee and the Director of Assessment will collaborate on the assessment report for the academic year.

6

6


Faculty Volunteer Scorers Fall 2014Michael Martin (Arts and Humanities)*Michele Malach (Arts and Humanities)

Gordon Cheesewright (Arts and Humanities)*Rebecca Austin (Social Sciences)*

Brad Clark (Social Sciences)Cathy Hartney (Honors Program)

Gary Gianniny (Sciences)Melissa Knight-Maloney (Sciences)

Erich McAlister (Sciences)*Paul Clay (Business)

*are individuals also serving on the assessment committee or the liberal arts core council

7

7


III. Submissions, Findings, and Modifications Fall 2014 Liberal Arts Core Assessment

Submissions Tracking and ImprovementsFor Fall 2014 liberal arts core assessment, work samples were collected from 14 courses. All faculty from whom the assessment committee requested work samples submitted work samples, but through the process of requesting work samples, it was clear that a system was needed to track submissions. A submission tracking system would enable us to determine which faculty had not yet submitted work samples, issues that may arise in collecting the work samples and the reasons for these issues, and the total number of submissions and students represented by these submissions (some work samples were group assignments).

Area Participation for Fall 2014 Work SamplesArea Total #writing samples # selected samplesArts and Humanities 10 10Business 11 11Social Science 47 15Science 44 15Teacher Education 0 0

The number of writing samples is affected by the 1) number of students/courses taught 2) whether assignments are completed individually or as groups and 3) if suitable assignment is included in the capstone course, i.e. writing, in English, critical thinking and problem solving can be assessed.

The primary issues that arose in collecting Fall work samples were identifying which faculty needed a reminder to submit work samples and determining if work sample submissions from a faculty represented all student work in a course. To improve on the submission process for Spring 2015, we developed spreadsheets shared by the Assessment Coordinator and the chair of the Assessment Committee. The spreadsheets made it possible to review submissions weekly and send reminders to faculty who still needed to submit work samples. For many faculty who did not submit work samples on time, they did not understand if they needed to submit work samples. Familiarity with the process of assessment we are designing will resolve this issue.

The issue of incomplete submissions was corrected by including on the cover sheet (see appendixes) that faculty list the total number of students in the course, the number of work samples they are submitting, and clarification for the difference. For example, faculty might have 14 students in a course, but only submit 10 work samples if 2 students did not complete the assignment and some worked in a group. Alternatively, work sample submissions may be incomplete because an email may not have gone out. For liberal arts core assessment, faculty submission of work samples was done through email, but it is our intention to shift this process to a different system such as task stream or canvas.

8

8


Area Participation for Spring 2015 Work SamplesArea Total #writing

samples# students # selected

Arts and Humanities 64 78 20Business 12 34 12Social Science 112 151 20Science 86 123 20Teacher Education 0 0 0

Our improved system for work sample submissions allowed us to track the total number of writing samples and the number of students this includes (some students worked in groups). More work samples were submitted in spring as more programs teach an appropriate capstone course in spring terms. We will also increase the number of selected samples given the greater number of submissions.

In spring 2015, we also created a system to track reasons that work samples were not submitted from senior seminar / capstone courses in order to improve the process for requesting and receiving submissions. While nearly all requested work samples were submitted, we can improve through planning for scoring in Spanish, through familiarity among faculty with the process, and earlier notification by assessment committee to faculty that work samples are needed in an electronic format.

Findings and Modifications for Liberal Arts Core AssessmentIn our first round of assessment of the liberal arts core, the assessment committee chose to have faculty select two components of critical thinking (of 5 in the LEAP rubric) and one component of problem solving (of 6 in the LEAP rubric). This was chosen as assignments were not intentionally designed for these rubrics and some components may not be well matched to the assignment. Through our rubric norming session for the faculty volunteer scorers, we learned that this approach had some limitations as well. Many faculty who provided their selection of components were either unfamiliar with the rubric components or the written explanation of the components is not sufficient to determine if it is appropriate for an assignment.

During the rubric norming session, the 10 scorers became more familiar with the rubrics and identified approaches to use the rubrics for scoring assignments in their areas. For example, science faculty scorers created a system to match components of critical thinking and problem solving to common sections of scientific papers, which was the format for most submissions in the sciences. We agreed that all rubric components, not just those selected by the instructor, should be scored and that a 0 would be entered for any components for which there was no evidence. No evidence was often a result of the assignment design or student understanding of the assignment not corresponding with a component of the rubric. For Spring 2015 liberal arts core assessment (to be completed Fall 2015), we will again use all components of the rubrics

9

9


for Critical Thinking and Problem Solving and have not asked faculty to select components. For data analysis, we did not include scores of 0 in the analysis as these do not reflect an assessment of student work.

Using the LEAP rubrics for critical thinking and problem solving, Fort Lewis College students demonstrated a milestone-level of proficiency. Mean (dashed line) and median (solid line) scores for each were near 2.5, where scores of 2 and 3 represent milestone proficiency (see figure below). Scores represent the mean score across components of critical thinking or components of problem solving for each student, thus data show individual-level performance. Scores among students varied with 25 to 75% of students scoring between 1.8 and 2.9 for critical thinking and between 2.0 and 3.0 for problem solving (represented by the boxes). Fewer than 5% of the students scored above 3.5 or below 1.5 for critical thinking and problem solving competencies, indicating few students demonstrated either capstone or benchmark levels of performance.

Fort Lewis College Liberal Arts Core Assessment 2014-2015mean and variation for Critical Thinking and Problem Solving

Mean scores across components within each

Critical Thinking Problem Solving

Score (4 is capstone)

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Student performance within components of critical thinking indicate that at least some students can perform capstone-level work for each component (see figures that follow), although no students demonstrated capstone-level work on all components (i.e. no 4’s in the previous figure). Mean and median scores for each component ranged from 2.0 to 2.8,

10

10


representing milestone performance. Among the components of critical thinking, students could best demonstrate evidence for CT1: Explanation of issues and CT5: Conclusions and related outcomes (fewer scores under 2.0). We expect that intentional assignment design will enable students to perform better on other components of critical thinking.

Student performance within components of problem solving show some students can perform capstone-level work, but this is achieved less often than for critical thinking. Students can best define problems (PS1). The greater number of scores for this component also provides evidence that assignments incorporated this component of problem solving (ex. a score of 0 would have been entered if there was no evidence and would not be tallied). There was least often evidence for assignments designed for students to evaluate outcomes (PS6). Mean and median scores for each component ranged from 2.1 to 2.7 (excluding PS6 due to few observations), presenting milestone performance. Similarly, we expect that intentional assignment design is necessary to evaluate problem-solving. As an example, scientific writing often does not require stating challenges of the research and how they were overcome. Even more so, faculty may specifically state this should not be included in a formal scientific report. Alternative assignments would enable students to demonstrate competency with problem solving.

11

11


Fort Lewis College Liberal Arts Core Assessment 2014-2015mean and variation for Critical Thinking scores

Critical Thinking Criteria

CT1 CT2 CT3 CT4 CT5


1.0

1.5

2.0

2.5

3.0

3.5

4.0

Fort Lewis College Liberal Arts Core Assessment 2014-2015mean and variation for Problem Solving scores

Problem Solving Criteria

PS1 PS2 PS3 PS4 PS5 PS6


1.0

1.5

2.0

2.5

3.0

3.5

4.0

12

12


Fort Lewis College Liberal Arts Core Assessment 2014-2015Frequency for Critical Thinking scores

Scores (4 is capstone)

1 to 1.9 2 to 2.9 3 to 3.9 4

Frequency of score (for 51 senior theses)

0

5

10

15

20

25CT1: explanation of issuesCT2: evidence

CT3: context and assumptionsCT4: student's position

CT5: conclusions and outcomes

Fort Lewis College Liberal Arts Core Assessment 2014-2015Frequency for Problem Solving scores

Scores (4 is capstone)

1 to 1.9 2 to 2.9 3 to 3.9 4

Frequency of score (for 51 senior theses)

0

5

10

15

20

25PS1: define problemPS2: identify strategiesPS3: propose solutionsPS5:evaluate potential solutionsPS5: implement solutionPS6: evaluate outcomes

13

13


Procedure for faculty service for liberal arts core assessment For Fall 2014 liberal arts core assessment, faculty scorers were asked to serve by members of the assessment committee with the assurance that a system would be developed to fairly distribute this responsibility among faculty. In March 2015, the chair of the Assessment Committee presented a summary of the method for assessment and schedule to Faculty Senate. Faculty Senate had tasked the Assessment Committee to develop the methods and schedule for the 2014-2015 academic year. At this meeting, the chair of the Assessment committee also proposed four options for recruiting faculty scorers for liberal arts core assessment. These options were developed by the Assessment Committee and are listed below).

• Option 1: Volunteer• Option 2: Requirement for faculty serving on LAC Council and Assessment Committee• Option 3: Jury-service type system (AC recommendation)

• All faculty after completing 2nd year service become eligible for selection• Random selection, document service completed, not called again to serve until

all other faculty have served• Can’t be called in term when instructing for a course being assessed• Can defer by one term• ~110 faculty eligible, ~20 scorers needed per year, service about 1x every

5 years• Option 4: Service requirement for LAC assessment is included in faculty expectations

statement; chairs document this is completed.

Discussion continued to the end of the meeting and resumed at the following faculty senate meeting, one week later. At that time, senators proposed a fifth option, which is that faculty are selected for service as scorers and notified that they have been selected. At that time, faculty may prepare for and complete service or may opt-out of service providing a statement for the reason they are opting out. Faculty that opt-out will remain in the pool of faculty from which scorers are selected. Service as a scorer will remove faculty from this pool until 50% of the faculty have served as scorers. Faculty statements will be kept to determine how the process is working and if modifications are needed. Faculty senate voted to approve this option and this opt-out approach for selecting faculty scorers will be implemented in Fall 2015 for Spring 2015 assessment.

Planning for an approach to select scorers and seeking faculty senate approval for the process is a primary reason that spring scoring was not done in May 2015. There was not enough time after the procedure was approved by senate to plan for and notify selected faculty, since many faculty teach intensive summer courses or travel in May.

14

14


Appendixes

I. Letter to colleagues requesting Spring 2015 work sample submissions

Hello colleagues,PLEASE READ this email to learn about AY 2014/2015 assessment of Liberal Arts Core (LAC)Assessment of LAC is required by the Higher Learning Commission. This year, LAC assessment will score work samples from senior seminar/thesis courses across all majors for critical thinking and problem solving. Fall courses have been assessed, and what we learned is being used to inform assessment for spring term courses as evidence for developing a process of LAC assessment at FLC.

As a faculty teaching a senior seminar/thesis course this term, the AC will need you to submit the following:

1) digital copies of student work for an assignment completed within your course this term that includes critical thinking and problem solving. Details are listed below:

a) work samples could be any written assignment in your course, ex. senior thesis report, essay, or other assignment. If possible, it is preferred that the assignment is one completed by individuals rather than groups. Assignments need to be in English at this time.b) if your course does not include an appropriate assignment, email to provide an explanation, ex. art portfolio or assignments are in Spanish; email to ask questions about selecting an assignment from your course.c) assignments can be submitted as Word documents or PDFs or other text files; Word documents are preferred.

2) the instructions/rubric for the assignment as a digital file, and

3) a completed cover sheet (see attachment) with information needed to track submissions. Submit by email as individual files or a zipped folder (preferred) to Val Skarbek, FLC assessment coordinator, at [email protected]

If you only have paper copies and cannot request students provide digital copies, provide the paper copies to Val in EBH Room 161. She will scan these and return the copies to you.Student work samples from your course need to be submitted by 5:00pm on May 8, 2015.

For LAC assessment, 20% of the work samples will be randomly selected from within each of the areas of study at FLC (ex. Sciences, Social Sciences, Arts Humanities, SOBA and Teacher Education), selected work samples will be de-identified, and scoring will be completed for critical thinking and problem solving. This assessment will not be linked to individual students, courses, or faculty, as it is designed to assess LAC.

15

15

mailto:[email protected]


We would be glad to answer questions by email, phone or in person. Contact Heidi Steltzer, chair of the assessment committee ([email protected], 247-7450) or Lisa Snyder, director of assessment ([email protected], 247-7543).

Thanks,The Assessment CommitteeHeidi Steltzer, chair

16

16




II. Cover sheet for information needed to track submissions (Spring 2015)

Fort Lewis CollegeGeneral Education Assessment Spring 2015Assignment Cover Sheet

Department:

Faculty Name:

Course Number and Section:

Course Name:

Submission Instructions:

1. To track receipt of the submissions, we need to know:a. How many students are in your course? b. How many papers you are submitting?c. Please comment on the difference – ex. students worked in groups or the number of students

earning incompletes or failing to complete the assignment.

2. Were papers completed by groups or individuals (please circle) Groups Individuals

3. Were students allowed to submit multiple revisions of this paper prior to turning it in for a grade? (please circle) Yes No

4. What is the sequence of courses that constitutes your capstone?

Course Credit

The above information is collected for tracking purposes only. All identifying information will be removed prior to the student work being forwarded to trained faculty scorers for

assessment at the institution level using the corresponding VALUE rubric. Assessed work will not be traceable to a student, course, or faculty member. Be assured the assessment results

from this project will be reported in aggregate only.

17

17

2015-2016/Fe… · Web viewFort Lewis College. Assessment Committee. Office of Assessment ....

Documents

Transcript of 2015-2016/Fe… · Web viewFort Lewis College. Assessment Committee. Office of Assessment ....