AssessingAssessing StudentStudent

LearningLearning

Section 6Section 6

Program Program AssessmentAssessment

Program Assessment

This section will explore program assessment by beginning with a scenario to gain perspective when writing program SLOs, then discussing an important fundamental question concerning defining and designing programs. The faculty role in General Education assessment, which may represent either program or institutional level assessment (depending upon the institutional culture), will also be explored in some detail. This section of training will conclude with sample program assessment tools and reports, as well as the role of program review in assessment. There are many options available for program assessment. Some institutions have created assessment days or weeks for program and institutional assessment, others have embedded common questions or an essay into regular course exams and finals. The tools and techniques are similar to course assessment, but require a slightly broader perspective.

Workshop Student Learning Outcomes

After completing this workshop material, participants should be able to:Create a set of DRAFT SLOs for a program through faculty dialogue.Plan for and implement assessment for one or two program SLOs each year.  

Linkage of course outcomes to program and institutional level outcomes is essential. The philosophy behind this website is based upon the concept that training faculty to articulate outcomes and develop assessment practices in their favorite courses will equip them to apply that expertise at program and institutional level assessment. Program assessment is a more comprehensive endeavor and must be clearly focused on learning to produce improvement, rather than a report of the program details which is commonly the extent of Program Review activities. We will begin this section by looking at things from a different and broader perspective and then working through some of the details of program assessment in community colleges.

2

Course Assessment

Program

Assessment

Institutional

Assessment

Let’s start with a brief look at learning outcomes for a program that is not part of a community college, but is well defined, and has articulated both a mission and an outcome. The program is a grant-funded women and children's shelter called The Respite. Funding is dependent upon the annual report of the program outcomes. (This scenario was developed with Lynn Krausse, a BC business professor and CPA, based upon experiences consulting for non-profit organizations.)

Scenario – Respite Women and Children’s Shelter Background: If all the women victimized by domestic violence in a given year joined hands, the line would stretch from New York to beyond Los Angeles. (Women's Shelter Inc. statistics for 2001)

The Respite Women and Children's Shelter is a program that provides shelter and protection for women and children that are caught in domestic violence. In addition to shelter, there are hotlines, counseling and support groups, and a program to assist women to live independently and secure jobs. The shelter is supported by outside donations and grants. In order to renew the major grant to fund the program, a report must be submitted that assesses how well the shelter has met its purported outcomes.

The volunteers at the shelter have had years of experience, are fully aware of, and most concerned about changing the cycle of violence and abuse. They have vision beyond just providing refuge and protection. The mission statement for the shelter reads:

To provide safety and shelter to women and children who are victims of domestic violence through crisis intervention, therapeutics, and education and to reduce societal violence against women and children.

An overarching outcome that they have defined reads - Following the services and therapeutics of the women's shelter, battered women will make a choice to leave their violent relationships as a major strategy to protect themselves and their children.

How would you critique this outcome in relation to the shelter's mission? Consider the SLO checklist (in Appendix A) and evaluate the Outcome as it is written.

Take a moment and write down your thoughts.

3

Setting the Stage - Discussion about the Shelter SLO

Program Outcome- Following the services and therapy provided by the women's shelter, battered women will make a choice to leave their violent relationships as the major strategy to protect themselves and their children.

This outcome does focus on what the woman can do following the program. This outcome defines an action - "make a choice to leave" The outcome is a measurable expectation. You could find out how many

people from the shelter have left their violent situations after a given time period.

There has been dialogue. The shelter board and participants know this is an important answer to promote real health and safety.

However, the next points provide a new perspective from the role of program outcomes.

Is this outcome valid and appropriate for the shelter? Is this outcome understandable to the intended audience?

Thoughts:There is no doubt that this outcome is valuable and would contribute to solving the problem and meeting their vision of reducing domestic violence in our society. The outcome suggests an easily measured and valued assessment method - Determine how many clients left their violent relationships last year?

However, this is where a problem occurs - most victims of domestic violence chose NOT to leave their relationships. In fact, many people in the shelter come only to receive temporary refuge with no desire to leave a relationship; their main desire is that the violent person will change their behavior. Therapists struggle with this aspect of domestic violence; few have succeeded in producing substantive behavioral changes in spite of therapy. Basing income (grant funding) upon an outcome that is ideal, but difficult or impossible to achieve is problematic; this may be an excellent goal or vision, but it is an unpredictable and perhaps improbable outcome.

It is more realistic to base program outcomes upon things that the program actually does, and outcomes that the participants care about. Below is a sample year-end report. Notice that this shelter has accomplished some terrific results, worthy of their effort and continued financial support.

4

Respite Women and Children’s Shelter statistics for 2003 (This is real data from an actual shelter.)

Sheltered a total of 412 women and 463 children. Provided community advocacy and criminal justice intervention to 2,525

women and legal advocacy to 2,259 battered.  Helped 1,382 battered women through crisis line support.  Received 12,299 calls from friends, family, and others concerned about a

battered woman they know.  Conducted 139 presentations and 2,333 hours of professional training for

4,757 participants.  Thirty-seven battered women received assistance specific to immigration.  Fifty-four individuals lived in the Respite’s transition accommodations.  Thirty-five children attended the Respite Children's School. Two hundred fifty-three children participated in Children's Program

activities.

These statistics are very impressive and indicate excellent work. For our scenario, which uses an outcomes approach, the strategy is to look at the product in order to improve. Just sheltering more women or giving more presentations may not get us to the program goal- to reduce societal violence against women and children. Using these statistics the shelter needs to reconsider what they are able to do, what they do well, and what outcomes the funding does support. It is laudable to have values of improving society, but these are not the direct outcomes of this program. (In the same way, a local food bank program does not reduce world hunger.)When writing program outcomes:

state the program purpose or mission consider other areas or programs that feed into or interact with your program analyze community expectations for the program survey program descriptors and accomplishments review the components of the program and determine participant expectations

Good program assessment begins with a clearly stated Mission or Goal that defines the program's reason for existence. Based upon this purpose, assessable outcomes

5

represent a mechanism to determine how well the program is achieving its goal and provides feedback on how to improve. Using the information reported for the shelter above, can you construct an assessable and realistic outcome for this program that could be used as a basis for a grant-funded outcomes report? The statistics above prove that help, refuge, advocacy, and training are being provided (in the same way that course content is delivered). What observable outcomes will be the product of the participants encounter with the shelter? Remember to consider the three domains for Cognitive, Psychomotor, and Affective outcomes. (Use the SLO checklist in the Appendix A.)

Can you suggest any assessment tools to gather evidence concerning the outcome?

In assessing the outcomes, surveys concerning self-assessment from the women would be helpful. Baseline information concerning their status before using the services would help, as would evidence of change over time in attitudes and skills. These same issues relate to academic programs, so let’s move our thinking to our own programs.

Defining Programs

The primary requirement for writing SLOs and designing program assessment is a clearly defined program with a written mission statement. Mission statements are not hard to create and the conversations are exceedingly useful. Appendix B has a brief tutorial for writing mission statements for programs.

During the budget crises, our campus conducted an institutional audit; we identified 72 different instructional, support, and administrative programs, a nearly unmanageable number. Each program was required to create a mission statement and describe how the program contributed to IMPROVED learning on campus. Programs wanted to explain how they contributed to learning, but the assignment was to describe how they contributed to IMPROVED learning. This audit included all instructional programs, as well as administrative and support services programs, such as the cafeteria, bookstore, Chicano student center, and the president's office. This began an exciting shift in our perspective as defined by the learning institution paradigm. (Don't envision sudden transformation, but do imagine great dialogue.)

The audit instruments are available online at http://online.bakersfieldcollege.edu/courseassessment/Section_6_Program%20Assessment/Section6_3definingprogram.htm

6

This audit process generated an important question for Bakersfield College, "What is an assessable program?" We had always defined programs by departments and disciplines, or geographic locations, e.g. the biology department, physical science, humanities, the book store, and counseling. Viewing it from the student's perspective we began to see that a program might be a pathway. For instance, the biology program really contained three pathways which were programs of study ending in or contributing to terminal degrees.

the pathway or program for biology majors- requiring some pre- and co- requisites (math, chemistry, physics)- taking numerous interrelated courses with a discipline focus - typically transferring to a four year institution

the pre-allied health program- requiring pre-requisites- taking a lock-step series of courses to prepare for a profession- concluding with a vocational program and eventual board exam

the general education program- requiring only collegiate level reading- serving as the only science portion to many student's education- concluding in a liberal studies degree (potential teachers) or as transfer degree in another discipline field or vocation

Before the campus begins to create new program outcomes, review the campus structure and culture to determine whether the existing structure works well and is learning-centered, or whether robust conversation needs to occur concerning structures and program definitions. Share information between programs; some existing programs have well-defined outcomes and assessment practices in place, particularly vocational or grant-funded programs. (Section 8 discusses the use of an assessment audit to save time and energy.) Finally, a discussion concerning programs must consider cross-disciplinary programs or degrees. This material will go into some detail concerning the General Education program, but consider other cross-disciplinary programs such as Chicano Studies. For pathways or programs such as a pre-allied health biology program, this entails discussions with the Math department, the Chemistry department, and the nursing or x-ray department. This represents a unique, but stimulating challenge, which could greatly benefit students (and is somewhat reminiscent of learning communities).

*Warning: These discussions take time and examine the fabric of institutional organization and governance structures. However, the discussions provide a rationale for why things exist as they do, and an opportunity to review them concerning learning centered strategies. Allow time and be inclusive when examining these issues.

7

Writing Program SLOs

Articulating the program goals, and coordinating the appropriate course SLOs, are important foundations in finalizing draft program SLOs. It is also important to consider external requirements or expectations after a program or course of study. This would include an analysis of: 1) the community or employer expectations, 2) professional standards and expectations, 3) alignment between course, program, and institutional outcomes, 4) student expectations and needs, and transfer institution expectations.

Program SLOs

Community expectations

Program vision, mission, and goals

Parent institution mission vision and goalsStudent needs and goals

Course

SLOs

Overlapping Course SLOs Become Program SLOs Course SLOs

Course SLOs

Course SLOs

Course SLOs

Related professional expectations

8

The goal is to explicitly state overarching outcomes that represent skills, knowledge, and abilities the students attain as a result of the program of study. This may include activities beyond course work (field trips, internships, volunteer experiences). Once again the SLO checklist should be useful.

Instructional Programs Outcomes

Student Services Programs Outcomes

Name a specific course or programName a specific program or service (e.g. counseling, financial aid, DSPS, Admissions and Records, International Studies, etc.)

 

Are there any prerequisites for this course?

This program must interface with what other key areas?

Does this course serve as a prerequisite for any other courses?

Course Purpose or MissionPurpose, Mission, or Function that best describes this programs role in student learning:

 

Is the General Education Program assessed as an institutional assessment, or as a multi-disciplinary program? It depends upon the campus culture and organization.

Regardless of the organization, WASC requirements for General Education (GE) Programs are very explicit and it is important to review the implications of these standards. The next portion of this section involves GE assessment and then returns to traditional program assessment. Skip the next few pages if GE is not a concern for you. An example of linking courses to GE outcomes is in Appendix D.

9

General Education Program Assessment

General Education refers to that course of study often called Liberal Studies or Inter-Disciplinary Studies and represents the majority of AA degrees granted in California Community Colleges and the set of core studies for most discipline specific degrees. Some combinations of these courses are also referred to as CSU Breadth Requirements, IGETC - Intersegmental General Education Transfer Curriculum, or the local institution's general education requirements. (Look in your institution's catalogue for specific courses approved for transfer and degrees.)Some campus-wide assessment initiatives start by assessing the General Education Curriculum, which cuts across nearly every department or program. Initiatives at this level require intensive cross disciplinary discussion and participation and clearly defined Student Learning Outcomes for the General Education Program of study. *Once again I suggest this conversation occurs most productively if faculty have been trained and had the experience of doing assessment in their favorite courses first.

General Education requirements often define the unique character of US higher education institutions. Program outcomes will be highly variable depending upon program goals, the community expectations, the student population, and the institutional mission. ACCJC-WASC describes very specific standards which must be present in the core of all GE programs, I have summarized them below - the link in the resource section contains them in detail.

Summary of GE Standards ACCJC-WASC Standard II.A.3.a-c

The GE program must publicize a philosophy and must include courses as part of the program based upon SLOs.

GE outcomes must include:

1. Basic discipline content and methodology for humanities, fine arts, natural sciences and social sciences.

2. Productive individual and life-long learner skills including oral and written communication, computer literacy, scientific and quantitative reasoning, critical analysis/logical thinking

3. Recognition of ethics and effective citizenry such as: interpersonal skills, appreciation of diversity, historical and aesthetic sensitivity, and civic responsibility.

Exact Standards Language is available at http://www.accjc.org

10

General Education Outcomes Topics from Sample Institutions

This list represents the topics from a variety of schools, that represent what every student will know or be able to do when they graduate with any degree from that college. Specific outcomes can be written within each discipline area based upon competency in these general areas. These areas, while unique to each institution and reflecting the values and mission of that institution, overlap in the general areas of Oral Communication, Written Communication, Analytical Skills, Problem Solving or Critical thinking, and Social responsibility.

Alverno CollegeThe specific abilities identified by our faculty as central to our approach to liberal arts and professional education are:

Communication Analysis Problem Solving Valuing in Decision-Making Social Interaction Developing a Global Perspective Effective Citizenship Aesthetic Engagement

Alverno's unique emphasis on learning the abilities needed to put knowledge to use -- commonly called "ability-based education" -- has gained national praise.

University of North Carolina Analytical Skills Oral Communication Skills Written Communication Skills Teamwork and Project Management Skills Information Competency Technology Skills Self-Learning Skills Ethics and Professionalism Systems Thinking

Creative Problem Solving Ability skills.

UNC has a useful link to many assessment resources

St Scholastica Problem Solving Value-based decision making Social Responsibility Effective Communication Disciplinary Understanding Aesthetic Response Living with Diversity 

Babson College Rhetoric – ability to communicate Numeracy – Effectiveness in quantitative work Ethics and Social responsibility International and Multicultural Perspective Leadership/Teamwork/Creativity

Hutchinson Community College Demonstrate the ability to think and make

reasonable judgments by acquiring, analyzing, combining, and evaluating quantitative and non-quantitative information.

Demonstrate the skills necessary to access and manipulate information through various technological and traditional methods.

Demonstrate effective communication through writing and speaking.

Demonstrate effective interpersonal 

South Seattle Community College Communication Computation Human Relations Critical Thinking and Problem Solving Technology Personal Responsibility Information Literacy

 

California State University NorthridgeStudents will be able to do many or all of the following:

California State University San Bernardino Appreciation of the Diversity of Cultures and

11

interpret and critically assess different texts, including those that reflect multicultural images and perspectives, making connections and comparing ideas among these texts;

use clear, focused, unified, and coherent organization;

use the conventions of Edited American English (for example, with respect to syntax, mechanics, grammar and diction);

use logical support—including informed opinion and fact, as well as their interpretations—to develop ideas, avoiding fallacies, biased language, and inappropriate tone;

use multiple drafts to develop various purposes, styles, and voices for specific audiences;

incorporate ideas derived from a variety of sources (for example, the World Wide Web, library resources, and interviews) and document them correctly;

write in a variety of modes (e.g., autobiography, reports, editorial, case study, inquiry, and research) with complexity and appropriate detail;

use various computer applications such as word processing, email, and HyperNews to facilitate writing.

Intellectual Points of View Understanding of Ethical Issues and

Responsibilities Ability to Work Effectively with Others Skills Useful to Function Effectively as a

Professional in Their Fields Skills Consistent with Continuing

Development and Learning in Their Field Skills Necessary to Communication Effectively

with Others Meaningful Experiences Involving the

University Community or the Community outside the     University

Mastery of Higher-order Objectives (i.e. Problem Solving Skills) in his/her field

Growth and Mastery in Subject Matter Knowledge of Discipline

Awareness of and Knowledge about Impact and Effect of Computer Technology on Society

Professional Skill necessary to promote social and Economic Justice and Combat Institutional Discrimination

Appreciation of Values of Democracy, Equal Opportunity, Work Ethic, Ongoing Personal Growth and Renewal, and Satisfactions derived from Personal Success

University of Illinois at Urbana-Champaign Write effectively Speak effectively Understand and apply scientific principles and

methods Understand and apply mathematical reasoning Think critically. Locate, screen, and organize

information. Draw conclusions after weighing evidence, facts, and ideas

Create original ideas and/or products Effectively use technology Understand and appreciate cultural and ethnic

differences. Get along with people whose attitudes and opinions are different from theirs

Work cooperatively in groups Work in a global/societal context Understand values and ethical standards Learn independently.

California State University Monterey Bay Community Participation Creative/Artistic Expression Culture and Equity Democratic Participation English Communication Ethics Language Literature/Popular Culture M athematics Communication Science Technology/Information United States Histories Vibrancy

(Each term is linked to explanations.)

 Lane Community CollegeCore Ability Outcomes —students completing general education will:

Communicate effectively

Austin Community College General Education Possess sufficient literacy skills in writing,

reading, speaking, and listening to communicate effectively above the 12th grade

12

Think critically and solve problems effectively Increase understanding of the relationship

between self and community, including self-awareness, personal responsibility, and the development of cultural competence

Explore academic disciplines

 

level. Understand numerical data and their

implications for daily living Possess consciousness of our society. Think

and analyze at a critical level Appreciate multi-cultural, multi-ethnic

contributions to our country Understand our technological society Possess basic skills in the use of computers.

San Joaquin Delta College

GENERAL EDUCATION OUTCOMES.

Speak and Listen Effectively Critical Thinking Access, Analyze and Use Information Citizenship Interact Productively with Others Diversity Quantitative Literacy Wellness Aesthetics Technology International/Intercultural

 

Louisiana State University

Statements of Expected Student Learning Outcomes:

Specifically, these courses are designed to produce students who have developed:

an effective command of written and spoken English;

an informed appreciation of the roles of the arts and the humanities;

a familiarity with the nature and function of the social sciences;

an appreciation of the methods of critical inquiry;

an ability to deal with moral and ethical issues;

a rational basis for selecting a vocation;

an understanding of other cultures and other times; and

a comprehension of how knowledge is acquired and applied.

Notice the similarity, yet the unique nature, amongst these various institution’s general education areas. Reviewing these sample general education areas in light of the ACCJC-WASC standards provides good guidelines for creating General Education Outcomes. Remember, courses become part of the General Education Programs based upon the alignment of course and GE outcomes.

13

Assessment Methods for General Education Programs

Assign or create a committee to spearhead General Education AssessmentNichols (1995)  reported that every successful assessment program had established assessment committees – but the roles and levels of authority were highly variable depending upon the campus.

Use broad interdisciplinary representation on the committee.Palomba & Banta (1999) described successful general education program assessment as led by strong interdisciplinary committees that cooperatively assessed, interpreted, and recommended action for improvement.

 Emphasize the importance and scope of General Education assessment,

but try not to place blame in any one department.General Education Assessment is crucial because all programs require a core of general education courses (Palomba & Banta, 1999; Tickle, 1999). Gail Mee of Mesa college in Arizona reported that it was important not to blame the English department when written communication assessment was lower than expected. General Education outcomes are taught by some classes but must be reinforced in all General Education curriculum.

Consider locally developed assessment tools.Locally developed assessment tools created by interdisciplinary general education assessment committees were very common, however the similarity stopped there. Case studies displayed a compilation of highly individualized assessment strategies that were reflective of campus culture and values (Banta, Lund, Black, & Oblander, 1996; Nichols, 1995; Tickle, 1999). Developing tools locally required extra time and effort from faculty generally necessitating stipends or workload adjustments.

Evaluate commercially developed standardized tests.There are some commercial standardized assessment tools for limited aspects of general education, such as critical thinking and math competency. The variety of goals for accessing General Education made standardization difficult. Another issue involves cost.

Define the assessment goals and pilot test the tool.Whether locally developed or standardized tests were used, it was important to pilot the test, and determine how the data would be used and who would interpret and act on the data.

Apply good assessment principles.General education case studies invested time to validity and reliability.

Make General Education assessment manageable.  1. SAMPLE, SAMPLE, SAMPLE! Don't attempt to test all students. Determine a representative sample.2. Don't assess all General Education outcomes every year. Select one or two to

14

concentrate in each year.3. Consider using course embedded assessment to guarantee quality student involvement.

15

Program Assessment Tools - Direct Standardized Options

Refer to Section 4 to review the types of assessment tools and the importance of validity and reliability when selecting a tool. Program assessment tools are used to compare outcomes over a series of courses and a variety of course sections. Many programs consider standardized assessment tools to assess cognitive skills, thereby decreasing the time involved in creating a tool and achieving agreement. Standardized tools allow programs to compare themselves across institutions but are sometimes limited to cognitive skills and require close examination for content validity. Many commercial sources for these tools exist including professional organizations and the Educational Testing Service.

One useful standardized technique than can be applied with minimal additional cost and effort is to retest a sample set of students using the pre-collegiate assessment tools or placement exams (such as math, English, or General Education). Be sure that using an exam truly tests the information relevant to the SLOs and that the data provides information that can be used to improve learning.

Vocational programs often have the advantage of standardized licensing exams as terminal assessments. These exams are excellent sources of assessment; however translating the results into useful modifications for improving learning is a challenge.

In addition to the terminal licensing exam, Bakersfield College Nursing program uses an assessment test for entering students to assess weaknesses and determine areas requiring tutoring. The exam includes math, chemistry, anatomy, physiology, microbiology, and other relevant pre-requisite knowledge. The biology department hoped to use this standardized testing data as a summative assessment for the pre-allied health students. Upon examination we discovered the exam only asked 3 or 4 questions in each topic area; students received scores of 33%, 66% or 100% in some areas. This information was too general and too dependent upon content and question construction to be useful to the biology department needs, but provides excellent baseline information for the nursing program.

Homegrown assessment tools can be designed for program assessment in the same way that it can be developed for courses. Review to section 4 of these materials for a review of sample tools, as well as advantages and disadvantages. Appendix F contains a sample assessment tool checklist.

16

Program Assessment Tools- Indirect Program AssessmentBest practices in assessment indicate that multiple methods and a variety of data should be built in to the assessment plan. Direct measures provide excellent information about the learned curriculum, coupling information from indirect assessment data often provides information on the effect of those changes and how well students are able to integrate and apply information outside of the classroom.Methods of program assessment include:

External reviewers - sharing information and artifacts with people in other institutions, professional organizations, or advisory committees.

Student Satisfaction Surveys

Exit Interviews

Alumni Surveys

Employer Surveys

Graduate School Records

Data on awards granted, retention, success and other overall program data

Samples of Locally-Developed Program Assessment ToolsProgram assessment provides a unique opportunity to assess learning over time, integrated learning. For this reason many programs use embedded course assessment, portfolios, performance assessments, capstone or senior projects, and capstone courses to assess program outcomes. Well-articulated SLOs will suggest a form of assessment that closely approximates real-life experiences. While development of homegrown tools can be time intensive, the dialogue and customized feedback are invaluable to improving programs. In programs it is important to check the assessment tool out using sample student artifacts, use trial populations to check the tool and the feasibility of its administration. Review the assessment tool on an annual basis. (Use the assessment tool checklist as a guide.) The sample program assessment methods below have been used at a number of institutions successfully.

Embedded Course Questions or ProblemsSeveral institutions have reported successful use of embedded questions to assess program outcomes across a number of sections. This entails cooperation to develop valid and reliable questions or problems relevant to the program SLOs that are then embedded within context of routine course assessment throughout the program. There are several advantages to this technique: assessments are relevant to the specific course, program, and institutional goals, data collection does not require extra time for students or faculty, student buy-in is greater because the assessment is part of the course work, and immediate formative feedback provides diagnostic improvement.

17

PortfoliosPortfolios were developed based upon the art portfolio model that displays the student's abilities through a collection of artifacts. Many institutions use portfolio projects to provide a unique opportunity to assess development and change over time. Portfolios benefits student metacognitive growth and result in a resume-like product which students can use beyond their schooling. Difficulties include managing the variability between portfolios, storing the physical products, and assessing the work. Some institutions use electronic student portfolios that are commercially available (see links to the right). Assessing the portfolio work is a challenge, requiring detailed rubrics, norming, and time outside of the normal faculty workload. Instructions to the students must be explicit, based upon the purpose and uses of the portfolio.

Performance AssessmentAssessment of student performance provides a unique opportunity to assess skills and abilities in a real-time situation. While performance assessment appears a natural tool for fine arts, it has also been used in the humanities in the form of debates or re-enactments. "High-quality performance as a goal, whether at the course or program level can make the curriculum more transparent, coherent, and meaningful for faculty and students alike. Clarity and meaningfulness, in turn, can be powerful motivators for both faculty and students, particularly if the performance is a public one. And public performances provide models for other students" (Wright, 1999). Performance assessments, like portfolios, require well-designed instruments, criteria, rubrics, and norming between reviewers.

Capstone ProjectsMany institutions have developed senior projects to assess the integrated skills, knowledge, and abilities of students in programs over a program of study. A variety of sample senior projects (capstones) are linked in the resources section. These may be individual or team projects. The advantage of this kind of assessment is that it can be developed to exemplify authentic working conditions. Some programs use outside evaluators to help assess the student work.

Capstone CoursesSome institutions have developed capstone courses for programs which integrate an entire sequence of study. Capstone courses, where the course itself is an assessment instrument, provide unique and challenging opportunities for students to integrate and demonstrate their knowledge, skills, and abilities. Capstone courses provide ample and focused formative time to synthesize and cement specific skills and competencies. Capstone courses are a significant learning experience as well as a powerful assessment tool.

Student Self-AssessmentStudent self-assessment can provide powerful information that can not be accomplished by any other means of assessment. Student self-assessment can provide insight into affective development and metacognitive growth that other assessment can not. The goal of the self-assessment and the rubric to evaluate the self assessment should be explicit. It is wise to ask the students to provide evidence of any conclusions they make; this may include artifacts to support these conclusions.

18

Aligning Courses to Program SLOs

In the same way that we created a matrix in section 5 to evaluate our course activities with regards to SLOs, it is helpful to create a course matrix for the program SLOs.

After writing the Program SLOs, an analysis of where those SLOs are formatively (F) and summatively (S) addressed are plotted in a matrix.

Pre-Allied Health Program SLOs Matrix Course SLO1 SLO2  SLO3 SLO4 SLO5Math D          Chemistry 11 F   F F FBiology 14 F     F FBiology 15   S F S  Biology 16 S   S    SMedical Terminology

F       F

Examining SLOs using a matrix ensures that students have been introduced to the outcome, had formative feedback, and are summatively assessed concerning successful student learning. This process is somewhat more complicated when looking at GE outcomes across many disciplines, but essential.

Can you identify potential problems inherent in this matrix?

Comments:1. Although math is a requirement for this pathway, and necessary for chemistry and biology, the material is either not necessary or not relevant to any SLO for the program. This does not necessarily mean that math does not belong in the program, but the content should be reassessed. Perhaps students could demonstrate a math competency without taking the course. Perhaps the program needs to look at the prerequisite rationale and incorporate more aspects requiring math skills into the other courses. The linkage of the program pre-requisites need to be re-examined in light of the SLOs. If math is a necessary aspect of the pre-allied health skills, then the SLOs may need to be revised.

2. SLO 2 is never formatively assessed. If the students are not given an opportunity develop this outcome with feedback to improve, then it may not be the outcome of THIS program.

3. SLO 1 and SLO 5 have an odd sequence for assessment if these courses are in the typical order in which they are taken by students. It is useless to formatively assess a student once the final or summative assessment has occurred. The program should examine the sequence of courses to determine if Medical Terminology belongs earlier in the sequence.

4. Biology 16 appears to summatively assess several of the SLOs. The department may want to consider the creation of a capstone course, or capstone project as a program assessment technique.

19

Program Review and Program Outcomes Assessment

Typically the program review process is a periodic evaluation involving an intensive self-study. Components of program review may include:

documentation of the program activities requests for additional funding

assessing present and projected staffing needs

describing facilities

cataloging equipment

verification of the program's values and effectiveness

accounting how the program supports the overall institutional missions and goals.

Program review is an excellent reflective process that easily lends itself to the outcomes reporting process. In fact, some institutions have noted significant redundancies in program review, outcomes assessment reporting and periodic accreditation reviews. Thoroughly reviewing these process and aligning the reporting for each of these processes provides a considerable time savings, increases the efficiency of the reporting, and the effectiveness of the processes contributing to each final product.

Program review provides an opportunity to align courses with program outcomes and overall program outcomes with the accreditation standards. Several institutions have revised these governance processes to reduce the additional work to faculty and administration. Kennesaw State, Riverside Community College, and Bakersfield College, among others, have been revising their program review processes to streamline accreditation, program review and outcomes assessment reporting. Drafts of their work are linked at http://online.bakersfieldcollege.edu/courseassessment/Section_6_Program%20Assessment/Section6_9ProgramReview.htm

Aligning these three processes will determine what the outcomes reports look like. The next page provides examples of program assessment reports.

20

Sample Program Assessment Reports

Data from program assessment should be aggregated to protect individual student privacy, individual faculty member's privacy, and individual course sections. The aggregated information is used when preparing unit, department, and institutional assessment reports. Use of the data must be clearly explained. (Will it effect grades, program completion, certificate or degree awards, or only be used for information?)

Program Assessment Reports are most useful when they are aligned with and incorporated into reports that can be integrated into program review and accreditation reports. A program assessment report should include aggregated data and be condensed to minimally include the essential elements shown in the template below.

Sample Program Assessment Report Template

Title of Program:

Program Mission or Goal:

Student Learning Outcomes:

Measurement Methods and Processes:

Data Summary (overview):

Use of Results (in decision making, curricular changes, etc.):

Timeline for Program Modifications or Response to Data:

Appendixes for supporting data

Samples of model reports can be found at :Courtland State University in the SUNY System http://www.cortland.edu/oir/assmtreports.htmlUniversity of Washington Program Assessment Reports http://www.washington.edu/oea/BackList

21

While some of these reports have varying length depending on the amount of supportive data included as evidence and the space required to write the SLOs, the reports are executive summaries of less than one page to a maximum of two pages in length excluding appendixes and Student Learning Outcomes statements. Also see Southern Missouri State University’s Busy Chairpersons Guide for Departmental Assessment Reporting at the SMSU website http://www2.semo.edu/provost/assmt/1Some institutions are using a simple table to report on each SLO as shown below.

Program Assessment Report        Department:Program SLO

Method of Assessment

Summary Data

Interpretation of Results

Positive Findings

Areas for Improvement Timeline

SLO1            SLO2             etc            

Ultimately the overall report should be focused and indicate evidenced based modifications subsequent to the assessment. Program assessment, like course assessment, should focus on 1 or 2 SLOs per year. It is better to close the loop on one SLO that to partially complete assessment and reporting for each SLO.

After completing this section the participants should be able to:

Create a set of DRAFT SLOs for a program through faculty dialogue.

Plan for and implement assessment for one or two program SLOs each year.  

Take a moment to consider and write down aspects of your campus program review process and accreditation preparation that might align with program assessment.

List any people that could begin the discussion to streamline these processes.

Please take a moment to complete the survey on the next page and provide feedback to improve this training process.

22

Assessing Student Learning in Community Colleges Evaluation1. Check each of the following sections you have completed

Section 1 Purpose and Introduction

Section 2 Background and Rationale for Assessment

Section 3 Student Learning Outcomes (SLOS)

Section 4 Assessment Tools and Data

Section 5 Course Assessment

Section 6 Program Assessment

Section 7 Closing the Loop

Section 8 Implementing Assessment Training on Campus

2. Please rate the following aspects of the website design and presentation.

Excellent Good Average Poor

a. Ease of Navigation

b. Time for Website to Load

c. Ease in Downloading Word Documents

d. Links to Resources

e. Mechanics of Surveys

3. Please evaluate the following aspects of the course content in relation to equipping you to do assessment.

Very Useful

Somewhat Useful

A waste of time

Not applicable to completed sections

a. Course outcomes

b. Course organization

c. Course content

d. Course activities

e. Resources

f. Quizzes and surveys

g. Hands-on assignments

h. Downloadable documents

23

How well did the materials in this course address: Thoroughly Adequately Inadequately Not Applicable to completed sections

a. Basic assessment terminology

b. Faculty roles and responsibilities in assessment

c. Benefits of assessment

d. The learning paradigm and new learning strategies

e. Types of assessment tools

f. Guidelines and good practices in assessment

g. Faculty concerns about assessment

h. The importance of measurable SLOs

i. The assessment loop

j. The importance of dialogue

k. Assessment of deep learning

Rate your experience and knowledge concerning the following prior to doing the course material and then after completing the course material. If you have not completed a section simply rate your experience and knowledge prior to doing the course material and select NA for not applicable after the course.

Expert Experienced Beginner Novice Totally Unfamiliar

NA (not applicable)

4. Student Learning Outcomes (before course)Student Learning Outcomes (after course)

5. Course Assessment(before course)Course Assessment(after course)

6. Program Assessment (before course)Program Assessment (after course)

7. Ability to mentor new faculty on assessment (before course)Ability to mentor new faculty on assessment (after course)

I would recommend this course to a friend Yes

No

Not sure

Comments:

24

Resources for Section 6Allen, M.J. (2004). Assessing academic programs. Bolton, MA: Anker Publishing

Bers, T. (n.d.) Assessment at the Program Level. California Assessment Website at http://cai.cc.ca.us/workshops/Prog Level Assessment by Bers.doc 

Creel, D.W. (n.d.). Northern Virgina Community College General Education Assessment http://www.nv.cc.va.us/assessment/VAG Gen Ed/VAG Gen Ed.PPT

Educational Testing Services at http://www.ets.org/ for standardized testing.

Erwin, T.D. (2000). The NPEC sourcebook on assessment, volume 1: Definitions and Assessment methods for critical thinking, problem-solving, and writing. Download document from http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2000195

Moskal, B.M. & Blake, B.B. (2000). Developing a departmental Assessment plan: Issues and concerns. In The department chair 11(1). Bolton, MA: Anker Publishing. Also available online at http://www.acenet.edu/resources/chairs/docs/Moskal_and_Bath.pdf

Southern Missouri State University. Busy Chairpersons Guide for Departmental Assessment Reporting at the SMSU website http://www2.semo.edu/provost/assmt/1

Upcraft, M.L. & Schuh, J.H. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass.

Student Services Program Assessment ResourcesCollege Student Experiences Questionnairehttp://www.iu.edu/~cseq

Community College Survey of Student Engagement http://www.ccsse.org/

Educational Testing Service (ETS)http://www.ets.org

National Center for Higher Education Management Systems (NCHEMS)http://www.nchems.org

National Survey of Student Engagement (NSSE)http://www.indiana.edu/~nsse

Noel-Levitzhttp://www.noellevitz.com

The Student Learning Imperative: Implications for Student Affairs  http://www.acpa.nche.edu/sli/sli.htm

25

Appendix A

Student Learning Outcome Checklist Yes No

Do the SLOs include active verbs?

Do the SLOs suggest or identify an assessment?

Do the SLOs address the expected level of learning for the course using Bloom’s Taxonomy as a guideline?

Do the SLOs address more than one domain (cognitive, psychomotor, and affective)?

Are the SLOs written as outcomes rather than as objectives? Language indicates an important overarching concept versus

small lesson or chapter objectives. Outcomes address what a student will be able to do at the

completion of the course. SLOs address student competency rather than content

coverage.

Are the SLOs appropriate for the course? Consistent with the curriculum document of record Represents a fundamental result of the course Aligns with other courses in a sequence, if applicable Represents collegiate level work

Will students understand the SLOs?

26

Comments or suggestions:

27

Appendix B - Writing a Mission Statement for a Community College Program

A mission statement explains what the program goal or goals are in a sentence or two. It is a simple statement that encapsulates the direction or purpose of the program.

Getting started:These statements are best written with representatives from as many program participants as possible. Students, staff, faculty, and deans should meet to create this statement.

Step 1:Brainstorm the activities of the program. List what the program does and how it does things.What makes the program unique? Why does this program exist? What are the defining characteristics of this program?

Step 2:Begin by writing:who you are what you dofor whomand why

For example:DRAFT Mission statement for the Pre-allied Health Biology program

The pre-allied health biology program prepares students for vocational health programs by providing rigorous courses and hands-on experiences in human biology.

Step 3: Take this statement and modify it by asking questions.Does this represent us?Is this really what we do?Is this all we do?Does this include our unique features?Does this include the aspects that make us successful?Does everyone involved in the program understand the statement?Does the statement suggest a vision and mission for the future?

Step 4:Compare your statement to other mission statements on line or other departments on your campus and modify again. Refine the statement so that it is clear and succinct.

The pre-allied health biology program prepares students for vocational health programs through rigorous coursework, hands-on lab experiences, and relevant field trips in human biology. Students learn to seek out and apply the appropriate information, think critically, develop communication skills, and value others.

Modify the statement often so that it represents your program and the people in it.

28

Appendix C  Links and Summaries of Sample General Education Assessment Tools

Mesa Community College General Education Assessment - http://www.mc.maricopa.edu/organizations/employee/orp/assessment/FAQ.htmlThe Mesa Community College (MCC) assessment program is faculty driven, owned and operated. The faculty created learning outcomes for general education, career and technical programs, and basic skills and then developed homegrown assessment tools to assess the outcomes. MCC has set aside one week annually to conduct campus-wide assessment. For general education, the performance of a sample of students who are just beginning their general education is compared to that of a sample who have completed at least 30 hours of general education. For career and technical programs, students completing specific programs of study leading to AA/AS degrees or certificates are assessed. These students participate in workplace skills assessments and also may be assessed on program-specific outcomes at the end of their programs as part of the program review process. Developmental education students in English, mathematics, and reading are assessed when they complete the final course in a developmental sequence (i.e., below 100-level courses). Assessments are conducted annually during a spring Assessment Week.Oklahoma State University General Education Assessment - http://www.okstate.edu/assess/OSU created specific General Education Learning Outcomes.

The following are the primary learner goals for general education. General education courses must identify one or more of these as learner goals in the course: 1. Students will communicate original thought in written composition, speech, and graphic representations.2. Students will identify, evaluate critically, and seek solutions to complex problems.3. Students will use the tools of mathematics and physical or biological sciences to solve problems and interpret the results.4. Students will have knowledge of the relationship between historic and contemporary issues and will understand contemporary issues with sensitivity to a rapidly changing, diverse, and complex world.5. Students will have an understanding of the beliefs that guide human actions and their consequences.6. Students will have an understanding of how the content knowledge from general education courses applies to the practice of their discipline and to other disciplines, to society, and to their own lives.In 2001-2002 the campus assessed outcomes 1 (communication) and outcome 3 (analytical problem solving). Artifacts were collected from a variety of class and the writing was scored by 6 faculty using a rubric. The data revealed that students writing improved over the four years at the university and that native students out-performed transfer students. Three faculty developed a rubric and evaluated math artifacts that were collected from a variety of classes.

Pellissippi State Technical College -http://www.pstcc.edu/departments/coe/assess-ed-outcomes.html

PSTCC uses Angelo and Cross Classroom Assessment Techniques to assess General Education, however this appears to be done on the individual class level and not at the institutional or department level.

29

Appendix D Linking General Education SLOs to Courses

Bakersfield College has not defined GE course SLOs yet, although the conversation has been initiated. However, in anticipation of these GE SLOs I have selected common outcomes found on a majority of campuses and built-in SLOS reflecting these outcomes and anticipating methods of embedded assessment.

General Education Outcomes for Microbiology B16

These general education outcomes are assessed throughout the course using various projects, case studies, lab assignments, quizzes, and test questions.

Area OutcomesUpon completing the BC Microbiology Course the student will be able to:

Analytical SkillsApply the metric system using standard laboratory equipment.  Systematically collect, organize, and present appropriate data in graphs, tables, or figures.  Assess the validity of the data and interpret it correctly.

Ethics and Professionalism Integrate knowledge and make informed judgments based upon sound assessment of data balanced with concern for individuals.

Information Competency  and Technology Skills

Access and evaluate microbiological data from a variety of sources including the Internet.  Use technology to email, produce documents and create graphs and tables for reports.

Oral and Written Communication Skills

Exhibit oral and written communication competency through case studies, projects, and lab presentations.

Self-Learning Skills Describe the preferred learning style and strategies to improve learning.  Assess their own learning and list steps to improve their learning.

Systems Thinking and Creative Problem Solving Ability

Evaluate and analyze simulated and real patients by developing a differential diagnoses, identifying key factors, and determining a treatment strategy.

Teamwork and Project Management Skills

Effectively work in teams, managing time and tasks, sharing results and analyses, and arriving at a final collaborative product.

30

Appendix E General Education Outcomes WorksheetReview the WASC areas. Combine areas and rename them relevant to your institutional goals and culture (e.g. Some people combine oral and written communication.)Following the completion of general education (also called liberal studies or interdisciplinary studies) program students will be able to:

ACCJC-WASCList

What does this look like, or how is this skill defined at your institution?

Create an outcomes describing an observable knowledge, skill, or ability assessable student be able to do that

How might this be assessed?

Life long learning skills

Oral Communication

Written Communication

Computer LiteracyScientific ReasoningQuantitative ReasoningCritical Analysis

Ethics

Effective Citizenry

Personal Communication SkillsAppreciation of DiversityHistoric SensitivityAesthetic SensitivityCivic ResponsibilityGE Outcome unique to your institution.

31

Appendix G Student Services Assessment

Sample SLOsObjective: To facilitate opportunities for the academic, personal, and social development of students. Outcomes:

1. By the time they graduate, students will be satisfied with the student counseling center.  

2. By the time they graduate, most students will have participated in one student organization or club.  

3. By the time they graduate, students will recognize diversity amongst the student population and identify and provide support for needs of different student groups.  

4. By the time they graduate, students will demonstrate social awareness and historical perspective to thrive in a changing world.

Sample Student Services SLO from Jerry Rudman's presentation (used with his permission)http://cai.cc.ca.us/SLOworkshops/Strand3/1

32

     Appendix F Assessment Tool Checklist1. Does the assessment adequately evaluate academic performance relevant to the

desired outcome? (validity)

2. Does this assessment tool enable students with different learning styles or abilities to show you what they have learned and what they can do?

3. Does the content examined by the assessment align with the content from the course? (Content validity)

4. Does this assessment method adequately address the knowledge, skills, abilities, behavior, and values associated with the intended outcome? (Domain validity)

5. Will the assessment provide information at a level appropriate to the outcome? (Bloom’s)

6. Will the data accurately represent what the student can do in an authentic or real life situation? (Authentic assessment)

7. Is the grading scheme consistent; would a student receive the same grade for the same work on multiple evaluations? (Reliability)

8. Can multiple people use the scoring mechanism and come up with the same general score? (Reliability)

9. Does the assessment provide data that is specific enough for the desired outcomes? (alignment with SLO)

10. Is the assessment summative or formative - if formative does it generate diagnostic feedback to improve learning?

11. Is the assessment summative or formative - if summative, is the final evaluation built upon multiple sources of data? (AAHE Good practice)

12. If this is a summative assessment, have the students had ample opportunity for formative feedback and practice displaying what they know and can do?

13. Is the assessment unbiased or value-neutral, minimizing an attempt to give desirable responses and reducing any cultural misinterpretations?

14. Are the intended uses for the assessment clear? (Grading, program review, both)

15. Have other faculty provided feedback?

16. Has the assessment been pilot-tested?

17. Has the evaluation instrument been normed?

18. Will the information derived from the assessment help to improve teaching and learning? (AAHE Good Practice)

19. Will you provide the students with a copy of the rubric or assignment grading criteria?

20. Will you provide the students examples of model work?

33

34