Program Evaluation in a Concept Based CurriculumDr. Linda Miles

Chair, School of Nursing

Daytona State College

Objectives

• Differentiate between assessment and evaluation

• Describe the process for developing a program evaluation plan

• Describe how the evaluation plan directs activities in the nursing education program

• Describe the different aspects of a program evaluation plan for a concept based curriculum

Evaluation

Effort to increase human effectiveness through systematic data-based inquiry.

Systematic examination of accomplishment and effectiveness in program and services.

Patton, M.

Assessment: focuses on learning and teaching. It provides information to make changes in the learning environment. It is a systematic process of collecting and interpreting information to base decisions (formative).

Evaluation: focuses on grades and may reflect classroom components other than course content or mastery level. It is a value judgment that assigns meaning to the data obtained through assessment (summative).

Components of Educational Instruction

P L A N N I N G

Instructional Objectives

Learning Outcomes

Instructional Activities

A S S E S S M E N T Brookline,1999

Evaluation

To improve- should provide information that can be used to determine whether or not desired outcomes are being achieved and how programs can be improved.

To inform- should inform faculty and other decision makers about relevant issues that can impact the program and student learning.

To prove- should summarize and demonstrate what the program is accomplishing and students are learning.

Should Answer These Questions:

What are you trying to do?

How well are you doing it?

Using the answers to the first two questions, how can you improve what you are doing?

How can the student learning experience be improved?

Purpose of Evaluation

• New program

• Curricular revision

• Educational initiatives

• Justification

• Quality of education

• Validation

Purpose of an Evaluation Plan

• Provides a framework for assessment− Provides a timeline for activities− Identifies responsible parties

• Guides the daily assessment activities

• Aligns assessment activities with the accrediting standards

• Documents program development and activities

Components of an Evaluation Plan

• Levels of Achievement• Responsible Party or Person• Timeframe or Frequency• Evaluation Methodologies• Data (3 years)• Results and Actions

Process for Developing an Evaluation Plan

Best practices

Full faculty involvement

Reflects current accrediting standards

Suitable for your institution and program

Systematic and ongoing

Food for Thought

Outcomes• Drive curriculum and development of evaluation

methodologies

Activity

1. What type of evaluation methodologies would you use to assess this outcome?

2. How would this be reflected in the program evaluation plan?

Program Learning Outcomes:

Nursing Process

The graduate of the associate degree nursing program will utilize the nursing process and principles of best practices to meet the needs of individuals, families, groups and communities across the lifespan.

 

Cultural Competence

The graduate of the associate degree nursing program will provide caring, spiritual and culturally sensitive nursing care to individuals, families, groups and communities across the lifespan.

Evaluation Methodology

Summative Evaluation of the Program:

How well did the associate degree nursing program prepare you for the following nursing aspects? (Likert)

Nursing Process

• Utilize the nursing process and principles of best practices to meet the needs of individuals, families, groups and communities across the lifespan.

• Evaluate the effectiveness of pharmacological interventions for clients with recurring healthcare needs.

 

Cultural Competence

• Provide caring, spiritual and culturally sensitive nursing care to individuals, families, groups and communities across the lifespan.

• Create an interdisciplinary plan of care reflective of client’s cultural, spiritual, and physical care.

 

Evaluation Methodology

Agency Evaluation of Graduates:

XX nursing graduates provide safe and technically competent nursing care to individuals, families, groups or communities across the lifespan.

XX nursing graduates demonstrate competency in caring for individuals, families, groups or communities across the lifespan.

Evaluation Methodology

Standardized Testing:

(Crosslink test plan with outcomes)

 

85% of students will achieve the 80% score reflective of the national norm

Evaluation Methodology

Clinical Evaluation Tool:

Students demonstrate competency providing nursing care to clients with recurring health care needs:

 

Nursing Process

• Applies nursing process to maximize the function of clients with chronic or recurring health needs through the lifespan.

• Evaluates client outcomes from pharmacological interventions.

 

Cultural Competence

• Performs nursing care in a manner that is respectful of client’s individuality.

• Evaluates the effectiveness of interdisciplinary plan of care with patient outcomes.

6.4 Graduates demonstrate achievement of competencies appropriate to role preparation. Nursing ProcessThe graduate of the associate degree nursing program will utilize the nursing process and principles of best practices to meet the needs of individuals, families, groups and communities across the lifespan.Cultural CompetenceThe graduate of the associate degree nursing program will provide caring, spiritual and culturally sensitive nursing care to individuals, families, groups and communities across the lifespan.

Expected Level of Achievement: 100% of graduates will demonstrate achievement of the student learning and program competencies appropriate for the role of an associate degree nurse. (Clinical Evaluation and Final Course Requirements.) 75% of students will achieve the RN Test Score Equal or Above the National Mean.

75% or greater of students will agree or strongly agree on the program summative evaluation that they attained the program learning outcomes. 75% or greater of employers will agree or strongly agree on the agency satisfaction survey that graduates attained the program learning outcomes.

Assessment Methods: Graduates will be assessed for attainment of competencies and student learning outcomes through the use of a variety of evaluation methodologies (ATI, quizzes, case studies, learning activities, clinical evaluations, course grades, and graduate surveys).

Results: 2010-2012Clinical Satisfaction: 100% of students who graduated were satisfactory in clinical.Course Satisfaction: 100% of students who graduated met all course exit requirements. Standardized TestSpring 2012 RN Test Score 70.2Fall 2011 RN Test Score 68.1%

Student Summative Evaluation: 2011-12 (Each of the calculations reflects the percentage of students by item that agreed or strongly agreed that outcomes were met by year.Nursing Process (five outcomes)84%/64%/74%/64%76%-December89%/76%/80%/80%/82%-MayCultural Competence (two outcomes)94%/93%-December81%/77%-May

Agency Satisfaction 2011-12 GraduatesNursing Process97% agreed or strongly agreed that graduates attained the outcomeCultural Competence100% agreed or strongly agreed that graduates attained the outcome

Timeframe: Fall and spring Actions:

Evaluation Plan for a Concept Based Curricular Approach

Describe evaluation methodologies

Describe the process to align evaluation methodologies with outcomes

Describe assessment measures• Formative-Summative• Quantitative-Qualitative• Triangulation

Classroom-clinical linkage

Compare and Contrast Program Evaluation by Curricular Model

Curriculum Building• Mission• Philosophy• Conceptual Framework• Outcomes

Learning Outcomes

What are Institutional Learning Outcomes (ILOs)?

The knowledge, skills, abilities, and attitudes that students are expected to develop as a result of their overall experiences with any aspect of the college, including courses, programs, and student services.

What is the purpose of ILOs?

Institutional learning outcomes (ILOs) are designed to help guide individual departments and disciplines in the development of outcomes for programs, courses and services, and to help shape the decision-making processes of the college.

How are ILOs different from Program and Course Learning Outcomes?

They are inter‐related. ILOs are the collective expression of the learning environment the college offers to any enrolled student. Program and course learning outcomes focus on the more particular skills, knowledge, and attitudes that students learn in programs and courses. 

Nine Principles of Assessment

The assessment of student learning begins with educational values.

Assessment is most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time.

Assessment works best when the programs it seeks to improve have clear, explicitly stated purposes.

Assessment requires attention to outcomes but also and equally to the experiences that lead to those outcomes.

Assessment works best when it is ongoing, not episodic.

Assessment fosters wider improvement when representatives from across the educational community are involved.

Assessment makes a difference when it begins with issues of use and illuminates questions that people really care about.

Assessment is more likely to lead to improvement when it is part of a larger set of conditions that promote change.

Through assessment, educators meet responsibilities to students and to the public. American Association for Higher Education (http://www.aahe.org/assessment/principl.htm)

Ten characteristics of an effective program to assess student academic achievement

Successful assessment flows from the institution's mission and educational purposes.

Successful assessment emerges from a conceptual framework.

Successful assessment is marked by faculty ownership and responsibility.

Successful assessment has institution-wide support.

Successful assessment relies on multiple measures.

Successful assessment provides feedback to students and the institution.

Successful assessment is cost-effective.

Successful assessment does not restrict or inhibit goals of access, equity, and diversity established by the institution.

Successful assessment leads to improvement.

Successful assessment includes a process for evaluating the assessment program. (The North Central Association Commission on Institutions of Higher Education)