1

Core Assessment: Core Assessment: Attributes of a Successful Program Attributes of a Successful Program

at James Madison Universityat James Madison UniversityTAIR TAIR

Professional Development Professional Development WorkshopWorkshop

September 2004September 2004

Dr. Dena PastorDr. Dena Pastor

Center for Assessment and Research StudiesCenter for Assessment and Research Studies

2

James Madison University (JMU)James Madison University (JMU)

Location: Harrisonburg, Location: Harrisonburg, VAVA

Students: Students: ~15,000 undergraduate~15,000 undergraduate ~700 graduate~700 graduate

State-funded university State-funded university operated by its own operated by its own board of visitorsboard of visitors

3

JMU Infrastructure for AssessmentJMU Infrastructure for Assessment

Center for Assessment and Research Center for Assessment and Research Studies (CARS)Studies (CARS) Supports JMU assessment endeavorsSupports JMU assessment endeavors

General EducationGeneral Education Programs/MajorsPrograms/Majors Student AffairsStudent Affairs AlumniAlumni

99 full-time faculty members & 3 full-time faculty members & 3 administrative staffadministrative staff Faculty split their time evenly between Faculty split their time evenly between

assessment duties & teaching/researchassessment duties & teaching/research Expertise in IRT, SEM, Validity, Computer-Expertise in IRT, SEM, Validity, Computer-

Based Testing, HLM, Examinee Motivation, Based Testing, HLM, Examinee Motivation, Performance Assessment, Student Performance Assessment, Student DevelopmentDevelopment

Offers both M.A. and Ph.D. programs in Offers both M.A. and Ph.D. programs in Assessment & MeasurementAssessment & Measurement 10-15 students10-15 students

4

JMU Infrastructure for JMU Infrastructure for General EducationGeneral Education

Dean of General EducationDean of General Education Cluster CoordinatorsCluster Coordinators

Faculty given release time for this taskFaculty given release time for this task Steering CommitteeSteering Committee

Broad faculty representationBroad faculty representation Positions allocated to departments to teach Positions allocated to departments to teach

General Education and major coursesGeneral Education and major courses Each cluster undergoes a review process Each cluster undergoes a review process

every 5 yearsevery 5 years

5

General Education at JMUGeneral Education at JMU 41 credit hours of courses organized into 5 41 credit hours of courses organized into 5

clustersclustersCluster 1: 9 credits

Skills for the 21st CenturyCluster 2: 9 credits

Arts & HumanitiesCluster 3: 10 credits

The Natural WorldCluster 4: 7 credits

Social & Cultural Processes

Cluster 5: 6 credits

Individuals in the Human Community

Critical Thinking, Written & Oral Communication, Information Literacy

Culture, Philosophy, Fine Arts & Literature.

Quantitative & Scientific Reasoning

Global & American: History, Govt., Economics, Anthropology

Wellness, Psychology & Sociology

General Education

6

General Education General Education Course OfferingsCourse Offerings

GHTH 100: Personal Wellness

GKIN 100: Lifetime Fitness and Wellness

GEIC 101: The Wellness Dimension:Individual Perspectives

GPSYC 101: General Psychology

GPSYC 160: Life Span Human Development

GSOCI 240: Individual in Society

GEIC 102: The Sociocultural Dimension:Community Perspectives

Wellness: Choose one

Sociocultural: Choose one

Example from Cluster 5’s Wellness & Sociocultural Domains

7

Assessment Steps

EstablishObjectives

Use Information

Select/Design

Instruments

Collect Informatio

n

Analyze Informatio

n

MaintainInformatio

n

8

Cluster Learning ObjectivesCluster Learning Objectives

Created by cluster faculty with Created by cluster faculty with assistance from CARSassistance from CARS

Start by asking:Start by asking: ““What should a student What should a student knowknow and and be be

able to doable to do as a result of completing their as a result of completing their coursework in this cluster?”coursework in this cluster?”

What the student knows & is able to do What the student knows & is able to do should should NOTNOT be tied to be tied to specific coursesspecific courses taken within the cluster.taken within the cluster.

9

Example of Cluster Learning ObjectivesExample of Cluster Learning Objectives

After completing Cluster Five students should be able to:After completing Cluster Five students should be able to: Understand the dimensions of wellness, the Understand the dimensions of wellness, the

various factors affecting each dimension, and various factors affecting each dimension, and how dimensions are interrelated. how dimensions are interrelated.

Recognize an individual's level of health and Recognize an individual's level of health and wellness and understand how these levels wellness and understand how these levels impact quality of life. impact quality of life.

Discriminate between ethical and non-ethical Discriminate between ethical and non-ethical practices in the social/behavioral sciences. practices in the social/behavioral sciences.

Example from Cluster 5’s Wellness & Sociocultural Domains

10

Match Learning Goals With CourseworkMatch Learning Goals With Coursework Have instructors rate the extent to which the material Have instructors rate the extent to which the material

necessary to achieve each of the goals is emphasized in their necessary to achieve each of the goals is emphasized in their coursecourse

This will help ensure that students are able to achieve goals This will help ensure that students are able to achieve goals regardless of which combination of courses they takeregardless of which combination of courses they take

GHTH 100: Personal Wellness

GKIN 100: Lifetime Fitness and Wellness

GEIC 101: The Wellness Dimension:Individual Perspectives

GPSYC 101: General Psychology

GPSYC 160: Life Span Human Development

GSOCI 240: Individual in Society

GEIC 102: The Sociocultural Dimension:Community Perspectives

A student taking A student taking GHTH100 & GHTH100 & GPSYC101 should GPSYC101 should have the same have the same chance of chance of achieving goals achieving goals as student who as student who takes GKIN100 & takes GKIN100 & GSOCI240GSOCI240

11

Assessment of Learning ObjectivesAssessment of Learning Objectives

Faculty within each cluster decide Faculty within each cluster decide how to assesshow to assess the extent to which the extent to which students are mastering the cluster’s students are mastering the cluster’s learning objectiveslearning objectives

Most clusters create a Most clusters create a multiple-multiple-choice test from scratchchoice test from scratch, with , with continual improvements made on the continual improvements made on the instrumentinstrument

12

Assessment ToolsAssessment ToolsCluster 1 ISST - Information Seeking Skills Test

Basic Technology Skills Test: Word, Excel PowerpointWriting Portfolio Assessment

GCOM: Oral Communication TestCTCT: Comprehensive Test of Critical Thinking

Cluster 2 Arts & Humanities Test

Cluster 3 QRQ: Quantitative Reasoning TestNAW: Natural World/ Scientific Reasoning Test

Cluster 4 AMEX: American History & Government TestGLEX: Global Experience Test

Cluster 5 KWH: Knowledge of Wellness & HealthHWQ: Health & Wellness Questionnaire

SDA: Sociocultural Domain Assessment

Almost all assessment tools are

multiple-choice tests created by JMU faculty.

13

Matching Assessment Tools to Matching Assessment Tools to Learning ObjectivesLearning Objectives

Learning Objective # items % of testMake plausible interpretations about behavior in social contexts

5 11.90%

Identify implications of taking action regarding social/ behavioral issue

6 14.29%

Use evidence to develop and evaluate positions regarding social/ behavioral issue

13 30.95%

Discriminate between ethical and nonethical practices in the social/ behavioral sciences

12 28.57%

Identify relevant contributions of sociocultural/ psychological variables to a perspective

6 14.29%

Total # Items 42 100.00%

Example from Cluster 5’s Sociocultural Domain Assessment

14

Administration of Assessment ToolsAdministration of Assessment Tools Two institution-wide Assessment DaysTwo institution-wide Assessment Days

Fall (August):Fall (August): Incoming freshmen tested at orientation Incoming freshmen tested at orientation Spring (February):Spring (February): Students with 45-70 credits Students with 45-70 credits

(sophomore/junior) tested (sophomore/junior) tested

All undergraduate classes are cancelled on this All undergraduate classes are cancelled on this dayday

All students are required to participateAll students are required to participate course registration blocked for those who do not course registration blocked for those who do not

participateparticipate

Students are assigned to testing rooms by last 4-Students are assigned to testing rooms by last 4-digits of their id #s where a particular series of digits of their id #s where a particular series of instruments are administered instruments are administered

15

Assessment Day Assessment Day Data Collection Scheme:Data Collection Scheme:

Repeated MeasuresRepeated MeasuresFall 2002Spring 2003

Fall 2003Spring 2004

Fall 2004Spring 2005

Fall 2005Spring 2006

COHORT 1

COHORT 2

COHORT 3

Students in each Students in each cohort are tested cohort are tested twicetwice on the same on the same instrument – once as instrument – once as incoming freshmenincoming freshmen and again in the and again in the second semester of second semester of thethe sophomore sophomore year. year.

16

With this data, we can…With this data, we can…

1.1. Compare pre-test scores Compare pre-test scores to post-test scores (same to post-test scores (same students) of Cluster students) of Cluster “Completers” .“Completers” .

2.2. Compare Incoming Compare Incoming Freshmen with Sophomore Freshmen with Sophomore Students Students (different students).(different students).

3.3. Examine the Effect of Examine the Effect of Number of Courses.Number of Courses.

4.4. Correlate Scores and Correlate Scores and Course Grades.Course Grades.

Sophomores should Sophomores should outperform incoming outperform incoming freshmen.freshmen.Students who have Students who have

completed a larger # of completed a larger # of courses within a cluster courses within a cluster should outperform should outperform students who have students who have completed a smaller # of completed a smaller # of courses.courses.

There should be a There should be a positive correlation positive correlation between test scores and between test scores and the grades received by the grades received by students in the cluster students in the cluster courses.courses.

Students completing their Students completing their cluster courses should cluster courses should score higher on the post-score higher on the post-test as sophomores than test as sophomores than they did on the pre-test they did on the pre-test as incoming freshmen.as incoming freshmen.

17

1. Compare pre-test scores to post-test 1. Compare pre-test scores to post-test scores (same students) of Cluster scores (same students) of Cluster “Completers” .“Completers” .

45.3941.31

0

10

20

30

40

50

60

70

80

Fall 2002 Spring 2004

Assessment Period

AM

EX

Raw

To

tal S

core Average Score as

Incoming Freshmen

Average Score as2nd semester Sophomore

Fall 2002Spring 2003

Fall 2003Spring 2004

Example from Cluster 4’s American Example from Cluster 4’s American Experience TestExperience Test

18

2. Compare Incoming Freshmen with 2. Compare Incoming Freshmen with Sophomore Students Sophomore Students (different students).(different students).

26.8527.45

29.05 29.20 29.05

15.00

20.00

25.00

30.00

35.00

Freshman 2000 Freshman 2001 Sophomores2000

Sophomores2001

Sophomores2002

Av

era

ge

To

tal S

co

re

Sophomores Freshmen

Example from Cluster 3’s Natural World Example from Cluster 3’s Natural World AssessmentAssessment

19

3. Examine the Effect of # of Courses.3. Examine the Effect of # of Courses.

30.0529.2028.90

27.4526.10

15.00

20.00

25.00

30.00

35.00

None One Two Three Four or more

Av

era

ge

To

tal S

co

re

Example from Cluster 3’s Natural World Example from Cluster 3’s Natural World AssessmentAssessment

20

4. Correlate Scores and Course Grades.4. Correlate Scores and Course Grades.

Course 2004 2002GHIST 225 0.36 0.38GPOSC 225 0.40 0.45GANT 195 0.33 0.49GECO 200 0.23 0.27GGEO 200 0.31 0.28GPOS 200 0.59 0.75GSOC 210 0.13 0.37

Example from Cluster 4’s Global Example from Cluster 4’s Global Experience TestExperience Test

21

The GCOM test is a Computer-Based Test embedded within a course: This test serves as the final exam in all sections of the Gen. Ed. communication course.

These are both Computer-Based Tests administered throughout the school-year at a computer lab on campus. All students must exceed the passing score set by the faculty within their first year.

Assessment Information Assessment Information NotNot Collected on Assessment DayCollected on Assessment Day

Cluster 1 ISST - Information Seeking Skills TestBasic Technology Skills Test: Word, Excel, Powerpoint

GCOM: Oral Communication TestWriting Portfolio Assessment

A random sample of ~400 students taking the Gen. Ed. writing course are asked to submit a portfolio of their best work which is rated according to a 4 point rubric by the faculty.

22

Other Assessment Information Other Assessment Information Previously Collected on Previously Collected on

Assessment DayAssessment DayScale Assess the student's…AOT interests and behaviors in specific situations.AMS reasons for choosing to attend college. ATL attitudes toward learning and performance.CCTDI beliefs, expectations, and perceptions.CCAI ability to live and interact effectively in another culture.CI curiosity. EIS self-image. SID intellectual development.SSLI sense of responsibility.SDTLA activities, feelings, attitudes, aspirations, and relationships.SOS opinions about assessment day testing.WPI motivation toward coursework.

23

How does General Education How does General Education Assessment Benefit JMU Assessment Benefit JMU FacultyFaculty??

Collaboration among faculty in different Collaboration among faculty in different disciplinesdisciplines

Opportunity to communicate with one Opportunity to communicate with one anotheranother

Feel ownership over assessment tools, Feel ownership over assessment tools, more invested in the resultsmore invested in the results

Data may help support resources they Data may help support resources they would like to acquirewould like to acquire

Gain skills that they can take back into the Gain skills that they can take back into the classroomclassroom

Opportunities for scholarshipOpportunities for scholarship

24

How does General Education How does General Education Assessment Benefit JMU Assessment Benefit JMU StudentsStudents??

Receive instruction that is more Receive instruction that is more tailored to their needstailored to their needs

Receive instruction that is more Receive instruction that is more consistent across sectionsconsistent across sections

Held accountable for knowledge/skills Held accountable for knowledge/skills (ISST, Tech Level tests, GCOM final (ISST, Tech Level tests, GCOM final exam)exam)

25

How does General Education How does General Education Assessment Benefit Assessment Benefit the Universitythe University??

Enhanced public relations and visibilityEnhanced public relations and visibility Can communicate information to parents, Can communicate information to parents,

possible applicants, stakeholders and say possible applicants, stakeholders and say “this is what a students will know and be “this is what a students will know and be able to do as a result of completing the able to do as a result of completing the Gen Ed courses at JMU”Gen Ed courses at JMU”

Enhanced grant funding opportunitiesEnhanced grant funding opportunities Marketable assessment tools can be a Marketable assessment tools can be a

source of incomesource of income

26

JMU’s Marketable JMU’s Marketable Assessment ToolsAssessment Tools

Computer-Based Multiple-Choice TestsComputer-Based Multiple-Choice Tests ILT: Information Literacy Test (60 items)ILT: Information Literacy Test (60 items) QR: Quantitative Reasoning Test (45 items)QR: Quantitative Reasoning Test (45 items) SR: Scientific Reasoning Test (80 items)SR: Scientific Reasoning Test (80 items)

Marketable Tests Being DevelopedMarketable Tests Being Developed Oral CommunicationOral Communication Critical ThinkingCritical Thinking

27

What are Other Institutions Doing?What are Other Institutions Doing?

Approaches to Gen Ed assessment Approaches to Gen Ed assessment (Palomba & Banta, 1999) :(Palomba & Banta, 1999) : Individual Course-Based ApproachIndividual Course-Based Approach Multi-Course ApproachMulti-Course Approach Noncourse-Based ApproachNoncourse-Based Approach

28

Individual Course-Based ApproachIndividual Course-Based Approach

Instructor submits a document Instructor submits a document showing how goals are being showing how goals are being addressed in their particular courseaddressed in their particular course May include syllabus, quizzes, tests, May include syllabus, quizzes, tests,

assignments, activities, examples of assignments, activities, examples of students’ workstudents’ work

May be judged by other faculty for the May be judged by other faculty for the extent to which course is covering goalsextent to which course is covering goals

29

Individual Course-Based ApproachIndividual Course-Based Approach

College of William and Mary is taking College of William and Mary is taking this approachthis approach

Also using faculty and student Also using faculty and student surveyssurveys

Different Gen Ed “clusters” assessed Different Gen Ed “clusters” assessed in different yearsin different years

Office of Assessment oversees Office of Assessment oversees processprocess

http://www.wm.edu/wmoa/index.htm

30

Individual Course-Based ApproachIndividual Course-Based Approach

AdvantagesAdvantages Maximizes faculty involvementMaximizes faculty involvement Assessment results may help identify changes that need to Assessment results may help identify changes that need to

be made in a particular coursebe made in a particular course May ensure that goals are being addressed in the May ensure that goals are being addressed in the

classroomclassroom DisadvantagesDisadvantages

Can be labor intensive for facultyCan be labor intensive for faculty Doesn’t necessarily involve faculty collaborationDoesn’t necessarily involve faculty collaboration Faculty may only choose information that makes them Faculty may only choose information that makes them

“look good”“look good” Not necessarily measuring whether or not student has Not necessarily measuring whether or not student has

mastered the goal – just measuring whether or not student mastered the goal – just measuring whether or not student was exposed to material that would allow them to master was exposed to material that would allow them to master the goalthe goal

Some information from Palomba & Banta, 1999

31

Multi-Course ApproachMulti-Course Approach

Faculty from different disciplines join Faculty from different disciplines join together to decide upon assessment for together to decide upon assessment for their “cluster”their “cluster”

Winthrop University takes this approachWinthrop University takes this approach Collect assessment data via course-embedded Collect assessment data via course-embedded

assessments: common essays, tests, assessments: common essays, tests, assignments across courses that are graded assignments across courses that are graded according to a common rubricaccording to a common rubric

Also administers the National Survey of Also administers the National Survey of Student Engagement (NSSE) and the College Student Engagement (NSSE) and the College BASE to freshmen and seniors (repeated BASE to freshmen and seniors (repeated measures)measures)

Office of Assessment oversees processOffice of Assessment oversees processSome information from Palomba & Banta, 1999http://www.winthrop.edu/assessment/gened.htm

32

Multi-Course ApproachMulti-Course Approach

AdvantagesAdvantages Encourages faculty collaborationEncourages faculty collaboration Focuses on student experience in a Focuses on student experience in a

“cluster”, not in a particular course“cluster”, not in a particular course Shows evidence of student learningShows evidence of student learning

DisadvantagesDisadvantages When course-embedded, dictates what When course-embedded, dictates what

assignments/tests, etc. must be assignments/tests, etc. must be administered in a particular course, which administered in a particular course, which may be a turn-off to some instructorsmay be a turn-off to some instructors

33

Noncourse-Based ApproachNoncourse-Based Approach

Campus-wide assessment activities Campus-wide assessment activities focused on individual student or focused on individual student or groups of students, not a particular groups of students, not a particular coursecourse

Assessments administered outside Assessments administered outside classroom to all students or a sample classroom to all students or a sample or studentsor students

Assessment tools may be locally-Assessment tools may be locally-developed (like JMU) or standardized; developed (like JMU) or standardized; may include surveysmay include surveys

34

Noncourse-Based ApproachNoncourse-Based Approach AdvantagesAdvantages

Assessment removed from classroom, Assessment removed from classroom, instructors are given more freedominstructors are given more freedom

Faculty collaboration encouragedFaculty collaboration encouraged Focuses on students’ experience in a “cluster”Focuses on students’ experience in a “cluster”

DisadvantagesDisadvantages Logistics of collecting dataLogistics of collecting data Student motivation to take assessment seriouslyStudent motivation to take assessment seriously Mechanisms need to be in place:1) to ensure Mechanisms need to be in place:1) to ensure

that students are exposed to materials within that students are exposed to materials within courses that will enable them to meet courses that will enable them to meet objectives; 2) to match instrument to objectives objectives; 2) to match instrument to objectives

35

Assessment Information on InternetAssessment Information on Internet

James Madison UniversityJames Madison University http://www.jmu.edu/assessment/http://www.jmu.edu/assessment/

University of North TexasUniversity of North Texas http://www.unt.edu/vpaa/Assessment/Assessment_Home.hhttp://www.unt.edu/vpaa/Assessment/Assessment_Home.h

tmltml American Association of Higher EducationAmerican Association of Higher Education

http://www.aahe.org/http://www.aahe.org/ National Center for Postsecondary ImprovementNational Center for Postsecondary Improvement

http://www.stanford.edu/group/ncpi/index.htmlhttp://www.stanford.edu/group/ncpi/index.html NC State: NC State: Internet Resources for Higher Education

Outcomes Assessment http://www2.acs.ncsu.edu/UPA/assmt/resource.htmhttp://www2.acs.ncsu.edu/UPA/assmt/resource.htm