1 Core Assessment: Challenges Facing New Programs TAIR Professional Development Workshop September...

11

Core Assessment: Core Assessment: Challenges Facing New ProgramsChallenges Facing New Programs

TAIR TAIR Professional Development Professional Development

WorkshopWorkshopSeptember 2004September 2004

Dr. Dena PastorDr. Dena Pastor

Center for Assessment and Research StudiesCenter for Assessment and Research Studies

2

What is Assessment?What is Assessment?Conceptualize Core Curriculum What is our intention? How will it be structured?

Create learning goals and objectives for students

Decide upon assessment methods

Collect and analyze data

Interpret data – what does this mean?

Use what was learned from assessment process to make improvements to core curriculum

3

Core CurriculumCore CurriculumAssessment in VirginiaAssessment in Virginia

►1987:1987: State Council of Higher Education in State Council of Higher Education in Virginia (SCHEV) mandated that public Virginia (SCHEV) mandated that public institutions:institutions: Define learning objectivesDefine learning objectives Create an assessment scheme Create an assessment scheme

►A report by SCHEV in 1999 notes:A report by SCHEV in 1999 notes: ““Many institutions struggled in their initial efforts to Many institutions struggled in their initial efforts to

respond.”respond.”

4

Why is Assessment of the Core Why is Assessment of the Core Curriculum So Challenging?Curriculum So Challenging?

► There are many questions that an There are many questions that an institution must answer:institution must answer:

Why assess?Why assess? What to assess?What to assess? When to assess?When to assess? Who to assess?Who to assess? How to assess?How to assess?

1.1. Stakes of assessmentStakes of assessment2.2. How to collect the assessment dataHow to collect the assessment data3.3. Assessment MethodAssessment Method4.4. Reporting of ResultsReporting of Results5.5. Who’s going to oversee core curriculum assessment?Who’s going to oversee core curriculum assessment?

5

Why assess? Why assess? (1)(1)

It is a It is a good ideagood idea to keep in mind that the state may to keep in mind that the state may want such information for the purposes of resource want such information for the purposes of resource allocation…allocation…but,but, it is a it is a better ideabetter idea to keep in mind to keep in mind that assessment can bring about improvement in that assessment can bring about improvement in the curriculum, which in turn may lead to a higher the curriculum, which in turn may lead to a higher quality education for the students.quality education for the students.

The state told me to.

To improve the quality of

my core curriculum program.

6

Assessment ToneAssessment Tone

► Assessment is often met with resistance by Assessment is often met with resistance by faculty and/or administratorsfaculty and/or administrators Make them Make them wantwant to do it, to overcome the feeling to do it, to overcome the feeling

of “I of “I havehave to do it”. to do it”. Create an “assessment culture” at the university Create an “assessment culture” at the university

that has a positive attitudethat has a positive attitude► Remind them that they…Remind them that they…

have more have more controlcontrol over the assessment process over the assessment process compared to teachers/administrators in K-12compared to teachers/administrators in K-12

can use the assessment process to provide can use the assessment process to provide themselvesthemselves with the kind of information that with the kind of information that theythey will find usefulwill find useful

7

Institutions’ Orientations Institutions’ Orientations to Student Assessmentto Student Assessment

►Proactive Proactive Institutions engaging in student Institutions engaging in student

assessment assessment prior prior to state mandateto state mandate►Retain a sense of control over implementation Retain a sense of control over implementation

and utilization of assessmentand utilization of assessment►Internal improvement remains primary goal of Internal improvement remains primary goal of

assessment, external accountability is assessment, external accountability is secondarysecondary

Based on 1997 information from National Center for Postsecondary Improvement

8


►Responsive Responsive Institutions engaging in student Institutions engaging in student

assessment assessment as a resultas a result of state mandate of state mandate►Decisions regarding implementation and Decisions regarding implementation and

utilization of assessment shared by internal utilization of assessment shared by internal and external constituentsand external constituents

►Internal improvement and external Internal improvement and external accountability are the dual goals of accountability are the dual goals of assessmentassessment


9


►Reactive Reactive Institutions engaging in Institutions engaging in minimalminimal student student

assessment assessment as a resultas a result of state mandate of state mandate►External accountability remains primary goal of External accountability remains primary goal of

assessment, internal improvement is secondary assessment, internal improvement is secondary or non-existentor non-existent

► Institutions’ orientation may be Institutions’ orientation may be predictive of institutions’ predictive of institutions’ supportsupport for for assessment & assessment & utilizationutilization of assessment of assessment informationinformation


10

State-Level State-Level Student Assessment InitiativesStudent Assessment Initiatives

►Research says that state-level Research says that state-level initiatives that initiatives that promote greater promote greater institutional support for & engagement institutional support for & engagement in student assessment in student assessment are those that:are those that: emphasize institutional improvement as emphasize institutional improvement as

their primary purpose, not external their primary purpose, not external accountabilityaccountability

set broad guidelines for student set broad guidelines for student assessment, but leave the specifics up to assessment, but leave the specifics up to the institutionthe institution


11

What to assess?What to assess?

►What to assess should be directly tied to the What to assess should be directly tied to the learning objectiveslearning objectives for the core curriculum for the core curriculum

► Learning objectives should be an answer to Learning objectives should be an answer to the question:the question: What should a student know and be able to do as What should a student know and be able to do as

a result of completing their core curriculum a result of completing their core curriculum coursework?coursework?

►Measuring # of faculty with Ph.D.’s, faculty Measuring # of faculty with Ph.D.’s, faculty to student ratio, # students completing the to student ratio, # students completing the course, etc. will not help you understand if course, etc. will not help you understand if students are meeting the learning objectivesstudents are meeting the learning objectives Need to measure skills, knowledge, behavior or Need to measure skills, knowledge, behavior or

values of the studentsvalues of the students

12

When to assess?When to assess?

► Preferable to measure a single student on Preferable to measure a single student on multiple occasionsmultiple occasions using the same using the same assessment method assessment method More is better!More is better!

► Preferable to have a Preferable to have a pre-measure pre-measure of of knowledge, skills, behavior to serve as knowledge, skills, behavior to serve as baselinebaseline What do the students know coming in?What do the students know coming in? What kind of /amount of knowledge do they gain?What kind of /amount of knowledge do they gain?

13

Who to assess?Who to assess?► Unit of analysis should be congruent with purpose of Unit of analysis should be congruent with purpose of

assessmentassessment Are you using assessment to hold student accountable for Are you using assessment to hold student accountable for

knowledge/skills? (Assess student).knowledge/skills? (Assess student). Are you using assessment to evaluate a curriculum? Are you using assessment to evaluate a curriculum?

(Aggregate student data).(Aggregate student data).

Data are collected at the student level, but results are Data are collected at the student level, but results are often only reported at the program leveloften only reported at the program level If it is helpful, results might be reported by course, semester, If it is helpful, results might be reported by course, semester,

or subgroup (e.g., males vs. females)or subgroup (e.g., males vs. females) NotNot advisable to report results by faculty member advisable to report results by faculty member

14

Who to assess?Who to assess?

►Collecting the data:Collecting the data: Preferable to assess Preferable to assess allall studentsstudents If not feasible, assess a If not feasible, assess a random samplerandom sample

(the more the better)(the more the better) Could ask for volunteers (good luck!), but Could ask for volunteers (good luck!), but

your sample will be biasedyour sample will be biased

15

1.1. Should stakes be associated with a Should stakes be associated with a students’ performance on the assessment?students’ performance on the assessment?

2.2. How, logistically, are we going to collect How, logistically, are we going to collect the assessment data?the assessment data?

3.3. What kind of method are we going to use What kind of method are we going to use for assessment?for assessment?

4.4. How should the results of assessment be How should the results of assessment be reported and who should they be reported reported and who should they be reported to?to?

5.5. Who’s going to oversee core curriculum Who’s going to oversee core curriculum assessment?assessment?

How to assess?How to assess?

16

HIGH STAKESHIGH STAKES

PROSPROS► Students will Students will

be more likely be more likely to take to take assessment assessment seriously, to seriously, to prepare and prepare and to put forth to put forth maximal maximal effort.effort.

High Stakes for the Student: Must pass in order to High Stakes for the Student: Must pass in order to graduate, register, fulfill course requirements, etc.graduate, register, fulfill course requirements, etc.

CONSCONS► Security of the assessment materials is Security of the assessment materials is

now of concern (may need more now of concern (may need more proctors, multiple test forms, secure proctors, multiple test forms, secure databases).databases).

► How to handle students who don’t How to handle students who don’t pass? Should multiple attempts be pass? Should multiple attempts be allowed?allowed?

► Legal concerns: What about Legal concerns: What about students/parents of students who want students/parents of students who want to contest a score? Are the instruments to contest a score? Are the instruments of sufficient quality to bear the weight of sufficient quality to bear the weight of the consequences?of the consequences?

17

► Requiring students to pass in Requiring students to pass in order to graduate, register, order to graduate, register, or fulfill course requirements.or fulfill course requirements.

► Reporting student’s Reporting student’s assessment scores on their assessment scores on their transcripttranscript

► Reporting assessment scores Reporting assessment scores to the students themselves to the students themselves via e-mail/e-campusvia e-mail/e-campus

►Not reporting the assessment Not reporting the assessment scores to the students at allscores to the students at all

Stakes & Examinee Stakes & Examinee MotivationMotivation

HIGH STAKES

NO STAKES

HIGH EXAMINEE MOTIVATION

QUESTIONABLE EXAMINEE MOTIVATION

18

Collecting the DataCollecting the Data

► Course-Embedded Course-Embedded AssessmentAssessment Examples: common final Examples: common final

across sections, common across sections, common portfolio of students’ workportfolio of students’ work

► Computer-Based TestComputer-Based Test Examples: Test administered Examples: Test administered

in computer lab or over the in computer lab or over the web; students have a window web; students have a window of time in which to take the of time in which to take the examexam

► Assessment DaysAssessment Days Particular days set aside for Particular days set aside for

assessment purposesassessment purposes

19

Method of AssessmentMethod of Assessment► Commercially-available test (standardized tests)Commercially-available test (standardized tests)

Do a backwards translation to ensure that content of test Do a backwards translation to ensure that content of test matches up well with learning objectivesmatches up well with learning objectives

Could be costlyCould be costly► Faculty-created tests (locally-developed) Faculty-created tests (locally-developed)

Time-consuming & challenging for faculty Time-consuming & challenging for faculty May better ensure a match between test content and May better ensure a match between test content and

learning objectiveslearning objectives Faculty may feel “ownership” over test and may be more Faculty may feel “ownership” over test and may be more

committed to revisions and enthusiastic about resultscommitted to revisions and enthusiastic about results Can use test-development process as a professional Can use test-development process as a professional

development opportunity for faculty (perhaps even pay development opportunity for faculty (perhaps even pay them or give them release time to participate)them or give them release time to participate)

If high-stakes for students, then multiple forms of test If high-stakes for students, then multiple forms of test may be necessary thus requiring more item-writing from may be necessary thus requiring more item-writing from facultyfaculty

20

Arguments for Arguments for Locally-Developed InstrumentsLocally-Developed Instruments

► Research supports that the use of locally-Research supports that the use of locally-developed instruments over state-mandated developed instruments over state-mandated standardized tests:standardized tests: Is more helpful for institutional improvementIs more helpful for institutional improvement Increases institutional support for assessment Increases institutional support for assessment

and institutional ownership over assessmentand institutional ownership over assessment► Problem: How to compare institutions to one Problem: How to compare institutions to one

another?another? But is that the goal? But is that the goal? May be better to compare each institution to its May be better to compare each institution to its

own objectives, previous performance, or own objectives, previous performance, or standardsstandards

21

Setting StandardsSetting Standards► May want to use standard setting procedures to establish “cut-offs” for proficiency on the May want to use standard setting procedures to establish “cut-offs” for proficiency on the

testtest► Could be that students are gaining knowledge/skills over time, but are they gaining Could be that students are gaining knowledge/skills over time, but are they gaining

enough?enough?

30

55

0

10

20

30

40

50

60

70

80

90

100

Incoming Freshmen Core Curriculum "Completers"

Te

st

Sc

ore

STANDARD SET BY FACULTY

22

Method of AssessmentMethod of Assessment► Performance AssessmentPerformance Assessment

Student portfolio of their work (e.g., collection of essays) Student portfolio of their work (e.g., collection of essays) that is rated by faculty according to a rubricthat is rated by faculty according to a rubric

An actual performance by the student (e.g. speech) that is An actual performance by the student (e.g. speech) that is rated by faculty according to a rubricrated by faculty according to a rubric

► Pros & ConsPros & Cons May be more authentic, but…May be more authentic, but… With a large # of students, may be costly in terms of With a large # of students, may be costly in terms of

faculty time/resourcesfaculty time/resources► Perhaps use a random samplePerhaps use a random sample

► Preferable to use multiple methodsPreferable to use multiple methods► To supplement performance assessment or test:To supplement performance assessment or test:

Surveys of faculty/studentsSurveys of faculty/students Focus groupsFocus groups

May be best when wanting to measures attitudes, values and behavior; not gains in knowledge or skills

23

Student SurveysStudent Surveys

► May ask student if they feel that their core May ask student if they feel that their core curriculum courses helped them master curriculum courses helped them master learning goals and objectiveslearning goals and objectives They may feel that they did, but wouldn’t you They may feel that they did, but wouldn’t you

rather know if they did??rather know if they did??► Or may have student self-reflect on their Or may have student self-reflect on their

learning and developmentlearning and development May be a very useful exercise for the student; May be a very useful exercise for the student;

really involves student in assessment; but self-really involves student in assessment; but self-reflections are NOT hard, unbiased datareflections are NOT hard, unbiased data

► Surveys are best used as supplementary Surveys are best used as supplementary materialmaterial

24

In Defense of In Defense of Multiple-Choice Multiple-Choice

TestsTests

► Can create multiple-choice items that are Can create multiple-choice items that are cognitively complexcognitively complex Challenging, but do-able!Challenging, but do-able! Easy to scoreEasy to score Can use item analysis to improve itemCan use item analysis to improve item Most practical with a large number of examinees Most practical with a large number of examinees Direct measure of student learningDirect measure of student learning

25

Discriminate between association and Discriminate between association and causation, causation,

and identify the types of evidence used and identify the types of evidence used to establish causationto establish causation

For the following pairs of events, determine whether:For the following pairs of events, determine whether:

a. one event caused the othera. one event caused the other

b. the two events are related, but no causation can be b. the two events are related, but no causation can be determineddetermined

c. the two events are unrelatedc. the two events are unrelated

1. A man goes to use his telephone, only to find it 1. A man goes to use his telephone, only to find it dead, and;dead, and;

The lights flicker and go out.The lights flicker and go out.

26

Reporting ResultsReporting Results

► Results should be Results should be reported reported andand discussed discussed by by allall stakeholders (e.g., faculty, students, stakeholders (e.g., faculty, students, administration, state higher education council)administration, state higher education council)

► Keep it Keep it simplesimple: descriptive : descriptive statistics/tables/graphs over inferential statistics/tables/graphs over inferential statisticsstatistics

► Keep in mind the reliability/validity of the Keep in mind the reliability/validity of the assessment tool, it’s match to learning assessment tool, it’s match to learning objectives, the timing of the assessment, objectives, the timing of the assessment, characteristics of the sample (generalizability & characteristics of the sample (generalizability & size)size)

27

Link Results to ImprovementLink Results to Improvement

AssessmeAssessment nt Technique Technique or Processor Process

Findings/Findings/ResultsResults

Use Made of Use Made of FindingsFindings

Evidence of Evidence of Effect of Effect of ImprovemenImprovementt

1. Writing 1. Writing portfolios portfolios scoredscored

Many portfolios Many portfolios contained contained materials that materials that faculty did not faculty did not think were think were appropriate for appropriate for the course; the course; assessment data assessment data indicated that indicated that such portfolios such portfolios were rated more were rated more harshlyharshly

Instructors of Instructors of the course met the course met to discuss what to discuss what were were appropriate appropriate assignments for assignments for the course and the course and what were not – what were not – ideas were ideas were shared and shared and guidelines for guidelines for assignments assignments were developedwere developed

Writing portfolio Writing portfolio scores scores increased over increased over three next three next years and fewer years and fewer portfolios were portfolios were found including found including inappropriate inappropriate materialsmaterials

28

Ways to Use ResultsWays to Use Results1.1. Continuous ImprovementContinuous Improvement

2.2. Program ReviewProgram Review• Self-studies and peer reviewSelf-studies and peer review

3.3. Planning and BudgetingPlanning and Budgeting• Resource allocation, Needs assessmentsResource allocation, Needs assessments

4.4. Teaching and LearningTeaching and Learning• Changing teaching methods, faculty distribution, Changing teaching methods, faculty distribution,

curriculum, etc.curriculum, etc.

5.5. Improving AssessmentImproving Assessment• Revise process for the futureRevise process for the future

29

VA Assessment VA Assessment Reporting GuidelinesReporting GuidelinesAny scientist would Any scientist would require the samerequire the same

►Presented in quantitative & non-anecdotal Presented in quantitative & non-anecdotal formform

►Evidence of reliabilityEvidence of reliability►Evidence of validityEvidence of validity►Description of data collection context Description of data collection context

(sampling procedures, etc.)(sampling procedures, etc.)►Evidence of the use of assessment resultsEvidence of the use of assessment results

30

Who’s going to oversee core Who’s going to oversee core curriculum assessment?curriculum assessment?

Responsibility of some or all of the following:Responsibility of some or all of the following:► Core curriculum committeeCore curriculum committee

Assessment sub-committeesAssessment sub-committees►Separate committees for different areas consisting of Separate committees for different areas consisting of

faculty from across the different departments that may faculty from across the different departments that may be teaching in that areabe teaching in that area

► Academic AffairsAcademic Affairs► Student AffairsStudent Affairs► Institutional ResearchInstitutional Research► Separate Office of AssessmentSeparate Office of Assessment

At the At the very leastvery least Director or Coordinator of Director or Coordinator of AssessmentAssessment

31

State Resources for State Resources for Institutions’ Assessment EffortsInstitutions’ Assessment Efforts

►Budget linesBudget lines to fund assessment to fund assessment activitiesactivities

►Permission to Permission to charge student feescharge student fees for for assessment activitiesassessment activities

►Offering Offering grants grants or or technical assistancetechnical assistance

► If they offer $$ for assessment, take it!If they offer $$ for assessment, take it!

32

Institutions’ Assessment Institutions’ Assessment Support StrategiesSupport Strategies

►Have an explicit and visible student Have an explicit and visible student assessment planassessment plan

►Use a series of incremental planning Use a series of incremental planning stepssteps

►Examine assessment practices at other Examine assessment practices at other institutionsinstitutions

►Pilot assessment strategiesPilot assessment strategies►Encourage broad participation in the Encourage broad participation in the

planning process, particularly faculty & planning process, particularly faculty & administratorsadministrators


1 Core Assessment: Challenges Facing New Programs TAIR Professional Development Workshop September...

Documents

Transcript of 1 Core Assessment: Challenges Facing New Programs TAIR Professional Development Workshop September...