Nationally Normed Testing via ACS Exams:

Research and Practice

Thomas Holme Iowa State University

ACS DivCHED Examinations Institute

ChemEd 2012, Adelaide

A fundamental challenge

• Teaching is, at once, inherently personal and inescapably corporate.

• At present, the corporate interests in student learning are often articulated in terms of assessment.

Exams Institute?

• How is it that chemistry in the US has an Exams Institute?

A Short History

1921

Division of Chemical Education

A Short History

1930 Committee on Examinations and Tests Formed to construct or exploit the development of

objective test construction and train teachers how to construct tests for their own classes

A Short History

1930

Committee on Examinations and Tests

“…only specialists in the field can write good tests because they are the persons who know what is significant and important, rather than the test experts who know the forms and techniques of test construction but not the subject matter”

Ted Ashford on Ralph Tyler’s position on testing

A Short History

1930

Committee on Examinations and Tests

The Committee was subsidized by:

General Education Board of the Cooperative Test Service

Carnegie Foundation for the Advancement of Teaching

Dr. Ben Wood

A Short History

1934

A group of five Committee members released the first general chemistry test in three forms

A Short History

1946

Ted Ashford appointed as Chair of the Committee

A Short History

1984

Committee on Examinations and Tests renamed to Examinations Institute

Board of Trustees appointed to oversee operation of the Institute

A Short History

1987

Dwaine Eubanks appointed Director

Examinations Institute moves to Oklahoma State University

A Short History

2002

Tom Holme appointed Director

Examinations Institute moves to University of Wisconsin – Milwaukee

- (2008) Moves to Iowa State

Reason Exams Institute exists?

• ~ P.T. Barnum

• There’s a sucker born every 2 or 3 decades

• “on average”

The key constituencies

• Practitioners

– Often motivated by practicality

• Chem Ed Researchers (TUES grant recipients)

– Often motivated by validation challenges

Exam development

• Chair is named • Committee is recruited • First meeting - sets content coverage • Items are written and collated • Second meeting - editing items, setting trials • Trial testing in classes - provides item stats • Third meeting - look at stats and set exam • Meetings are held in conjunction with ACS National

Meetings (or BCCE) – Partial reimbursement to volunteers

Gen Chem Exams

• Full Year Exam (2009, 2011)

• First Term Exam (2005, 2009)

• Second Term Exam (2006, 2010)

• 1st Term Paired Questions (2005)

• 2nd Term Paired Questions (2007)

• Conceptual (1st term, 2nd term, full year)

• Full year - brief exam (2002, 2006)

• All exams carry secure copyright – Released – not published

Norms and reporting

• Norms are calculated on voluntary return of student performance data

• We have an interactive web site for score reporting for exams that do not yet have enough data to report a norm.

• People often use norm (percentile) to help students who transfer to other programs.

Exams as artifacts of teaching

• Because ACS Exams have been around for a long time, they provide an additional artifact of what the community values.

• Consider Organic Chemistry Exams – Analyze cognitive demands

• Vast majority of items on organic exam qualify as conceptual understanding

– Analyze chemistry content. • Content coverage shows modest fluctuation, but few

fundamental shifts in the past 20 years.

Then and now

• In addition to historic value, the current efforts of the Exams Institute also shed light on research and practice.

• First a bit on curricular practice.

• Then some a research example.

Criterion referencing for program assessment

• Requires criteria

• At the college level, they don’t exist.

• Build a consensus content map.

• Similar to using backward design1.

1: Understanding by Design, Grant P. Wiggins, Jay McTighe

Anchoring Concept

• Use “big ideas” or anchoring concepts to organize content across disciplines.

• Build levels with finer grain size down to the point where exam items are generally written.

Levels of criteria map

• Anchoring Concept

• Enduring Understanding

• Sub-disciplinary articulation

• Content details

Process for setting map (so far)

• Begin from EMV conference ideas • Focus Group (Mar08): Level 1 + Level 2 • Workshop (Jul08): Level 2 + Level 3 (General) • Focus Group (Aug08): Level 2 + Level 3 (Organic) • Workshop (Mar09): Level 3 + Level 4 (General) • Focus Group (Aug09): Level 2 + Level 3 (Organic) • Workshop (Mar10): Alignment (General) • Focus Group (Mar 10): Level 2 + Level 3 (Physical) • Focus Group (Jul 10): Level 3 Organic • Focus Group (Dec 10): Level 3 + Complexity Organic • Focus Groups (Mar 11): Level 3 (Analytical, Biochem, Physical) • Focus Groups (Aug 11): Level 3 + Complexity (Organic) • Focus Groups (Mar 12): Level 4, Alignment, Complexity (General,

Organic)

Example of comparing content

Access?

• The Gen Chem version has been published. – Uses Authors Choice

so it should be downloadable.

• Organic is expected to be publishable by fall of this year.

Item Alignment

• Look at current items from ACS Exams and align them to Level 3/4

• Process guided by psychometric experts.

• Can include both skills and content

• Ultimately can help define specifications for future ACS Exams.

Comparison of gen and org

Enjoy the ride…

• This project shows how discussions around testing/benchmarking could ultimately lead to curricular forcing.

• The discussions remain “grass roots”

• The Exams Institute provides the playground and occasionally has to decide what rules to follow, but doesn’t push an “ACS” agenda.

But, testing…are you sure?

• Are we really that confident in our measurements?

Research: Teachable moments

• Because a sizeable fraction of the Chem Ed community uses (or at least trusts) ACS Exams, the characterization of the exams allows an avenue to educate about assessment issues.

Recently taught topics

• Role of item complexity

• Item characteristic curves

• Item Order Effects

• Answer Order Effects

• Differential Item Functioning (DIF)

• Partial credit / polytomous scoring

Content vs. construct

• Tests demand that students complete tasks

• Each item is a task

• Students need knowledge within the content domain (chemistry)

• Students need knowledge about how to organize their efforts (test taking)

• Cast this understanding in terms of item complexity.

The Information Processing Model

• Consider the task in terms of the Information Processing Model of Johnstone and coworkers.

Johnstone, CERP, 7, 49-63 (2006)

Estimating Task Complexity (Johnstone and coworkers)

• Count up the pieces of information needed to accomplish the task

• Compare to student performance

Estimating task complexity elsewhere

Paas & Van Merriënboer’s 9-point scale (1994).

1 ~ very, very low mental effort 2 ~ very low mental effort 3 ~ low mental effort 4 ~ lower than average mental effort 5 ~ average mental effort 6 ~ higher than average mental effort 7 ~ high mental effort 8 ~ very high mental effort 9 ~ very, very high mental effort

Data for each chemistry exam item

• Performance data (difficulty index)

• Expert-rated objective complexity

• Mental effort (hypothesized to represent the subjective complexity)

Three constructs of task complexity

• Complexity treated as a psychological experience. – Subjective complexity

• Complexity treated as a function of objective task

characteristics. – Objective complexity

• Complexity treated as an interaction between task and

person characteristics.

Principle component analysis

Communalities

1.000 .700

1.000 .629

1.000 .704

error rate

complexity (rating)

mental effort (rating)

Initial Extraction

Extraction Method: Principal Component Analysis.

Total Variance Explained

2.033 67.768 67.768 2.033 67.768 67.768

.537 17.910 85.678

.430 14.322 100.000

Component

1

2

3

Total % of Variance Cumulativ e % Total % of Variance Cumulativ e %

Initial Eigenv alues Extract ion Sums of Squared Loadings

Extract ion Method: Principal Component Analy sis.

Component Matrixa

.837

.793

.839

error rate

complexity (rating)

mental ef fort (rating)

1

Component

Extraction Method: Principal Component Analysis.

1 components extracted.a.

Factor analysis

• Factor Analysis finds a single factor with all PCA loading factors above 0.75

• Hypothesis: This factor represents the complexity of multiple-choice chemistry items.

• Principal axis factoring and maximum likelihood factoring both reveal a single factor as well.

Take home message

• Depending on the model used for factor analysis the amount of variance changes (from 51% to 67%).

• Nonetheless, for Gen Chem half of the variance of student performance can be explained by the latent variable of task complexity.

• Not arguing against complex content – arguing for awareness of complexity when making measurements

• Current work is looking at Organic. – Key challenge is assigning task complexity.

What can researchers use?

• High quality individual instruments

– DUCK

– Paired question exams

– On-line laboratory assessment

• Includes full motion video and novel item constructs

• A support system for program assessment.

– Criterion referencing

What’s a DUCK

• Diagnostic of Undergraduate Chemistry Knowledge.

• Fundamentally inter-disciplinary

• Scenario Based.

• 15 Scenarios, each with 4 items.

• Taken at/near the end of the undergraduate curriculum. – Sometimes in a capstone course, sometimes as an

extra.

What have we learned?

• Analysis of performance from US students on current DUCK.

Other advantages?

• Having the Exams Institute provides a venue for looking at the interaction of measurement and learning in chemistry.

• Complexity

• Interdisciplinarity

• Measurement factors – Item order effects

– Differential Item Functioning

– Coming: Human-computer interaction

Acknowledgements

• Kristen Murphy (UWM)

• Jeff Raker (ISU)

• Kim Linenberger (ISU)

• Mike Slade (ISU)

• Heather Caruthers (ISU)

• Anna Prisacara (ISU)

• Jessica Reed (ISU)

• John Balyut (ISU)

• April Zenisky (Umass)

• Karen Knaus (UC-Denver)

• Jacob Schroeder (Clemson)

• Mary Emenike (Rutgers)

• Megan Grunert (W. Michigan)

• Chris Bauer (New Hampshire)

NSF: DUE-0618600, 0717769, 0817409, 0920266

taholme@iastate.edu @Tom_Holme

Current Collaborators Prior Collaborators