1

Planning for Successful Student Learning Outcomes

Assessment for the Voluntary System of

Accountabilitywith

© 2009 ACT, Inc. All Rights Reserved.

Presentation for Fitchburg State

January 13, 2009

2

Webinar Agenda

• CAAP Overview• CAAP and the VSA• CAAP Usage Strategies• Plan for Success• CAAP Resources

3

Collegiate Assessment of Academic Proficiency

4

The standardized assessment program from

ACT that enables postsecondary institutions to

assess, evaluate, and enhance the outcomes of

their general education programs.

5

Selected as one of the options for measuring

student learning outcomes for the Voluntary System of

Accountability.

6

CAAP Benefits

Systematically measure your students’ achievement

levels and growth in your general education program

7

CAAP Benefits

Institution and individual student level data to help you identify opportunities

to enhance student learning

8

CAAP Benefits

Help satisfy the need for greater accountability in higher education

9

CAAP Benefits

Help meet accreditation self-study requirements

10

CAAP Benefits

Help evaluate individual student readiness for

upper division coursework

11

CAAP Benefits

Not to evaluate faculty.

Not as a sole evaluation in high stakes environments.

12

CAAP Features

• Modular format (6 content areas)• Flexible administration• Comprehensive reports• Extensive national user norms• Value-added measurement

through linkage to COMPASSTM and the ACT® test• Identification of action items

with Content Analysis Reports

13

CAAP Test Modules

•Writing Essay

•Writing Skills

•Mathematics

• Reading

• Critical Thinking

• Science

14

CAAP Administration

• Convenient in-class administration• 40 minutes per test• Pencil and paper• Proctored• Can be embedded in required

freshman or capstone course• 2-week test window• Nine local questions can be added

(except for VSA testing)

15

CAAP Standard Reporting Package• Institutional Summary Report

• Mean scores for each demographic and subscores in math, reading, & writing skills

• Student Score Reports – 2 copies• One for institution and one for student

• Certificate of Achievement• For students scoring at or above the

national mean

• Student Roster Report• Listing of each student with their CAAP

results

16

CAAP User Norms

to

17

CAAP User Norms

• Norms for each test module (and subscore)

• Broken out by:

• Institution type

•Year of students tested

• Institution ownership

• Three-year rolling norms

• Custom groupings available

18

CAAP and the VSA• Voluntary System of Accountability

• A joint initiative developed in partnership between NASULGC and AASCU

• Designed to help institutions• Demonstrate accountability and

stewardship to the public• Measure educational outcomes to

identify effective educational practices• Assemble information that is

accessible, understandable, and comparable

• Currently over 60 NASULGC/AASCU institutions use CAAP for their outcomes assessment

19

CAAP and the VSA

• VSA College Portrait• Consumer Information• Student Experiences and Perceptions• Student Learning Outcomes

• VSA participants measure critical thinking, analytical reasoning, and written communication

• CAAP• Measures critical thinking and written

communication using two test modules• CRITICAL THINKING• WRITING ESSAY

20

VSA College Portrait Pilothttp://www.voluntarysystem.org http://www.collegeportraits.org

• VSA Web Site• College Portrait data elements• College Portrait prototype• VSA Background/development• VSA Enrollment

21

CAAP and the VSAStudent Learning OutcomesFreshman to Senior Learning

Gains

• Critical Thinking

• Writing Essay

• The increase in learning on the performance task is “_____” what would be expected at an institution with students of similar academic abilities.

• “well above”, “above”, “below” or “well below” what would be expected

22

College Portrait Learning Outcomes Reporting

College Portrait Prototype

23

CAAP Writing Essay

• Two independent prompts

• 20 minutes each

• Essays scored at ACT by two trained raters

• ACT scoring rubric 1-6• http://www.act.org/caap/tests/holistic.html

24

CAAP Writing Essay

• Formulating an assertion about a given issue

• Supporting that assertion with evidence appropriate to the issue, position taken, and a given audience

• Organizing and connecting major ideas

• Expressing those ideas in clear, effective language

25

Writing Essay Sample

Your college administration is considering whether or not there should be a physical education requirement for undergraduates. The administration has asked students for their views on the issue and has announced that its final decision will be based on how such a requirement would affect the overall educational mission of the college. Write a letter to the administration arguing whether or not there should be a physical education requirement for undergraduates at your college.

26

CAAP Critical Thinking

• 32 items/4 passages• Sample questions on-line• Clarifying, analyzing,

evaluating, and extending and argument • Subject matter—Social

Sciences, Natural Sciences, Humanities• Formats include:• Case Studies

• Dialogues

• Debates

• Editorials

• Statistical Arguments

• Experimental Results

• Overlapping Position

27

CAAP and the VSA• Sample Size

• ACT requires a minimum 200 freshman and 200 seniors tested on each module

• Test Dates• Freshmen must be tested early in the fall semester

• Seniors must be tested within 6 months of graduation

• Both groups tested during the same academic year (September-May)

• ACT (or SAT) Score Required• Institutions must provide data file with CAAP

tested students’ ACT or SAT scores – see CAAP VSA Guidelines for more details

28

CAAP and the VSA

• Random selection• Institution determines what constitutes

a random representative sample

• Value-added VSA Model• Cross-sectional

• Longitudinal Model– Consider the minimum number of

records needed at the time of post-testing and then calculate back to establish the number for pre-test, and test 3-4 times that number

29

CAAP Scoring• Return materials to ACT for scoring

and reporting

• Critical Thinking • ACT proprietary scanning

• Writing Essay• Two ACT-trained scorers

• Three-week turn-around time for standard reporting package

30

CAAP VSA Report• Provided upon completion of all freshmen

and senior testing – late summer

• Provided upon receipt of your institution data file (with student matched ACT or SAT and CAAP scores)

• Report will include:

• Average scores for both freshmen and seniors on both test modules

• Student learning gains on each test module

• Average ACT (or SAT) for students tested

31

32

CAAP Administration for VSA

• Administer at least once every 3 years

• Report within 4 years of becoming VSA participant

• Updated at least every 3 years

33

Additional CAAP Test Modules

• CAAP VSA Materials Order Form provides option for additional modules

• Administer to a different group of students than those tested on the VSA modules

• Option for local questions for only those who are not testing for VSA

34

Any questions so far?

35

Usage Strategies

36

CAAP Usage Strategies

• Outcomes Measurement

• Value-Added Measurement

• Student Readiness Evaluation

37

Outcomes Measurement

38

Outcomes Measurement

• Establish benchmarks – (Scale scores, percentiles)

• Identify relative strengths and weaknesses– (Subscores, Content Analysis

Report)• Evaluate issues• Propose solutions - set goals• Implement changes• Track results

39

TEST MODULE C.A.R. DOMAINS

Writing Skills- Usage/Mechanics- Rhetorical Skills

Punctuation Grammar Sentence Structure Strategy Organization Style

Mathematics- Basic Algebra- College Algebra

Prealgebra Elementary Algebra Intermediate Algebra Coordinate Geometry College Algebra Trigonometry

Reading- Arts/Literature- Social/Nat. Sciences

Reasoning Skills Referring Skills

Science Analysis Generalization Understanding

Critical Thinking Analysis of Elements of Arguments Evaluation of Arguments Extension of Arguments

Content Analysis Report

40

41

Content Analysis Report (Math Sample)

42

43

Value-Added Measurement

44

Value-Added Measurement

59.3

61.9

62.462.7

60.7

62.4

57

58

59

60

61

62

63

Fresh Senior

Inst. X

Inst. Y

Nat. Mean

45

Value-Added Measurement

• Longitudinal (CAAP to CAAP)

• Longitudinal (Linkage)

• ACT to CAAP

• COMPASS to CAAP

• VSA Model – Cross-Sectional – Freshman to Senior Learning Gains

46

Value-Added Measurement

Longitudinal (CAAP to CAAP)

• Test cohort with CAAP upon program entry

• Test same cohort with CAAP upon completion of program

• Change in scores is a direct measure of “value-added” performance gain

• Evaluate results in terms of expectations

• Investigate opportunities for improvement

47

Value-Added Measurement

Longitudinal (ACT or COMPASS to CAAP)

• Use cohorts’ ACT or COMPASS scores as a benchmark

• Test cohort with CAAP upon completion of general education coursework

• Linkage report evaluates cohorts’ results in terms of “expected gain”

• Evaluate results in terms of expectations

• Investigate opportunities for improvement

48

Value-Added Measurement

Cross-sectional - VSA Model

• Test a cohort of incoming freshmen at the beginning of the fall term

• Test a similar group of seniors within 6 months of graduation

• Differences between the two group mean scores may indicate performance gain

• Challenge of this strategy is matching student characteristics across groups

49

Student Evaluation

50

Student Evaluation

• Determine student skill expectations

• Establish appropriate cut scores• Propose appropriate

remediation

51

PLAN FOR SUCCESS

52

Plan for Success

#1 Confirm the fundamentals

• Review your institutional mission statement.

• Evaluate the objectives of your institution’s general education program.

• Evaluate your institutional general education core curriculum.

53

Plan for Success

#2 Specify usage of results

• Who will review and analyze the data? • How will the results be used to

influence program evaluation and planning?

• Will results be tracked on a longitudinal basis?

54

Plan for Success

#3 Define population

• Specify characteristics of the student body that is the focus of the study.

• Specify characteristics of subgroups.

55

Plan for Success

#4 Sampling• Test entire population or sample?

• How will sample be selected?

• How will subgroup sample(s) be selected?

• Is sample random?

• Is sample representative of student population?

56

Plan for Success

# 5 Faculty motivation• Specify plans to communicate with

faculty.

• Specify plans to enlist faculty support.

• Specify plans to share results with faculty.

57

Plan for Success

#6 Student motivation• Specify plans to communicate with

students.

• Specify plans to motivate students.

• Specify plans to share results with students.

58

Plan for Success

• Submit CAAP-VSA Participation Agreement

• Confirm test modules, if additional*• Confirm optional local questions*• Confirm reporting requirements• Confirm test period (two-week window)• Confirm test venues• Confirm students to be tested• Confirm proctors• Schedule training for proctors• Order test materials at least one month

prior to test dates

• *For non-VSA test modules only

#7 CAAP Test Administration

59

• Critical Thinking - $13.50/student• Writing Essay - $13.50/student

– Includes Standard Reporting Package and CAAP VSA Report

• Additional CAAP Modules– Writing Essay - $13.50/student– Single Objective Test - $13.50/student

• 501+ - $12.70/student

– Multiple Objective Tests - $ 13.50/student• 501+ - $19.20/student

CAAP Testing Costs

60

CAAP RESOURCES

61

CAAP Planning Workbook available online at

http://www.act.org/caap/pdf/planguide.pdf

Portions of Guide are available at

http://www.act.org/caap/materials.html

62

Portions of this can be found at http://www.act.org/caap/techhandbook

63

Norms available on-line at http://www.act.org/caap/norms/

64

Available on-line at http://www.act.org/caap/pdf/userguide.pdf

65

http://www.act.org/caap http://www.act.org/caap/vsa

66

• CAAP Overview• CAAP and the VSA• CAAP Usage

Strategies• Plan for Success• CAAP Resources

Recap

67

ACT Contact Information

ACT Postsecondary Assessment Services 500 ACT Drive P.O. Box 168 Iowa City, Iowa 52243-0168

Phone: 800-294-7027Sandra Stewart

Sandra.Stewart@act.orgDirect # 319-337-1051

FAX: 319/337-1790

E-mail: outcomes@act.org