Laura Renninger Shepherd University

Laura RenningerShepherd University

Overview

CLA Approach

CLA Administration

CLA Measures

CLA Scoring and Shepherd’s CLA Results

CLA Additional Data and Next Steps

CLA Approach

Holistic assessment of important skills Critical Thinking Analytic Reasoning Written Communication Problem Solving

Measurement of value-addedInstitution as unit of analysis

CLA AdministrationThe CLA is administered by the Council for Aid to Education

(CAE), a non-profit organization based in New York City.

Reporting Products Institutional Presentation Institutional Report Student Data File

Results are not reported publicly Schools can share data within consortia of peer institutions

CLA Administration

We participated in a cross-sectional study, in which growthbetween freshmen and seniors is estimated by testing samplesof students, not the entire class.

Students take the CLA online in proctored settings. Testing timeis approximately 90 minutes.

150 “native”, first-year students were selected at random and tested at Orientation in August 2004, 2005 and 2006

150 “native” seniors were selected via a stratified random sampling process and tested during the spring 2005, 2006 and 2007 semesters

Tests were proctored by members of the Assessment Task Force: Dr. Gordon DeMeritt, Dr. Barri Tinkler, Dr. John Adams, Shannon Holliday, Laura Renninger and John Sheridan

CLA Measures

Analytic Writing Task Make-an-Argument Critique-an-Argument

Performance Task

Analytic Writing Task: Make-an-Argument

“In our time, specialists of all kinds are highly overrated. We need more generalists -- people who can provide broad perspectives.”

Directions: In 45 minutes, agree or disagree and explain the

reasons for your position.

Analytic Writing Task:Critique-an-Argument

“Butter has now been replaced by margarine in Happy Pancake House restaurants throughout the southwestern United States. Only about 2 percent of customers have complained, indicating that 98 people out of 100 are happy with the change. Furthermore, many servers have reported that a number of customers who still ask for butter do not complain when they are given margarine instead. Clearly, either these customers cannot distinguish margarine from butter, or they use the term "butter" to refer to either butter or margarine. Thus, to avoid the expense of purchasing butter, the Happy Pancake House should extend this cost-saving change to its restaurants in the southeast and northeast as well.”

Directions: In 30 minutes, discuss how well-reasoned you find the argument.

Performance Task

Performance Tasks place students in a real-world scenario.

In the following case, students have 90 minutes to advise the

mayor on crime reduction strategies and evaluate two potential

policies:

1. Invest in a drug treatment program or

2. Put more police on the streets.

Students are provided with a Document Library, which includes

different types of information sources, such as…

Performance Task

A MEMO by a private investigator that

reports on connections between a

specific drug treatment program and

a vocal critic of placing more police on

the streets.

Performance Task

CRIME STATISTICS that compare thepercentage of drug addicts to thenumber of crimes committed in thearea.

Performance Task

Crime and community DATA TABLESprovided by the Police Department.

Performance Task

A NEWS story highlighting a rise in localdrug-related crime.

Performance Task

A RESEARCH BRIEF summarizing ascientific study that found the drugtreatment program to be effective.

Performance Task

A CHART that shows that counties witha relatively large number of policeofficers per resident tend to have morecrime than those with fewer officers perresident.

Performance Task

WEB SEARCH results of other studiesevaluating the drug treatment program.

Performance Task

•Performance Tasks require students to use an integrated set of critical thinking, analytic reasoning, problem solving, and written communication skills. •There are no “right” answers. The goal is to stimulate students’ abilities to make reasoned, reflective arguments.

Performance Task

Students are expected to evaluate evidence by:

1. Determining what information is or is not pertinent

2. Distinguishing between fact and opinion

3. Recognizing limitations in the evidence

4. Spotting deception and holes in the arguments of others

Performance Task

Students are expected to analyze and synthesize the evidence by:

1. Presenting his/her own analysis of the data

2. Breaking down the evidence into its component parts

3. Drawing connections between discrete sources of data

4. Attending to contradictory or inadequate information

Performance Task

Students are also expected to draw conclusions by:

1. Constructing cogent arguments rooted in data rather than speculation

2. Selecting the strongest set of supporting evidence

3. Avoiding overstated or understated conclusions and suggesting additional information to complete the analysis

CLA Scoring and our CLA Results

CLA scores for a school represent the average (or “mean”) score for all students that completed a CLA task and who also have an SAT score (or ACT score converted to the SAT scale) on file with the registrar. The CLA scale approximates the SAT scale.

Mean SAT Scores (on the horizontal x-axis) are used to control for incoming academic ability.

Put another way, it allows for a level playing field when comparing performance across all CLA schools.


This blue dot represents the mean CLA score and mean SAT score for the 78 freshmen we sampled in 2006.


These blue circles represent mean CLA and SAT scores at the other 114 schools testing freshmen in fall 2006.

Once again, the unit of analysis is institutions, not students.


The diagonal blue line shows the typical relationship between academic ability and mean CLA scores of freshmen across all participating institutions.


Points along the line represent expected CLA scores for a school testing freshmen across the range of mean SAT scores.


The focus is on the difference between a college’s actual and expected CLA scores—graphically, the vertical distance between the dot and the line.

These differences are reported in both CLA scale points and standard errors.


Colleges with actual mean scores between -1.00 and +1.00 standard errors from their expected scores are categorized as being “At Expected.”


Institutions with actual mean CLA scores greater than one standard error (but less than two standard errors) from their expected scores are in the “Above Expected” or “Below Expected” categories (depending on the direction of the deviation).


The schools with actual scores greater than two standard errors from their expected scores are in the “Well Above Expected” or “Well Below Expected” categories.


Based on the average SAT

score (1005) of the 78

freshmen we sampled, their

expected average CLA score

was 1038. Our freshmen

scored 972, which is Below

Expected

(-66 CLA scale points and -1.6

standard errors from the line).


Repeating the process for seniors, this solid red square represents the mean CLA score and mean SAT score for the 79 seniors we sampled in spring 2007.


These red squares represent mean CLA and SAT scores at the other 105 schools testing seniors in spring 2007.


The diagonal red line shows the typical relationship between academic ability and mean CLA scores of seniors across all participating institutions.


Points along the line represent the expected CLA score for a school testing seniors across the range of mean SAT scores.


Based on the average SAT

score (1042) of the 79 seniors

we sampled, their expected

average CLA score was 1148.

Our seniors scored 1150,

which is At Expected

(2 CLA scale points and 0.0

standard errors from the line).


Based on the average SAT scores of our freshmen and seniors, we would expect a difference of 110 CLA scale points.


So how did we do?The difference between how our seniors and freshmen scored was 178 points, which places us in the decile group 10; we performed better than 90 percent of institutions.

(68 CLA scale points and 1.6

standard error units from expected).


CLA Data and Next StepsInstitution-level CLA results operate as a signaling tool of overall

institutional performance that we can compare with other

outcomes, such as retention and graduation rates.

Outcome Your School Expected Value Deviation Score Performance LevelFirst-year retention rate 69.0 65.0 0.6 At4-year graduation rate 16.1 14.2 0.2 At6-year graduation rate 40.2 35.8 0.5 AtFreshmen CLA score 972 1038 -1.6 BelowSenior CLA score 1150 1148 0.0 At

Table 10: Comparison of observed and expected outcomes at your school

CLA Data and Next Steps

Student-level CLA results are also provided for us to link with

other data sources (e.g., course taking patterns, grades,‐

portfolio assessments, student satisfaction and engagement,

major-specific tests, etc.) so we can identify correlations,

begin to explain our results and formulate additional questions

for investigation.


In-depth sampling focuses on specific populations transfers versus “native” students fields of study academic majors students living on/off campus work-study students financial aid recipients athletes

Longitudinal studies track the same students over time Students tested as freshmen, rising juniors and seniors Cross-sectional sample of seniors tested in first year to establish baseline

for performance

Finally, the Performance Task described earlier in this presentation will be released publicly in fall 2007 as an instructional tool, complete with a scoring guide.

This will provide faculty with the chance to work with students to understand why they achieved the scores they did, and what to do next to improve their skills.

This initiative is called “CLA in the Classroom”.

How can you help?


Questions?

MAPP Test• “Measure of Academic Proficiency and Progress”

• The MAPP is a standardized measure of college-level reading, mathematics, writing, and critical thinking in the context of the humanities, social sciences, and natural sciences. The exam is designed by the Educational Testing Service.

• According to ETS, “The MAPP test is designed for colleges and universities to assess their general education outcomes, so they may improve the quality of instruction and learning. It focuses on the academic skills developed through general education courses, rather than on the knowledge acquired about the subjects taught in these courses.”

• Multiple Choice Exam

MAPP Administration• Shepherd University “native” sophomores (students

who have completed between 25 and 56 semester hours of work).

• 100 sophomores randomly selected

• Spring 2007 N = 97; mean GPA = 2.80; median GPA = 2.84

• Paper and pencil test proctored by members of the Assessment Task Force

MAPP Proficiency Levels• MAPP Proficiency Classifications:

• Reading/Critical Thinking• To be considered Proficient at level 1 a student should be able to• Recognize factual material explicitly presented in a reading passage • Understand the meaning of particular words or phrases in the context of a reading passage

• To be considered Proficient at level 2 a student should be able to• Synthesize material from different sections of a passage • Recognize valid inferences derived from material in the passage • Identify accurate summaries of a passage or of significant sections of the passage • Understand and interpret figurative language • Discern the main idea, purpose, or focus of a passage or a significant portion of the passage

• To be considered Proficient at level 3 a student should be able to• Evaluate competing causal explanations • Evaluate hypotheses for consistency with known facts • Determine the relevance of information for evaluating an argument or conclusion • Determine whether an artistic interpretation is supported by evidence contained in a work • Recognize the salient features or themes in a work of art • Evaluate the appropriateness of procedures for investigating a question of causation • Evaluate data for consistency with known facts, hypotheses or methods • Recognize flaws and inconsistencies in an argument

MAPP Proficiency Levels• Writing Skills• To be considered Proficient at level 1 a student should be able to• Recognize agreement among basic grammatical elements (e.g., nouns, verbs, pronouns and conjunctions) • Recognize appropriate transition words • Recognize incorrect word choice • Order sentences in a paragraph • Order elements in an outline

• To be considered Proficient at level 2 a student should be able to• Incorporate new material into a passage • Recognize agreement among basic grammatical elements (e.g., nouns, verbs, pronouns, and conjunctions) when these

elements are complicated by intervening words or phrases • Combine simple clauses into single, more complex combinations • Recast existing sentences into new syntactic combinations

• To be considered Proficient at level 3 a student should be able to• Discriminate between appropriate and inappropriate use of parallelism • Discriminate between appropriate and inappropriate use of idiomatic language • Recognize redundancy • Discriminate between correct and incorrect constructions • Recognize the most effective revision of a sentence

MAPP Proficiency Levels• Mathematics

• To be considered Proficient at level 1 a student should be able to• Solve word problems that would most likely be solved by arithmetic and do not involve conversion of units or proportionality. These problems can be

multi-step if the steps are repeated rather than embedded. • Solve problems involving the informal properties of numbers and operations, often involving the Number Line, including positive and negative

numbers, whole numbers and fractions (including conversions of common fractions to percent, such as converting "1/4" to 25%). • Solve problems requiring a general understanding of square roots and the squares of numbers. • Solve a simple equation or substitute numbers into a algebraic expression. • Find information from a graph. This task may involve finding a specified piece of information in a graph that also contains other information.

• To be considered Proficient at level 2 a student should be able to• Solve arithmetic problems with some complications, such as complex wording, maximizing or minimizing, and embedded ratios. These problems

include algebra problems that can be solved by arithmetic (the answer choices are numeric). • Simplify algebraic expressions, perform basic translations, and draw conclusions from algebraic equations and inequalities. These tasks are more

complicated than solving a simple equation, though they may be approached arithmetically by substituting numbers. • Interpret a trend represented in a graph, or choose a graph that reflects a trend. • Solve problems involving sets; the problems would have numeric answer choices.

• To be considered Proficient at level 3 a student should be able to• Solve word problems that would be unlikely to be solved by arithmetic; the answer choices are either algebraic expressions or are numbers that do

not lend themselves to back-solving. • Solve problems involving difficult arithmetic concepts such as exponents and roots other than squares and square roots and percent of increase or

decrease. • Generalize about numbers, e.g., identify the values of (x) for which an expression increases as (x) increases. • Solve problems requiring an understanding of the properties of integers, rational numbers, etc. • Interpret a graph in which the trends are to be expressed algebraically or in which one of the following is involved: exponents and roots other than

squares and square roots, percent of increase or decrease. • Solve problems requiring insight or logical reasoning.

Shepherd MAPP Scoresspring 2007

Skill Dimension Proficiency Classification

Proficient Marginal Not Proficient

Reading, Level 1 71% 19% 10%

Reading, Level 2 35% 28% 37%

Critical Thinking 6% 12% 81%

Writing, Level 1 70% 23% 7%



Mathematics, Level 1 43% 35% 22%



Comparative Data

Laura Renninger Shepherd University

Documents

Transcript of Laura Renninger Shepherd University