Assessed: 5 Cycles 2006, 2009, 2010, 2011, 2013.

Assessed: 5 Cycles2006, 2009, 2010, 2011, 2013

General Education RequirementsRequired course in Intermediate

Composition and Critical ThinkingRequired course in Quantitative ReasoningAdditional lower and upper division General

Education requirements Preparation for the Major Requirements Core Courses in the BSBA Major Advanced Courses in the BSBA Major

The refinement of the process over five cycles of assessment.

Case analyses completed for the college-wide CBA capstone course (MGT 405) were examined.

CBA’s former Director of Assessment developed a rubric after extensive review of existing critical thinking instruments

2 CBA faculty worked with the former Director of Assessment to rate the case analyses using the rubric following a training and norming session

Exceeds Expectations

(4)

Meets Expectation

s (3)

Approaches Expectation

s (2)

Fails (1)

Issue Identification

Explicitly identifies the key issue(s)

Implicitly identifies (discusses) the key issue(s)

Identifies subsidiary issues as key

Fails to identify issue(s) or question(s)

Use of evidence and data

Interprets/analyzes data in a way that improves understanding of case

Cites data and uses it to analyze case

Mentions/cites data but fails to apply it to case issues

Fails to use data provided; provides little to no support for analysis

Models and Frameworks

Explicitly applies models or frameworks to case analysis

Analyzes case using concepts from models or frameworks

Uses models/frameworks inappropriately or incorrectly

Uses no models or frameworks to analyze case

Conclusions & Recommendations

Recommends and defends a conclusion based on the analysis

Recommends a solution congruent with the analysis

Recommends a solution not congruent with the analysis

Does not offer a specific recommendation or conclusion

Final exams completed for the college-wide CBA capstone course (MGT 405) were examined.

Course Instructor and Director of Assessment developed a rubric after reviewing the 2006 rubric and others from the critical thinking literature.

Rubric was widely distributed to students during the semester.

Instructor & Director of Assessment rated the final exams using the revised rubric following training and norming sessions.

7 Point scale from Weak to Excellent Expectations determined post-hoc based

on examination of means for dimensions. Dimensions:1. Clear understanding of the question and issues to be addressed.2. Understanding of the relevant concepts and frameworks.3. Effective application of relevant concepts to address

question/issues.4. Ability to effectively use case data (depth & breadth) to illustrate a

position.5. Ability to think strategically (i.e. integrate across internal &

external environment; across functional areas; take a general management view).

6. Ability to arrive at logical and well reasoned conclusions based on the discussion.

Dimension

Fail

(1)

Below

(2 to

Expectations

3)

Approaches Expectations

(4)

Meets Expectations

(5)

Exceeds

(6 to

Expectations

7)

1 1 2 3 4 5 6 7

2 1 2 3 4 5 6 7

3 1 2 3 4 5 6 7

4 1 2 3 4 5 6 7

5 1 2 3 4 5 6 7

6 1 2 3 4 5 6 7

Final exams completed for the college-wide CBA capstone course (MGT 405) were examined.

Course Instructor and Director of Assessment refined 2009 rubric based on mapping & experience.

Rubric was widely distributed to students during the semester.

Two independent CBA faculty members rated the final exams using the revised rubric following training and norming sessions.

7 Point scale from Weak to Excellent retained. Expectations determination post-hoc based on

examination of means for dimensions retained. Dimensions from Cycle #2 Rubric were

reduced by one and wording modified in some cases:

1. Clear understanding of the question and issues to be addressed.2. Knowledge of case facts & ability to effectively use case data to

illustrate position.3. Effective understanding and application of the relevant concepts and

frameworks.4. Ability to think strategically (i.e. integrate across internal & external

environment; across functional areas; holistically from firm perspective).

5. Ability to arrive at logical and well reasoned conclusions/recommendations.

Dimension

Fail

(1)

Below

(2 to

Expectations

3)

Approaches Expectations

(4)

Meets Expectations

(5)

Exceeds

(6 to

Expectations

7)

1 1 2 3 4 5 6 7

2 1 2 3 4 5 6 7

3 1 2 3 4 5 6 7

4 1 2 3 4 5 6 7

5 1 2 3 4 5 6 7

Process adopted for Cycle #3 retained. MGT 405 final exams used Two independent CBA faculty members rated.

Additional rubric revision/refinement in Cycle #4. Descriptions of levels added to each dimension. “Approaches Expectations” dropped in favor of 3 point scale

to provide consistency across assessment measures:▪ Exceeds Expectations▪ Meets Expectations▪ Below Expectations

Task Below Expectations Meets Expectations Exceeds Expectations Total

Demonstrates a clear understanding of the question

(10% of total)

Response does not address all aspects of the question.

(0-6 points)

Response is linked to all aspects of the question, but is not to the point. A significant portion of the response has peripheral link to the question.

(7-8 points)

Response directly addresses all aspects of the posed question; response demonstrates a in-depth understanding of the required response.

(9-10 points)

Demonstrates effective application of relevant concepts

(20% of total)

Response does not use relevant concepts to address question; response does not demonstrate a clear understanding of the relevant concepts; response discusses irrelevant concepts.

(0-13 points)

Response uses relevant concepts to address question; response demonstrates an understanding of the relevant concepts; however, the response is not succinct and directly informed by the concept. Some extraneous concepts summarized and discussed.

(14-17 points)

Response is informed directly by relevant concepts; response demonstrates a clear and in-depth understanding of the concepts; no extraneous concepts invoked.

(18-20 points)

Demonstrates effective use of case data

(15% of total)

Limited use of quantitative and qualitative case data to support position.

(0-10 points)

In-depth use of selective quantitative and qualitative case data to support position.

(11-13 points)

Comprehensive and in-depth use of appropriate quantitative and qualitative case data to support position.

(14-15 points)

Demonstrates ability to think strategically

(20% of total)

Response does not demonstrate an ability to think holistically; does not integrate internal and external factors and strategy; does not take an organizational perspective; considers functional areas in isolation and takes an operational perspective.

(0-13 points)

Response demonstrates some ability to think holistically; integrates a few internal and external factors and some connection to strategy and integrate across a couple of functions. Some evidence of taking an organizational perspective.

(14-17 points)

Response demonstrates a clear ability to think holistically; integrate several internal and external factors to strategy and integrate across multiple functions; clear evidence of an organizational perspective with an understanding of operational issues.

(18-20 points)

Demonstrates ability to arrive at well reasoned conclusions

(15% of total)

Response does not arrive at logical conclusions and inferences. Statements are made and not supporting logic is offered.

(0-10 points)

Response arrives at logical conclusions and inferences. Statements are made and some supporting logic is offered.

(11-13 points)

Response arrives at very logical conclusions and inferences. Position is well argued and tightly presented with supporting logic.

(14-15 points)

Demonstrates ability to communicate effectively

(20% of total)

Series of individual paragraphs that are not well connected nor well presented; paragraphs do not have strong lead sentences; sloppy paper with numerous spelling and grammatical errors.

(0-13 points)

Individual paragraphs that are reasonably well connected and cogent; reasonably strong lead sentences; generally competent paper marred with a few spelling, punctuation and grammatical errors.

(14-17 points)

Very well organized response with inter-connected paragraphs. Strong lead sentences; craftsmanship and style exhibited throughout the entire report.

(18-20 points)

Total

Process adopted for Cycle #3, used in Cycle #4 retained. MGT 405 final exams used Two independent CBA faculty members rated.

Rubric revision in Cycle #5: Descriptions of levels as added in Cycle #4 retained. 3 point scale added in Cycle #4 retained. Point ranges within three point scale dropped in favor of:

▪ 1 – Below Expectations; 2 – Meets Expectations; 3 – Above Expectations▪ However, ½ and ¼ point judgments acceptable when appropriate.

Dimensions reduced to two to better align with Critical Thinking SLOs.▪ Third dimension rating writing used, results not included in this report.

Dimension Below Expectations

1

Meets Expectations 2

Above Expectations 3

Collect and organize critical data and information to solve a problem.

Response does not use relevant concepts to address question; response does not demonstrate a clear understanding of the relevant concepts; response discusses irrelevant concepts. Limited use of quantitative and qualitative case data to support position.

Response uses relevant concepts to address question; response demonstrates an understanding of the relevant concepts; however, the response is not succinct and directly informed by the concept. Some extraneous concepts summarized and discussed. In-depth use of selective quantitative and qualitative case data to support position.

Response is informed directly by relevant concepts; response demonstrates a clear and in-depth understanding of the concepts; no extraneous concepts invoked. Comprehensive and in-depth use of appropriate quantitative and qualitative case data to support position.

Analyze information and follow logical steps to reach an effective decision.

Response does not demonstrate an ability to think holistically; does not integrate internal and external factors and strategy; does not take an organizational perspective; considers functional areas in isolation and takes an operational perspective. Response does not arrive at logical conclusions and inferences. Statements are made and not supporting logic is offered.

Response demonstrates some ability to think holistically; integrates a few internal and external factors and some connection to strategy and integrate across a couple of functions. Some evidence of taking an organizational perspective. Response arrives at logical conclusions and inferences. Statements are made and some supporting logic is offered.

Response demonstrates a clear ability to think holistically; integrate several internal and external factors to strategy and integrate across multiple functions; clear evidence of an organizational perspective with an understanding of operational issues. Response arrives at very logical conclusions and inferences. Position is well argued and tightly presented with supporting logic.

Ability to communicate effectively in writing.

Series of individual paragraphs that are not well connected nor well presented; paragraphs do not have strong lead sentences; sloppy paper with numerous spelling and grammatical errors.

Individual paragraphs that are reasonably well connected and cogent; reasonably strong lead sentences; generally competent paper marred with a few spelling, punctuation and grammatical errors.

Very well organized response with inter-connected paragraphs. Strong lead sentences; craftsmanship and style exhibited throughout the entire report.

Cycle #1: Fall Semester 2006 & Spring Semester 2007 in MGT 405 (International Business Strategy & Integration):▪ Sample Size: 175

Cycle #2: Fall Semester 2009 in MGT 405▪ Sample Size: 124



Cycle #5: Data collected In Fall Semester 2012 in MGT 405, analyzed Spring 2013 ▪ Sample Size: 122

Updated through here . . .

SLO #1: Collect and organize critical data and information to solve a problem. Cycle #1: Issue Identification Cycle #2 - #4: Clear understanding of the question and

issues to be addressed. Cycle #1: Use of Evidence & Data Cycle #2: Ability to effectively use case data (depth &

breadth) to illustrate a position. Cycles #3 & 4: Knowledge of case facts & ability to

effectively use case data to illustrate position. Cycle #5: Single dimension to assess SLO #1 as

indicated on rubric.

SLO #2: Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. Cycle #1: Models & Frameworks Cycle #2: 1) Understanding of relevant concepts & frameworks; 2) Effective

application of relevant concepts to address question/issues Cycle #3 & #4: Effective understanding and application of the relevant concepts &

frameworks. Cycle #1: Conclusions & Recommendations Cycle #2: Ability to arrive at logical and well reasoned conclusions based on

discussion. Cycle #3 & #4: Ability to arrive at logical and well-reasoned

conclusions/recommendations. Cycle #2: Ability to think strategically (i.e. integrate across internal & external

environment; across functional areas; take a general management view. Cycle #3 & #4: Ability to think strategically (integrate across internal & external

environments, functional areas, holistically from firm perspective). Cycle #5: Single dimension to assess SLO #2 as indicated on rubric.

BENCHMARKS: 85% of our students should meet or exceed

expectations for critical thinking skills 50% of our students should exceed

expectations for critical thinking skills

Cycle #1

53%

23%

13% 11%

“85% should meet or exceed expectations”; 83% actually did“50% should exceed expectations”; 53% actually did

22%

43%

27%

8%


14%17%

3%

65%


26%

45%

23%

7%


“85% OF OUR STUDENTS SHOULD MEET OR EXCEED

EXPECTATIONS”

NO Issue

Identification Use of Evidence &

Data Models &

Frameworks Conclusions &

Recommendations

“50% OF OUR STUDENTS SHOULD EXCEED EXPECTATIONS”

YES Issue

Identification NO

Use of Evidence & Data

Models & Frameworks

Conclusions & Recommendations

4 recommendations were made by the assessment team regarding ways to improve the analytic and critical thinking skills of our students. Faculty should encourage students to make clear problem

statements without hedging. Students should be asked to show their use of models and tools and

to demonstrate a clear connection between those models and their analysis.

Faculty should model the use of data in developing solutions to cases and problems.

Students should be given opportunities to learn how to support recommendations with evidence by writing and revising these sections of their papers.

An additional recommendation was to urge faculty to grade assignments with a rubric and to distribute the rubric to students prior to assignment submission

A Memo was sent to all CBA faculty detailing the recommendations and strongly urging them to adopt them

A number of faculty members across the college expressed interest in learning more about the use of rubrics and rubric development.

The CBA Assessment Committee developed a short “rubric primer” which was distributed to the CBA faculty along with examples of rubrics available online and rubrics currently being used in the college.

Cycle #2

39%

53%

5% 3%


19%

52%

11% 18%


27%

50%

10% 13%


24%

47%

17% 12%


21%

63%

7% 9%


21%

53%

15%11%



EXPECTATIONS”

YES Understanding question &

issues NO

Understanding relevant concepts & frameworks

Application of relevant concepts to issues

Effective use of case data Ability to think

strategically Ability to arrive at

conclusions


NO Understanding question

& issues Understanding relevant

concepts & frameworks Application of relevant

concepts to issues Effective use of case

data Ability to think


conclusions


EXPECTATIONS”

NO SLO #1: Collect and

organize critical data and information to solve a problem.

SLO #2: Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision.





SLO #1

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Cycle #1

Cycle #2

SLO #2

0%

10%

20%

30%

40%

50%

60%

Cycle #1

Cycle #2

* Dimensions combined to form single SLO measure

Our students appear to have improved slightly in meeting expectations between Cycles #1 & #2. The benchmark was not achieved for any

dimension in Cycle #1 but was achieved on one and very nearly on a second dimension in Cycle #2.

Overall, however, benchmarks were not met for either SLO.

Our students continue to struggle with critical thinking.

The Undergraduate Committee was heartened by the improvement between Cycles #1 & #2 albeit small.

A decision was made to reassess Critical Thinking in one year to determine if the improvement was the beginning of a trend prior to making additional “Loop Closing” decisions.

Cycle #3

0%

10%

20%

30%

40%

50%

60%

70%

80%

ExceedsExpectations

MeetsExpectations

ApproachesExpectations

BelowExpectations

Clear understanding of the question and issues to be addressed.

17.6%

67.2%

10.1% 5%


0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

ExceedsExpectations

MeetsExpectations


BelowExpectations

Knowledge of case facts & ability to effectively use case data to illustrate position.

16.8%

57.1%

16.8%9.2%


0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

ExceedsExpectations

MeetsExpectations


BelowExpectations

Effective understanding and application of the relevant concepts & frameworks.

10.1%

52.1%

21%16.8%


0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

ExceedsExpectations

MeetsExpectations


BelowExpectations

Ability to think strategically

11.8%

57.1%

18.5%12.6%


0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

ExceedsExpectations

MeetsExpectations


BelowExpectations

Ability to arrive at logical and well reasoned conclusions/recommendations.

6.7%

56.3%

18.5% 18.5%



EXPECTATIONS”

YES Understanding question

& issues NO

Understanding & application of relevant concepts & frameworks

Effective use of case data Ability to think


conclusions


NO Understanding

question & issues Understanding &

application of relevant concepts & frameworks

Effective use of case data


Ability to arrive at conclusions


EXPECTATIONS”








Although the number of students meeting expectations continues to increase, there was a significant decline in those exceeding expectations between Cycles #2 & #3.

The percent of students failing to meet expectations for SLO #2 was particularly discouraging.

The UG curriculum map indicated that virtually all faculty members believed they were delivered critical thinking skills in their courses.

The Undergraduate Committee and the Assessment Committee discussed and concluded that faculty needed specific models of exercises and assignments that address critical thinking.

A “Best Practices in Critical Thinking” workshop for faculty was planned and implemented.

Cycle #4

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Exceeds Expectations Meets Expectations Below Expectations

Clear understanding of the question and issues to be addressed.

41.7%

54.2%

4.2%

“85% should meet or exceed expectations”; 95.8% actually did“50% should exceed expectations”; 41.7% actually did

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%


Knowledge of case facts & ability to effectively use case data to illustrate position.

15.8%

74.2%

10.0%


0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%


Effective understanding and application of the relevant concepts & frameworks.

10.8%

76.7%

12.5%


0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%



13.3%

78.3%

8.3%


0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%


Ability to arrive at logical and well reasoned conclusions/recommendations.

23.3%

70.0%

6.7%



EXPECTATIONS”

YES Understanding question

& issues Understanding &






NO Understanding

question & issues Understanding &






EXPECTATIONS”

YES SLO #1: Collect and







SLO #1

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

Cycle #1

Cycle #2

Cycle #3

Cycle #4

SLO #2

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%

Cycle #1

Cycle #2

Cycle #3

Cycle #4

* Dimensions combined to form single SLO measure

The improvement in the % of students meeting and/or exceeding expectations was heartening.

Improvements in the % of students exceeding expectations were noted but it is acknowledged that the % remains well below the established benchmark. The UG Committee believes that with 4 cycles

of data complete it may be time to reassess the “Exceeding” benchmark.

Cycle #5

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%


19.7%

74.6%

5.7%


0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%


15.6%

77.9%

6.6%



EXPECTATIONS”

YES SLO #1: Collect and







SLO #1

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%

Cycle #1

Cycle #2

Cycle #3

Cycle #4

Cycle #5

SLO #2

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%

Cycle #1

Cycle #2

Cycle #3

Cycle #4

Cycle #5

* Dimensions combined to form single SLO measure** “Approaches” dropped after Cycle #3

Slight improvement in the % of students meeting and/or exceeding expectations for both SLOs. Not significant.

Percent of students exceeding expectations for SLO #1 remained constant from Cycle #4 however 9 % decline in % of students exceeding expectations for SLO #2.

Acknowledged that percent exceeding expectations continues to remain well below the established benchmark. The UG Committee believes that with 5 cycles of data

complete it may be time to reassess the “Exceeding” benchmark.

Assessed: 5 Cycles 2006, 2009, 2010, 2011, 2013.

Documents

Transcript of Assessed: 5 Cycles 2006, 2009, 2010, 2011, 2013.