A Rubric-based Approach for Operationalizing the

Entrepreneurial Mindset: An exploratory investigation

J-D YoderProfessor and ChairMechanical Engineering

Rob KleineAssociate Professor of MarketingPharmaceutical Business Program Coordinator

We’ve got rubrics, and we’re here to help you.

Ohio Northern University• Approximately 3500 students among five

colleges• College of Engineering is about 450

students across 5 majors• Largely traditional, residential students

Kern Entrepreneurship Education Network (KEEN)

• A program of the Kern Family Foundation

• Started in 2005• Private universities with ABET

accreditation• Focus on the entrepreneurial

mindset, not entrepreneurship

The KEEN Network

Baylor University (TX)Boston University (MA)Bucknell University (NY)Gonzaga University (WA)Kettering University (MI)Lawrence Technological University (MI)Mercer University (GA)Milwaukee School of Engineering (WI) Ohio Northern University (OH) St. Louis University (MO)

Santa Clara University (CA)Union College (NY)University of Dayton (OH)University of Detroit-Mercy (MI)University of Evansville (IN)Villanova University (PA)Western New England University (MA)Widener University (PA)Worcester Polytechnic Institute (MA)

The Entrepreneurial Mindset (KEEN)1. Effectively collaborate in a team setting2. Apply critical & creative thinking to ambiguous

problems3. Construct & effectively communicate a customer-

appropriate value proposition4. Persist through and learn from failure5. Effectively manage projects through appropriate

commercialization or final delivery process6. Demonstrate voluntary social responsibility7. Relate personal liberties and free enterprise to

entrepreneurship

Motivation

• Many schools in the network have created engineering-mindset-related activities.

• How do we know which ones work?• How can we leverage “best in class”

activities?• Note – don’t want it to be relative to student

feelings about the activity, but on the outcomes.

Can you answer these questions?

• How effectively is my PROGRAM cultivating the entrepreneurial mindset in our students?

• How effective is each ELEMENT in my program at cultivating intended aspects of the entrepreneurial mindset (whatever the unit: minor, course, project, in-class activity, etc.)

Our effort:

• Develop a common set of evaluation tools.

• Rubric-based.• Evaluate student work.

A Rubric-based Approach for Operationalizing and Assessing the Entrepreneurial Mindset

Source: Kleine & Yoder (2011), JEEN

The process: Activity

StudentArtifact

RubricsData Analysis

ReflectionRevision?

Exploratory Investigations Utilizing the Rubric-Based Approach

• Faculty training protocol• Faculty using and testing the

rubrics

A faculty training protocol for rubric-based assessment Preparation (Kleine & Reid 2012)

1. A student learning outcome of interest2. A Rubric3. A student activity that generates student

work (artifacts) relevant to the learning outcome

4. At least two examples of student work generated by the activity:

– High proficiency – Low proficiency

Materials distributed to each participant

1. The rubric that will be applied. 2. A score sheet for each artifact to be

evaluated.3. Details about the activity (assignment)

that generated the student work. 4. Student work examples (artifacts) that

illustrate high and low proficiency (each on a separate page.

Example Artifact Score SheetArtifact Score Sheet

(Please Refer to the Effective Collaboration Rubric)

Does Not Meet Expectations (1) Developing (2)

Meets Expectations (3) Proficient (4)

Not Applicable

Contributes to Team Meetings

Facilitates the Contributions of Team Members

Fosters Constructive Team Climate

Responds to Conflict

Activity: A Reflection Paper A one-page “Team reflection” should be written to describe the results of your “Team evaluation” rubric. This memo should allow you to reflect on whether your team has been effective, especially considering qualities of effective teams we have discussed. You may also include any concerns about team dynamics (these concerns will not be shared with your teammates without your consent).

Sample Artifact Low Proficiency (Student JJ)Team LL has had a rough start. After having six team meetings the group has failed to have a hundred percent attendance to a single meeting (AA has missed 4, DD 3, RR 1, and JJ 0.) Without the whole group present, our decision making and brainstorming has been continuously pushed back. When we do have meetings they are somewhat productive, however, for the following meeting we must repeat what was said previously to ensure everyone is on the same page. Our team meetings usually last one hour, during which I do most of the talking while everyone else just agrees with me. Being the team leader I understand that it is my job to run the meetings and to keep everyone on track, but it becomes a nuisance when no one else has any suggestions and must be told to help. The poor attendance is a result of poor communication skills. Now that email addresses and phone numbers were exchanged I hope that communicating will no longer be a problem.

Sample Artifact Low Proficiency (Student JJ)After having six team meetings … When we do have meetings they are somewhat productive, …

… Our team meetings … during which I do most of the talking while everyone else just agrees with me. Being the team leader I understand that it is my job to run the meetings and to keep everyone on track, but it becomes a nuisance when no one else has any suggestions and must be told to help.

The poor attendance is a result of poor communication skills. Now that email addresses and phone numbers were exchanged I hope that communicating will no longer be a problem.

Sample Artifact High Proficiency (Student MM)PW Inc. is made up of four members: NN, KK, MM, and SS. The team has worked well together and

all the members get along with each other very well. All the individuals on the Team have contributed to the project equally so far. This is very beneficial for the team because all the assignments are getting done in a reasonable amount of time. The culture identity was divided up so that each person had to research certain information. As a team everything has been evaluated and everyone has come to a consensus on team assignments so that no individual’s opinion is less important than another. Each person has pulled his or her weight on assignments and/or a task assigned to them, which shows that everyone has respect for each other and wants to succeed. The key characteristic of the PW Inc. team is our ability to effectively communicate. The team has a lot of meetings to insure that everything can get done in a timely manner, so communication is key and so far it has been excellent. In addition to communication, attendance at each team meeting is valuable to getting things accomplished in a timely manner which the group has done a great job of so far. The team doesn’t waste time on unimportant things, but rather focuses on the duties that need to get done, such as deciding the company name, working on the cultural identity, and creating the company.

One aspect I feel like the team has not done well is in group dynamics. I believe that some group members don’t always participate in discussions because he or she doesn’t want to be wrong or cause a distraction. My feeling is we are mature enough to not get offended if someone else has a better idea or an opinion that we don’t necessarily agree with at the time.

Working together and communication are the two main keys to a successful team and so far I believe that PW Inc. is building for success. We have followed these two keys and have accomplished all the tasks assigned to the group.

Sample Artifact High Proficiency (Student MM)As a team everything has been evaluated and everyone has come to a consensus … Each person has pulled his or her weight on assignments and/or a task assigned to them, which shows that everyone has respect for each other and wants to succeed. The key characteristic of the PW Inc. team is our ability to effectively communicate.

… communication is key and so far it has been excellent.

… The team doesn’t waste time on unimportant things, but rather focuses on the duties that need to get done, …

… the team has not done well is in group dynamics. I believe that some group members don’t always participate in discussions because he or she doesn’t want to be wrong or cause a distraction. My feeling is we are mature enough to not get offended if someone else has a better idea or an opinion that we don’t necessarily agree with at the time.

Working together and communication are the two main keys to a successful team … I believe that PW Inc. is building for success.

Example Artifact Score SheetArtifact Score Sheet

(Please Refer to the Effective Collaboration Rubric)

Does Not Meet Expectations (1) Developing (2)

Meets Expectations (3) Proficient (4)

Not Applicable

Contributes to Team Meetings

Facilitates the Contributions of Team Members

Fosters Constructive Team Climate

Responds to Conflict

Distribution of Ratings on the “Fosters Constructive Team Climate” row of the “Effective Collaboration in a Team Setting” rubric

Does Not M

eet Exp

ectations

Developing

Meets

Expecta

tions

Proficie

nt

Not Applic

able02468

1012

JJ MM

NOTE: n=23. The distributions are statistically distinct (Chi-squared=27.5, df=4, p<.001).

“Effective Collaboration in a Team Setting” Aggregate Ratings

Does Not M

eet Exp

ectations

Developing

Meets

Expecta

tions

Proficie

nt

Not Applic

able05

10152025303540

JJ MM

Faculty Using and Testing the Rubrics

• ONU Faculty in Colleges of Engineering and Business

• Faculty at other KEEN institutions

Rubric Testing Protocol

Pick a KEEN learning outcome

(or two)

Identify a class activity you do

already (or create a new one)

Apply the rubric(s) to one or more examples of

student work generated by the

activity

Provide feedback on how we can

improve the rubric(s)

Work in progress

• Intend to use in additional courses at ONU.

• Expand to other KEEN schools.• All schools are welcome to participate.• Use feedback to modify rubrics as

needed.• Inventory entrepreneurial mindset

activities by learning outcome

Mapping Activities into Entrepreneurial Mindset Student Outcomes

Feb. 2012 KEEN Student Outcomes (KSO) Contact: ___________________________

1 2 3 4 5 6 7

Effectively collaborate in a team setting

Apply critical & creative thinking to ambiguous problems

Construct & effectively communicate a customer-appropriate value proposition

Persist through and learn from failure

Effectively manage projects through appropriate commercialization or final delivery process

Demonstrate voluntary social responsibility

Relate personal liberties and free enterprise to entrepreneurship

For-Credit Courses Provide course number and title

  Co-curricular Activities

Extra-curricular Activities

Opportunities

• Data Aggregation• Integrating with ABET• Integrating Entrepreneurial Mindset and

General Education assessment systems

Conclusion

Our goal is to answer the question empirically:

How effective are our efforts to cultivate the Entrepreneurial Mindset in our students at each of the following levels:

• KEEN network • Individual KEEN school • Program • Course • Assignment/activity

Thank You!

Questions?

Misc debris follows

Connect General Education and the Entrepreneurial MindsetONU Gen Ed Learning Outcomes1. Effective communication 2. Critical and creative thinking3. Scientific and quantitative literacy 4. An understanding of diverse cultures

and their effects on human interaction5. Integration of concepts across

disciplines6. Informed and ethical responses to

personal, civic, and global needs7. Informed responses to aesthetics in

art or nature

KEEN Learning Outcomes1. Effectively collaborate in a team setting2. Apply critical & creative thinking to

ambiguous problems3. Construct & effectively communicate a

customer-appropriate value proposition

4. Persist through and learn from failure. 5. Effectively manage projects through

appropriate commercialization or final delivery process

6. Demonstrate voluntary social responsibility

7. Relate personal liberties and free enterprise to entrepreneurship

Motivating vision

• Cultivate the entrepreneurial mindset in every ONU student.

• Support those students interested in becoming entrepreneurs

Entrepreneurial Mindset for All

Supporting Future Entrepreneurs

Zach Ferres

Chris Corrado

Entrepreneurship Education at ONU Circa 2004

Entrepreneurship education at ONU Circa 2004

A Multi-Pronged Approach to Engaging the Entrepreneurial Mindset

Curriculum

Faculty

Development

Extra/Co-Curricular

Activities

Assessment

Entrepreneurial Mindset Touch

Points across the Four Years

Highlights: Student Engagement

Polar Innovation Exchange Competitions

Student competitions

• Polar Innovation Exchange

• Elsewhere• IIT Idea Chase • Innovation Chase at LTU

Highlights: Speakers that Inspire the Entire Campus Community

Paul Polak author of Out of Poverty and co-founder International Development Enterprises

Bob Peterson, Pixar Studios

Entrepreneurship education at ONU Circa 2004

Thanks

• The Deans• Dr. Baker and Dr. DiBiasio• A variety of faculty, especially Dan

Ferguson and Tammy Schakett• Fantastic students• KEEN and NCIIA• External allies