2017 Colorado Leadership Alliance Summit Report · 2017 Colorado Leadership Alliance Summit Report...

2017 Colorado Leadership Alliance Summit Report

January 28th, 2017 at Metro State University, Denver, Colorado

Full report available at cpd.colostate.edu

Report completed by the Center for Public Deliberation Research Team

Colorado State University

Center for Public Deliberation 2

About the Center

The Colorado State University Center for Public Deliberation (CPD) serves as an impartial resource to the northern Colorado community. Working with students trained in small group facilitation, the CPD assists local government, school boards, and community organizations by researching issues and developing useful background material, and then designs, facilitates, and reports on innovative public events. The interpretations and conclusions contained in this publication have been produced by CPD associates without the input of partner organizations to maintain impartiality.

About the Foundation

The Colorado Leadership Alliance unites collegiate leadership programs from across Colorado, exposing students to community issues outside their campus at an annual summit and connecting program directors to share best practices. The Colorado Leadership Alliance is part of the Denver Metro Chamber Leadership Foundation, whose mission is to educate and inspire people who want to make a difference as leaders in their community.


Report on the 2017 Colorado Leadership Alliance Summit

A partnership between the Colorado Leadership Alliance and the Colorado State University Center for Public Deliberation

Event held on the 28th of January, 2017 at the Tivoli Student Union at Metro State

University in Denver, Colorado

Report Completed in Collaboration by: • Elizabeth Blythe • Julie Mann • Allie Salz • JR Santa Rosa • Nathan Sawkins • Cramer McGinty

Colorado State University Center for Public Deliberation Spring 2017 Research Team

Table of Contents

• Executive Summary 4 • Summary of Key Findings 4 • Deliberative Tensions and Takeaways 5 • Explanation of the Process 6 • Who was at the Event? 7 • Pre-Survey Results 8 • Dr. Carcasson’s Talk about the Nature and Psychology of

Polarization 13

• Values Ranking Exercise 16 • Facilitated Small Group Conversation Exercise Description 19 • Data Analysis Methods 22

o Approach 1 (Maximize Student Choice) Results 23 o Approach 2 (Give College Departments Greater Control

over their Students’ Overall Curriculum) Results 25

o Approach 3 (Improve the Effectiveness of the Current Model with Important but Smaller Changes) Results

27

o Approach 4 (Adapt the Core Curriculum to Better Address 21st Century Needs) Results

29

• Core Curriculum Action Steps from Dot Exercise 32 • Facilitative Leadership Talk and Debrief from Group Discussions 34 • Process Reflections and Survey Responses 35


Executive Summary On January 28th, 2017, student participants of the Colorado Leadership Alliance and Colorado State University’s Center for Public Deliberation gathered at the Colorado Leadership Summit to discuss leadership on college campuses. For this event, 192 student participants traveled from twelve different colleges and universities to learn about how deliberative techniques can improve communication in college leadership. This partnership’s purpose was to provide an example of deliberation in order to show its fruitful connection to the future of campus leadership. Through this exploration of leadership and deliberation, students would learn the skills and processes necessary to implement this revolutionary approach to leadership on each of their own campuses. The topic of the deliberative example was the purpose of core curriculum classes; however, this topic served solely as a vehicle to the actual goal of the Summit—to gain new skills in order to improve leadership on college campuses. Discussion of the exemplary deliberative topic was preceded and followed by talks presented by Martin Carcasson, the founder and director of the Center for Public Deliberation. Dr. Carcasson first introduced the values and processes of deliberation and then illustrated its essential connection to leadership following the students’ participation in the process. Summary of Key Findings

• During the “Values Ranking Exercise” participants ranked equality (on average) as the most important value to them (M = 3.693, #1: 41, #2: 34, #3: 28). Conversely, participants ranked security (on average) as one of the least important values to them (M = 5.641, #8: 51), while also overwhelmingly ranking consistency/tradition as the least important value to them (M = 8.198, #9: 129). The rest of the values (progress/innovation, individual responsibility, justice, community, diversity, and freedom) were fairly evenly distributed in terms of importance to the event participants.

• Throughout the discussion there were many common themes that came up among multiple groups. The most prevalent theme among participants was a general favorability toward the university core curriculum and general education requirements. Participants expressed interest in creating a greater variety of options within the core curriculum, with some courses specifically geared toward addressing some of the issues and topics relevant to the 21st century. Additionally, many students thought that core curriculum courses need to address things like life skills and personal finance, and should perhaps do so by going outside of the classroom (i.e., service learning). Furthermore, there were multiple instances where participants noted the need for a greater emphasis placed on teaching and instructor evaluation to make sure that courses are taught effectively.

• Participants were given a worksheet to evaluate each approach. Participants indicated that they were largely in favor of ensuring student choice, whether that is the belief that students are the customers who are entitled to curriculum customization, or the idea that students should be free to develop their own values and beliefs. Conversely, participants were also in favor of making changes to the core curriculum such as continuous course evaluation, adding courses that address contemporary needs, and making sure that some courses are tailored and relevant to specific majors.

• In a survey about the CPD process, 132 of 148 survey respondents indicated that they were either “Satisfied” (78; 54.16%) or “Very satisfied” (54; 37.5%) with the forum.


Deliberative Tensions and Takeaways One of the goals of deliberative civic engagement is to have a meaningful conversation where all voices are heard so that we as citizens can take greater ownership over our political processes. It is the responsibility of deliberative practitioners to not only present the results of deliberative discussions, but to also make suggestions about how to most effectively move the conversation forward in similar contexts. Throughout the report, there are sections that outline the deliberative tensions and trade-offs of each approach to improving core curriculum. Each of those sections is provided at the end of every analysis of an approach (for approach one, see page 24, for approach two, see page 26, for approach three, see page 28, and for approach four, see page 30). In addition to a discussion of the deliberative tensions and takeaways within each approach, there is also a discussion of the deliberative tensions and takeaways derived from the conversation as a whole. The aforementioned discussion is provided below, as well as on page 34. Participants frequently discussed many of the same underlying deliberative tensions and trade-offs throughout the conversation, regardless of which approach was being discussed. Some of the tensions were:

• Freedom (i.e., extent to which students choose their own classes) vs. Consistency (i.e., extent to which all students receive a common education)

• The purposes and assumptions of higher education: o Higher education as a business vs. Higher education as an institution for transformation o Students as customers vs. Students as pupils o Higher education as job training vs. Higher education as vehicle for personal growth

• Major change (i.e., creating a whole new system) vs. Minor change (i.e., making slight changes to our existing system)

• Increased costs for change vs. Rearranging of current funds for change Future conversations about how to improve core curriculum should focus on a few things:

1. There needs to be further discussion about what students think the purposes of higher education are, and should be. Additional discussion about how their perceptions differ from faculty would be helpful.

2. How should increasing costs of higher education affect, if at all, the core curriculum? 3. What would it look like to create an entirely new system in place of core curriculum versus what it

would look like to make minor changes to the existing system?


Explanation of the Process The Colorado Leadership Alliance Summit was held on Saturday, January 28th at the Tivoli Student Union on the campus of Metropolitan State University, in Denver. This was an all-day event, though the sessions hosted by Center for Public Deliberation (CPD) were only from 9am to 2pm (schedule below; details in separate sections). Participants arrived between 7am and 8am to get settled in, eat breakfast, and complete ice-breaker activities. At 9am, representatives from the Colorado Leadership Alliance introduced Dr. Martín Carcasson, Director of the CPD, to speak about wicked problems and the need for deliberative engagement. Overall, the day’s schedule was designed to give participants an introductory lecture to deliberation, with hands-on activities and discussions during and after the introductory lecture to expose participants to deliberation and facilitative leadership. The day concluded with a lecture from Dr. Carcasson about how to utilize and perform facilitative leadership based on the day’s activities. The first and third sessions were held in a large room with 32 tables of roughly 6-7 participants and one CPD facilitator. The second session was held in small, breakout rooms throughout the building, with 26 groups consisting of roughly 9 participants, one facilitator, and one note-taker. For this exercise, some groups were accompanied by an observer. The schedule for the event was as follows: 8:00 – 9:00 Participants arrive, eat breakfast, and introduction by CLA is presented.

9:00 – 9:30 Talk 1 (Dr. Carcasson’s Talk about the Nature and Psychology of Polarization): Martín Carcasson is introduced and gives an introductory talk about deliberation.

9:30 – 9:45 Values Ranking Exercise: A values exercise run by CPD students is done at the tables.

9:45 – 10:15 Talk 1 (continued): Martin continues his talk on deliberation.

10:15 – 10:30 Break: Students were given 15 minutes to get to their assigned breakout rooms.

10:30 – 12:00 Facilitated Small Group Conversation Exercise): CPD students run small group discussions in breakout rooms.

12:00 – 12:55 Lunch: Event participants and staff take a break to eat lunch.

12:55 – 2:00 Facilitative Leadership Talk and Debrief from Group Discussions: Everyone returns back to main room. Martin presents about the process and leadership with some participant small group work debriefing the process (conversation facilitated by CPD students). Participants fill out final surveys.

2:00 All CPD-related activities are finished and the rest of the Summit continues.


Who was at the Event? Originally, there were more than 200 participants registered to attend the event. However, not all who were registered actually attended the event. Due to the fact that the CPD did not get a formal count of all participants in attendance, the number of “Values Worksheets” from the “Values Ranking Exercise” (see “Values Ranking Exercise” section below) was used to get an approximation of the number of event participants, of which there were 192.

Demographic data was not formally collected from participants, however, a pre-survey (see “Pre-survey Results” section below) that was sent exclusively to those attending the event indicated various demographic characteristics of event attendees. Participants represented eleven different colleges across Colorado. All of the students were undergraduates participating in their college’s leadership program. Students attended the event to learn more about the values that they embody by being a part of the Colorado Leadership Alliance and to explore how deliberative techniques can influence their roles as leaders in their campus communities.

Figure 1 Based on the survey results (see “Figure 1”), there were indications of a large proportion of attendees from the University of Denver and the University of Colorado (Boulder). Figure 2 indicates that more than half (90 students; 71.42%) of the students who took the pre-survey (and most likely attended the event) were in their first few years of higher education as Freshmen or Sophomores.

Figure 2

2724

18 17

129 8

4 42 1

0

5

10

15

20

25

30

What university do you attend? (126 Responses)

40.50%

31%

14.30%

14.30%

What year of college are you currently in? (126 Responses)

Freshman (51)

Sophomore (39)

Junior (18)

Senior (18)


Pre-Survey Results Prior to the event, members of the Colorado Leadership Alliance (who planned on attending the event) were sent a survey regarding the discussion topic of the 2017 Leadership Alliance Summit: core curriculum reform within colleges and universities. The survey was sent roughly two months prior to the actual event and received 126 responses. However, not all participants responded to every question, which the reason for the discrepancy in the number of responses for each question. Data from the survey was used to create the discussion materials for the deliberative group discussions (see “Facilitated Small Group Conversation Exercise” section for further details). After roughly three weeks, associates from the CPD compiled all of the data collected from the 126 respondents (no additional responses were collected after this). A preliminary report of all the responses was then compiled and analyzed form common themes and responses. The data and common themes were discussed amongst a group of CPD associates, which then informed the four approaches that were created for the group discussion (see “Facilitated Small Group Conversation Exercise” section for further details). Below are the questions that were asked in the survey and their corresponding responses. Note that the questions “What university do you attend?” and “What year of college are you currently in?” were not included in this section (despite the fact that they were part of the pre-survey) because they were included in the previous section “Who was at the Event?” for demographic purposes.

Figure 3

Participants were asked about their current overall satisfaction with their college experience (see “Figure 3”). All 126 participants responded to this question, with the overwhelming majority of them indicating that they were either “Satisfied” (67) or “Strongly satisfied” (48) with their overall college experience. Participants were also asked to indicate their major (see “Figure 4”). Due to the high variability in qualitative responses, a member of the CPD research team created categories of themes that were similar. The most common majors were either “Science” (34 responses) or “Business” (26 responses) majors.

Majors Themes Key A Science (biomedical studies, biochemistry,

neuroscience, chemistry, geospatial, physics, environmental, zoology, general science)

B Business (accounting, marketing, finance) C Social science (philosophy, sociology, history, social

legal studies, family studies) D Government, civics, politics, international relations E Engineering F Communication/language G Health and exercise science/ physiology/ public health H Nursing I Psychology J Art (performing and visual) K Math L English

Figure 4

38.10%

53.20%

7.90% 0.80% 0%

What is your current level of overall satisfaction with your college experience? (126 responses)

Strongly satisfied(48)Satisfied (67)

Neutral (10)

Dissatisfied (1)

Stronglydissatisfied (0)

34

26

1613 13

10 9 8 85 3 1

05

10152025303540

A B C D E F G H I J K L

Num

ber o

f Res

pons

es

Response Themes

Types of Majors of Participants


Figure 5

When asked what level of satisfaction participants had with the core curriculum and/or general education requirements, the majority (82) indicated that they were “Satisfied.” Many participants (26) also indicated that they felt a “Neutral” level of satisfaction with their core curriculum. The rest of the participants indicated that were either “Strongly satisfied” (12), “Dissatisfied” (4), or “Strongly dissatisfied” (2). Note that the majority of respondents identified as “Freshmen” [51] or “Sophomores” [39] meaning that their exposure to core curriculum courses is rather small compared to “Juniors” or “Seniors” (see “Figure 2” for details).

The following five questions in the pre-survey were in the format of free-response (rather than multiple choice/Likert scale) so that qualitative data could be gathered. In order to organize all of the data collected from the following questions, CPD associates performed thematic coding by categorizing each of the responses into separate categories. Note that not all responses were categorized due to their lack of conformity to a category.

Figure 6

Overall Thoughts Theme Key A Courses are valuable and important to overall education B These courses are unnecessary or waste of time C Courses are necessary (despite their drawbacks) D Create more well-rounded students E Nonspecific positive responses F Makes students take classes they would not have otherwise G Courses are reasonable and easy to accomplish H Not executed or implemented well I These courses are boring or uninteresting or not relevant J Too many core courses, reduce only to essential K Nonspecific neutral responses L These courses take up time on schedule M The professors need to take time to keep it interesting and

better explain the value of course N Not challenging enough O These courses have too many students which not conducive

for learning P Courses provide students freedom to explore interests

Participants were asked “Overall what do you think about core curriculum/ gen ed courses?” Respondents indicated both positive and negative feelings toward the core curriculum. For example “I think they are necessary, but not as many should be required.” This case and other similar responses it were tallied under multiple themes. The most prevalent positive reaction theme (theme A; 15 responses) was, “The courses are valuable and important to overall education.” An example of a response in that theme is “I think they are an important part of college and education in general.” The most prevalent negative response theme (theme B; 15 responses) was “these courses are unnecessary or a waste of time.” An example of a response in that theme is “I think it is unnecessary to take classes such as ‘Art Appreciation’ and other Humanities when I am a Mechanical Eng. Major.”

9.50%

65.10%

20.60%

3.20% 1.60%

What was your satisfaction with the core curriculum/gen ed courses you took? (126

responses)

Strongly satisfied (12)

Satisfied (82)

Neutral (26)

Dissatisfied (4)

Strongly dissatisfied(2)

15 1514

12 1211

108 8

76

4 43

2 2

02468

10121416

A B C D E F G H I J K L M N O P

Num

ber o

f Res

pons

es

Response Themes

Overall thoughts about Core Curriculum/ General Education Courses

(122 responses)


Purpose Theme Key A To provide a general base knowledge to

prepare for further schooling B To create well-rounded students and citizens C To encourage exploration of interests D To prepare for future beyond schooling E To create greater more diverse perspectives,

critical thinking and communication skills F Did not answer correctly G Institution is required to meet certain

educational requirements H To recognize interdisciplinary connections

Figure 7 Additionally, participants were asked “How would you describe the primary purpose of the core curriculum/ general education?” The most prevalent theme (theme A; 46 responses) was that the purpose of these courses is to provide students with general base knowledge to prepare them for further education. An example of a response in this theme is, “They are important base to take in order to progress to more advance classes.” The second most prevalent theme (theme B; 37 responses), of responses stated the primary purpose of core curriculum and general education requirements is to create well rounded students.

Figure 8

Favorite General Education Course Theme Key A English (literature and composition) B Social science (philosophy, anthropology,

psychology, sociology, history) C Science (physics, chemistry anatomy biology

physiology, geography) D Humanities (art, language, communication,

leadership, religion) E Math F Cultural/identity diversity G Did not answer correctly H Economics I Government, civics, law, politics J Environment K Business L Engineering M Technology

The survey also asked questions about specific core curriculum courses so that we could get a better idea of what is currently working and what needs to be improved. Participants were asked “What was your favorite core curriculum/gen ed class? Why?” Of the 124 responses, the most common favorite course indicated by participants (theme A; 21 responses) was “English” (i.e., composition and/or literature). The second most favorite course indicated by participants (theme B; 20 responses) was a course in the field of social sciences (i.e., philosophy, anthropology, history, sociology, and psychology). Some of the reasons respondents indicated favorability to specific courses included: “we had an incredible teacher who actually used and applied the knowledge,” and “because the content was very interesting and overall was presented in a compelling way,” and lastly, “because it provided meaningful and engaging discussions.”

4637

12 10 8 6 4 30

10

20

30

40

50

A B C D E F G H

Num

ber o

f Res

pons

es

Response Themes

What is the Purpose of General Education/ Core Curriculum Courses?

(121 responses)

21 2018

1412 11

85 5 4 4

1 10

5

10

15

20

25

A B C D E F G H I J K L M

Num

ber o

f Res

pons

es

Response Themes

Favorite General Education Courses (124 responses)


Least Favorite General Education Course Theme Key A Science (physics, chemistry, anatomy, biology.

physiology, geography) B English (literature and composition) C Math D Social science (philosophy, anthropology,

psychology, sociology, history, mythology) E Did not answer correctly F Humanities (art, language, communication,

leadership, ethic studies) G Government, civics, law, politics H Economics I Business J Technology K Cultural/identity diversity

Figure 9

Participants were also asked “What was your least favorite core curriculum/ general education course? And why?” The most prevalent type of class (theme A; 25 responses) that was indicated as respondents’ least favorite course was science (i.e., physics, chemistry, anatomy, biology, physiology, geography). For example, one student said “My least favorite would have to be the general chemistry courses. Chemistry can sometimes be overwhelming.” The second most prevalent type of class (theme B; 24 responses) that was indicated as respondents’ least favorite course was English (i.e., literature, composition). For example one respondent said, “writing, because I have taken so many writing classes in high school already that I felt it was unnecessary and redundant.” Some other reasons for disliking general education courses included, “had already taken it in high school,” “course was too difficult,” “lecture too big,” and, lastly, “professor taught course poorly and wasn’t relevant to major.”

Figure 10

Power to Choose Theme Key A Life skills/ how to adult 5 Did not answer correctly B Non-main theme responses C Personal finance/ financial literacy D Culture E Diversity/ inclusivity based, social problems/

dialogue F Business (accounting, human resources,

management)) G Social science (Philosophy, sociology) H Arts visual and performing- creativity based I Politics and civics J Did not answer correctly K Personal health (physical and mental) L Public speaking M Science (physics, biology, anatomy) N Language

Furthermore, participants were asked “If you had the power to add a course to the core curriculum that everyone would be required to take? Why?” The most prevalent theme (theme A; 18 responses) was that students should be required to take a life skills/how to adult class. Some responses that are included in this theme are, “I would add a life class, a class that teaches students about taxes, loans, job search, etc., so, they are more prepared for life after college.” Many other participants made more specific, but similar comments about the perceived need for a financial literacy course, which created a completely separate category (theme C; 15 responses).

25 2420

1612 11

3 3 3 2 2

0

5

10

15

20

25

30

A B C D E F G H I J K

Num

ber o

f Res

pons

es

Response Themes

Least Favorite General Education Courses (121 responses)

18 1715 14

11 108 7 6 5 5 4 4 3

0

5

10

15

20

A B C D E F G H I J K L M N

Num

ber o

f Res

pons

es

Response Themes

Courses that would be Required if Participants had Power to Choose (121

responses)


Participants were then asked a few questions about the overall purposes and underlying assumptions of higher education. For the purposes of creating a common understanding among event participants, the eight purposes of higher education outlined in Derek Bok’s (former President of Harvard University) book, Our Underachieving Colleges: A Candid Look at How Much Students Learn and Why They Should Be Learning More, were used as part of the pre-survey and event discussion materials.

Survey respondents were first asked to rank the eight purposes of higher education on a scale from “Most important to me” to “Not critical to me.” Participants were allowed to rank each purpose any way they wanted, but were instructed to only choose one purpose as “Most important to me”. The purpose with the highest number of “Most important to me” was “Learning to Think” (63 votes) meaning that respondents thought that learning to think was one of the most important purposes of higher education. Participants also thought that “Learning to Communicate” (81 votes) and “Building character” (75 votes) were “Very important” purposes of higher education.

Figure 11

Figure 12

Participants were also asked to indicate which purposes their specific universities needed to focus on addressing more via the core curriculum and/or general education requirements. Respondents indicated that “Learning to communicate” (35 votes), “Learning to think” (35 votes), and “Living with diversity” (32 votes) were the most important purposes of higher education that needed to focused on more by their universities. Conversely, respondents indicated that “Acquiring broader interests” (60 votes) and “Preparing for a career” (59 votes) were somewhat important for their universities to focus on.

0

20

40

60

80

100

120

140

Opinions about the 8 purposes of higher education

Not sure

Not critical to me

Should be anavailable option,but not required

Somewhatimportant

Very important

Most importantto me

0

20

40

60

80

100

120

140

Which purpose do you think your university needs to focus on more or less, particularly in terms of the gen

ed requirements?Not sure

Not critical to me

Should be anavailable option,but not requiredSomewhatimportant

Very important

Most important tome


Dr. Carcasson’s Talk about the Nature and Psychology of Polarization To kick off the event, Dr. Martín Carcasson, the Director of the Center for Public Deliberation and Professor of Communication Studies at Colorado State University, lectured about the nature of political polarization, and how leaders can use this information to change the way we have conversations. Carcasson began his process by explaining several factors of society that contribute to political polarization. Based on the things we have learned from social psychological research, most of the factors mentioned arise from our nature as human beings. We tend to crave certainty and consistency and therefore, we inherently create a “good vs. bad” scenario in which we portray ourselves as the “good guys” while the opposition is the “bad guys” so that we don’t actually have to do any of the hard work of problem solving. We are also very tribal; we tend to surround ourselves with those who think alike. We have the tendency to only seek out the information that supports our own views (or “confirmation bias”). Ultimately, we avoid dilemmas involving values, tensions, and making tough choices.

Stages of motivated reasoning What and who we expose ourselves to

selective exposure/echo chambers

How we interpret new evidence

confirmation bias

How we make attributions and tell stories

egoism, illusory correlation, negativity bias

How we make decisions

heuristics, self-serving bias, social proof

What we remember availability bias Figure 13

Carcasson then talked about how our polarizing human characteristics are further embedded in both our nature and society by talking through the “Stages of Motivated Reasoning.” There are five different stages: 1) what/who we expose ourselves to, 2) how we interpret new evidence, 3)how we make attributions and tell stories, 4) how we make decisions, and 5) what we ultimately remember. These stages contribute to why our society is polarized, as it shows the problems in how we obtain certain information and how it is utilized. However, Carcasson pointed out the fact that not everything about our human nature is bad or at odds with collaborative problem-solving. He highlighted that humans are inherently social beings and as a result, we seek purpose and community. We also generally value being empathetic, pragmatic, and creative, and ultimately, we have the ability to overcome our bad tendencies to build better habits. The big takeaway message from this section was that the most powerful way we can overcome our biases and build community is to have a “genuine conversation with someone we respect and trust.” After talking about polarization on a mostly micro-level, Carcasson then transitioned to talk polarization on a macro political level. Carcasson pointed out that our current system is adversarial, as it people are primarily focused on winning disputes and demeaning the opposition, rather than actually solving problems. Instead of taking accountability, individuals in an adversarial system focus on blaming others. Carcasson asserted that the problem with our current adversarial system is that our processes for public engagement and community problem solving revolve around tearing others down instead of building each other up because of things like our two-party system, the politicized media, and simply just bad listening skills.


The Vicious Cycle of False Polarization Figure 14

How Do We Change the Conversation? After outlining the current issues with how we solve problems, Carcasson proposed a possible solution by advocating for the adoption of a “wicked problems” mindset. Essentially, wicked problems are problems that do not have a clear “right” answer (e.g., abortion, tax reform, etc.). Methods to solve these problems tend to create new problems, and often require adaptive changes. To address them, the public must be a part of the solution. Adopting a wicked problems mindset was described by Carcasson as a way of looking beneath the surface of problems we face to truly examine the underlying assumptions and values behind each perspective, and recognizing that every perspective has inherent trade-offs.

Along with adopting a wicked-problems mindset, Carcasson asserted that we must also build our individual skills for collaboration. These skills were described by Carcasson as working to minimize the impact of negatively motivated reasoning by becoming more comfortable with uncertainty and conflict, while also making sure that we are open to actively listening and refining our opinions. Carcasson then had students engage in a values exercise to help them get a better sense of how wicked problems work. The first part of the exercise asked students use electronic clickers to identify and vote on which values – taken from the Constitution – were the most important (see “Figure 15”), and the least important (see “Figure 16”).

Figure 15 Figure 16 Equality was the most popular important value among participants. Conversely, Carcasson highlighted the fact that even though equality was voted the most important by the majority, it was deemed as the least important to others. He further pointed out that our values ultimately filter the information we receive and gather so that our views are reinforced.

3322

70

33 27

01020304050607080

Which is most important to you? (choose only one)

27

55

29

4332

0102030405060

Which is least important to you? (choose only one)

Individually developed subconscious biases

Negative interaction effects

The Russell effect Purposeful partisan manipulation

Media focus on conflict

Impact of the internet


The values exercise ultimately pointed out the fact that we have more in common than we think, however, Carcasson argued that we cannot begin to bridge our divides unless we change our public processes to focus on deliberative engagement. Carcasson talked about deliberative engagement as an approach to public engagement in which citizens are deeply involved in public decision making along with experts and/or politicians. After giving an overview of the need for deliberative engagement in helping to solve and talk about wicked problems, Carcasson introduced the topic of the event: how to improve the core curriculum and general education requirements at Colorado universities. The topic was chosen for its contemporary relevance and because it is a topic that all university students could discuss. While the subject matter of the conversation was important, the underlying goal was to expose members of the Colorado Leadership Alliance to conversation facilitation and deliberative discussion.

What is deliberative engagement? Deliberative democracy

Community problem-solving Participatory decision-making

Slow democracy Strong democracy

Multi-stakeholder dispute resolution Public participation

Democratic governance Collaborative governance

Organic or community politics Consensus building or seeking

processes Organic politics

Figure 17

The 8 purposes of higher education according to Derek Bok (see “Pre-survey Results” for more details) were introduced to participants to help them start thinking about core curriculum within universities. Participants were asked to vote on which purposes were important to them, as well as which purposes they believed universities need to focus on more. Figure 18 illustrates which purposes were ranked as the most important, while Figure 19 illustrated which purposes universities need to focus on more. Learning to Think, Learning to Communicate, and Preparing for a Career were deemed as most important, while Learning to Communicate, Learning to Think, and Preparation for Citizenship were deemed as ones that universities need to focus on more.

8 Goals of Higher Education Questions Key

A Learning to Communicate

B Learning to Think C Building Character D Preparation for

Citizenship E Living with Diversity F Preparing for a Global

Society G Acquiring Broader

Interests H Preparing for a Career

Figure 18

Figure 19

219272

15667 73 99

29

184

0100200300

A B C D E F G H

Rank the following key aims of higher ed in terms of their importance to you (Pick your top 3 in order)

171 173 150111

170132

87 102

0

100

200

A B C D E F G H

Rank the following key aims of higher ed in terms of what universities need to focus on more (Pick your top 3 in order)


Values Ranking Exercise

During Carcasson’s talk, students were invited to engage in an activity at their tables to explore some of the concepts he had discussed. The activity – known as the “Values Ranking Exercise” – is one that the CPD typically uses to introduce tensions between values that fellow participants hold. Participants were each given a stack of nine value terms. The value terms were: consistency/tradition, progress/innovation, equality, individual responsibility, justice, community, diversity, freedom, and security. Participants were then asked to rank the values from most important to least important.

After ranking their values from most important to least important, the participants were asked to give a short account of why they ranked their top value as most important and their bottom value as least important. These discussions took place between either the whole table or groups of two or three participants at each table; the format was up to the discretion of the facilitator. The participants were then asked some or all of the following questions (depending on time) pertaining to the values:

1. Can you imagine a situation where you get too much of one of your top 3 values? Can it dominate the others too much?

2. Can you make the positive case for a value in your bottom 3? 3. Do you see in particular tensions between the values in your top 3 or 4? If we focus on one, do we

tend to get less of another?

At the end of this exercise, the participants were given a worksheet with all nine values listed on it. They were then instructed to number the values as they had ranked them in the activity. There was also room on the worksheet for the participants to add another value that they believed to be essential and explain why it should be added. There were 192 worksheets collected from participants, and all worksheets were completed correctly (i.e., participants followed the directions of the assignment).

Figure 20 Overall, students who participated in this exercise rated equality as their most important value. This is indicated by the mean response rate of 3.693, meaning that on average, students ranked equality in their top 3-4. While there were still some students who rated it lower on the scale, over half of the students rated equality in their top three values, and 42 ranked it as their top value, the highest frequency for the top value.

Figure 21 Diversity was a value that tended in the middle rankings of students’ importance. The most common rating (30 students) for this value was a 5 out of 9. Some students ranked diversity as their highest or lowest, but the majority of ranks exist at or near the middle of the spectrum of importance.

42

3428 26

18 16

6

15

7

0

10

20

30

40

50

1 2 3 4 5 6 7 8 9

Res

pons

e R

anki

ng F

requ

ency

Numerical Value Ranking

Equality

9

25

20

29 3026

23

1614

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9Res

pons

e R

anki

ng F

requ

ency


Diversity


Figure 24 Students ranked the value of security all across the board. However, 25 percent of students rated Security as their second least important value. This means that security was ranked just behind consistency/tradition as the least important value to the students that did the values exercise.

Figure 25 Freedom was mostly ranked of middle importance to the students. There are spikes in the value rankings of 3, 5, and 7, none of which are at polar ends of the ranking. Very few participants (5), ranked freedom as their lowest value, but not that many (12), ranked it as their highest either.

1015

2216

1923

26

51

10

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9

Res

pons

e R

anki

ng F

requ

ency


Security

12

17

32

23

29

22

30

21

5

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9

Res

pons

e R

anki

ng F

requ

ency


Freedom

Figure 22 For the most part, students ranked the value of individual responsibility very differently. The only distinguishing factor of this value is that very few students (5), ranked it as their least important value. The highest number of students ranked individual responsibility as 6 out of 9 on the scale of importance.

Figure 23 Similar to individual responsibility, community was ranked very differently by the students who participated in this activity. Thirty participants, the highest frequency, ranked it as a 6 out of 9. However, very few (8) ranked it as their least important value.

23

1719 20 20

3028 28

5

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9

Res

pons

e R

anki

ng F

requ

ency


Individual Responsibility

1922

14

2628

3028

17

8

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9

Res

pons

e R

anki

ng F

requ

necy


Community


Figure 26 Students’ ranking of this value was overwhelmingly the lowest. Two thirds, 129 students, of the total participants ranked this value as the least important to them. There was no other value that was ranked as the least important as frequently as consistency/tradition. This shows that students at the Colorado Leadership Summit view consistency/tradition as the least valuable out of all nine that were offered.

Figure 27 Progress/innovation was ranked neither the students’ most, nor least important value. 31 percent of students ranked progress/innovation in their top three values, but 29 percent also ranked it in their bottom three. There was not any overwhelming pattern in students’ ranking of this value; however, it is important to note that only 5 out of 191 students ranked this value as their lowest.

Figure 28

Means for Values Ranking Exercise Equality 3.693 (Highest) Diversity 4.979 Individual responsibility 4.492 Community 4.859 Security 5.641 Freedom 4.89 Consistency/Tradition 8.198 (Lowest) Progress/Innovation 4.98 Justice 4.078

Overall, students at the 2017 Colorado Leadership Summit ranked equality (on average) as the most important value to them (M = 3.693, #1: 41, #2: 34, #3: 28). Conversely, participants ranked security (on average) as one of the least important values to them (M = 5.641, #8: 51), while also overwhelmingly ranking consistency/tradition as the least important value to them (M = 8.198, #9: 129). The rest of the values were fairly evenly distributed in terms of importance to the event participants.

Students ranked justice all across the board, 75 percent of the students during this activity ranked it in their top five, while only 25 percent ranked it in their bottom four. Furthermore, only 3 students ranked Justice as their least important value, which was the lowest frequency for the ranking of the least important value during this activity. Thus, justice was ranked relatively high overall by the students in the values ranking exercise, resulting in a mean response rate of 4.078.

2 2 3 3 2 620 25

129

0

20

40

60

80

100

120

140

1 2 3 4 5 6 7 8 9

Res

pons

e R

anki

ng F

requ

ency


Consistency/Tradition

12

17

32

23

29

22

30

21

5

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9

Res

pons

e R

anki

ng F

requ

ency


Progress/Innovation

2629 28

31 30

1614 15

3

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9

Res

pons

e R

anki

ng F

requ

ency


Justice


Facilitated Small Group Conversation Exercise During this section of the leadership conference, participants were sent to different rooms to have a discussion. Each group was strategically designed to ensure that participants were matched up with people representing different universities. There were 26 groups total, with roughly 7-8 participants per group. Additionally, each group had a facilitator from the Center for Public Deliberation to help guide the discussion, and a note taker (some were from the Center for Public Deliberation, while others were from the Colorado Leadership Alliance) to capture responses from participants. The conversation was focused around the issue of core curriculum in universities. The structure of the discussion followed the National Issues Forum Institute model of thinking about the issue from 3-4 general perspectives, with each perspective having tangible action steps, and inherent benefits and drawbacks. The approaches for this discussion were created from the pre-survey data discussed previously (see “Pre-Survey Results” section). A team of research associates analyzed the data in the pre-survey and identified four main themes to create the following approaches: 1) Maximize student choice, 2) Give college departments greater control over their students’ overall curriculum, 3) Improve the effectiveness of the current model with important but smaller changes, 4) Adapt the core curriculum to better address 21st century needs. Participants were provided with a placemat that outlined each of the four approaches to serve as a point of common reference throughout the discussion. The information from the placemat is provided below (however, due to spacing issues, the format is slightly different):

Approach 1: Maximize student choice Overview Core curriculum processes should be adjusted to provide students more flexibility and individual choice to design their own path. Specific Policy Options/Action Steps

• Reduce the number of requirements for core curriculum. Narrow the requirements to only what is deemed absolutely necessary for all students.

• Increase the number of classes that fit the various requirements. • Be more flexible with transfers. • Allow students the opportunity to test out of requirements if they have the necessary knowledge

from high school. Key Arguments

• With the rising cost of higher education, students should not be forced to take classes they would prefer not to take.

• Students are essentially the customer, and they should have the freedom to make their own decisions concerning their coursework.

• With greater freedom in choice afforded to students, students who want a diversified education can take a variety of classes while students who want a specialized education can take classes tailored to their specialization. According to this argument, “everyone wins.”

Concerns and Potential Trade-offs • This approach gives too much freedom and choice to students at a time they may not know what

the best courses are for them. • Too many choices can lead to confusion and frustration. A clearer set path is often preferable.

Unprepared students may flounder in this system. • Students are less likely to challenge their existing views if they are allowed to take any class they

choose. • Students may end up filling their schedules with classes they deem to be easy, rather than taking

classes that are intellectually rigorous.


Approach 2: Give college departments greater control over their students’ overall curriculum

Overview Core curriculum processes should primarily be designed by individual departments to develop the most efficient yet rigorous path to a career. Specific Policy Options/Action Steps

• Shift the control from university administration to college departments by providing individual academic departments the authority to design a specific path through the core curriculum that fits the needs of students in their major.

• Design core classes specifically for different majors (i.e. Public Speaking for Engineers or Science for Non-Science Majors).

• In some cases, a group of related majors can work together to build an initial set of core classes across them (a “meta-major”), then students would shift to their specific major later in their college education.

Key Arguments • University departments know which courses are the most relevant for their majors; therefore,

they should be the ones to decide which classes students need to take so that students are not wasting any time or money on irrelevant courses.

• Departments will likely still believe that their students need a well-rounded education, but will design a path that is more coherent and reasonable.

• Core curriculum classes will more clearly connect and inform the student’s chosen major. • Students that choose a major early and make progress on it quickly are more likely to stay on

track. Concerns and Potential Trade-offs

• This approach assumes that students will choose their major early and not change their mind. Those that decide to switch majors will be disadvantaged and will likely need to stay in college much longer.

• This approach puts too much of an emphasis on a student’s career rather than their overall education.

• This approach would require all departments to develop different classes, which will be much less efficient than one university wide core (i.e., departments would have to hire and/or train people to teach new specialized courses).

• Some departments may choose to require students to take classes not much different than their major.

Approach 3: Improve the effectiveness of the current model with important but smaller changes

Overview The current core curriculum works well, but could use some minor changes to ensure that all classes deliver a quality learning experience. Specific Policy Options/Action Steps

• Improve evaluation processes to ensure that core classes meet course objectives. • Work to make core curriculum classes smaller, with more focus on discussion and project work

rather than lecture and memorization. • Work with advisors and instructors to be clearer about the purpose of the core classes (make sure

students understand why each course is relevant). • Attract better teachers to teach core curriculum classes due to their overall importance to a

successful college experience. • Consider adding a financial literary course.


Key Arguments • A majority of students are supportive of the reasoning behind the current system, but do have

suggestions to improve implementation. • Core classes are in need of robust evaluation and continuous improvement to better achieve their

purposes. • Requiring a university-wide core curriculum exposes students to a broader range of courses and

makes them a more well-rounded student. • Requiring a university-wide core allows students the time to explore different disciplines before

having to choose a major. Concerns and Potential Trade-offs

• Doesn’t do enough to change a system with which some students are strongly dissatisfied. • Many core classes are too easy or repeated from high school and are simply not time well spent. • It will be difficult to have smaller class sizes for core classes or have the best teachers teach core

classes without taking away from more advanced classes or increasing the cost of higher education.

• Proper evaluation takes significant resources (time and money) to do well. • What constitutes a quality course or instructor can be difficult to establish for evaluation.

Approach 4: Adapt the core curriculum to better address 21st century needs

Overview The core curriculum should be expanded or transformed in order to achieve goals tied to the broader purposes of higher education and meet the needs of our rapidly changing society. Specific Policy Options/Action Steps

• Develop core courses that effectively teach critical thinking, particularly in the face of current challenges regarding polarization and the loss of respect for key institutions.

• Re-evaluate which classes and categories of classes should be included in the core curriculum and possibly add core curriculum requirements in areas such as inclusivity, diversity, globalization, and/or sustainability to meet current needs of our communities.

Key Arguments • A university education must be more than job training, and a robust core curriculum is the best

tool to ensure that all graduates have the skills, knowledge, and values necessary for modern life. • As society changes, we should re-evaluate what we want every student to know via the core

curriculum. • More classes in critical thinking are important to overcome the overspecialization in modern

universities that hamper problem solving. • The students that are in the greatest need of classes in topics like inclusivity and diversity are the

most likely to not take those classes unless they are required to do so. Concerns and Potential Trade-offs

• The process of continuously adapting the core curriculum to meet current needs will exhaust too much time and resources to do well.

• Adding core requirements tied to concepts such as diversity, sustainability, and globalization will cause conflict and will be seen as inappropriate indoctrination by some people.

• Supporting particular values and building character is the responsibility of families, not educational institutions.

• Students are already required to take a significant amount of core courses. Adding more would take away from the specialization that is needed in today’s job market.


Data Analysis Methods Forum Notes The data presented in this section are drawn from the notes taken by note takers in each of the discussion groups. Roughly half of the note takers were provided by the CPD, while the remainder were staff and academic administrators associated with the Colorado Leadership Alliance in some capacity. The latter were not formally trained on note taking; however, the data collected was consistent enough to make sense of, regardless of who took notes. After all of the notes were collected and compiled, associates from the CPD typed the notes into a Google form so that the data would be organized by approach number, and thus, easier to work with. Members of the CPD Research Team then performed a thematic analysis of the data by looking for the key themes within each of the approaches, and then tallying the frequency of each of those themes within each approach. Within each group’s response, there could have been multiple themes present, or even repeats of the same theme within the same group (meaning that participants within the same group both/all talked about the same theme, sometimes on more than one occasion). The graphs and charts below that have “Themes from Notes” in the title present the results of the thematic analysis for the group conversation notes. Due to the varying –often long – length of the theme names in the data, each graph’s x-axis contains letters that correspond to themes noted in accompanying charts. All of the charts displayed in the analysis of each approach have tables to serve as keys in order to adequately display the full title of each theme. Each theme in the key will often have either a “(-)” or “(+)” symbol next to the title indicating that the participants expressed either favor “(+)” or disdain “(-)” within that particular theme. Themes with no symbol next to the title were themes where it was unclear whether participants were expressing favor or disdain toward the approach with their comments. Post-forum Worksheets Additionally, during the group discussions, facilitators gave a worksheet to participants and instructed them to complete it by the end of lunch. The front of the work sheet asked participants what the most compelling arguments were in support of each approach, as well as what the biggest concerns were for each approach, totaling eight questions (four approaches, two questions each). Of the 192 total participants, 148 participants filled out the worksheet to varying degrees, with some participants responding to only one question and other participants responding to all eight questions. After all the data was compiled and organized, a member of the CPD research team typed the responses into a Google form so that all of the data could be organized and easier to analyze. Responses that were illegible or blank were subtracted from the total number of responses indicated for that question. Using the data in the Google form, a CPD staff member analyzed the data using thematic coding. All responses for a question were read, and as similarities in responses appeared, a theme was noted by the researcher. For example, for question one “What was the most compelling argument in favor of approach one?” responses included, “It’s their money, so it should be their choice”, “with the rising cost of an education students should be allowed to choose their classes. They are in a sense the customers” and “pay for education”. Though each response is unique, they showed similarities and were thus grouped under the theme “Students pay for their education and therefore should have more choice in their course work”. As new themes emerged, responses were categorized into existing and new categories. The researcher stopped creating new themes after it was clear that each response fit into an already existent category. Many responses indicated multiple themes and were tallied accordingly. However, there were some participants who did not answer the question correctly (i.e., indicated an argument in favor rather than a concern, doodled on the page, etc.), and their responses are noted in “Category J.” The graphs and charts below that have either “Most compelling arguments” or “Strongest concerns” in the title display data from the worksheets completed by participants.


Approach 1 (Maximize Student Choice) Results Overview: Core curriculum processes should be adjusted to provide students more flexibility and individual choice to design their own path (see page 18 for further details).

Approach 1 Themes from Notes Key A (-) Core curriculum is meant to broaden

students’ perspectives and challenge them B (+) Students need more curricular flexibility C (-) Students need direction when choosing a

major; this approach might not provide that D (+) Greater ease with transfer credits is good E (-) This approach isn’t applicable to all majors F (-) This approach creates burdens for schools G (-) Increased cost resulting from more choice H This approach would create less

standardization among students I (-) Without core curriculum, students will not

have their views challenged J (-) Repeat classes with lack of core curriculum K (-) Creating more classes to increase choice

would be a waste of resources L (-) Core curriculum helps foster development of

life skills M More data is needed about which

major/program requirements are needed

Figure 29 One major theme (Theme “A”) that appeared in 22 of the 26 group discussions was the idea that core curriculum is set in place in order to broaden perspectives and ultimately challenge students. As a result, core curriculum was generally discussed with favorability when this idea was brought up. Another major theme (Theme “B”) that came up in 19 of the 26 group discussions of approach one was the idea that students need to have flexibility in choosing their curriculum. This idea was in slight contrast to the other largely prevalent theme of the core curriculum being used to broaden perspectives and challenge students (Theme “A”).

3532

22

18

129 9

6 6 5 5 5 4

0

5

10

15

20

25

30

35

40


Num

ber o

f Men

tions

Response Theme

Approach 1 Themes from Notes

Figure 30

Approach 1 Compelling Arguments Themes Key A Students pay for their education and therefore should

have more choice in their course work B Greater student freedom and choice C Customizability and specialization meets student

specific needs and interest D Greater opportunity to test out of courses E More flexibility for transferring credits F Saving time and money by reducing core curriculum

requirements G Freedom to choose classes that are tailored to degree

and career H Freedom to change mind and explore other interest I Did not answer the question correctly J Expand thinking style and diversified education

41

31

18

10 10 9 73 3 2

05

1015202530354045

A B C D E F G H I J

Num

ber o

f Res

pons

es

Response Themes

Most Compelling Arguments in Favor of Approach One


The worksheet asked participants to indicate the most compelling argument in favor of approach one (maximizing student choice; see “Figure 30”). Of the 148 who filled out at least some portion of the work sheet, 124 participants provided a response to this question. The most common theme (41 responses) from this question was that students pay for college and are the customer of the university, therefore they should have more say in what courses they take and what subjects they learn. Some examples of student comments that fell in this theme are “It’s their money, so it should be their choice”, “with the rising cost of an education students should be allowed to choose their classes. They are in a sense the customers” and “pay for education.” The second most prevalent theme (31 responses) was “greater student freedom and more choice”. Examples of the responses that fell under theme A include “student choice of classes” and “students have more choices to choose their class.”

Participants were also asked to indicate the strongest concern that they had with approach one (maximizing student choice; see “Figure 31”). Of the 148 who filled out at least some portion of the work sheet, 120 participants provided a response to this question. The most common theme (theme A; 32 responses) found in their responses was the concern that students may not know what courses are best for their education and may become misguided. Examples of their responses from this theme included “Sometimes students don't know what classes would be best for their development.” The second most prevalent theme (theme B; 29 responses) was that students will have too much freedom and too many choices. One example of a response that fell under this theme was “choice over load for students”. Theme C was the third most common theme (26 responses) was that students will enroll in easy or familiar courses instead of challenging themselves. An example response of this theme was “Students may take advantage of choosing easier courses.” Deliberative Tensions and Takeaways The clearest trade-off for approach one was the desire for more student freedom and class choice, with the realization that student freedom may not be beneficial for all students and majors. There was a definite tension between those that felt like a college education should be designed by the students paying for it, and those that felt that a college education should require students to take a common set of courses from a list designed by the university (i.e., the core curriculum). Some expressed favorability towards aspects of this approach such as greater ease with transfer credits, while others expressed disdain for other aspects of this approach such as the potential loss of challenge that would come with increased class choice. Future discussions on this approach to improving core curriculum should note the trade-offs that come with greater student freedom, and practitioners should dig deeper with participants about exactly how much choice and freedom would be beneficial for all students, and which areas of higher education could be changed to reap the benefits (e.g., greater curricular flexibility) of increased student choice without the consequences (e.g., lack of direction).

Approach 1 Strongest Concerns Themes Key A Student may be misguided or may not know

what courses are best for their education B Too many choices and too much freedom C Students will enroll in easy or familiar

courses instead of challenging themselves D Result in close minded and not well

rounded students E Too narrowly focused or specialized F Did not answer question correctly

Figure 31

3229

26

18

13

8

0

5

10

15

20

25

30

35

A B C D E F

Num

ber o

f Res

pons

es

Response Themes

Strongest Arguments for Approach One


3731

15 139 7 7

4 3

0

10

20

30

40

A B C D E F G H I

Num

ber o

f Res

pons

es

Response Themes

Most Compelling Arguments in Favor of Approach 2

Approach 2 (Give College Departments Greater Control over their Students’ Overall Curriculum) Results Overview: Core curriculum processes should primarily be designed by individual departments to develop the most efficient yet rigorous path to a career (see page 19 for further details).

Approach 2 Themes from Notes Key A (-) This approach would limit exploration B (-) This approach requires major changes and

thus corresponding challenges C Departments need to act as a guide for

students D (+) This would better prepare students for a

career E (+) Unnecessary courses would be taken away F (-) Approach is burdensome to schools G (-) This approach’s actions are unnecessary H (-) Increased costs due to new classes I (-) Too much power would be given to

departments

Figure 32

Within the discussion for the Approach 2, one major theme (Theme “A”) that appeared in 20 of the 26 group discussions was the idea that this approach would limit the exploration and expansion of academic/professional interests. Participants often made comments about this approach’s inability to allow students to explore all of the options that are offered within universities because they would have to quickly choose their major if they did not want to waste any time or money.

Figure 33

Approach 2 Compelling Arguments Themes Key A Having classes that are specific to major B Departments know what’s best for their students

and should have more control of course work C Having classes relevant to the future and career D It is a streamline system for students with a clear

and definite educational path E Did not answer question correctly F Professors playing a larger role G More structure H Response that did not match main themes I Life lesson integrated into core curriculum

The worksheet also asked participants to indicate the most compelling argument in favor of approach two (give college departments greater control over their student’s overall curriculum; see “Figure 33”). Of the 148 who filled out at least some portion of the work sheet, 130 participants provided a response to this question. The most common theme (theme A; 37 responses) was having classes that are specific to major. A participant whose response aligned with this theme was “design core classes to particular majors”. The second most prevalent theme (theme B; 31 responses) was that departments know what’s best for their students and should have more control of course work. One participant response that aligned with this theme said “college departments can provide a more rigorous education to their majors.”

49

30

14 12 9 7 6 62

0

10

20

30

40

50

60

A B C D E F G H I

Num

ber o

f Men

tions

Response Themes



3732

21

8 8 7 6 4 4 3

05

10152025303540

A B C D E F G H I J

Num

ber o

f Res

pons

es

Response Themes

Strongest Concerns of Approach Two

Approach 2 Strongest Concerns Themes Key A Discourages students from switching their majors B Strict structure that is too specialized and lacks

student choice C Student will not be well rounded and will lack a

diversity of education D Drastic change may be difficult to implement and

is logistically hard on departments E Too much power to department and its professors F Incompatible system for undeclared students G Too much of an emphasis on a career H Incompatible system for interdisciplinary students I Did not answer the question correctly J Incompatible system for transfer students

Figure 34 Participants were also asked to indicate the strongest concern of approach two (give college departments greater control over their student’s overall curriculum; see “Figure 34”). Of the 148 who filled out at least some portion of the work sheet, 117 participants provided a response to this question. The most common theme (theme A; 37 responses) was that this approach discourages students from switching their majors. Some examples of the participant’s responses under this theme include “hard to change majors, doesn’t account very well for nontraditional students” and “not being able to change your major, manipulated.” The second most prevalent theme (theme B; 32 responses) is strict structure that is too specialized and lacks student choice. An example of a response for that theme is “too much specialization” and “they will be too specific.” Deliberative Tensions and Takeaways The clearest trade-off indicated by participants within this approach was the recognition that allowing departments to choose the path for students would eliminate unnecessary courses and would better prepare students for a career, while at the same time eliminating the ability for students to choose the courses they are paying for. One of the biggest tensions that came out of this approach was the disagreement about the purpose of higher education. Some participants viewed higher education as a time and place for personal exploration (and therefore did not view this approach favorably), while others viewed higher education as a place for job training and career preparation (and therefore viewed this approach favorably). Future discussions on this approach to improving core curriculum should note and further investigate the underlying assumptions participants have about the purpose of higher education. Practitioners could expand on the discussion of this approach by asking participants about how implementing this approach could differ between academic majors, and if/how it should apply to all university departments.


Approach 3 (Improve the Effectiveness of the Current Model with Important but Smaller Changes) Results Overview: The current core curriculum works well, but could use some minor changes to ensure that all classes deliver a quality learning experience (see pages 19-20 for further details).

Figure 35

Approach 3 Themes from Notes Key A (+) We need to recognize and emphasize teaching

more than we currently do B (+) Greater curricular flexibility within core

curriculum is a good thing C (+) Core curriculum provides a broad education D (+) Add classes to core curriculum that teach life

skills (financial literacy, credit, etc.) E (-) Current core curriculum is a waste of time F (+) We need core curriculum to challenge students G (-) This approach would create a strain on resources

(money for teachers, classrooms, etc.) H (+) This approach allows for greater ease with

transfer-credits I (+) Core curriculum enhances critical thinking skills J There needs to be a communication of real-world

application K (-) Reduces freedom for students

One major theme (Theme “A”) from the discussion on Approach 3 was that students feel the need for colleges and universities to emphasize teaching more than they currently do in order to increase the quality of classes taken in higher education. Another major desire from students was that they want more diversity in class options and more flexibility in choosing classes.

Figure 36

Approach 3 Compelling Arguments Themes Key A Continuously evaluate system to allow for

incremental change and improvement of university B Having classes relevant to the future and career C Well-rounded students D Current system is adequate and this approach

requires little to no change E Allows student more freedom of choice and ability

to explore majors F Smaller class sizes with more discussion G Better professors H Good balance between approaches right amount of

structure and flexibility I Logistically easy to implement J Teach students why course is relevant and help them

better understand the purpose and objectives of course

K More responsibility on university L Did not answer the question correctly M Focus on critical thinking

20

14 1413

1110

6 64 4 4

0

5

10

15

20

25

A B C D E F G H I J K

Num

ber o

fMen

tions

Response Theme


19

1615

1211

98 8

6 65

43

0

2

4

6

8

10

12

14

16

18

20


Num

ber o

f Res

pons

es

Response Themes

Most Compelling Arguments in Favor of Approach Three


Following the same format of the first two questions, participants were asked to indicate the most compelling argument in favor of approach three (improve the effectiveness of the current model with important but smaller changes; see “Figure 36”). Of the 148 who filled out at least some portion of the work sheet, 113 participants provided a response to this question. The most common theme (theme A; 19 responses) was to continuously evaluate the core curriculum system, allowing for incremental change and improvement of the university. An example response for that theme is “This approach takes a growth based approach, universities can improve.” The second most prevalent theme (theme B; 16 responses) was to have classes relevant to the future and to students’ careers. An example of a response for this theme is “include more relevant core courses and give students appreciation for content.” Theme C was another common theme (15 responses) that stated core curriculum would make more well-rounded students. An example response for this theme is “Requiring a university-wide core curriculum exposes students to a broader range of courses and makes them more well-rounded student.”

Approach 3 Strongest Concerns Themes Key A Not enough to fix issues and the core

curriculum needs much greater reform B Many core classes are too easy or repeated

from high school C Did not answer the question correctly D It may be too difficult to evaluate, what

constitutes a quality course and instructor E Responses that did not match main themes F May not be plausible to implement and will

take too many resources G May not affect all universities the same H Students may not be able to change majors

easily I No guarantee that university will listen to

student to student and make the change Figure 37 Additionally, participants were asked to indicate their strongest concern with approach three (improve the effectiveness of the current model with important but smaller changes; see “Figure 37”). Of the 148 who filled out at least some portion of the work sheet, 111 participants provided a response to this question. The most common theme (theme A; 33 responses) was that core curriculum did not do enough to fix issues and needs much greater reform. An example response for this theme is “Common core needs greater reform and each school needs to be clear about their overall intent and the intent of the common core.” The second most prevalent theme was a concern that many core classes are too easy or repeated from high school. Some examples of what participants said in their responses are “Too many core classes are repeated from high school” and “classes are too easy.” Deliberative Tensions and Takeaways The clearest trade-off within this approach was the recognition that this approach advocates for keeping the core curriculum relatively the same, which not all students are happy with, while also recognizing that this approach would require few resources to make the changes laid out in the issue guide. One of the most evident tensions with this approach was the fact that many students advocated for the small changes outlined in the issue guide (e.g., improving core curriculum teaching), while others maintained that the core curriculum was a waste of time and money. Future discussions about this approach to improving core curriculum should focus on what specific changes could be made to core curriculum to make those opposed to it more likely to tolerate it. Practitioners can and should use this approach to talk about the specific costs of changing various aspects of core curriculum.

33

26

1311 11

85

3 3

0

5

10

15

20

25

30

35

A B C D E F G H I

Num

ber o

f Res

pons

es

Response Themes

Strongest Concerns of Approach Three


Approach 4 (Adapt the Core Curriculum to Better Address 21st Century Needs) Results Overview: The core curriculum should be expanded or transformed in order to achieve goals tied to the broader purposes of higher education and meet the needs of our rapidly changing society.

Approach 4 Themes from Notes (Key) A (+) We need to emphasize curricular diversity B (+) There need to be more classes that go

outside the classroom to learn and apply knowledge

C (+) Schools need to teach students practical skills for the real world

D (+) We need to emphasize teaching and learning values

E (-) It would hard for teachers to teach some of the content proposed in this approach

F (-) This approach removes student freedom G (-) Schools need to provide a career-focused

education for students H (+) Teaching morals can be a good thing I (+) We need to find better ways to foster

creative teaching

Figure 38

In the discussion for Approach 4, there were a variety of prevalent themes with the most frequent theme (Theme “A”) being an expressed desire for curricular diversity. Students also desired an educational experience that emphasized practical skills like critical thinking and financial literacy (Theme “C”), as well as classes that go outside the classroom to apply knowledge, such as service-learning courses (Theme “B”). One major push back to approach 4 is that the 21st Century skills are taught outside the classroom by parents and other experiences.

Figure 39

Approach 4 Compelling Arguments Themes Key

A Adaptive to current and relevant needs B Adds diversity and inclusivity courses to

core curriculum C Adds critical thinking courses to core

curriculum D Did not answer the question correctly E Adds globalization and International

relations courses to core curriculum F Easy to implement G More well-rounded people H Adds sustainability courses to core

curriculum I More than just job training

1614

1210 10

98

6

3

02468

1012141618

A B C D E F G H I

Num

ber o

f Men

tions

Response Themes


47

22

16 15

9 7 5 4 3

05

101520253035404550

A B C D E F G H I

Num

ber o

f Res

pons

es

Response Themes

Most Compelling Arguments in Favor of Approach Four


Furthermore, participants were asked to indicate the most compelling argument in favor of approach four (adapt the core curriculum to better address 21st century; see “Figure 39”). Of the 148 who filled out at least some portion of the work sheet, 116 participants provided a response to this question. The most common theme (theme A; 47 responses) was that core curriculum is adaptive to current and relevant needs. Some participant’s responses that fell within this theme include, “adapts to time” and “relevance in new time.” The second most prevalent theme (theme B; 22 responses) was that it adds diversity and inclusivity courses to course curriculum.

Approach 4 Strongest Concerns Themes Key A Students should develop their own values and

beliefs; this is an overreach by the university and its professors

B Too many core courses already and this approach adds even more. Too many courses to choose form

C Expansion of core curriculum could take too much time and resources

D Too broad and vague, it will be difficult to implement

E Too modern focused F Did not answer the question correctly G Approach may be met with serious resistance H May not be good for all students, can be too

costly or ineffective

Figure 40

Finally, the worksheet asked participants to indicate their strongest concerns with approach four (adapt the core curriculum to better address 21st century; see “Figure 40”). Of the 148 who filled out at least some portion of the work sheet, 111 participants provided a response to this question. The most common theme (theme A; 32 responses) was that students should develop their own values and beliefs, and therefore, this approach is an overreach by universities and its professors. A participant’s response that falls within this theme includes, “Is it job of school to present values to students?”. The second most prevalent theme (theme B; 26 responses) was that there are too many core courses already and this approach adds even more. One participants said in regard to this theme “Could create too many require classes”. The third most prevalent theme (theme C; 24 responses) was that the expansion of core curriculum could take too much time and resources. An example of a participant’s response for this theme was “too much resources and time to dedicate” Deliberative Tensions and Takeaways The clearest trade-off indicated by participants within this approach was the recognition that while this approach would provide an avenue for new ways for students to learn (e.g., service-learning courses), while also taking away the freedom that students have to choose which courses they take as part of their higher education. One of the most evident tensions was the underlying assumptions people had about the purpose of higher education. Some thought our current societal problems warrant the teaching of values and discussion of topics like diversity, while others did not think it was the place of higher education to teach people what they should value and/or think. Future discussions about this approach to improving core curriculum should further prompt participants to think about what the underlying purposes of higher education can and should be. Practitioners should begin to dig deeper with participants about which topics they would be comfortable having professors teach if they were required to take those particular classes as part of their core curriculum.

32

26 24

8 7 6 4 4

05

101520253035

A B C D E F G HN

umbe

r of R

espo

nses

Response Themes

Strongest Concerns of Approach Four


General Forum Reflections As with most surveys administered regarding a wicked problem, participants did not express any clear consensus about the best approach or course of action to take. However, there were notable trends across the data that was collected. Participants were largely in favor of maximizing and ensuring student choice, whether that is the belief that students are the customers who are entitled to curriculum customization, or the idea that students should be free to develop their own values and beliefs. Conversely, participants were also in favor of making changes to the core curriculum such as continuous course evaluation, adding courses that address contemporary needs, and making sure that some courses are tailored and relevant to specific majors. In addition to notes taken on each approach, note takers also took notes on questions asked by facilitators toward the end of the discussion. Reflection questions were asked to help participants consider all that had been said.

Figure 41

Themes from Reflection Question Notes (Key) A There need to be more life skills classes in

the core curriculum B Students should be given choice in their

curriculum C Higher education should create better

citizens D Students need a diverse education E We need to create more interactive classes F Credit should be given for extracurricular

activities

From the reflection questions as well as the additional comments after the approaches we found these major themes:

• Students expressed that they feel like they should have more choice over what classes they take because they are paying for them.

• Students also want life skills when they graduate and want their curriculum focused on general life skills like financial literacy and critical thinking.

16

129

64 3

0

5

10

15

20

A B C D E F

Themes from Reflection Question Notes


Core Curriculum Action Steps from Dot Exercise At the end of the core curriculum discussion participants were asked, “What is the most important thing you would want college administrators in Colorado to do to improve how the core curriculum/general education requirements work?” Each group brainstormed suggestions in response to the question on an easel paper, and then spent time discussing them. Afterwards participants were given three dots to identify how they would prioritize these ideas. Participants were asked to pick their top three by sticking dots next to their favorite action steps. Participants were allowed to put one or two dots next to their choices, but were asked to not put all three on one single choice. After all the data was collected, a member of the CPD research team typed all of the responses into a document organized by group. A member of the CPD research team then performed thematic analysis by grouping together action steps that were alike among different groups. For example, the response (or action-step), “continuously reevaluate which classes are not relevant” and the response “re-evaluate the impact of individual programs to ensure their effectiveness related to current needs” were both grouped together under the theme “Conduct more regular assessment of relevance of courses in core curriculum” (Theme “E”). The numbers reflected in the chart below indicate the total number of dots (votes) for that theme from all participants, not the number of groups that produced a similar theme.

Figure 42

Core Curriculum Action Steps Theme Key A Create greater breadth of core curriculum options B Include 21st century and holistic courses in core curriculum C Require work study, internship or create job training based courses in core curriculum D Require life skills courses in core curriculum E Conduct more regular assessment of relevance of courses in core curriculum F Implement higher standards for hiring and tenuring professors G Require financial literacy course in core curriculum H Implement a greater focus on critical thinking in core curriculum courses I Implement core curriculum courses that are aligned with and specific to majors J Responses that did not match the main themes K Implement more transparency and greater communication between all levels from administration to students L Engage student opinion when making decisions regarding core curriculum M Make core curriculum courses discussion based with smaller class sizes N Better communicate the core curriculum objectives O Increased course flexibility for nontraditional students P Make core curriculum criteria based on growth not performance

5350

36 3431 30 29 27 25

20 19 18 17 15 1411

0

10

20

30

40

50

60

A B C D E F G H I J K L M N O P

Num

ber o

f Res

pons

es

Response Themes

Core Curriculum Action Steps


The most common proposed action steps are demonstrated in themes A-D which account for 174 (41%) out of 429 dots placed in total. Some examples of theme A before it was created as a thematic category include “more choice with in core classes within broad categories” and create a “broader selection of core classes”. Theme B was similar to theme A, but was thematically differentiated due to its focus on 21st century skills. Some proposed actions steps under theme B before it was noted as an overarching theme were, “more class/ options for the 21st century” and “add 21st century skills”. Theme C was the third most common theme from the proposed action steps and was titled “require work study, internship or create job training”. One example of this theme from the data (before the overarching theme was noted) was, “balance education with hands-on experience among professors”. With 34 dots, Theme D was the fourth most common theme, which suggested requiring life skills courses within the core curriculum. An example of this theme before it was noted as an overarching category was “real-world how-to adult classes”. Note that theme P was a category created for action steps that were not common amongst other groups, and were therefore given their own category.

Summary

Throughout the discussion there were many common themes that came up among multiple groups. The most prevalent theme among participants was a general favorability toward the university core curriculum and general education requirements. Participants expressed interest in creating a greater variety of options within the core curriculum, with some courses specifically geared toward addressing some of the issues and topics relevant to the 21st century. Additionally, many students thought that core curriculum courses need to address things like life skills and personal finance, and should perhaps do so by going outside of the classroom (i.e., service learning). Furthermore, there were multiple instances where participants noted the need for a greater emphasis placed on teaching and instructor evaluation to make sure that courses are taught effectively.

Deliberative Tensions and Takeaways from the Overall Discussion Participants frequently discussed many of the same underlying deliberative tensions and trade-offs throughout the conversation, regardless of which approach was being discussed. Some of the tensions were:

• Freedom (i.e., extent to which students choose their own classes) vs. Consistency (i.e., extent to which all students receive a common education)

• The purposes and assumptions of higher education: o Higher education as a business vs. Higher education as an institution for transformation o Students as customers vs. Students as pupils o Higher education as job training vs. Higher education as vehicle for personal growth

• Major change (i.e., creating a whole new system) vs. Minor change (i.e., making slight changes to our existing system)

• Increased costs for change vs. Rearranging of current funds for change

Future conversations about how to improve core curriculum should focus on a few things:

1. There needs to be further discussion about what students think the purposes of higher are, and should be. Additional discussion about how their perceptions differ from faculty would be helpful.

2. How should increasing costs of higher education affect, if at all, the core curriculum? 3. What would it look like to create an entirely new system in place of core curriculum versus what it

would look like to make minor changes?


Facilitative Leadership Talk and Debrief from Group Discussions Carcasson began debriefing by reiterating the stages of motivated reasoning and presenting key components of deliberative engagement. Some of these key components were overall deliberative framing, safe places to gather, diverse groups, and time to talk and analyze results. A call to action was also presented for public processes and what those processes should aim to accomplish, as illustrated in Figure 35.

What we need public processes to do • Build capacity for collaborative action

and co-creation • Spark collaborative learning and the

refinement (not just expression) of opinion • Positively manage conflict, build mutual

understanding, and develop respect • Support listening and genuine interaction • Provide opportunities for voice and public

input Figure 43

Carcasson then transitioned to talk about a deliberative model known as the “The Groan Zone”. This deliberative model, from Sam Kaner’s Facilitator’s Guide to Participatory Decision-Making, is used to explain the dynamics of group decision-making. The model begins by recognizing the reality of divergent thinking and opinions. In fact, divergent thinking is encouraged, because of Kaner’s assertion that not allowing enough divergent opinions can attract strong opposition and possibly unsustainable decisions. To avoid this, Kaner argues that communities need better processes to ensure adequate divergent thinking, and to ultimately make sure all voices are heard before a decision is made. After voices have been heard, the next step is the groan zone, which is the primary area where deliberative engagement happens. Essentially, the groan zone refers to the process of recognizing the inherent values and trade-offs with every possible perspective. Rushing through the groan zone may result in false polarization because people’s perspectives could become misunderstood. Conversely, spending too much time in the groan zone may lead to paralysis by analysis, where no decisions are made and the public begins to get frustrated. In short, communities need better processes for collaboration, prioritization, and moving from talk to action in order to avoid these problems. Carcasson’s talk was focused on just that: improving public processes through deliberative engagement and facilitative leadership.

Figure 44

The necessary skills – such as critical thinking and analytical reasoning – needed to partake in deliberative engagement are also skills that employers are looking for in students. These skills can be earned through deliberative engagement and facilitative leadership, as these skills are also necessary to collaborate with, and lead others through the “solving” of wicked problems. Facilitative leadership has similarities to the traditional leadership; however, traditional leaders mobilize the like-minded, while facilitative leaders collaborate between broad perspectives. Carcasson argued that this facilitative type of leadership is needed to address the polarization of today’s society. Hopefully, students walked away with a better sense of how to engage and lead others through facilitative leadership based on the activities and talks led by Carcasson and the CPD.


Process Reflections and Survey Responses On the back side of the work sheet (see “Participant Feedback on Most Compelling Arguments for and Against Each Approach” section for further details), participants were asked to fill out a survey regarding the overall process. The process survey administered to participants was a standard survey the CPD distributes at most events. The purpose of the survey is to provide the CPD with information on the effectiveness of its processes. After all the data was compiled and organized, a member of the CPD research team typed the responses into a Google form organized by question. The survey asked participants seven Likert Scale questions and a space to allow for additional comments.

Figure 45

When asked “How would you rate your overall satisfaction with today’s forum?” 132 out of 144 responded that they were satisfied or very satisfied. Some comments from the additional comments section (last chart) reflect this data. For example, one participant said, “Loved being able to have diverse conversation with a group of students from different backgrounds/schools”. A small portion of students (3) felt dissatisfied with the forum. One student commented: “the topic was boring. we all pretty much agreed. maybe do something on the cost of education next year.”

When asked “How much did you learn from participating in today’s forum?” 75 out of 143 participants felt they learned some things and another 56 felt they learned a great deal. With only one participant who respond feeling that they felt they learned nothing. One participant left the comments “I LOVED that there would be an outcome of our group discussions! Hugely impactful to me-tangible results/takeaways”.

Figure 46

Figure 47

When participants were asked “Would you say that you had sufficient opportunity to express your views today?”, 141 of 144 (97.9%) responded with “probably yes” or “definitely yes”. Only three participants selected that they probably did not have sufficient opportunity express views.

0 39

78

54

0102030405060708090

Verydissatisfied

Dissatisfied Neutral Satisfied Verysatisfied

Num

ber o

f Res

pons

ee

Response

Overall Satisfaction with Forum

111

75

56

0

20

40

60

80

Nothing A little Some things A great dealNum

ber o

f Res

pons

ee

Response

Amount Learned from the Forum

0 3 0

41

100

020406080

100120

Definitelyno

Probablyno

Unsure Probablyyes

Definitelyyes

Num

ber o

f Res

pons

ee

Response

Sufficient Opportunity to Express Views


Figure 49

Additionally, participants were asked “How often did you feel that other participants treated you with respect?” 118 participants (81.9%) felt they were “Almost always” treated with respect, with an additional 21 participants (14.6%) reporting that they were “Often” treated with respect.

When participants were asked how satisfied they were in the facilitators ability to conduct the forum in an unbiased way, 123 of them were either satisfied (38) or very satisfied (85). Another 17 participants felt neutral, with four participants feeling dissatisfied.

Figure 50

0 1 421

118

0

20

40

60

80

100

120

140

Never Rarely Occasionally Often Almostalways

Num

ber o

f Res

pons

es

Response

Treated with Respect by other Participants

0 417

38

85

020406080

100

VeryDissatisfied

Dissatisfied Neutral Satisfied VerysatisfiedN

umbe

r of R

espo

nses

Response

Facilitators Ability to Remain Unbiased

Participants were asked “When other participants expressed views different from you own today, how often did you CONSIDER CAREFULLY what they had to say?” 137 participants (95.8%) answered “Often” (64) or “Almost always” (73). From the additional comments below, some participants expressed the following, “great conversation, learned a lot and appreciated everyone’s opinions” and “My views have been broadened to consider the opposing view”. Only six participants expressed “Occasionally” (5) or “Rarely” (1) considering other participants views with whom they disagreed. Figure 48

0 1 5

6473

01020304050607080

Never Rarely Occasionally Often Almostalways

Num

ber o

f Res

pons

es

Response

Careful Consideration for other Participants Views


Figure 51 When asked “did you change your opinion as a result of the discussion, or are your views mostly the same?” 83 out of 144 (57.6%) participants felt that their views had changed somewhat. 53 participants felt that their views mostly stayed the same, which is reflective in the comments left by participants, including, “I gained more perspective but my core views did not change” and “I have new perspectives on the same opinion”.

Figure 52 The last survey question asked participants if they had any additional comments. Because the question was open ended, the data was organized and processed using thematic coding (unlike the quantitative data in the chart above). 12 participants gave positive responses to the discussion (thus creating a theme) with comments like, “Discussion was great! Very relevant - good topic!” and “Discussion and interactions were awesome.” Similarly, comments such as “Great workshop” and “great session. Thank you!” created the theme “Appreciative and positive reaction to forum”. Conversely, the theme “Negative response to discussion” included the comment “the topic was boring. we all pretty much agreed. maybe do something on the cost of education next year.” The responses to this question are used in the analysis of chart above.

5

53

83

30

102030405060708090

My views are entirely thesame as before

My views are mostly thesame as before

My views changedsomewhat

My views changedcompletely

Num

ber o

f Res

pons

ee

Response

Change in Opinion as a Result of Discusion

12

9

3 2 13

5

02468

101214

PositiveReponse toDiscusion

Appreciativeand positivereaction to

forum

Positiveexperiencewith other

particiapntsperspectives

Negativeresponse todiscusion

NegativeResponse to at

least oneaspect of event

Not enoughconnection to

leadership

OtherNum

ber o

f Res

pons

es

Response Themes

Additional Comments

2017 Colorado Leadership Alliance Summit Report · 2017 Colorado Leadership Alliance Summit Report...

Documents

Transcript of 2017 Colorado Leadership Alliance Summit Report · 2017 Colorado Leadership Alliance Summit Report...