Incorporating Oral Presentations into Electrical and ... · Incorporating Oral Presentations into...

Paper ID #8823

Incorporating Oral Presentations into Electrical and Computer EngineeringDesign Courses: A Four-Course Study

Ms. Nabila A. Bousaba, University of North Carolina, Charlotte

Nabila (Nan) BouSaba is a faculty associate with the Electrical and Computer Engineering Departmentat the University of North Carolina at Charlotte since 2008; she is the senior design instructor for thedepartment, additional courses taught include Basic Circuit for non- majors, and Technology Innovationand Entrepreneurship course ECGR4090/5090. Nan Earned her BS and Master Degrees in ElectricalEngineering (1982, 1986) from North Carolina Agricultural &Technical State University. She mentoredDepartmental sponsored projects such as UNCC Parking team, IEEE Hardware competition teams, indus-try sponsored projects from Microsoft, NASA teams and special Innovation and Entrepreneurship teams.She published and presented papers in ASEE conferences in June 2009, 2010, and 2011. Prior to hercurrent position at UNC- Charlotte, Nan worked for IBM (15 years) and Solectron (8 years) in the area oftest development and management.

Dr. James M. Conrad, University of North Carolina, Charlotte

James M. Conrad received his bachelor’s degree in computer science from the University of Illinois,Urbana, and his master’s and doctorate degrees in computer engineering from North Carolina State Uni-versity. He is currently a professor at the University of North Carolina at Charlotte. He has served asan assistant professor at the University of Arkansas and as an instructor at North Carolina State Uni-versity. He has also worked at IBM in Research Triangle Park, North Carolina, and Houston, Texas; atEricsson/Sony Ericsson in Research Triangle Park, North Carolina; and at BPM Technology in Greenville,South Carolina. Dr. Conrad is a Professional Engineer, Senior Member of the IEEE and a Certified ProjectManagement Professional (PMP). He is also a member of ASEE, Eta Kappa Nu, and the Project Manage-ment Institute. He is the author of numerous books, book chapters, journal articles, and conference papersin the areas of embedded systems, robotics, parallel processing, and engineering education.

Ms. Jean L. Coco, University of North Carolina, Charlotte

Jean Coco is the Acting Director of the Communication Across the Curriculum Program at the Universityof North Carolina at Charlotte.

Prof. Mehdi Miri, University of North Carolina, CharlotteProf. Robert W Cox, University of North Carolina, Charlotte

c©American Society for Engineering Education, 2014

Improving Oral Presentation in an Electrical and Computer Engineering Department: A Four Course Study

Abstract One measure of continuous improvement in the Electrical and Computer Engineering Department (ECE) at the University of North Carolina at Charlotte is survey feedback from alumni on their workplace readiness. In a recent survey, alumni highlighted oral communication as an area of weakness in the curriculum. When a group of faculty teaching design courses learned about the University’s Communication Across the Curriculum (CAC) program, they formed a pilot team to focus on improving student oral presentation skills in the design courses. The CAC program focuses on the oral and written communication as playing an integral role in teaching students reasoning, critical thinking, and problem solving skills. And as faculty development program, it seeks to develop a communication enhanced curriculum (CEC) at the departmental level. The CAC program hosts an annual institute during which departmental teams gain professional development in the areas of oral and written communication. At the 2010 CAC Institute, a team of ECE faculty set the strategic goal to improve ECE students’ ability to clearly convey technical information orally during design presentations. The ECE team then met with the CAC coordinator to develop an oral communication rubric for uniform implementation, thus establishing a cohesive assessment across the department’s design sequence. Our goal with this approach was to improve oral communication skills among our graduates to increase their opportunities for success in their professional careers. We focused on three important skills in oral presentation: audience analysis, message coherence / focus, and message delivery. A team of five faculty--four from ECE and the CAC director--worked together to develop a rubric to evaluate students oral presentation skills in the sophomore design (ECGR 2252), junior design (ECGR 3157) and senior design (ECGR3253 and ECGR3254) courses. The implementation of the process began by using the rubric in Appendix (a) to evaluate student and team presentations in each of the four courses above. We videotaped the presentations for students to review later so they could learn from their mistakes. We followed teams of students from the sophomore design in the spring 2012 to the Senior Design in the fall 2013, and we tracked and observed their progress from sophomore design to senior design. Our hope was that the results would justify full implementation into other ECE courses by the fall of 2014. This paper describes the process we followed to implement this emphasis on oral communication. This paper also presents a comparison of oral communication performance before and after the emphasis on oral communications was implemented. Data collected is from measurement tools put in place six years ago for ABET Student Outcome reporting. Introduction The Electrical and Computer Engineering (ECE) department and the Communication Across the Curriculum (CAC) program worked together on a three-year research project to study the impact of instructors’ written feedback and students’ written reflections on electrical engineering students’ speaking skills. Four design courses—sophomore, junior, and two senior design

classes—provided the project’s framework. The research involved assessing the presentations of a select group of project students and an equal number of control group students, beginning with the sophomore design class and continuing through the two senior design courses. The project students received feedback via an analytic rubric. The benefits of using rubrics are shown in Conrad et al 7. The Project students viewed their videotaped presentations and wrote a reflective paper on their performances. The control group did not receive feedback, although their presentations were scored using the rubric. At the conclusion of the senior design class, a statistical analysis of the data was expected to support the project’s overall objective: that student’ speaking skills would improve with multiple opportunities for practice and feedback.

The ECE department was invested in this project because communication skills are criteria by which the department is judged for accreditation. The university community stands to benefit from the knowledge created because our findings address oral communication goals stated in the 2008 UNC tomorrow report, a system-wide visioning document. This project has the potential to increase student engagement in the discipline, and we hope it will become a campus-wide model of how pedagogical revision can speak to the objectives of the Quality Enhancement Program that is part of the SACS assessment. The strategic goal of this project was to improve the oral communication skills of all undergraduate electrical engineering students. The research question was whether this strategic goal could be achieved via instructors’ post-performance feedback and students’ reflective writing. We investigated this question through a statistical comparison of the effectiveness of the oral presentations made by the project students with those made by a control group of students. The assessment data needed for this comparison was based mostly on the blind judgment of third-party evaluators in the second senior design class. Project research overview The objective of this joint research project was to test the hypothesis that students’ speaking skills would improve with multiple opportunities for practice, self-reflection, and instructor feedback. The methodology was to statistically compare the speaking effectiveness of a select group of project students with an equal number of control group students after all students were given multiple opportunities for practice. However, the presentations of only the project students were videotaped for their self-assessments, and only the project students received feedback from the instructors. Four curriculum-required designs courses—namely, sophomore, junior, and two senior design classes, provide the project’s framework. The project concluded in fall 2013, when third-party evaluators judged all student presentations in the Senior Design class oral presentation without knowing who the project students were. Project Narrative A. Specific Aims The overall purpose of this research was to improve the oral communication skills of approximately 350 undergraduate electrical engineering students by providing multiple opportunities for practice and feedback. The project’s objective was to determine whether or not the quality of students’ oral presentations improved after post-performance feedback and reflective writing. We determined this through a statistical comparison of the control group and the select group of project students.

The proposed project addressed the following research questions: a. Is the level of audience awareness and interaction (aai) higher for the project students than for the control group? b. Is the level of message coherence and focus (mcf) higher for the project students than for the control group? c. Is the level of message delivery effectiveness (mde) higher for the project student than for the control group? These questions generated the criteria by which we would evaluate the students’ oral presentations. Both the questions and the criteria were generated in a July 2011 meeting, during which, after much discussion, the faculty team determined that audience awareness and interaction, message coherence and focus, and message delivery effectiveness became priority criteria because students demonstrated weaknesses in these areas. The rationale for the project was partially driven by the Accreditation Board for Engineering and Technology’s recent addition of communication standards for accrediting engineering programs (ABET 3)6. The current research on oral communication in electrical engineering has grown as a result of this development, along with the realization that oral presentations are frequently utilized in professional engineering practice. Hence, the ECE faculty has created programmatic student learning outcomes that address the need for students to practice communicating their ideas orally to both professional and lay audiences. To link the project to professional workplace readiness, the design faculty will continue the current practice of asking a panel of local engineers to evaluate the students’ final presentations in the second senior design class. To plan this project, the ECE design team and the CAC coordinator met twice during the summer of 2011 to develop a standardized analytic rubric for use during the study. We then tested the rubric during a senior design presentation in October 2011 and revised it to improve its usability. The impact goal of the proposed project was the creation of new pedagogy that is more effective in imparting oral communication skills to electrical engineering students in order to prepare graduates for oral presentations required for employability and professional advancement. The CAC program seeks to use the knowledge gained to assist other departments across campus who seek to improve their students’ oral communication skills. B. Literature Review Past and current research speaks to the need for a pedagogical shift in the general engineering curricula from a purely technical focus to one that integrates written and oral communication. Darling and Dannels, in “Practicing Engineers Talk about the Importance of Talk,” note that there has been a “disparity between the perceived importance of communication and the respective preparation students receive on communication related tasks” in engineering and the need to provide students with practice and preparation in speaking1. Currently, scholars and teachers are working with engineering departments to respond to this disparity in a variety of theoretical, curricular, and pedagogical ways. Incorporating public speaking requirements into the curriculum and aligning oral communication assignments with workplace expectations are two examples of this shift, Darling and Dannels1. Based on survey and interview data, one recent study recommended adjusting the engineering curricula “to include the practice of cooperative problem solving, to make evaluation of oral communication competence a component of grades”

Vest et al2 and specifically targeted audience adaptation, language use, and style as important criteria in grading communication assignments. These findings led us to select audience awareness as a student learning outcome.

Deepening Student Engagement with Oral Communication While graduates’ workplace readiness is a compelling professional reason for integrating oral presentations, researchers have found that there are intellectual benefits, too. During the development and execution of an oral presentation, student engagement with content deepens as students analyze, synthesize, and create knowledge; thus, they are not merely transferring information (Winsor 223)12. Furthermore, a study of chemical engineering graduates’ workplace preparation noted that one’s deep understanding of technical content is reflected in the genre of oral presentations and that “technically sound” presentations, executed by confident engineers, were the most effective (Martin et al 173)11. Finally, an extensive study of the design presentation in engineering concluded that because students learn how to situate new knowledge for an audience and how to negotiate what was legitimate for presentation, the use of oral presentations in the classroom had “clear epistemological implications far beyond the realm of delivery” (Dannels 166)3. We viewed the “negotiating what was legitimate” for presentation as part of message coherence and focus. Rubric development and iterations At the outset, we knew we would employ a standardized rubric to both assess the students’ performances and to capture data since the design course teachers were already using rubrics. In addition, the literature we reviewed supports the idea that rubrics assist students in setting performance goals, while helping them make specific revisions and/or corrections to reflect improvement (Reddy et al 437)5 . In a study conducted in a Business Management course, Petkov and Petkova discovered that the mean percentage grade for the section that used rubrics in oral presentations was higher than the comparison group (505). And, based on research conducted by C.A. Reitmeier, L.K. Svendsen, and D.A. Vrchota, integrating rubrics into an oral communication assignment shifts the evaluation protocol from “subjective observations to specific performances” (2004, 18). The project team recognized that the three oral presentation rubrics currently in use in the design sequence needed to be collapsed into one standard rubric for the following reasons: to create a consistent, reliable and efficient rating process for the purpose of methodology; to help students internalize the criteria early in the study; and to develop a common vocabulary for post-presentation discussions. After multiple iterations, the penultimate version of the rubric was tested in the sophomore design course; subsequent revisions focused on reorganizing criteria into three categories: audience, content, and delivery, all of which were bolded and capitalized to increase usability during the presentation. We also provided additional space at the bottom for the evaluator comments. The final version of the rubric is in Appendix IV. Promoting Deep Learning with Reflective Writing The justification for implementing reflective writing in the study was grounded in research conducted by Kathleen Blake Yancey and Jane Bowman Smith on the efficacy of this practice. As Yancy and Smith note, reflection records a “student’s process of thinking about what she or he is doing while in the process of that doing” (170)8. Both argue that self-assessment and

reflection are essential to the learning process because they are a “method for assigning both responsibility and authority to a learner” (170)8. In the discipline of engineering, Case and Gunstone (2003)9 have identified two styles of approaching engineering problems: deep and surface learning. Deep learning is encouraged by metacognitive learning activities, such as reflective writing, while the latter is consistent with "plug and chug" (symbolic) approaches. Case and Gunstone (2003)9 and Case and Marshall (2004)10 have identified writing-to-learn activities as key to promoting deep learning and its associated metacognitive properties. C. Methods The sophomore, junior, and senior design classes are required for all electrical engineering students, and each course requires student teams to give oral presentations about their design projects. The table below specifies the courses and the semesters during which the study was conducted.

Sophomore Design Spring 2012 Junior Design Fall 2012 Senior Design I Spring 2013 and Fall 2013 Senior Design II Fall 2013 and Spring 2014

The sophomore design class has an enrollment of approximately forty-five during the spring semesters. Students are divided into three design teams. Each team is required to design a product to satisfy specific end-user needs, and each team is required to give three oral presentations with these three elements included: Product design specifications, conceptual designs and a detailed design/product demonstration. We divided the sophomore design class into two groups: a project group and a control group, with similar profiles in terms of ethnic, gender, and GPA diversity. To answer the three questions posed in Section A, the following student learning outcomes (SLO) assessed competencies believed to be essential for effective oral communication skills:

SLO. a) Students will demonstrate an awareness of the audience’s background knowledge and expectations by fielding questions and interpreting information in a way that is appropriate to the specific audience, be it the general public, an industry representative, or their academic peers.

SLO. b) Students will organize and focus technical material and graphics to deliver a coherent message about the new knowledge they have synthesized and produced.

SLO. c) Students will deliver the presentation in an audible voice, with minimum use of notes and filler words.

The levels of achievement of these outcomes by the project students were statistically compared with those by the control group, using direct assessment data from the four design courses. Table 1 summarizes the assessment method used. It shows the selected courses for each SLO, the metrics used to determine the levels of achievement of the SLO, and the statistical variable names analyzed in the evaluation phase.

To score the students, we used the analytic rubric shown in Appendix IV. This rubric was developed by the Electrical Engineering design faculty in collaboration with the Communication Across Curriculum (CAC) coordinator. The Electrical Engineering design faculty drafted the rubric during the May 2011 CAC Institute, and then rewrote and revised it during two meetings in July 2011 with the CAC coordinator. In October 2011 we conducted a usability test during the senior design presentation class, and then revised the test rubric to make it less cluttered, and thus more user friendly for scoring during live presentations.

The local industry engineers who evaluated the students’ final presentations were not informed of who the project students were, since they represented the audience with whom our graduates will have to communicate. In addition, they were unbiased because they have no stake in the study’s outcome. Therefore, the conclusions of the study was predicted to be more accurate if the rubric scores were awarded by the industry panel members were weighted more heavily than the scores from the course instructors. To achieve that weighting, each score from the panel members was given five times more weight in determining the outcome of the statistical study.

Table 1: A Summary of the Assessment Method

Courses used for assessment Metrics used Statistical variable name used

SLO.a

Sophomore Design (ECGR2252)

Junior Design (ECGR3157)

Senior Design (ECGR3253)

Senior Design II (ECGR3254)

Scores from rows 1 & 2 of the Analytic Rubric

aai

SLO.b



Senior Design I (ECGR3253)


Scores from rows 4 & 7 of the Analytic Rubric

mcf

SLO.c



Senior Design I (ECGR3253)


Scores from rows 3, 5 & 6 of the Analytic Rubric

mde

Actual Implementation After receiving IRB (Institutional Review Board for Research with Human Subjects) approval for the project, twenty students in the spring 2012 sophomore design class were selected with consent (see consent form in Appendix I) as the project students. The remaining nineteen students served as the control group. A share-drive at N:\uncc.edu\usr8\SOTL_Project was established on the Engineering computer network (known as Mosaic) for the project students’ videotaped presentations, instructors’ reflective writing prompts, students’ reflective essays, instructors’ feedback, and all project-related assignments. A teaching assistant videotaped and uploaded the recordings of the three presentations delivered by each of the five project teams. Following each presentation, the project faculty provided written feedback to each project team at the individual level. After viewing their recorded presentations, each team wrote a one-page reflective essay on their team’s performance, which included suggestions for improvement. Selection of Project Students Following IRB guidelines, the consent form shown in Appendix I was created and distributed to all forty students in the sophomore design class in spring 2012. The students were given one week to sign and return the forms indicating their willingness to participate as a project or a control group student. Twenty-five students volunteered to participate as the project students, and fifteen volunteered as the control group students. The extra five volunteers for the project group gave us the flexibility to select two groups of students with equal average GPA. Appendix II shows the list of all students with the students in Teams 1-5 being the project students, and students in Teams 6-10 being the control students. Sophomore Design teams oral presentation and student reflection In the sophomore design class, students were divided into design teams of four. Each team was assigned to design a product to satisfy specific end-user needs. Each team made three oral presentations according to the schedule shown in Table 2.

Table 2: Schedule & contents of the three oral presentations in the sophomore design class.

Date of Presentations Content of Presentations

First Oral Presentation:

Feb. 28, 2012: Control group students (Teams 6-10)

March 1, 2012: Project group students (Teams 1-5)

Analyze your target market (size, income, and why you think they would be interested in your product), the need statement, the objective statement, competitive benchmarking, and an estimate of the potential annual profit. The goal of presentation should be to convince the engineering managers that your proposed project will be profitable.

Second Oral Presentation:

April 10, 2012: Project group students (Teams 1-5)

April 12, 2012: Control group students (Teams 6-10)

Present the design specs for your product in a table. Follow with at least two conceptual designs in the block diagram format; use the Analytical Hierarchy Process that leads to a decision on conceptual design selection. The goal of your presentation is to convince the engineering managers that your decision on conceptual design meets the target mark requirements.

Third Oral Presentation:

May 8, 2012: All ten teams

Review the goals of the project including the need and objective statements. Review design specs, and the conceptual designs considered. Discuss your detailed design followed by cost estimation. Demonstrate the operation of your prototype at some point during your presentation. End with concluding remarks on whether all design specs were met or not. For each design spec that was not met, suggest design modifications that would help meet that spec. The goal of your presentation/demo is to convince the engineering managers that your team has followed sound design practices and has developed a reliable product that meets the needs of your target market.

Faculty Feedback The project faculty provided the project students with feedback on their oral communication skills using the rubric in Appendix IV. All such rubrics were scanned and uploaded to the project share-drive by the project TA. Each team had access only to its folders on the share-drive and could not see the faculty feedback provided for other teams.

Students’ Written Reflections At the outset, we intended to capture at least three reflective essays from each team of project students because we believed it would provide a window into the students’ problem solving as it related to their oral presentations. Therefore, the project teams were asked to watch the recording of their presentations and to then respond in writing to the reflective writing prompts developed by project faculty (see Appendix III). The reflective essays were then uploaded to a designated folder on the project’s shared drive. While we hoped that the reflective writing would generate most of the benefits stated in the literature review, acquiring reflective writing responses from students proved challenging. Because this writing assignment was only given to the project students, instructors could not award points for the written work since it would create inequity with the control group students, and hence students’ incentive to complete this low-stakes writing task was low. We also discovered that another barrier existed: bringing groups back together to watch the recording and then produce a written response outside of class posed scheduling challenges. Even so, the essays we collected demonstrate students’ increased awareness of the strengths and weaknesses of their presentations as evidenced by the following excerpts from their reflective essays in ECGR2252 (sophomore design), and in ECGR3157 (junior design) and in ECGR3253 (senior design). Please note that, in junior and senior designs, some teams included a combination of project and control students; hence their reflective writing essays were composed individually.

PROMPT: What did you notice about the group’s interaction with the audience? Group A: We’ve noticed from the video presentation that our team was scattered during the first presentation. Preferably, whoever is closest to the computer should dictate the

next slides, so as to refrain individual team members from consistently walking back and forth.

Group B: After having reviewed both the video and the reviewer’s notes, we as a team have come to the conclusion that as a team we need to engage the audience more as a whole, and work on the “speech” portion of the presentation so that is flows more smoothly and effectively. During our first presentation we used the highly technical terms of “ahh” and “umm” quite a few times during the speech portions of the presentation. We also tended to simply readdress the information that we had already put on the power point presentation.

Group C: We could implement the use of a “practice” panel to help ready ourselves for the actual presentation and to engage the audience more during the actual presentation. Use of question and answer tactics, direct eye contact, and moving around the room are also ways we could get the audience engaged. We also need to understand the difference between keeping the audience engaged and making the audience feel like that someone is trying to sell them time shares in Kansas. PROMPT: Make three or four directive statements recommending specific changes that the group/individual should make to improve the delivery component before the next presentation. What will you do to make those changes? What are your next steps?

Group A: We need to elaborate more in details with our responses to the questions that were asked. To effectively eliminate the “ahh” and “umms” out of our presentations, we should practice our presentations more effectively, the use of note cards, or other important point reminder techniques could also be implemented to help eliminate the unwanted pauses.

Group B: We need to spend more time preparing for the second presentation. One thing that will definitely help will be practicing in the room. Watching the video gave us a unique perspective on the presentation, so we might do this on our own before the next one. One thing this video showed us was that the map on the second slide was unreadable. There were a lot of ‘umms’ and other crutch words from everyone, so that can be addressed by having notecards or more preparation.

Group C: Not restating the information that is listed on the PowerPoint in front of the audience is probably the hardest technique we have to work on. The audience is capable of reading the information, so we do not want to simply “read” the PowerPoint to the audience. This also ties in to keeping the audience engaged. We want to be able to pass the information along to the audience in both a verbal and visual format in hopes of the audience retaining twice the amount of information as they would with just the visual or verbal presentation alone.

Junior Design implementation Students in junior design are divided into teams consisting of three or four members. Each team must complete four projects. As part of the second and third project modules, each team is required to present its design to the instructor during a design-review session. Both sets of design-review presentations were evaluated using the rubric in Appendix IV during the fall 2012 semester. It is important to note that 17 of the 20 project students and 11 of the 19 control-group students enrolled during that semester. Five of the 19 control group students ultimately changed their major (three to computer Engineering and two to Computer Science) and were thus not required to enroll in junior design. Three of the project students and three of the control-group students took junior design during summer or fall 2013. Since these students had significantly more experience or were concurrently enrolled in Senior Design I, their results were not included. Given these logistical constraints, it was physically impossible to keep the students on the same ten teams used in sophomore design. In some cases, project students and control-group students were on the same team. Recordings of the fall 2012 presentations were made available to the project students. Unfortunately, the audio quality was relatively poor, and the student responses to the reflective prompts were sparse. Among the seven responses received, however, there was a unanimous sentiment that participation in the project had changed the students’ approach to presentation. All seven respondents noted that they were more mindful of their audience and the quality and structure of their presentations. Given the detailed senior design results presented in the next section, it is interesting to note that the student responses seem to indicate an improvement with respect to SLO.a and SLO.b and provide minimal evidence of improvement with respect to SLO.c. It is also possible, however, that the poor audio quality provided students with less of an opportunity to evaluate SLO.c. Scores from the presentation rubrics in junior design were normalized to sit on a scale of 0 to 10, as each represents 10 of the 100 points available on project modules two and three. The average presentation score for the 17 project students was 9.1; the average grade for the 11 control group students was 8.2. Senior Design implementation and results In the senior design course, the students were distributed across 18 projects; we followed all thirty-seven students individually, two of the control group students changed their major to Computer Science and were thus not required to enroll in senior design. We used the same rubric as displayed in Appendix IV to provide all project students and control students’ feedback. We also received several inquiries from control students asking for more feedback after the fact; because they observed the self-learning was tremendous. We kept the following research questions in mind while assessing all students’ reflective writing: Research Questions a. Is the level of audience awareness and interaction (aai) higher for the project students than for the control group? b. Is the level of message coherence and focus (mcf) higher for the project students than for the control group?

c. Is the level of message delivery effectiveness (mde) higher for the project students than for the control group? Evaluation: The data collected from the senior design class student presentation for both project and control groups were stored on the share drive. We assumed that the standards of deviations for the two populations (project and control) were equal and the three Pooled t-tests[1] were conducted to test the following hypothesis for each pair of variables such as aai_project(µ1) and aai_control(µ2):

:

: With a 0.05 level of significance, the p-values were used to make inferences about the population means µ1 and µ2 in each of the three tests. Tables 3, 4, and 5 show the results of the t-tests for the three variables. The p-values from the one-tail tests were compared with α = 0.05 level of significance and inferences were made following each table. Table 3: Comparing "Audience Awareness & Interaction t-Test: Two-Sample Assuming Equal Variances (α=0.05) aai_project aai_control

Mean 2.736486486 2.4609375

Variance 0.207006664 0.36749752

Observations 74 64

Pooled Variance 0.281351693

Hypothesized Mean Difference 0

df 136

t Stat 3.043269092

P(T<=t) one-tail 0.00140488

t Critical one-tail 1.656134988

P(T<=t) two-tail 0.00280976

t Critical two-tail 1.977560777 p-value = 0.00140488 < α = 0.05 indicates that the null hypothesis is rejected and that there is

strong evidence that for the aai sample populations. That is, the project students have shown a higher competency in this category.

Table 4: Comparing "Message Coherence & Focus" t-Test: Two-Sample Assuming Equal Variances (α=0.05) mcf_project mcf_control

Mean 2.700704225 2.450413223

Variance 0.220070423 0.289187328

Observations 142 121



df 261

t Stat 4.031203291

P(T<=t) one-tail 3.64367E-05


P(T<=t) two-tail 7.28734E-05

t Critical two-tail 1.969094724 p-value = 3.64367E-05 < α = 0.05 indicates that the null hypothesis is rejected and that there is

strong evidence that for the mcf sample populations. That is, the project students have shown a higher competency in this category. Table 5: Comparing "Message Delivery Effectiveness" t-Test: Two-Sample Assuming Equal Variances (α=0.05) mde_project mde_control Mean 2.837837838 2.7265625

Variance 0.124028138 0.229600694

Observations 74 64



df 136

t Stat 1.567567305

P(T<=t) one-tail 0.059652985


P(T<=t) two-tail 0.11930597

t Critical two-tail 1.977560777

p-value = 0.059652985 > α = 0.05 indicates that the null hypothesis cannot be rejected and that

there is no evidence that the alternate hypothesis is true for mde. That is, the project students have not shown a higher competency in this category. Discussion and Conclusions: What did we learn from this three-year study? The stated objective of the project was to test the hypothesis that student’ speaking skills would improve with multiple opportunities for practice, reflection, and instructor feedback. Based on the data collected we found the following results:

1. There is strong evidence that the project students have developed a higher competency relative to audience awareness and interaction (aai).

2. There is strong evidence that the project students have developed a higher competency relative to message coherence and focus (mcf).

3. There is no evidence of higher competency of project students group over control group relative to message delivery effectiveness (mde).

The first and second findings support our hypothesis; however, the third finding does not. Even though the project students’ scores on delivery reflect a slightly higher mean, the difference is statistically insignificant. After some discussion, we arrived at some possible reasons why this occurred. Design presentations have been in place in ECE since 1978, when it was first implemented in senior design. In the mid 1990’s the practice was integrated into sophomore and senior design. Because the practice has been in place in all three courses for almost 20 years, it has become institutionalized as a disciplinary genre in oral communication. Student familiarity with the expectations of the presentation—the team approach and the prescribed time limit of 20 minutes—may explain the minimal difference between project and control students’ performances. In addition, when we examine all of the factors enumerated in the delivery criterion, we acknowledged that they are more explicit, compared to the other two criteria, and within the student’s control:

• Voice is clear and audible • Rarely reads from notes/slides • Rarely uses filler words: “um,” “uh,” “like,” “well,” etc. • Adheres to time limit of 20 minutes • Effective pacing

In a discussion of the results, the sophomore design professor commented that “the rubric provided a priori feedback on delivery to all students.” And even though we videotaped the students in each class, the poor audio quality in junior design impeded their ability to self-evaluate, thus which leveled the “delivery” playing field between the control and project students. While it may seem that the delivery results would point to a failure in terms of the project’s objective, we recognize that audience awareness and interaction, and message coherence and

focus, are sites where significant revision and negotiation of content, both oral and visual, occurred after students received evaluators’ feedback. Anecdotally, project faculty observed improvements in project students’ speaking skills from their first presentation to their third. And it is important to note that the students’ reflective writing is the site where students express their metacognitive awareness of that revision and negotiation. Overall, both project faculty and students recognized that oral presentations are iterative, and practices such as peer and faculty review have a positive impact on the quality of the group’s presentation. In junior design the professor noted that the project students not only produced more thoughtful visuals, but that their presentations were “far more organized” and that they engaged the audience “in a far superior manner.” The senior design professor reported that the control students recognized that their performances were weaker than project students. She also pointed out that she received several inquiries from control students asking for more feedback after the fact; because they observed that their peers’ self-learning “was tremendous.” In terms of actionable data, the findings justify the need for providing students with multiple opportunities for feedback over the course of the three-year design sequence. The junior design professor reported that “prior to this process, I viewed the teaching of such ‘soft skills as somewhat difficult and that the ability to master them was something that would come naturally to the more ambitious students who were driven to succeed. I now feel that such skills cannot be taught effectively without some mechanism for self-reflection.” Currently, we are drawing up plans to integrate scaffolded oral communication assignments, beginning with sophomore design. To link the project to professional workplace readiness, the design faculty has continued the current practice of asking a panel of local engineers to evaluate the students’ final presentations in the senior design II class. The CAC Program has recently secured the assistance of an oral communication consultant from the Communication Studies department to focus on this work. And, on a broader level, the CAC Program will use the findings to assist other departments across campus that seek to improve their students’ oral communication skills. As a team of academics, we look forward to doing the good work of arming our graduates with the skills they need to succeed as professionals, both in ECE and across our campus. References 1. Darling, A. L. and Dannels, D.P. “Practicing Engineers Talk about the importance of talk: A

report on the role of Oral Communication in the Workplace.” Communication Education. 52.1, 2003, 1-16.

2. Vest, D., Long, M., and Anderson, T. “Electrical Engineer’ Perceptions of Communication Training and Their Recommendations for Curricular Change: Results of a National Survey” IEEE Transactions on Professional Communication. Vol. 39, No. 1, March 1996.

3. Dannels, D. J. “Teaching and Learning Design Presentations in Engineering: Contradictions between Academic and Workplace Activity Systems.” Journal of Business and Technical Communication. 17, 2003, 139-69.

4. University of North Carolina at Charlotte - Scholarship of Teaching and Learning Grant Proposal. “A Study of the Effect of Instructor Feedback and Students’ Written Reflections on the Oral Communication Skills of Electrical Engineering Students.” October 2011.

5. Reddy, Y. M., and Andrade, H. “A review of rubric in higher education.” Assessment & Evaluation in Higher Education. Vol. 35, No.4 July 2010, 435-448.

6. ABET: 2011-2012 Criteria for Accrediting Engineering Program. Retrieved 13 October 2011 www.abet.org .

7. Conrad, J.M., BouSaba, N., Hoch, D., Heybruck, W., Schmidt, P., and Sharer, D. “Assessing Senior Design Project Deliverables.” Proceedings of the 2009 ASEE Conference, Austin, TX, June 2009.

8. Smith, J.B., and Yancey, K.B. Self-assessment and development in writing: a collaborative inquiry. Cresskill, N.J.: Hampton Press, 2000.

9. Case, J., and Gunstone, R.F. Approaches to learning in a second year chemical engineering course. International Journal of Science Education. 25(7), 2003, 801-819.

10. Case, J., and Marshall, D.. Between deep and surface: procedural approaches to learning in engineering education contexts. Studies in Higher Education. 29(5), 2004, 605-615.

11. Rosanna, R., Maytham, B., Case, J., and Fraser, D. “Engineering graduates’ perceptions of how they were prepared for work Industry.” European Journal of Engineering Education. Vol. 30, No.2, May 2005, 167-180.

12. Winsor, D. A. “Genre and Activity Systems: The Role of Documentation in Maintaining and Changing Engineering Activity Systems.” Written Communication. 16, 1999, 200–224.

http://www.abet.org/

Appendix I Informed Consent for

A Study of the Effect of Instructor Feedback and Students’ Written Reflections on the Oral Communication Skills of Electrical Engineering Students

Project Title and Purpose: You are invited to participate in a research study entitled “A Study of the Effect of Instructor Feedback and Students’ Written Reflections on the Oral Communication Skills of Electrical Engineering Students”. The purpose of this study is to improve the oral communication skills of undergraduate electrical engineering students by providing multiple opportunities for practice and feedback. Oral communication skills combined with technical competence is essential for your professional development and it is the main ingredient for becoming successful engineers after graduation.

Investigator(s): This study is being conducted by the Electrical Engineering professors Mehdi Miri, Nan BouSaba, Jim Conrad, and Robert Cox in collaboration with Jean Coco, the Acting Director of the Communication Across the Curriculum Program.

Description of Participation: All students in this design class are required to give three oral presentations related to their design projects. If you choose to participate in the research project, your presentations will be videotaped. The recorded presentations will be posted on a secure share-drive accessible only by you and the project faculty. You will be asked to review your recorded presentations and reflect in writing on how you think you can improve your oral communication skills. All recorded presentations will be held confidential and will be deleted at the conclusion of the research project. Your participation in the research project is voluntary and will in no way effect your grade in this class. You will have the right to terminate your participation at any time during the project. I need 20 of you (about half of the class) to volunteer to participate as the project students and another 20 as the control group. You can volunteer for either of these two roles by signing the appropriate place at the bottom of this document and returning it to me. Please note that by volunteering to become a project student, you are giving your consent for your oral presentations to be videotaped and for your team-level rubric data to be used in the analysis phase of this research. By volunteering to become a control group student, you are giving your consent for your team-level rubric data to be used in the analysis phase of this research.

Length of Participation Your participation in this project will continue in the junior design (ECGR3157) and senior design (ECGR3253 and ECGR3254) classes. The nature of your participation in these classes will be the same as in this sophomore design class (ECGR2252). If you decide to participate, you will be one of the 20 participants in this study. Your last oral presentation in the ECGR3254

class will be observed by a panel from local industry and their feedback will be provided to you for your benefit.

Risks and Benefits of Participation: There are no risks to participation in this study. The benefit of participation in this study is that you will get constructive feedback from faculty and engineers from local industry to improve your oral communication skills. Volunteer Statement: You are a volunteer. The decision to participate in this study is completely up to you. If you decide to be in the study, you may stop at any time. You will not be treated any differently if you decide not to participate or if you stop once you have started.

Confidentiality versus Anonymity: Any information about your participation, including your identity, is confidential. The following steps will be taken to ensure this confidentiality: All recorded presentations will be posted on a secure share-drive accessible only by you and the project faculty. The recordings will not be shown to anyone else and will be deleted at the conclusion of the spring 2014 ECGR3254 class. In case the results of this study are published, the data collected by the oral presentation rubrics (see attached) shall be anonyms and will not contain any identifying information or any link back to you or your participation in this study. Fair Treatment and Respect: UNC Charlotte wants to make sure that you are treated in a fair and respectful manner. Contact the University’s Research Compliance Office (704.687.3309) if you have any questions about how you are treated as a study participant. If you have any questions about the project, please contact Mehdi Miri (704-687-8416, [email protected]).

Participant Consent:

________________________________ ________________________ ___________

Project Participant Name (PLEASE PRINT) Project Participant Signature DATE

__________________________________ ______________________ ___________

Control Group Participant Name (PLEASE PRINT) Control Group Participant Signature DATE

______________________________________ ___________________________

Investigator Signature DATE

mailto:[email protected]

APPENDIX II

APPENDIX III

Reflective Writing Prompts (Please write a one-page response per team in sophomore design, individually in Senior

Design and upload to subfolder named “Self-Reflections”)

By reflecting on your presentation performance and by articulating that self-assessment of the performance, you can gain some understanding of where you are now as a presenter, what has challenged you in this mode, and what you have accomplished at this point. Thus, the purpose of responding to these reflective prompts is to describe that understanding and those accomplishments to your professor, and to provide a starting point that will document how your oral presentation skills develop over time with practice and feedback.

Read these questions before you view the video of your performance, and then take notes while you watch the video to gather evidence and detail to develop the reflective piece.

AUDIENCE AWARENESS

How prepared were you to interpret information for the stated audience? What was challenging about this aspect of the presentation, especially the question and answer portion?

What did you notice about your group’s interaction with the audience when you viewed the video? Review the Q&A portion of the presentation. Critique the responses given and suggest how they might be improved.

How did the physical arrangement of the presentation impact the group’s ability to interact with the audience? If there were barriers to communication, how might you address them? DEVELOPMENT AND ORGANIZATION OF KNOWLEDGE Consider the transitions from speaker-to-speaker in the video—how effective are they? How might they be improved? How did planning and developing the oral presentation and the accompany graphics help you deepen your understanding of the technical content or “new knowledge” you have gained? How was your performance impacted by your level of interest in the specific aspect you spoke about as a member of the team?

DELIVERY Make three or four directive statements recommending specific changes that the group/individual should make to improve the delivery component before the next presentation. What will you do to make those changes? What are your next steps?

Write out at least two things that you think were particularly strong about the presentation. Back up your comments with specific examples from the video.

Appendix IV Actual rubric used to assess the team presentation.

Analyt ic Rubric for Oral and Visual Communication

Name _________________ Evaluator________________________ Category 3. Excellent 2. Satisfactory 1. Deficient Score

(1-3) AUDIENCE Awareness of audience’s prior knowledge and needs

Excellent awareness of audience’s prior knowledge & needs Appropriate dress

Adequate awareness of audience’s prior knowledge & needs Suitable dress

Lack of awareness of audience’s prior knowledge and needs Inappropriate dress

Interaction with audience

Engages audience with enthusiasm

Maintains eye-contact Conducts Q&A with clear answers & explanations

Engages the audience with some enthusiasm

Makes some eye contact Conducts Q&A with adequate explanations

Lacks engagement with audience

Makes little eye contact

Lacks knowledge to conduct Q&A

CONTENT Visual depiction of ideas

Superior visuals facilitate message delivery Frequently employs prototype

Average visuals facilitate message delivery Sometimes employs

prototype

Weak visuals detract from message delivery Rarely employs prototype

Focus

Focuses presentation by providing context

Provides some context to focus presentation

Fails to focus presentation by providing a context

Organization

Sequences ideas logically

Compelling introduction Strong, clear conclusion

Acceptable sequence of ideas Suitable introduction

Adequate conclusion

Neglects to sequence ideas

Weak introduction No clear conclusion

Quality of technical content

Technical content is clear and accurate

Proper, accurate references

Technical content is satisfactory Uses adequate references

Technical content is lacking

Fails to use proper, accurate references

DELIVERY Projection Elocution Filler word usage

Voice is clear and audible

Rarely reads from notes/slides Rarely uses filler words: “um,” “ uh,” “like,” “well,” etc.

Voice usually clear, audible

Sometimes reads from notes/slides Uses some filler words

Voice is unclear and inaudible

Reads from notes too often Filler words interfere

Time and Pacing

Adheres to time limit Effective pacing

For 20 min. allocated, team breaches limit by +/-3min. At times pace is too fast or too slow

For 20 min. allocated, team breaches limit by +/-5min. Pace is uneven: too fast or too slow

Evaluator’s Comments: Total: / 24

Paper ID #9303

Effect of Student Model Presentations from a Speaking Contest on the Devel-opment of Engineering Students as Speakers

Ms. Maryellen Meny OverbaughMr. Michael Alley, Pennsylvania State University, University Park

Michael Alley is an associate professor of engineering communication at Pennsylvania State University.He is the author of The Craft of Scientific Presentations (2nd ed.) and faculty advisor for Utree: Under-graduate teaching and research experiences in engineering.

Ms. Christine Haas, Engineering Ambassadors Network


Effect of Student Model Presentations from a Speaking Contest on the Development of Engineering Students as Speakers

Introduction

Because of TED.com, many high quality models exist of professional engineers and scientists presenting. However, high quality examples of students presenting are lacking. Such high quality models by engineering students are important because many engineering students cannot project themselves presenting in the same manner as TED speakers, who are experts in their fields [1]. For instance, engineering students simply cannot generate the same level of original content as TED speakers do.

At Pennsylvania State University, we have tried to address this void by introducing a technical presentations contest for engineering students. Called the Leonhard Center Speaking Contest, this speaking contest draws contestants from several engineering sections of a required presentations course. The final round of the Contest features eight engineering undergraduates making 10-minute presentations that describe an engineering solution to a societal problem. The engineering undergraduates in the contest took a required presentations course the previous semester and were selected in a competition by their peers for this contest. The timing of the contest finals is such that it occurs toward the beginning of the next semester, when a new crop of 150 engineering students are beginning to take the required presentations course. For that reason, many of the 200 attendees of the finals are students who are just beginning this course.

In addition, we have hired the local public television station to film the event, using production lighting and a three-camera set-up. The station also performs the editing of the films. Although perhaps not quite the quality of TED films, our films particularly in our last two contests have a studio-quality feel. Several presentations and montages of presentations are now posted on the web so that students have access to these examples throughout the semester.

We hypothesize that we can accelerate how quickly less advanced engineering students learn presentation skills by having those less advanced engineering students view strong models of presentations by other engineering students. To test this hypothesis, we had students who are in the first weeks of an engineering presentations course attend the finals of a speaking contest that consists of students from the previous semester’s course. In our particular case, over two semesters, about 300 students in the early stages of their presentations course saw eight of the strongest presentations from the previous semester. Because we teach an uncommon approach to technical presentations (the assertion-evidence approach), we sought to determine whether the students noticed the approach, what they thought about the approach, and whether they would consider adopting the approach.

This paper first presents a literature review of universities and professional societies that have tried to highlight model presentations by engineering students to raise the level of presentations by other engineering students. Included at the end of this literature review is a description of our speaking contest as a means to expose the younger students to the strong presentation models from engineering students. Next we discuss two tests to determine whether the less advanced students learn presentation skills at a more accelerated rate than similar

students who did not see such models. Finally, we discuss the results of those tests and provide explanations of those results. Literature Review

Various universities and institutions have posted model presentations by engineering students as a means to accelerate the development of other engineering students as speakers. This section discusses those attempts with the following criteria in mind: (1) strengths of the models, (2) limitations of the models, and (3) quality of the films. Table 1 summarizes the student models in this review.

Table 1: Summary of Relevant Literature on Student Model Presentations

Model Student Presentations from Other Institutions. As a first source for student model presentations, British Columbia Institute of Technology (BCIT) has created a competition called “Presentation Idol for Engineering Students,” informally known as Idol [2]. The initial

Source of Models Student Population Affected

Analysis of Models

British Columbia Institute of Technology: [2]

Speaking contest that complements a technical communication course

+ 40 students in initial contestant pool; question session at the end of talk; contest sponsored by companies and the university. + Majority of students from civil, mechanical, and electrical engineering. − Single-camera viewing

Purdue: EPICS Course [3]

Posted films of course design projects

+ Detailed presentation on an entire project; question session at the end of talk; talks sponsored by companies. − Speakers are not visible − Single-camera viewing − Talks follow topic-subtopic defaults of PowerPoint

ASME: Old Guard Competition [4]

ASME’s Student Professional Development Conference

+ Representation of students around the world − Difficulty accessing models on ASME’s webpage − Single-camera viewing

Royal Academy of Engineering Global Grand Challenges Summit [5]

Student Day Presentations from the Global Grand Challenges Summit

+ Easy access on YouTube.com + Multi-camera view − Long-winded introduction by professor before student speakers begin

CART Showcase [6] Biomedical engineering students

+ Easy access on Vimeo.com − Unsteady camera − Single-camera view

Construction Institute of ASCE [7]

Speaking contest for civil engineering students

+ Easy access on Vimeo.com + Double-camera view − Students reading from notes

Pennsylvania State University

Speaking contest for College of Engineering that complements a technical presentations course

+ Professional venue; many engineering disciplines represented; both individual talks and montage of talks easily accessed on Vimeo.com + Assertion-evidence approach to presentations + Multi-camera view

contestant pool consists of 40 full-time or part-time engineering students who have completed one BCIT communication course. After a preliminary round, eight student finalists compete in Idol by giving a 6 to 10 minute filmed talk, followed by a question session. BCIT then posts the winner’s film on the Idol website as a model presentation.

The BCIT Idol competition produces strong student models. Because of the large initial pool size, the variety of topics strengthens the student models. With that being said, because the large majority of students study either civil, mechanical, or electrical engineering, the models lack a broad representation of other engineering disciplines. In addition to the learning that occurs during the competition itself, the contest films the model presentations, which adds to the effectiveness of the models. A single camera view captures the 6 to 10 minute talk, as well as a question session. The films not only benefit the students and faculty at BCIT, but their various sponsors as well.

As a second source of student model presentations, Purdue University has created student models through its Engineering Projects in Community Service, or EPICS, course. Students enrolled in the course design, build, and develop solutions to engineering problems that benefit the local community [3]. Upon completion of the project, each team presents their results, which are filmed and posted on the EPICS website. In general, the main strength of these presentations lies in the quality and interest of the design projects. In addition, the voices of the students reflect the enthusiasm that they have for the subject. Like BCIT, the EPICS models use a single camera view and include a question session in the film. One limitation of these models is that the speakers are not visible in the films. As a result, students cannot observe the speakers’ nonverbal cues, which can be beneficial.

Perhaps the biggest weakness of these presentations, though, is that the presentation slides follow the topic-subtopic organization of PowerPoint. In other words, the slides do not follow the cognitive load recommendations of Sweller [8], who states that audiences suffer cognitive overload in the verbal portion of the brain when they try to process too much written text and spoken words simultaneously. Given the bullet-ridden slides of these students, that overload condition often occurs. In addition to having too much text, the slides suffer from not following the Mayer’s multimedia learning principles [9], which states that people learn more from words (mainly spoken words) and relevant images than from words alone. More than 40% of the student slides at the EPICS web-site do not have any relevant images.

Another type of student model that is available is from ASME’s Old Guard Competition at the Student Professional Development Conference (SPDC). The competition emphasizes the importance of clear and concise oral communication, as well as recognizes exceptional student speakers [4]. Because the SPDC has global venues, the models are diverse. However, the models are not easily accessible on ASME’s website, making it difficult for engineering students to benefit from them.

While the student models mentioned previously are effective, accessing them can be a challenge for engineering students. In today’s generation, students might look to more popular websites, such as YouTube.com or Vimeo.com to find student models. For instance, a simple search on YouTube for engineering student presentations will direct you to the Royal Academy of Engineering’s presentation at the Global Grand Challenges Summit [5]. This high quality model consists of a multi-camera view, showing the speakers, slides, and audience. One

limitation of the model is that many students might not make it through the longwinded 6-minute introduction of the student speakers given by a professor.

Not only is YouTube.com popular among students, but Vimeo.com is also growing in popularity. Two student models that can be viewed on Vimeo.com are the CART Showcase and the Construction Institute of ASCE. The former model is a single-camera view limited by an unsteady camera [6]. The latter is a double-camera view, showing both the speakers and slides [7]. However, some of the students read from their notes, which is a practice frowned upon in engineering talks.

Our Student Model Presentations. At Penn State, the speaking contest, which originates in several engineering sections of a required presentations course, features eight engineering undergraduates who took the required presentations course the previous semester and who were selected in a competition by their peers. While the engineering sections of the course are distinguished by an emphasis of advanced presentation techniques such as the assertion-evidence approach [18], the fundamental objective of the contest is to create effective student models. The timing of the finals is such that it occurs during the beginning of the next semester, when a new crop of several hundred engineering students are beginning to take the required presentations course. For that reason, many of the 200 attendees of the finals are students who are just beginning this course.

Shown in Figure 1 is a scene showing a speaker on stage from the third contest, and shown in Figure 2 is a scene showing the speaker, the audience, and camera crew from the third contest. As can be seen from the view of the speaker and stage in Figure 1, the setting is professional. As can be seen from the view of the first contest’s audience in Figure 2, the contest was well attended. The second contest had similar numbers of attendees.

Figure 1. View of a speaker on stage for the third speaking contest. In selecting the room, we wanted a venue that was professional and would accommodate at least 200 visitors.

Figure 2. View from the back of the room of third speaking contest. The room for this contest held 200, and several people viewed from the doorways. Seen in the back are two of the three cameras for filming the contest as well as the control board for the sound system.

In addition, because films of many of the presentations are posted on the popular web site Vimeo.com, students in the technical presentations course have access to these examples during the semester. These films use multi-camera views of the speaker, slides, and audience. In addition, the films are produced as a single talk or as a montage of multiple talks. The montages include teaching messages such as “Own your content—do not read from cards or slides.” These messages were included so that the montages could be used as teaching tools in engineering courses. For instance, one montage communicates the principles of effective delivery (in an engineering presentation), while another communicates the specific principles of effective slides. For each message, one clip from the contest serves as supporting evidence. As a final note on the films, we have tried to locate the speaking contests in venues that look professional and that seat about two hundred attendees. In our first contest, the room’s lighting relied on fluorescents, which lowered the quality of the films. Methods

More than 500 students from a technical presentations course have attended three speaking contests that occurred: one in January 2013, a second in September 2013, and a third in February 2014. To determine the effectiveness of the student speaking contest on the acceleration of the development of students as speakers, we used two different methods: (1) surveys of 127 students attending the second and third contests, and (2) observations from faculty who taught the technical presentation course taken by those 500 students. These methods

provide both qualitative and quantitative data that can be analyzed to determine whether this speaking contest is a valuable tool for accelerating the learning of presentation skills. Student Surveys. Our first method was a survey taken by more than 125 students who attended the contest in Fall 2013 or Spring 2014. In the survey, students were asked to analyze one talk from the Contest in general and to comment specifically on entry point, slide design, and delivery. From these analyses, we sought to extract whether students noticed three recognized features [21] of the assertion-evidence approach, which the contest speakers followed: (1) increased focus, (2) better audience understanding, and (3) more confidence. For each feature, the student responses were scored from 0 to 1—with a 0 representing no mentioning of that feature and a 1 representing a clear indication of that feature.

In addition, from these analyses, we intended to determine qualitatively what influences the contest had on the students. The goal of this review was to find what practices the students liked best in the talks and whether those practices lined up with the practices taught in the course. The rationale here was that if the students liked a practice, then they were more likely to try to incorporate that practice in their presentations during the rest of the semester. In addition, if the practice lined up with the practices that the instructors were trying to teach, then the Contest had aided the instructor in the teaching of those practices.

Using a course management system, we distributed the surveys to students enrolled in the technical presentations course. The surveys were completed during the three weeks following the contest, and the student survey results were reviewed by the course instructor and course assistant. Faculty Observations: As a second source of evidence, we sought testimonies from four instructors who taught the technical presentations course taken by the 500 students who attended the contest. At Penn State, all students are required to take a 3-credit presentations course—the technical presentations course was an option for engineering students to fulfill that requirement. Two of these four instructors have taught the engineering presentations course for more than eight semesters, with six of those semesters occurring before the incorporation of the speaking contest. The main question to these instructors was how having the students view the presentations of the speaking contest has affected the rate at which those students learn the principles of the course. In other words, what principles of the course appeared to be learned more readily because of the contest? If the principles observed by the faculty matched the practices identified by the students in their surveys, it is logical that the contest influenced and possibly accelerated the student’s learning of those principles. Results and Discussion The following sections present the results from the three perspectives pursued to determine the effect of the speaking contest on the acceleration of the learning of presentation skills by students who viewed those presentations. The three perspectives were (1) quantitative analysis of surveys of students who viewed the contest presentations, (2) qualitative analysis of surveys of students who viewed the contest presentations, and (3) responses from faculty who taught those students.

Student Surveys: Quantitative Analysis. More than 125 students enrolled in a technical speaking class completed a survey about the techniques used by students in the speaking contest. For each of these surveys, we derived scores (from 0 to 1) on the following three attributes: the talk had an appropriate scope and depth; the talk targeted the audience; and the speaker was confident. These three characteristics are the proclaimed attributes of assertion-evidence approach [21], which the Contest speakers were taught and which they followed. Table 2 presents the results of this scoring. As seen in this table, the talks scored very high, especially for audience understanding. While these results are positive, two limitations exist. First, the students being surveyed had the option of choosing which speaker to analyze. Given that, most students likely chose their favorite speaker in the contest. A better test would be to assign the students a specific speaker to evaluate (and preferably one outside the surveyed student’s major). Second, we were extracting these qualities from the general analysis provided by the students. In some cases, such as the scope and depth, the students did not make comments—if asked directly about that quality, the score might have been higher. Nonetheless, the contest talks scored high on these three attributes.

Table 2. Quantitative Results of Surveys of Students Who Viewed the Contest (N=127).

Talk had appropriate scope and depth

Talk targeted the audience

Speaker was confident

Percentage 79.5% 96.5% 86%

Student Surveys: Qualitative Analysis. More than 125 students enrolled in a technical speaking class completed a survey about the techniques used by students in the speaking contest. More specifically, the students were asked to analyze in general the presentation of a Contest speaker and to analyze in particular three aspects of that presentation: entry point, slide design, and delivery. Because the students analyzed one of eight presentations, the results were generalized by identifying three of the most frequently acknowledged practices for each of the three specific categories. The practices that the students identified for entry point to engage the audience, slide design, and delivery are discussed below.

Entry Point to Engage Audience. To begin, three of the most frequently acknowledged practices for the entry point were the use of relatable topics, emotional appeal, and statistics as a means to engage the audience. Regardless of the contestant’s topic, students acknowledged that they were more inclined to listen to the rest of the presentation if they could relate to the entry point. For instance, an electrical engineering student began his talk [12] about TV white space zones by discussing the frustration that follows a Wi-Fi connection failure. While many students did not fully understand his solution at the start of the talk, they stated that they were more intrigued to listen after hearing about Wi-Fi connection failure, which occurs on a daily basis for many students. In addition to using relatable topics, the use of emotional appeal and statistics also resonated with the students. Depending on the subject of the talk, either practice could be used. Many students liked the use of emotional appeal, such as the appeal in one talk [14] about quality of life of an age-related macular degeneration patient, because it quickly created personal concern for the topic. On the other hand, students were influenced by the use of statistics, such as the statistic given in a talk [13] on number of children who will die from pediatric cancer in one

year. Because the course teaches all three of these entry points as options, the contestants served as strong models for creative and effective entry points.

Slide Design. In addition to entry points, the students also analyzed practices for slide design, such as blank slides, use of images (as opposed to bullets), and highlighting of key information in graphics. While slides can convey a great deal of information, the students thought that some of the most effective slides were blank ones. The use of blank slides forced the audience to listen to the speaker without any distractions. For instance, this technique was often used at the conclusion to draw in the audience’s attention for closure. A second identified practice of the slides designed by the contestants was to outline the content of their talk by using pictures, as opposed to bullets, for each main point. In addition, as a way to keep the audience focused on the appropriate image, when the contestant began to talk about each topic, the corresponding picture would transition from grayscale to color. A third identified practice of visual aids was the emphasis of important information in graphics. In technical presentations, graphical representation of data can confuse the audience if not explained properly. The students frequently said that they appreciated the use of boxes to emphasize important information in slides, such as specific columns on a bar graph. All three of these practices are taught in the course and are part of an assertion-evidence approach to slide design.

Delivery. Lastly, the surveyed students identified the following practices for delivery: deft handling of equipment, not exhibiting distracting movements, and having a conversational tone. All contestants used a hand-held slide advancer during their presentation. The majority of students who completed the survey took note of the effectiveness of the slide advancer, because there were no technical difficulties or confusion when using the device. Second, a lack of distracting movements strengthened the delivery of the talks, because the audience could concentrate on the words and slides of the presentation. Many students noted the tall posture that speakers maintained, as well as their neutral hand positions, such as keeping their arms at their sides or holding the slide advancer in front of their bodies. Finally, the last practice of delivery identified by the surveyed students was the conversational tone that the speakers used. Most of the students noted that the talk was not memorized, and in fact, seemed very natural. Some students even noted that the conversational tone seemed to make the presenters appear at ease on stage, because they were comfortable and confident with the content of the talk. All three of these practices are practices taught in the course, with the practice of speaking extemporaneously (a practiced talk in which the speaker fashions words on the spot) being the most emphasized.

Summary. The results of the student surveys not only revealed frequently acknowledged practices for three distinct categories (the entry point to engage the audience, slide design, and delivery), but also the attention paid by the contestants to the details of the presentations and their advanced techniques. In the surveys, students conveyed a sense of awe in regards to the contestant’s presentation skills, and many stated that they hope to emulate these techniques. Therefore, the qualitative analysis of student surveys revealed that contest has in fact influenced the students to add new techniques to their repertoire of presentation skills. Faculty Observations. This section presents observations by faculty about the effect of the Speaking Contest on their students. Providing those comments four instructors who taught the technical presentations course from which the contestants came and for which students attended or watched films on the web [15]. Two of these instructors had much experience in this course having taught several classes before the Contest became a part of the course. Overall, all four

instructors asserted that the Contest significantly accelerated the learning of presentation skills by the students who attended. In particular, the instructors asserted that the Contest has affected the acceleration of learning in three ways: (1) acceptance of the assertion-evidence approach, (2) willingness to present without a script or note cards, and (3) appreciation for importance of narrowing the topic to achieve depth.

First, the Contest has affected the acceptance by students on the assertion-evidence approach taught in the course. In this course, the instructors challenge the common practice of creating a talk based on topic supported by subtopics. Instead, the instructors call for students to view their talks as a series of assertions that need strong evidence as support. The difference between these two approaches is most readily seen in the slides of the two approaches. In the ubiquitous topic-subtopic approach, the slides follow the defaults of PowerPoint: a topic-phrase headline supported by a bulleted list. In the assertion-evidence approach [16–17], the slide is characterized by a sentence headline that states the main message of the slide. That message headline is then supported by visual evidence: photographs, drawings, diagrams, films, and graphs. In this structure, bulleted lists do not appear. Because the structure is so different from what engineering students have seen, students are reluctant to adopt it. However, all four instructors asserted that having the students view the eight presentations of the Contest (all of which follow the assertion-evidence approach) has made the students much more receptive to this strategy.

Second, the instructors asserted that having the students view the Contest presentations has made the students more willing to present without notes. In engineering, the expectation in most technical presentations is that the engineer will speak to the audience rather than read a script or speak from note cards. Although speaking extemporaneously is the expectation in engineering, the delivery style taught in many high school speech classes and in the non-engineering speech classes of Penn State is to rely on note cards that the speaker carries.

Third, the instructors asserted that having the students view the contest presentations has helped the students select topics that are more focused. Often students (and professionals) try to cover too broad of a scope in their presentations. Doing so prevents them from achieving a depth that satisfies the audience [18]. The instructors contended that viewing the contest presentations gave students a better idea for what type of topic could achieve depth in a 10-minute presentation. Moreover, students who viewed the contest presentations were more willing to adopt strategies to limit the scope (such as defining limitations or making assumptions). Conclusion and Plans for Dissemination Overall, having students in a technical presentations course view the presentations of the Speaking Contest appears to have accelerated the learning of presentation skills by those students. As indicated in the student surveys, students were impressed by the ways in which the contest presenters delivered their talks in a confident manner (without notes), the assertion-evidence approach for visual aids that the student presenters used, and the entry points that the student presenters used to motivate interest. In addition, the instructors of those students asserted that having students view the contest presentations early in the semester made the students more inclined to adopt key principles in the course, including the assertion-evidence approach, which cuts against the current common practice in engineering and science.

A question arises whether contest at Penn State can affect the learning of presentation skills by students who are not enrolled in the technical presentations course. In other words, can students in other engineering courses who view the online contest presentations be influenced to adopt the best practices of those presentations? To address this question, we are incorporating the contest presentations into a class period to teach presentation skills for undergraduate courses such as first-year design. These class periods are being taught by a new undergraduate engineering organization (UTREE: Undergraduate teaching and research experiences in engineering) that consists of students who have excelled both in their technical classes and in the technical presentations course [19]. Because only 25% or so of the engineering students in the College take the technical presentations, teaching the main skills of that course to the other 75% is a desired goal. Beginning in the Spring 2014 semester, the class periods on presentations taught by UTREE will include example films from the speaking contest. Viewing these films will be part of the preparation assignment for the class period and the class instruction. Through surveys of students and faculty in those classes, we intend to discern the effect of viewing the contest presentations on the students’ learning of presentation skills and adoption of best practices such as speaking to the audience, as opposed to speaking from note cards.

Finally, through the web, we are making available the contest presentations to engineering courses outside our institution. For instance, we have a web-site [15] that presents the most popular individual presentations from the Contest. Already a top Google.com listing for the search term model presentations, this Model Presentations web-site includes both the filmed presentations and the accompanying handout for each of those presentations. According to our web statistics, the Model Presentations web-site has been visited more than 2000 times from November 2013 through December 2013. In addition, according to statistics on Vimeo.com, one of those presentations [20] has been played more than 5000 times from February 2013 through December 2013, and another [14] has been played more than 4000 times from October 2013 through December 2013. The teaching montages are also receiving many views. For instance, the montage on slide design [11] was viewed more than 1000 times in January 2014, and the new montage on delivery [22] is already being adopted by engineering design courses as an instructional tool. These early statistics suggest that a significant number of people are showing interest in these films. Acknowledgments We wish to thank the Leonhard Center for the Enhancement of Engineering Education for their support of the Speaking Contest. References 1. “Inviting and Preparing Speakers,” TED: Ideas Worth Spreading. Accessed 19 November, 2013.

www.TED.com.

2. Katherine Golder, Deanna Levis, and Darlene Webb, “Engaging students in communication through an oral presentation competition with prizes: Presentation Idol for engineering students,” 2013 ASEE Annual Conference & Exposition (Atlanta, GA: ASEE, June 2013), http://www.idol.bcit.ca/idol

3. EPICS / Purdue, “Project Conceptual Review Presentation Template,” https://engineering.purdue.edu/EPICS/Projects/Teams (West Lafayette, IN: Purdue University, 2011).

4. ASME, “Old Guard Presentation Competition,” https://www.asme.org/events/competitions/old-guard-competitions/old-guard-prize-oral-presentation-competition (2013).

5. Royal Academy of Engineering Global Grand Challenges Summit, http://www.youtube.com/watch?v=VKM-ZUarsAw (2013).

6. CART Showcase, “Biomedical Engineering Student Presentation: CART Best Presentation Award Winner,” http://vimeo.com/20650652 (2010).

7. Construction Institute of ASCE, “2013 Heavy Civil Engineering Student Challenge Winning Presentation,” http://vimeo.com/72267917 (2013).

8. John Sweller, “Implications of Cognitive Load Theory for Multimedia Learning,” The Cambridge Handbook of Multimedia Learning, ed. by Richard A. Mayer (New York: Cambridge Press, 2005), pp. 19–30.

9. Richard E. Mayer, Multimedia Learning (New York: Cambridge, 2001).

10. Leonhard Center Speaking Contest, “Creating Effective Presentations in Engineering and Science,” https://vimeo.com/81810680.html (University Park, PA, 2013).

11. Leonhard Center Speaking Contest, “Creating Effective Slides in Engineering and Science,” https://vimeo.com/81809530.html (University Park, PA, 2013).

12. Timothy Hackett, “Using TV White Space (TVWS) Bands as a Secondary Network for WiFi Traffic,” presentation (University Park: Leonhard Center Speaking Contest, September 2013).

13. Holly Cardillo, “Research of Immunotherapy to Treat Cancer,” presentation (University Park: Leonhard Center Speaking Contest, September 2013).

14. Isaac Moore, “Addressing AMD Vision Loss: Telescope Prosthetics,” presentation (University Park: Leonhard Center Speaking Contest, September 2013).

15. “Model Presentations,” Speaking Guidelines for Engineering and Science Students, http://writing.engr.psu.edu/models.html (University Park: Leonhard Center Speaking Contest, 2013).

16. Traci Nathans-Kelly and Christine Nicometo, Slide Rules: Design, Build, and Archive Presentations in the Engineering and Technical Fields (New York: Wiley-IEEE Press, 2014).

17. Joanna Garner and Michael Alley, “How the Design of Presentation Slides Affects Audience Comprehension: A Case for the Assertion-Evidence Approach,” International Journal of Engineering Education, vol. 29, no. 6 (2013), pp. 1564-1579.

18. Michael Alley, The Craft of Scientific Presentations, 2nd ed. (New York: Springer-Verlag, 2013), pp. 59-68.

19. Victoria Vadyak, Michael Alley, Joanna K. Garner, and Christine Haas, “Undergraduate Teaching and Research Experiences in Engineering (UTREE): an Engineering Student Organization with a Communication Focus,” 2014 ASEE Annual Convention and Exposition (Indianapolis, IN: ASEE, 2014).

20. Mimi Overbaugh, “Osseointegration Prosthetics,” presentation (University Park: Leonhard Center Speaking Contest, January 2013).

21. “Tutorial for Assertion-Evidence Presentations: Increased Focus, More Confidence,” http://writing.engr.psu.edu/assertion_evidence.html, ed. Michael Alley (University Park, PA: Penn State, 2014).

22. Leonhard Center Speaking Contest, “Effective Delivery in Engineering and Science Presentations,” https://vimeo.com/86342823.html (University Park: Leonhard Center Speaking Contest, 2013).

Paper ID #9522

Final Results of Reliability Testing for the Norback-Utschig Presentation Scor-ing System and Implications for Instruction

Dr. Judith Shaul Norback, Georgia Institute of TechnologyDr. Tristan T. Utschig, Georgia Institute of Technology

Dr. Tristan T. Utschig is a Senior Academic Professional in the Center for the Enhancement of Teachingand Learning and is Assistant Director for the Scholarship and Assessment of Teaching and Learningat the Georgia Institute of Technology. Formerly, he was a tenured Associate Professor of EngineeringPhysics at Lewis-Clark State College. Dr. Utschig consults with faculty across the university aboutbringing scholarly teaching and learning innovations into their classroom and assessing their impact. Hehas regularly published and presented work on a variety of topics including assessment instruments andmethodologies, using technology in the classroom, faculty development in instructional design, teachingdiversity, and peer coaching. Dr. Utschig completed his PhD in Nuclear Engineering at the University ofWisconsin–Madison.

Mr. Anthony Joseph Bonifonte, Georgia Institute of Technology

Anthony Joseph Bonifonte is currently in his 3rd year of Georgia Tech’s PhD program in OperationsResearch in the Industrial and Systems Engineering Department. He attended Oberlin College as anundergraduate, majoring in math and biology. He has served as teaching assistant five times for mathand industrial engineering courses. He currently works as a graduate research assistant in Georgia Tech’sCenter for the Enhancement of Teaching and Learning (CETL) where he assists with assessment anddata analysis for ongoing CETL projects. His thesis research involves mathematical models and decisionmaking in cardiology.

Gloria J Ross, Georgia Institute of Technology

Gloria Ross is currently a PhD candidate in History and Sociology of Science and Technology at GeorgiaTech. Her research focuses on the spatial and demographic factors that shape urban food distribution sys-tems. She currently works as a graduate research assistant in Georgia Tech’s Center for the Enhancementof Teaching and Learning (CETL) where she assists with assessment and data analysis for ongoing CETLprojects.


Final Results of Reliability Testing for the Norback-Utschig

Presentation Scoring System

and Implications for Instruction

Abstract

In this paper we report the results of our final work completed to improve the reliability and

usability of the Norback-Utschig Presentation Scoring System developed at Georgia Tech, and

based on executive input. Our approach was the modified Delphi method, a multi-stage feedback

process used to generate consensus among diverse stakeholders. The method was used to collect

data for seven of the 13 presentation skills not yet having high reliability between raters: “initial

and final impressions,” “logical flow,” “key points,” “layout and design,” “graphics,” “vocal

quality,” and “personal presence.” Data was collected from a variety of individual stakeholders,

and modifications were made to the scoring system. For example, “initial connection” was

renamed to “first impression.” The definition of the following skills were clarified to make them

easier to understand: “logical flow,” “key points,” “layout and design,” “graphics,” “vocal

quality” and “personal presence.” Once the skills were modified, the new scoring system was

tested for reliability in three settings—industrial engineering teaching assistants and the two

developers of the scoring system, biomedical engineering teaching assistants and one developer,

and a class of nuclear engineering seniors. Results indicate that the reliability between raters of

all skills tested improved at a significant level. The revised skills now have good to high

reliability. Implications for instruction will be discussed.

Introduction

The Norback-Utschig Presentation Scoring System for Engineers is based on interviews with 72

executives, with engineering degrees, who work for a variety of companies employing engineers.

Faculty input has been gathered over the years as well. The skills identified were those essential

for a “stellar presentation.” In this paper we report on the results of our final steps taken at

Georgia Tech to revise and test the Norback-Utschig Presentation Scoring System to improve its

usability and reliability. We expand upon our previously published work, which is briefly

reviewed below for context. We also share implications for instruction: teaching tips for helping

students perform better on each skill1. Examples include “key points: central message clear

throughout by linking details to big picture,” and “personal presence: effectively combines

energy, eye contact, and movement.”

One year ago we reported on our work using a modified Delphi method to revise a 19-skill

presentation scoring system. The method is a multi-stage feedback process used to generate

consensus among diverse stakeholders2. In the earlier paper we outlined lessons learned from

discussing the use of the scoring system with users. We also described how we, first,

summarized feedback we had collected from a small alumni-funded study, second, distributed

the summary to stakeholders for their review and, third, modified the scoring system according

to the newest feedback. The result was a 13-skill presentation scoring system with enhanced

usability and clarity. Figure 1 summarizes changes made to the scoring system after round 1 of

the Delphi update. For example, “flow” was added to “vocal quality;” both “engaging graphics”

and “appropriate graphics” were combined into “graphics,” “first/last impression” and “audience

connection” were combined into “initial connection.” “Sequencing” was redefined as “logical

flow.” Additionally, Appendix A shows details for how the definitions for skills were updated as

a result of these changes.

Figure 1 – Delphi Round 1 changes

This second round of feedback and modifications completed our modified Delphi method

procedure. To complete this second round of feedback and modifications we first collected and

summarized a second round of feedback from a variety of individual stakeholders, second,

modified the scoring system once more based on the newest feedback and third, tested the

scoring system for inter-rater reliability in several different settings. The changes made in Delphi

round 2 include renaming “initial connection” to “first impression” and changing the definitions

of about half of the skills to address user feedback on clarity and ease of understanding those

skills. Additionally, two skills, “sensitivity to time” and “taking questions,” were moved to the

end of the survey.

Below we describe literature related to our work on the Norback-Utschig Presentation Scoring

System, within engineering and then in the expanded context of STEM. Then we discuss

information available on engineering written communication scoring systems. Next, we focus on

the two methods used: the modified Delphi system and the testing of the scoring system for inter-

rater reliability in several different engineering settings. We describe the result: a highly usable

and reliable set of 11 skills divided into four categories (customizing to the audience, telling the

story, displaying key information, and delivering the presentation) plus two additional skills

which can be used when appropriate at the end of a presentation. The two skills are “sensitivity

to time” and “taking questions.”

Finally, we share a few tips that have worked well in coaching capstone design students to

improve each of the 13 skills. One example consists of teaching students to be “definite!” in

order to enhance the first impression they make with their audience. Another example is having

students practice, throughout their presentation, using reminders of how details relate to the big

picture.

Literature Review

Based on our literature review, the publication of inter-rater reliability studies or other

development work for research-based rubrics or scoring systems is not common. As noted above,

we focused first on scoring systems used for evaluating both oral communication within and

outside of engineering settings and then we turned to scoring systems for written engineering

communication. From what we could see, other work may be going on in engineering, but it is

not frequently documented in publications. Here, we briefly review what we have found to date.

Earlier we found that the most comprehensive work to date in developing an oral presentation

rubric for engineering settings was done at Iowa State, and is now in use at several other colleges

and universities3,4

. We also found the TIDEE Consortium had studied inter-rater reliability of

student communication in team settings for engineering design coursework5. Many other oral

presentation rubrics are used elsewhere, but the theoretical or research-based underpinnings for

these rubrics are unclear. Other than finding Thambyah’s discussion of how to create rubrics for

engineering outcomes, we reviewed similar studies of reliability in communication rubrics—

expanding from the engineering setting to STEM fields6.

These include

Mott’s work on assessing scientific expression using media7,

LaBanca’s eight-item rubric primarily targeting high school science teachers8,

Barney’s study on the effects of student self-assessment of their learning outcomes when

using an oral presentation rubric9,

Anderson and Anderson’s description of elements of oral communication for students in

the workplace, that should be modeled by instructors in their lectures10

,

Battacharyya’s use of a research-based approach to identify four key traits that industry

looks for in technical presentations: technical competence, effective delivery skills, IT

competency, and cultural awareness11

, and

Morton and Rosse’s research focusing on identifying characteristics related to the use of

personal pronouns in defining persuasive oral presentations in engineering12

.

Although two of these studies relate to workplace communication specifically for engineers, it

does not appear that any of this work directly links criteria for scoring systems to workplace

preferences or traditional rhetorical approaches. To our knowledge, then, inter-rater reliability

studies for instruments measuring the quality of engineering oral communication are very

limited.

Because of the dearth of literature describing the development and testing of systems to rate oral

communication, we have begun to review written communication as well. We identified a

number of relevant articles. For example

Covill examines writing rubrics specifically in the context of self-assessment. She

includes a summary of existing empirical research on the effect of rubric use on writing

quality, but claims most rubric studies are based on middle school students13

.

Andrade and Du found that undergraduate students perceived the writing rubric as a

useful tool that clarified teacher expectations, structured their own approach to each

assignment, and provided a reference to review and edit their work14

.

In one study involving student use of written assignment rubrics, Rawson et al. found that

students who used the rubric were able to acquire and use new field-specific terminology

in appropriate ways15

.

In a study comparing writing rubrics, Morozov concluded that students viewed the more

detailed and extensive rubric more positively than less-extensive rubrics16

. In this study,

an effective rubric model emphasized skills, elaboration of skill, and critical thinking.

One recent study compared the reliability of two writing rubrics across three different

settings and reported moderate reliability for most skills represented in the two rubrics17

.

Multiple studies address the effect of Calibrated Peer Review (CPR) on student writing18,

19, 20. CPR involves the electronic evaluation of student writing by their peers. None of

these studies specifically address rubric development or inter-rater reliability. One study

indicates significant differences in the effect of feedback received from TAs as opposed

to feedback received from peers18

.

Methods

Two major processes are included in this study. The first is the completion of round 2 of the

Delphi method to systematically modify the Norback-Utschig Presentation Scoring System to be

more usable and reliable. The second is the testing of the scoring system for inter-rater reliability

in several different engineering settings.

Final (Round 2) Modifications Using the Delphi Method

To make the final round of modifications to our scoring system, we first developed a process for

collecting stakeholder input about the scoring system resulting from the first round of the Delphi

method. Our stakeholders included faculty, graduate and undergraduate students (mostly

teaching assistants or TAs), and executives. Our academic participants represented industrial

engineering, biomedical engineering, nuclear engineering, and a Center for the Enhancement of

Teaching and Learning. In industrial engineering, our participants were two female faculty

members, three undergraduate TAs (one female and two male), and four graduate student TAs

(one female and three males). From biomedical engineering, one male faculty member and six

graduate student TAs (five female, one male) contributed. One male faculty from the Center and

one male undergraduate from a class in nuclear engineering rounded out the academic input.

Finally, three executives (one female and two male), from a variety of settings, took part,

bringing the total number of stakeholders to 21.

Our process of collecting feedback from the stakeholders consisted of two parts. The first part

was a quantitative questionnaire for users to rate, first, the ease of use and, second, the clarity of

wording on a three-point scale for each skill. The second part was qualitative and involved

interviews, focus groups, and email communication. Users explained their ratings from the

quantitative section of the survey and provided other comments about their interpretation and use

of the thirteen skills when scoring presentations.

Once the stakeholder feedback was collected, it was systematically analyzed to provide a

summary that could be used to make modifications to the scoring system. The methods used

included summative coding and content analysis. Qualitative comments about the scoring system

were first grouped by skill number and reviewed by two graduate research assistants or “coders.”

The assistants identified common themes and keywords in the comments and tallied each

comment under a theme, with some themes having only one comment. To ensure reliability and

consistency in the coding analysis, the process was independently replicated by one of the

scoring system developers, and the two sets were merged by a fourth person (the other developer

of the scoring system) upon discussion with both coders. The themes with the most comments

represented several stakeholders who shared common feedback for how to modify the scoring

system. For example, for the skill on “taking questions,” five stakeholders suggested that the

skill definition needed further clarification and specific indicators. Once these themes were

verified and finalized, they were used to guide modifications of the scoring system.

Final Inter-Rater Reliability Testing

Data Collection

It should be noted that a number of skills were not modified at all because they were already

highly reliable. For these skills, changes to the supplemental instructional materials will be made

to reflect the suggestions provided by the scoring system stakeholders.

To analyze the inter-rater reliability of the final version of the scoring system, we have collected

scores from raters in three different contexts. The Institutional Review Board approved this

research project. Prior to each rating session, permission was obtained from each presenter and

rater to use their work in this research. Each of these contexts is representative of a common

setting where the rubric might be employed.

Setting 1 – “Industrial Engineering Session” – In this capstone design context students were

preparing and presenting several presentations to clients and to academic faculty. A mixture of

videotaped interim and final presentations was used for this session, where 20 presentations were

rated by seven TAs who had moderate familiarity with the scoring system (two hours of training

and about two weeks working with students) and one of the developers of the scoring system.

Setting 2 – “Biomedical Engineering Session” – In this capstone design setting, students were

presenting progress reports on their work to classmates and their instructors at a point about

seven weeks into a 15-week semester. Fifteen videotaped presentations were rated by four TAs

who had no experience with the scoring system, and by one of the developers of the scoring

system.

Setting 3 – “Nuclear Engineering Session” – In this upper-division engineering course

populated mostly by seniors, students presented results for their completed team projects at the

end of the term. Ten live presentations were rated by 30 student peers who had no prior

experience with the scoring system, and by one of the developers of the scoring system who was

an instructor in the course.

Analysis

For each of the 13 skills on the rubric, we analyzed the data collected from the two sessions

above using two types of calculated measures for reliability: pairwise matching and criterion-

referenced matching. Pairwise matching consists of comparing each combination of rater scores

for a skill to each other. The frequency of matches is then calculated. In criterion-referenced

matching, a “true” score is determined based on the scores given by the scoring system

developers. Once the “true” score is identified, each of the other raters’ scores was compared to

the “true” score. More detail appears below.

In pairwise matching, each combination of rater scores for a skill is compared to each other. In

this case the rubric developers are treated no differently than the other raters. The frequency of

matches was then calculated for two cases:

Pairwise 1-point score consistency (in percent) where raters matched within ±1 point of

each other on the five-point scale for the rubric.

Pairwise exact score consistency (in percent) where raters’ scores matched exactly.

In criterion-referenced matching, a “true” score is identified based on the scores of the rubric

developers. In this process, if the two rubric developers did not give the same score, then the

“true” score was determined from the average of the two scores. For these averages, rounding of

any decimal results was intentionally biased to round towards the score provided by the rubric

developer with more experience rating presentations. This rounding is necessary because only

discrete values are allowed for rubric ratings and, thus, comparisons cannot be made to decimal

results. Once this “true” score was found, each of the other rater’s scores was compared to this

“true” score. The frequency of matches was then calculated for two cases:

Criterion referenced 1-point score consistency (in percent) where rater consistency is

compared to within ±1 point from the “true” score.

Criterion referenced exact score consistency (in percent) where rater consistency is

compared for exact matches with the “true” score.

It should be noted that not all raters provided ratings for all the skills on the rubric in all

instances. First, raters were not required to score each skill during the relatively short

presentations. Second, the skill of “sensitivity to time” could not be rated since the videos

showed one speaker of a group of speakers. Third, the “taking questions” skill was not rated in

videos containing no questions from the audience.

Results

Final (Round 2) Modifications Using the Delphi Method

The total number of individual comments collected was 117 (including general comments about

the scoring system that did not necessarily pertain to a particular skill). The individual coding

results for these comments are shown below. In general, final coding results were collapsed to

represent the smaller number of themes between the two coders. As a result of this coding, the

changes made in Delphi round 2 include

renaming “initial connection” to “first impression”

changing the definitions of about half of the skills to address user feedback about clarity

and ease of understanding of those skills

moving two skills, “sensitivity to time” and “taking questions,” to the end of the survey.

A summary of the results for each skill shown in Table 1. Additionally, Appendix A displays the

original and modified wording of the definitions for each skill.

Table 1: Coding of feedback collected from stakeholders and resulting changes to scoring system

Count of themes by skill: Round 2 Changes to Scoring System

Skill # of Themes

Coder 1

# of Themes

Coder 2

First Impression 5 9 Renamed

Relevant Details 3 5 None

Appropriate

Language

2 7 None

Logical Flow 5 7 Definition clarified

Key Points 3 3 Definition clarified

Layout and Design 4 7 Definition clarified

Focused Content 3 2 None

Graphics 3 6 Definition clarified

Elaboration 1 3 None

Vocal Quality 2 2 Definition clarified

Personal Presence 4 4 Definition clarified

Taking Questions 1 4 None

Sensitivity to Time 2 5 None

General 10 11

Total 48 75

Final Inter-Rater Reliability Testing

In general, the skills comprising the rubric were found to be of moderate to high reliability when

used by different raters across our three settings. Our analysis of ratings for each of the 11 skills

(not counting “sensitivity to time” and “taking questions”) shows, in general, that the reliability

of the overall rubric is acceptable. However, there is some variation in the reliability of each

skill. In particular, for the overall results we see

a. High reliability for 10 of the skills,

b. Moderate reliability for an additional 1 skill.

Specific results for each skill and in each setting are displayed below.

Industrial Engineering Session inter-rater reliability

As shown in Table 2, in this session, for pairwise comparisons, every skill demonstrates very

reasonable inter-rater reliability (at 80% and above within 1 point) for this rater group, with the

exception of “sensitivity to time.” Over half of the skills show quite strong inter-rater reliability,

at over 90% matching within 1 point on our 5-point scale. The anomaly for “sensitivity to time”

makes sense in this setting where the videotaped presentations were one speaker’s part of a

group presentation. As a result, that skill was very rarely rated. For the criterion-referenced

comparison, every skill except “elaboration” demonstrated very reasonable inter-rater reliability.

Table 2 – Industrial Engineering session inter-rater reliability

Industrial

Engineering session Pairwise Criterion Referenced

N EXACT WITHIN_1 N EXACT WITHIN_1

First Impression 490 43% 92% 127 45% 89%

Relevant Details 518 42% 91% 131 40% 89%

Appropriate

Language 532 50% 98% 133 41% 98%

Logical Flow 525 37% 90% 132 37% 86%

Key Points 518 37% 89% 131 33% 84%

Layout and Design 532 33% 80% 133 32% 82%

Focused Content 532 41% 86% 133 44% 89%

Graphics 518 35% 84% 131 36% 86%

Elaboration 525 32% 84% 132 20% 78%

Vocal Quality 525 43% 92% 132 39% 91%

Personal Presence 525 44% 92% 132 45% 92%

Taking Questions 137 44% 93% 37 38% 95%

Sensitivity to Time 17 29% 76%

Biomedical Engineering Session inter-rater reliability

The BME session results, shown in Table 3, demonstrate slightly lower inter-rater reliabilities

than the industrial engineering session. This is true for both the pairwise comparisons and the

criterion-referenced matching. For pairwise comparisons, eight of the 11 skills demonstrate inter-

rater reliability above 80% within 1 point. For the criterion-referenced comparison, the “key

points” skill stands out as having rather low reliability, despite being reasonably reliable for the

pairwise comparisons.

Table 3 – BME session inter-rater reliability

BME session Pairwise Criterion Referenced




Appropriate

Language 162 35% 86% 63 35% 79%

Logical Flow 160 39% 92% 64 31% 94%

Key Points 162 31% 80% 63 14% 59%


Focused Content 162 39% 90% 63 37% 90%

Graphics 119 32% 76% 50 30% 80%

Elaboration 162 40% 77% 63 32% 75%

Vocal Quality 166 43% 84% 64 36% 69%

Personal Presence 166 38% 87% 67 42% 85%

Nuclear Engineering Session inter-rate reliability

Table 4 shows the very reasonable inter-rater reliability for the NRE session. For this session,

we are using a cleaned data set that is modified from the original ratings. Several students

recorded all ‘5’ ratings for every skill. With one exception, these "all 5" ratings were not

realistic representations for presentations observed, so all such results were removed from the

data set. Note this is a conservative adjustment, and does not improve our ratings. All pairwise

exact scores are at least 40%, and all but 2 skills have over a 90% matching within 1. Likewise

the criterion referenced comparison demonstrates very high inter-rater reliability, with all exact

scores at least 40%, and all but 1 skill over 90% matching within 1.

Table 4 – NRE session inter-rater reliability

NRE session Pairwise Criterion Referenced




Appropriate

Language 3220 48% 97%

247 48% 95%

Logical Flow 3292 47% 94% 250 51% 96%

Key Points 3223 48% 95% 247 51% 94%


Focused Content 3241 43% 92% 248 50% 92%

Graphics 3268 46% 90% 249 43% 84%

Elaboration 3292 40% 89% 250 45% 92%

Vocal Quality 3272 45% 92% 249 45% 95% Personal Presence 3292 40% 88% 250 46% 94%

Overall inter-rater reliability

Using the results presented above, an overall reliability for each skill was determined by

combining the results from two settings. The data above includes three (3) settings for two (2)

types of criteria (reference criterion and pairwise matching) and for two (2) levels of agreement

(exact and within 1 point). This gives twelve individual reliability measures (3 x 2 x 2) for each

skill. Among these individual reliability measures, an overall inter-rater reliability index was

assigned for each of the eight individual reliability measures for each skill. This index is used to

judge whether a skill is highly reliable, moderately reliable, or marginally reliable. The indices

assigned for each individual reliability measure are shown in Table 5.

Table 5 – inter-rater reliability index assignments for each reliability data point

Index Criteria

-1 Marginally reliable (below 30% exact or below 70% within 1)

0 Moderately reliable (30-50% exact or 70-90% within 1)

1 Highly reliable (above 50% exact or above 90% within 1)

Once indices were assigned, the number of times a skill was measured “high”, “moderate”, or

“marginal” was added up. This resulted in a scale of +12 to -12, where +12 indicates the skill

always displayed a high level of reliability, and -12 indicates the skill always displayed a low

level of reliability. The results of this indexing process are shown in Tables 6 and 7. Table 6 lists

all the skills that were modified based on feedback from the previous study, and Table 7 lists

those skills unchanged from the previous study. The reliability index for the previous study may

be fractional since the reliability index for some changed/redefined skills are the average of two

skills (see figure 1).

As shown in the tables, moderate to high reliability was achieved for all of the skills (0 or

higher). In Table 6, we observe that reliability of every skill that was modified for this study was

better in the current study than in the previous. An unpaired two-sample t-test concludes there is

significant improvement in the overall current reliability compared to the previous reliability for

those skills that were changed (p<.001). In Table 7, we see no significant difference between the

reliability of those skills that were unchanged between studies (p=0.76).

Table 6 – Overall inter-rater reliability for changed / redefined skills

SKILL

Current

Overall

Reliability

Index

Previous

Study

Reliability

Index

Logical Flow 5 -1

First Impression 4 -1

Personal Presence 3 -1.5

Vocal Quality 3 1

Graphics 1 -1.5

Layout and Design 1 -1.5

Key Points 1 -4.5

Table 7 – Overall inter-rater reliability for unchanged skills

Now that the reliability among raters for these skills has been established, implications and

suggestions for instruction need to be addressed. Table 8 includes some tips, for teaching each

skill, that have worked well in coaching industrial engineering capstone students to improve their

presentation skills2.

Table 8: Suggestions for instruction

Customizing to the Audience

1. First impression (Speaker grabs audience attention while defining purpose)

Ask the presenter to start their presentation with lots of energy and inflection to engage the

audience on a personal level. Say “I can be definite!” as an example, and then ask the speaker to

try it. Ask the team members for their feedback. If their feedback is “so-so”, ask the speaker to

try it once more. Give positive feedback when the speaker hits a confident note and then

emphasize keeping that up.

2. Relevant details (Uses concrete examples and details familiar to the audience)

Emphasize using real-life examples to make the talk engaging and to explain ideas and

processes, etc., to the audience. Discuss what could be a concrete example in one or two cases,

and then ask the students to come up with their own.

3. Appropriate language (Describes concepts at just the right level for particular

audience)

Discuss the audience’s technical background and whether acronyms, for example, can be used in

the talk. If the audience’s technical background matches the presenters’ then acronyms will be

recognized. However, if part of the audience has a technical background and part doesn’t,

emphasize replacing acronyms with the complete phrase or using the acronym on the slide but

describe it in words fully each time it comes up.

SKILL

Current

Overall

Reliability

Index

Previous

Study

Reliability

Index

Appropriate

Language 5 6

Focused Content 4 1

Relevant Details 1 3

Elaboration 0 2

Telling the Story

4. Logical flow (Links and transitions smoothly among different parts of the

presentation)

Ask the group to use storyboarding to create and then check their logical flow. To do this, use a

guide sheet with boxes on the page, and enter the title of each slide in each box. If several slides

have the same title, ask the students to help the audience know what’s coming next by adding a

descriptive subtitle to the title for each slide. Second, ask the students to check the logical flow

by making sure there are no surprises in the order of the information, and that enough

information has been provided ahead of each slide so the audience can understand the slide.

5. Key points (Central message clear throughout by linking details to big picture)

Remind students that if an audience is unfamiliar with their presentation, the audience may be

presented a number of details and, in the midst of the details, forget exactly what their purpose is.

Therefore, when the students are discussing several details, the presenter will need to regularly

remind the audience of how the details connect to the overall purpose or the big picture. Ask

students to practice including this reminder when they present details.

Displaying Key Information

6. Layout and design (Easy-to-follow organization, easily digestible amount of text,

compatible color)

Ask the students several questions about their slide layout: is the information on the slide

balanced and is it easy for the audience to follow? If not, how can it be improved? Is the

information set out in the same order that people use to read-- from left to right and top to

bottom? Have they avoided having an overwhelming amount of text on the slide? Have they

followed the general guideline of eight words per bullet and eight bullets per page? Do the colors

used clash with each other when shown on the exact projector and computer to be used during

the presentation?

7. Focused content (For each slide, information supports only one or two key points)

Discuss with the students the number of main points appearing on each slide (for example, the

executive summary includes one main point: a preview of the project). Then encourage

interaction about whether each slide includes enough information to support the one or two key

points, keeping in mind the audience’s background.

8. Graphics (Maps/charts/graphs/pictures easy to understand and clearly illustrate

points)

In the case of graphs, ask students whether they consider that the graphs on their slides include

easy-to-understand main points, titles, labels for each axis, and units. If they include multiple

graphs on one slide, ask why and whether the graphs can be split up on multiple slides. If there is

good reason (such as a comparison) for showing all the graphs on one slide, discuss animation

that would help the audience digest the information easily—for example, bringing in one graph

at a time.

Delivering the Presentation

9. Elaboration (Avoids reads slides and instead expands upon slide content)

Ask students to check each other’s eye contact with the audience while they describe the content

of the slides. Eye contact will keep the audience engaged and enable the presenter to determine

what the audience’s reaction is. Emphasize using the slides as cues to more discussion instead of

a document to read. Include a discussion of using notes and how, in many cases, the audience

will see this as a case of the presenter not bothering to adequately prepare.

10. Vocal quality (Uses volume, pace, and inflection to emphasize key points)

If the presenter is soft-spoken, ask them to try saying, “I can be definite!” as in skill 1. Remind

them that people in the back of the room will need to hear them. The presenter my say they feel

like they are shouting. If so, assure them that this is natural when they are not used to projecting.

After the group listens to a presenter give their portion of the presentation, ask them whether the

presenter emphasized important points by avoiding a monotone and instead using inflection to

keep audience interest.

11. Personal presence (Effectively combines energy, eye contact, and movement)

Ask the group to check the presenter’s energy and enthusiasm while presenting. Then discuss

various ways to show enthusiasm—lots of eye contact, more definite body movement (such as

opening your hands away from you) during main points, using engaging expressions and varying

facial expression. Advise the groups not to practice the presentation too much or they’ll get tired

of it or memorize it. They should change their wording each time they practice and then act as if

this is the most interesting project they can imagine.

Avoiding memorization is important for several reasons: first, the presenter may tend to sound

robotic. Second, if the presenter is interrupted, with a question, for example, they may have

trouble returning to the exact place in their script. As a result they may review again what they

have already stated—possibly to the point of restating several full sentences. Memorization can

also cause the speaker to start a perfectly fine phrase and then interrupt or “correct” it with a

different phrase—because they remember what is on their script and feel they need to stick to it.

Two skills that can be applied to either (1) an individual speaker or (2) the team as a whole

12. Taking questions (Adeptly answers questions to satisfy the audience)

Remind students that they need to be respectful of their questioner by letting them finish their

question. If the presenter doesn’t understand it, they should ask for clarification. The next steps

involve answering the question concisely and truthfully (for example, if you don’t know, don’t

talk until you think you’ve said enough so no one will notice you didn’t answer the question.

Instead, admit you don’t know, thank the questioner for making the point, and state that you will

look into the answer and get back to the questioner.) Once you’ve answered the question, look

for a nod from the questioner, or check with them by asking, “Is that clear?” or “Did I answer

your question?” or something similar.

13. Sensitivity to time (Stays within allotted time even with questions throughout

presentation)

Ask the presenters and their team to review their slides and brainstorm possible questions before

they present. Then discuss the total time and the estimated time they might allow for questions

(for example, depending on the situation, perhaps 5 minutes out of 20). During practice ask the

non-speaking group members to help you interrupt the speakers with questions so they can better

estimate the time required for the presentation including the questions. Prepare them for the

possibility of needing to skip a slide or two if, despite all their preparation, a question takes

longer to answer than expected. (If the question starts to delay the talk, the presenter, depending

on the situation, may be able to ask the audience member for a separate discussion after the talk

ends.) Ask speakers if they know whether, as an individual, they tend to speak more quickly or

more slowly in front of an audience (as compared to practice). Then, time each speaker to be

sure they stay within their expected time limit.

Conclusions

We have now concluded final revisions to the Norback-Utschig Scoring System for oral

presentations in engineering settings. The system has been reduced to 11 skills in four categories

plus two additional skills to be rated at the end of a presentation. Each of these skills has been

tested for inter-rater reliability, with good to highly reliable results. Finally, we have shared

instructional tips for helping students perform well on each skill.

Current work is now targeting ease-of-use of our instructional materials, which will be shared at

the presentation. Central to this work is, first, revision of the Teacher’s Guide for the Norback-

Utschig Presentation Scoring System and second, expanded descriptions for each skill regarding

what an excellent performance looks like. Future work includes the development of a proposal

for dissemination to a national audience and a study on the cognitive models used by students to

create graphs and tables.

References

1. Norback, J., Utschig, T., & Bryan, J. “Workforce Communication Instruction: Preliminary Inter-Rater

Reliability Data for an Executive-Based Oral Communication Rubric.” ASEE, 2012.

2. Norback, J., Utschig, T., & Bryan, J. “Insights into the Process of Building a Presentation Scoring System

for Engineers.” ASEE, 2013.

3. Payne, D., & Blakely, B. (Eds.). (2004-2008). Multimodal Communication: Rethinking the Curriculum.

Iowa City, IA: ISUComm at Iowa State University.

4. Payne, D., & Blakely, B. (Eds.). (2007). ISUComm Foundation Courses: Student Guide for English 150

and 250. Iowa City, IA: ISUComm at Iowa State University.

5. Davis, D. “Establishing Inter-rater Agreement for TIDEE’s Teamwork and Professional Development

Assessments”. ASEE Conference Paper, 2011.

6. Thambyah, A. “On the design of learning outcomes for the undergraduate engineer’s final year project.”

European Journal of Engineering Education 36.1 (2011): 35-46.

7. Mott, M.S., Chessin, D., Sumrall, W., Rutherford, A., & Moore, V. “Assessing Student Scientific

Expression Using Media: The Media-Enhanced Science Presentation Rubric (MESPR).” Journal of STEM

Education: Innovations and Research 12.1 (2011): 33-41.

8. LaBanca, Frank. “The 21st-Century Oral Presentation Tool Bag.” Science Teacher 78.7 (2011): 51-55.

9. Barney, Sebastian, et al. “Improving Students with Rubric-Based Self-Assessment and Oral Feedback.”

IEEE Transactions on Education 55.3 (2012): 19-325.

10. Anderson, Randy J., and Lydia E. Anderson. "Professorial Presentations: The Link between the Lecture

and Student Success in the Workplace." Academy of Educational Leadership Journal 14.1 (2010): 55-62.

Business Source Complete.

11. Battacharyya, Ena, Arun Patil, & Rajeswary Appecutty Sargunan. “Methodology in Seeking Stakeholder

Perceptions of Effective Technical Oral Presentations: An Exploratory Pilot Study.” Qualitative Report

15.6 (2010): 1549-1568.

12. Morton, J. & Rosse, M. “Persuasive presentations in engineering spoken discourse.” Australasian Journal

of Engineering Education, 17 (2011): 55-65.

13. Covill, Amy E. “College Students' Use Of A Writing Rubric: Effect On Quality Of Writing, Self-Efficacy,

And Writing Practices.” Journal Of Writing Assessment 5.1 (2012): 1-19. MLA International Bibliography.

Web. 20 Dec. 2013.

14. Andrade, H. G. and Du, Y. “Student perspectives on rubric-referenced assessment.” Practical Assessment,

Research, and Evaluation 10 (2005): 1-11.

15. Rawson, R. E., Quinlan, K. M., Cooper, B. J., Fewtrell, C., & Matlow, J. R. “Writing-skills development in

the health professions.” Teaching and Learning in Medicine, 17 (2005): 233-239.

16. Morozov, A. “Student attitudes toward the assessment criteria in writing-intensive college courses.”

Assessment Writing, 16 (2011): 6-31.

17. Utschig, T., Newton, S., Bryan, J. “Measuring Skills across the Profile of a Quality Learner and of a

Quality Engineer.” International Journal of Process Education 4.1(2012): 3 – 12.

18. Hartberg, Yasha, et al. (2008). Development of Student Writing in Biochemistry Using Calibrated Peer

Review. Journal of the Scholarship of Teaching and Learning. 2, 29-44.

19. Gunersel, Adalet Baris, and Nancy J. Simpson. Instructors' uses, experiences, thoughts and suggestions

regarding Calibrated Peer Review." Assessment & Evaluation in Higher Education 35.7 (2010): 771-781.

20. Berry, Frederick C. and Carlson, Patricia A. “Assessing Engineering Design Experiences using Calibrated

Peer Review.” International Journal of Engineering Education 26.6 (2010): 1503-1507.

Appendix A – Skill Definitions

The definitions for each skill before and after each round of edits to the scoring system are

displayed here. This paper primarily describes the second round of edits applied following the

collection of stakeholder feedback in round 2 of our modified Delphi method process. Prior edits

from round 1 of our modified Delphi process are described elsewhere2.

Skill Original Definition(s) Definition after

Delphi round 1

edits

Definition after

Delphi round 2

edits

*First

Impression

Audience Connection – Refers

directly to audience needs to

help define purpose/goals of

presentation

First/Last impression - Grabs

audience attention at beginning

and inspires them with the

closing

Grabs audience

attention while

defining

purpose/goals of

presentation

Speaker grabs

audience attention

while defining

purpose

Relevant

Details

Uses concrete examples and

details familiar to audience No change No change

Appropriate

Language

Describes concepts at just the

right level for particular

audience

No change No change

*Logical

Flow

Formerly called sequencing -

Links different parts of the

presentation and uses

appropriate transitions

Links different parts

of the presentation

and uses appropriate

transitions

Links and transitions

smoothly among

different parts of the

presentation

*Key Points

Context - Clearly illustrates

major points by linking to

additional relevant information

Key Points - Consistently refers

to how key points fit into the big

picture

Material presented is

clearly linked to

major themes/big

picture

Central message

clear throughout by

linking details to big

picture

*Layout and

Design

Amount of text - Uses an

appropriate amount of text to

describe essence of key point(s)

Layout and design - Information

is easily understood due to

layout and color is used

appropriately

Information is easily

understood due to

organization, color,

and amount of text

Easy to follow

organization, easily

digestible amount of

text, compatible

color

Focused

Content

For each slide, information

supports only one or two key

points

No change No change

*Graphics

Appropriate graphics -

Maps/charts/graphs/pictures/illus

trations used clearly support key

points

Engaging graphics - Graphics

are visually appealing, easy to

understand, include helpful

labeling

Maps/charts/graphs/

pictures easy to

understand

and clearly illustrate

points

No change

Elaboration

Avoids reading slides and

instead expands upon slide

content

No change No change

*Vocal

Quality

Flow - Knows material well

without memorization or

repeated hesitations, ums, etc.

Vocal Quality - Adapts tone,

volume, and pace to emphasize

key points

Uses tone, volume,

pace, & inflection to

emphasize key

points

Uses volume, pace,

& inflection to

emphasize key

points

*Personal

Presence

Stature - Uses good posture and

bearing

Personal presence - Effectively

combines energy, inflection, eye

contact, and movement

Effectively

combines energy,

eye contact, and

movement

No change

†Sensitivity

to Time

Begins/ends on time even with

questions throughout

presentation

No change

Stays within allotted

time even with

questions throughout

presentation

†Taking

Questions

Adeptly accepts and

satisfactorily answers audience

questions

No change

Adeptly answers

questions to satisfy

audience

* This skill was counted as a changed skill when comparing results between original and final

reliability indices.

† This skill was not tested in final reliability calculations due to video format of presentations

making these skills difficult or impossible to evaluate.

Paper ID #9549

Work-in-Progress: Undergraduate Teaching and Research Experiences inEngineering (Utree): An Engineering Student Organization with a Commu-nication Focus

Victoria VadyakMr. Michael Alley, Pennsylvania State University, University Park

Michael Alley is an associate professor of engineering communication at Pennsylvania State University.He is the author of The Craft of Scientific Presentations (2nd edition) and the faculty advisor of Utree.

Dr. Joanna K. Garner, Old Dominion UniversityMs. Christine Haas, Christine Haas Consulting


Work-in-Progress: Undergraduate Teaching and Research Experiences in Engineering (UTREE): An Engineering Student Organization

with a Communication Focus

Introduction In engineering, many undergraduate organizations foster the professional skills of the students. Such organizations seek to help students prepare for the next stage of their careers—be that stage as a graduate student or as a professional engineer in industry or government. Most organizations, such as the American Society of Mechanical Engineers, target students through the discipline that those students have chosen. Other organizations, such as Society of Women Engineers, target members because of gender, race, or ethnicity. This work-in-progress paper introduces an organization that recruits engineering undergraduates based in large part on how well they communicate engineering.

Such an organization has inherent value for the discipline of engineering because the set of skills needed to excel in writing a technical report or making a technical presentation are skills important for succeeding as a graduate student in engineering or as a professional engineer. For instance, creating an excellent technical report or presentation requires the ability to perform library research, to organize information in a logical manner, and to target an audience.

The organization UTREE (Undergraduate Teaching and Research Experiences in Engineering) assembles students with such skills, as well as high academic achievement in technical classes, and seeks to further develop those skills. That development mainly occurs through preparing those undergraduates to teach a small set of class periods and then having those students teach those class periods multiple times.

Two potential benefits exist for a college of engineering to have such an organization. First, the peer teaching provided by the organization has the potential to enhance the teaching in a college of engineering for a number of a reasons including decreasing instructor-to-student ratios [1–2]. Second, because the organization develops the professional skills of students who are excelling in their technical classes, the organization creates a pool of students who are primed to excel as graduate students or professional engineers.

As far as costs to our College of Engineering for the operation of this organization, they are relatively small, if the faculty member’s volunteered time to serve as advisor is excluded. The basic operating budget for the group is about $4000 each year to support the scheduling of the class periods (by one of the UTREE members), the upkeep of the web-site [3], food for meetings, and the professional activities for the members. From its teaching outside the University, the group earns about half that sum. In addition, the group has been successful at obtaining small grants to produce teaching films that are used in the class periods and to secure teaching assistantships for some members in upper-level courses.

Given how much teaching and service this organization performs each semester and how many active members the organization has, UTREE has shown itself to be a successful organization for the College of Engineering at Pennsylvania State University. The question arises whether such an organization would work at other engineering colleges. To determine that, a formal assessment would be needed that would answer the following questions:

(1) Does UTREE foster the professional growth of its members in a statistically significant way? (2) Is the teaching by UTREE members effective?

This work-in-progress paper outlines our plans to answer these two questions. First, to provide a sense of possible teaching, research, and service that such an organization could provide to a college of engineering, this paper provides an overview of those activities by UTREE at Pennsylvania State University. Second, to determine whether a formal analysis would even be warranted, this paper analyzes the results of two surveys. The first is a self-evaluation by the UTREE members of their own professional development, and the second is a survey by faculty about the effectiveness of the teaching by UTREE members. Third, this paper discusses what would be needed to make an organization such as UTREE a sustainable organization for the long-term at an institution. Interwoven in this discussion are the criteria that a college of engineering should meet before adopting such an organization. Overview of UTREE’s Activities Now in its third year, UTREE is an engineering student organization at Penn State that has twenty members representing eight different engineering disciplines. These students are chosen based on their overall academic achievement (the average GPA of the group is about 3.7 out of 4.0) and their particular achievement in a required speech course. UTREE’s mission is to leverage the technical and communication skills of the members to raise the professional skills of all students in the College of Engineering. UTREE accomplishes this mission through teaching, research, and service.

Teaching. With respect to teaching, UTREE mentors teach or assist the teaching of class periods about communication and teamwork in several engineering courses. In 2013, UTREE taught more than 60 class periods on communication and teamwork. Most of the students that UTREE teaching mentors instruct are first-year design students, but UTREE also teaches upper level classes and assists in the teaching of graduate student seminars. Table 1 shows a breakdown of the types of class periods taught.

One of the class periods that UTREE mentors teach concerns rethinking the topic-subtopic approach that most engineers and scientists follow for structuring their engineering presentations. In this class period, the mentors first discuss the weaknesses of the topic-subtopic structure, which is reflected in the presentation’s slides: a topic-phrase headline supported by a bulleted list of subtopics. Heavily influenced by PowerPoint’s defaults, this structure leads to presentations that are not well focused and do not communicate technical information in an effective manner [4]. Second, the mentors teach students an assertion-evidence approach to creating presentations. In such an approach, the presenter builds the talk on assertions, rather than topics, and supports these

assertions with visual evidence rather than with bulleted lists. In their teaching, Utree teaching mentors show the students one of the following slide structures that follow this assertion-evidence approach: the assertion-evidence structure [11] or pecha kucha [5].

Table 1. UTREE Class Periods Taught in 2013.

Class Period Offered Number of Class Periods Taught

Working in Teams 7

Group Presentation 15

Assertion-Evidence Structure of Presentation Slides

31

Pecha Kucha Design of Presentation Slides

11

In the assertion-evidence slide structure, students place their assertion (the main message of the scene) as the sentence headline at the top of the slide. Rather than using a bullet list to support this headline, students come up with visual evidence such as graphs, photos, or equations [10]. With pecha kucha, which is Japanese for “chit-chat,” students write down the main assertions for the talk and then develop a sequence of scenes to communicate those messages. This style typically calls for a sequence of 20 scenes (or slides) with each slide showing for 20 seconds. Because each slide appears for only 20 seconds, the scenes are primarily graphics with few words (if any) used. Although this style of presentation is not ideal for a number of engineering presentations, the resulting presentations are often a significant improvement over presentations that follow PowerPoint defaults [11].

Both class periods emphasize a rethinking of the common practice of visual evidence found in engineering presentations. One underlying assumption for assertion-evidence presentations is that the slides are meant to aid the audience in understanding content, rather than to assist the speaker in recalling information. Because the UTREE

mentors teach strategies that go against the common practice, this class period is a challenge [6]. In other words, because most students are accustomed to the ubiquitous topic-subtopic default of PowerPoint, they are naturally resistant to an alternative, no matter whether that alternative is introduced by a faculty member or a peer. For that reason, much work has gone into the design of this class period.

UTREE also offers a class period that seeks to improve the teamwork of engineering students. This “working in teams” class period introduces instructional videos created by UTREE mentors and then includes discussion about those videos. These online videos, such as one in which a slacker resides on the team [7], point out common

problems that arise in engineering teams as well as strategies for overcoming those problems. Sometimes in combination with this class period, UTREE mentors give an example of a group presentation to show best practices for teams taking on the task of a group talk.

UTREE does not limit its teaching to our own university. For instance, each semester, three UTREE teaching mentors help teach a video-conferenced class period about slide design to engineering students at five different Korean universities. Shown in Figure 1 is a photo that captures the projected teaching slide (shown on the left screen) and a UTREE mentor teaching five classrooms at different Korean universities (shown on the right screen). After this class period, which occurs in the evening for Penn State and in the morning of the next day for the Korean universities, each student team from these five Korean universities submits a set of slides that the UTREE teaching mentors critique.

Figure 1. Scene from video teleconference class taught by UTREE students at Penn State to engineering students at five Korean universities. Shown on the left screen is the teaching slide being discussed. Shown on the right screen is the Utree teaching mentor (upper left frame) and views of the five Korean classrooms: (two small frames at bottom and three small frames at the right).

In addition, in May 2013, six UTREE students and the group’s faculty advisor traveled to Northeastern University in Boston to teach a workshop to 40 graduate students in engineering. The UTREE students helped give a lecture about the assertion-evidence approach and then held individual consultations with the graduate students to help them with the slides for their next research presentation. On that same trip, the mentors traveled to the University of Massachusetts Medical School in Worcester to help teach two workshops on the assertion-evidence approach to members of administration, research faculty, post-docs, and graduate students at the university. Attending each of these two workshops were about 35 participants. As with the graduate students at Northeastern, the UTREE students held individual consultations with the workshop participants to critique a set of slides for their next presentation. Figure 2 shows a scene from one of those workshops. In the Spring 2014 semester, UTREE will help teach similar workshops to two medical schools that are within driving distance of Penn State.

Figure 2. UTREE mentor teaching slide design at University of Massachusetts Medical School in Worcester. About 35 medical faculty, researchers, and students were in the room. This illustration conveys a sense of the professionalism carried by the UTREE teaching mentors.

Research. With respect to research, UTREE helps undergraduates, including its own members, obtain undergraduate research experiences not only at Penn State but also at other institutions. One way that UTREE helps students is through helping teach a first-year seminar that has a focus on undergraduate research. In this course, UTREE students who have undergraduate research positions give tours of their labs and make presentations about their research. These students also share strategies for how they obtained their research positions. In its first three years, UTREE has been instrumental in helping undergraduates at Penn State obtain research experiences both at Penn State and at other institutions including Northwestern University, West Virginia University, and Texas A&M.

Service. In regard to service, the main responsibility of UTREE students is to run the Leonhard Center Speaking Contest each semester. The Leonhard Center Speaking Contest, which is shown in Figure 3, is a contest spanning two semesters that seeks to raise the level of presentations given by engineering students through the performance, videotaping, and posting of model presentations given by engineering students [8]. One UTREE student is responsible for organizing the semi-final rounds, which occur at the end of one semester, and another UTREE student serves as coordinator for the final round, which occurs toward the beginning of the next semester. In addition, UTREE students serve as emcees and preliminary round judges. Additionally, UTREE students act as speaker angels for the final round of the contest. Similar to the speaker angels that serve TED.com events, these UTREE students provide feedback for all finalists so that contestants have the most opportunity for success.

Figure 3. UTREE student introducing one of the speaker finalists at the inaugural Leonhard Center Speaking Contest. This illustration provides a sense of the number of students whom the UTREE teaching mentors serve. Shown are a portion of the 200 people who attended the contest.

As another example of service to the College, UTREE also served as a sponsor for a College of Engineering guest speaker Rick Gilbert, who is the author of Speaking Up [9]. In his talk, Gilbert gave practical tips for presenting to upper-level managers or executives, which is a topic not covered in classes. This talk was attended by more than 150 students in the College.

In addition to fostering the professional development of all engineering students in the College, UTREE members also participate in activities to foster their own professional development. For instance, each semester, UTREE students participate in team building exercises, such as rock climbing or traversing a high ropes course (shown in Figure 4). In addition, UTREE students have the opportunity to take a research writing course that serves as a substitute for the required technical writing course. In this research writing class, mentors learn how to write documents such as correspondence, proposals, posters, and research papers that occur often in engineering research.

Figure 4. UTREE students traversing a high ropes course. This illustration depicts the teamwork that is required for the professional development activities of the group.

Preliminary Assessment of the UTREE Program To determine whether a formal assessment of UTREE would be warranted, this section presents two preliminary surveys intended to evaluate the effectiveness of the UTREE program. The first survey called on UTREE members to self-assess their own professional development. The second survey called on faculty members who hosted UTREE teaching mentors in their classrooms to evaluate the effectiveness of the teaching of the UTREE teaching mentors.

Methods. To identify the professional development of the students, 26 UTREE members were surveyed and asked to self-evaluate the effect of teaching and service. As shown in the Appendix, students were asked a total of ten questions. Some questions asked students to discuss the most beneficial part of being in the group or to identify the ways in which UTREE is preparing students for the next stage of their careers. Other questions probed for specific details such as the number of UTREE activities that a student participated in or the number of class periods that a mentor has taught. Eighteen of those 26 members responded to this survey.

To identify the impact that UTREE classes have on the students who view the presentations, ten faculty members at Penn State were surveyed. Six of those ten responded. The questions of this survey addressed to the effectiveness of the UTREE class periods. As with the first survey, the actual questions of this survey appear in the Appendix.

The main audiences of the UTREE classes, including undergraduate students and graduate students, were not surveyed for this preliminary assessment because of the challenge of determining the effectiveness of the teaching by the mentors from surveys about a single class period of the mentors’ teaching. First-year students who have had little exposure to engineering presentations may view the teaching in a required class very differently from a graduate student who has chosen to attend a UTREE event for her or his own benefit.

Results and Discussion. This subsection presents the results of the two surveys to determine whether a formal assessment of UTREE is even warranted. To gain a sense of the professional development of the UTREE teaching mentors, we considered the results of the first survey: the self-assessment by UTREE members of their own professional development.

This first survey revealed that perhaps the most beneficial parts of being in UTREE was having opportunities to present, which increased the confidence of the members in their presentation skills. For instance, 15 out of 18 mentors discussed being able to further their presentation skills. One UTREE graduate said, “In networking, public, and work settings, I am less nervous to say something. It has become a strength that a lot of people notice right from the start.” This mentor felt that UTREE has prepared her for her career as a civil engineer by preparing her for mentoring and effectively communicating.

Since graduating, she has offered workshops and help sessions on tips for successful presentations to her colleagues.

As shown in Table 2, most responding members of UTREE participated in the group’s activities by teaching class periods on communication and teamwork. For instance, 16 of the 18 responding had taught class periods to first-year students, 12 had taught to class periods to sophomores, juniors, and seniors, and 8 had helped teach workshops to graduate students or professionals. Of those 8 students who had helped teach graduate students or professionals, 5 volunteered the comment that this type of teaching was the most valuable. In the survey, one mentor explained that one of the benefits of leading workshops for graduate students and professionals was that they “were the most critical of the content in the presentations, which was a situation most [students] do not get a chance to experience.” Certainly, such teaching situations were potential confidence builders for the UTREE members. In effect, because the UTREE members had presented successfully to graduate students and professionals, those members no longer perceived those types of presentations as “unattainable.”

Table 2. Types of teaching self-reported by UTREE mentors.

Type of teaching Number of respondents (percentage)

Taught presentations or teams to first-year students 16

(89%)

Taught presentations or teams to sophomores, juniors, or seniors

12

(67%)

Served as paid teaching assistant in a 3-credit course: Effective Speaking for Engineers

12

(67%)

Gave a research presentation in first-year seminar on undergraduate research

7

(39%)

Helped teach a presentations workshop for graduate students or professionals

8

(44%)

Given the positive responses of the UTREE teaching mentors to the self-assessment survey, we have decided that a formal assessment of the professional development of the UTREE teaching mentors is warranted.

To gain a sense of whether the peer teaching by the UTREE members was effective, we considered three inputs: (1) the results of the faculty survey, (2) the self-assessments of the UTREE members, and (3) whether faculty invited UTREE mentors to come back to teach in a later semester.

In the faculty survey, six of the ten faculty polled provided responses. All six of the responding faculty members considered the teaching by UTREE mentors to be successful. According to one faculty member, “The presentation was more effective because my students saw a peer perform at a very high level.” Three of the responding faculty had specific suggestions on how the mentors could improve their presentations or how the class period itself could be improved, but all of these suggestions fell into the category of continual improvement.

In addition, all six faculty members asserted that they believed their students benefitted from the class period. According to one faculty member who taught a senior design class and for whom a junior in UTREE student gave a sample assertion-evidence presentation, “My students commented on the quality of [the mentor’s] presentation. They unanimously agreed that he was very effective in conveying the importance of his topic. They also mentioned that they learned something from his presentation (a true mark of effectiveness).” Another faculty who taught first-year design students remarked, “[my students] were engaged in the material during that class, and without [me] reteaching the material they were able to create presentations at the end of the semester using the assertion-evidence structure.” Still a third faculty member, who taught upper-level students, remarked, “In their course reflection papers my students mentioned that it was very helpful to see the UTREE students give the group presentation. Many of them modeled the group dynamics and transitions demonstrated by UTREE students in their own presentations.”

In regard to their success as teachers, the UTREE mentors were harder on themselves than the faculty members were. The self-assessment survey revealed a split on the question whether the students had accepted their advice in the class periods. In other words, some mentors believed that their teaching was accepted, while an almost equal number questioned whether it was. One mentor provided a possible explanation, “Different settings have different types of audiences. Some freshman classes have a majority of students who do not want to be there so only a few students really get into the information we are presenting. Other settings, such as graduate seminars, involve people who are much more willing to take what we have and run with it.”

Perhaps the most compelling evidence for the success of the peer teaching provided by the UTREE mentors comes from the requests by faculty in subsequent semesters to have the mentors return to their classes. Almost all the faculty who have had UTREE mentors teach in their classes have requested return visits. This set of faculty includes the faculty members from Korea who had requested the videoconference class for five semesters in a row and the faculty at Northeastern University and the University of Massachusetts Medical School who have an open invitation for UTREE to return. In

regard to the last three sets of faculty, those requests have been accompanied by promises for stipends (about $1000 each) to cover travel expenses (in the case of the last two) and to help support the running of the organization.

Given the positive responses of the faculty survey and the continued requests for UTREE teaching mentors to return to classrooms, both on-campus and off-campus, we consider that a formal evaluation of the teaching effectiveness of the UTREE teaching mentors to be warranted. However, such an evaluation will require the input from the students themselves. Moreover, because first-year students who have little experience making engineering presentations or working in engineering teams will be involved, that input should be done from both a short-term and long-term perspective.

Sustainability of UTREE and Prospects for Dissemination to Other Colleges

Given that the previous section has recommended that the professional development and teaching effectiveness of UTREE teaching mentors be assessed in a formal way, this section discusses what is needed to make UTREE sustainable at an engineering college. Interwoven in this discussion are the criteria that should be met for a college of engineering to consider adopting UTREE.

As mentioned, from an operating cost perspective, UTREE at Penn State is run for less than $5000 per year. Given that our organization teaches about 30 class periods per semester, our organization provides the teaching equivalency (not including grading or office hours) of one instructor for a 3-credit course. So far, that argument has sufficed in the procurement of $2000 each year from our College to support the organization.

What this analysis does not include is the time expended by the faculty member who advises UTREE. In our case, the faculty member estimates that the time needed to advise this organization is about the same amount of time spent on a 3-credit course. Time requirements include attending and contributing to the weekly meetings, assisting in the training of new UTREE teaching mentors, attending and contributing to the service events such as the Speaking Contest, and organizing the professional development activities. While the student members contribute as much as they can to all of these activities, time and effort from the advisor is still required. At present, our College of Engineering does not formally compensate the advisor for this time. So what is in it for the advisor?

In our College of Engineering, the UTREE advisor is an engineering communication specialist, who teaches engineering communication classes and gives guest lectures on communication in engineering courses. Having UTREE mentors teach engineering communication class periods in first-year design and other courses offsets to an extent the time that the advisor spends on the organization. Given that, for a college of engineering to consider adopting UTREE, the college would want to consider having an advisor who would be in a similar position of providing a significant number of guest lectures on communication to engineering courses. For many large colleges, such personnel exist, as does the need for such guest lectures.

Another step to make UTREE sustainable would be to treat the organization as a startup company and apply the principles for startups that Stanford Professor Steve Blank advocates [12]. In short, Blank argues that startups originate because of an innovation, but that the creators of the innovation do not fully realize what the features of that innovation are until various customer segments begin using that innovation. Likewise, because UTREE is so new, the benefits to the students being taught, the faculty whose classes are served, and a college of engineering are not yet fully known. By putting UTREE through the startup process of identifying customer groups and testing value propositions for those groups, then this organization could gain a deeper understanding of its intrinsic value.

Conclusion This work-in-progress paper has introduced a new undergraduate organization in engineering that has a dual mission of providing professional development of its members and performing peer teaching for our College of Engineering. Indications are that the organization is meeting both of its missions. At Penn State, the College of Engineering is pleased enough with the organization to continue supporting it, both with the group’s small operating budget and with small grants for teaching films and teaching assistantships. This paper recommends that a formal assessment be performed to determine whether long-term sustaining and widespread dissemination should occur. At present, the organization seems better suited for large engineering colleges that have an engineering communication instructor who can serve as the faculty advisor. In such a situation, the class periods on communication taught by the organization can help offset the time spent by the faculty member to advise the group.

Acknowledgments We wish to thank the Leonhard Center for the Enhancement of Engineering Education, which in the College of Engineering at Penn State, for their support of UTREE. Appendix: Surveys to UTREE Members and to Faculty Survey I: Your Experience as a Member of UTREE

1. Why did you get involved with UTREE?

2. Identify the kinds of teaching that you have done for UTREE? Taught presentations or teams to first-year students Taught presentations or teams to sophomores, juniors, or seniors Assisted in the teaching of CAS 100A for Engineers Gave a research presentation in First-Year Seminar on Research Helped teach a presentations workshop for graduate students or professionals

Which of these were the most valuable and why?

3. Roughly, how many classes have you taught for UTREE? 0-4 5-9 10-15 15+

4. Typically, how do (or did) you prepare for teaching a class?

5. From your perspective, how do you think that the students you taught viewed you as an instructor? In other words, do you think that the students accepted the advice you gave them? Please back up your answer with observations/

6. Please list any other activities that you did for UTREE? Helped run the Leonhard Center Speaking Contest Participated in a Shaver’s Creek activity Traveled to another institution to teach a UTREE workshop off campus Helped create a teaching film

7. For the above question, which of these were the most valuable to you and why?

8. In what ways do you think the teaching of UTREE is preparing you (or has prepared you) for the next stage of your career?

9. For you, what are (or were) the most beneficial parts of being in UTREE?

10. How could the experience of UTREE, especially the teaching, be improved? Survey II: Request for Your Feedback on Class Visits by UTREE

1. Please describe how effective the UTREE students were at communicating the material of the class period(s) that they taught for you this semester? In particular, please describe their strengths as well as any ways in which their teaching could be improved.

2. How would your students describe the effectiveness of the UTREE students at communicating the material of the class periods? On what basis, do you make this evaluation?

References 1. Neal A. Whitman and Jonathan D. Fife (1988). Peer Teaching: To Teach Is to Learn

Twice. ASHE-ERIC Higher Education Report Number 4. Washington, DC: ERIC Clearinghouse on Higher Education.

2. K. J. Topping (1996). The effectiveness of peer teaching in further and higher education: A typology and review of the literature. Higher Education, vol. 32, pp. 321-345.

3. UTREE: Undergraduate Teaching and Research Experiences in Engineering (2013). http://writing.engr.psu.edu/utree. University Park, PA: Penn State.

4. Joanna Garner and Michael Alley (2013). How the Design of Presentation Slides Affects Audience Comprehension: A Case for the Assertion-Evidence Approach. International Journal of Engineering Education, vol. 29, no. 6, pp. 1564-1579.

5. Garr Reynolds (2008). Presentation Zen. Berkeley, CA: New Riders.

6. Traci Nathans-Kelly and Christine Nicometo (2014). Slide Rules: Design, Build, and Archive Presentations in the Engineering and Technical Fields. New York: Wiley-IEEE Press.

7. UTREE (2012). Slacker. http://www.youtube.com/watch?v=KfLDM6-iCiE. University Park, PA: Penn State.

8. Mimi Overbaugh, Michael Alley, Victoria Vadyak, and Christine Haas (2014). Effect of Model Student Presentations from a Speaking Contest on the Development of Engineering Students as Speakers. 2014 ASEE Annual Convention and Exposition. Indianapolis, IN.

9. Rick Gilbert (2013). Speaking Up: Surviving Executive Presentations. San Francisco: Berrett-Koehler Publishers.

10. “Tutorial for the Assertion-Evidence Approach: Sharper Focus, More Confidence,” http://www.writing.engr.psu.edu/assertion_evidence.html, ed. by Michael Alley (University Park, PA: Penn State).

11. Michael Alley (2013). The Craft of Scientific Presentations, 2nd ed. New York: Springer, p. 192.

12. Steve Blank and Bob Dorf, The Startup Owner’s Manual: The Step-by-Step Guide for Building a Great Company (Pescadero, CA: K&S Ranch Publishing Division, 2012).

Paper ID #9686

Long-distance collaboration, international perspective, and social responsi-bility through a shared interdisciplinary engineering design course

Dr. Jodi Prosise, St. Ambrose University

Jodi Prosise is an assistant professor at St. Ambrose University in the Department of Engineering andPhysical Science. She earned her PhD in Biomedical Engineering at University of Minnesota and herBachelor of Science in Mechanical Engineering at Iowa State University. She teaches courses in bothIndustrial and Mechanical Engineering at SAU, focusing in Engineering Graphics, Manufacturing, theEngineering Sciences, and Design. She is the PI of an NSF S-STEM grant and helps to direct the un-dergraduate research program at SAU. She leads a study-abroad trip for engineering students to Ilheus,Bahia, Brazil every-other-year.

Prof. Hank Yochum, Sweet Briar College

Hank Yochum is the Director of the Margaret Jones Wyllie ’45 Engineering Program and Professor ofPhysics and Engineering at Sweet Briar College. Sweet Briar is one of two women’s colleges in theUnited States with an ABET accredited engineering degree. He earned his BS in Physics from the Collegeof Charleston and PhD in Physics from Wake Forest University. Prior to joining Sweet Briar, he was aMember of Technical Staff at OFS Specialty Photonics in the Optical Amplifier Development Group.


Long-distance collaboration, international perspective, and social

responsibility through a shared interdisciplinary engineering

design course

Abstract

Today’s societal characteristics are compelling engineering graduates to have a broader range of

skills rather than the highly focused technical repertoire demanded of engineers in the past,

including teamwork and communication skills1, as well as an awareness of the effects of

technologies on cultures, societies, and economies2. In order to meet these needs, an

undergraduate engineering design course has been developed as a collaborative effort between

faculty members at two small liberal arts institutions separated by more than 800 miles. Each

institution offers an ABET accredited engineering degree (Engineering Science and Industrial

Engineering) and graduates ~7-12 engineers per year. In the shared course, engineering student

virtual teams design and implement assistive technologies for persons with disabilities and

underprivileged individuals for both local and global clients. The course is required for

engineering majors from both institutions and is usually taken in the sophomore or junior year as

a pre-capstone experience. Sharing expertise, capabilities, and faculty time are important

considerations in developing the course because of the very small size of each school’s

departments.

Overcoming the challenges of communicating long-distance with teammates as well as project

collaboration at a distance are important aspects of the course. Long-distance collaboration is

particularly important today as many engineers in industry now work at a distance with

colleagues. In this paper, we will describe our course and describe a variety of lessons learned

about this type of course and collaboration. A comparison of the course with and without the use

of virtual teams suggests that students rate their multidisciplinary teamwork skills as being much

more developed in the version of the course with virtual teams as compared to when team

members are all at one campus.

Introduction

Corporations and national leaders have expressed disappointment in engineering graduates’

abilities to fulfill the needs of the workplace, emphasizing their low levels of communication and

teamwork skills2, 3

, as well as a deficiency in social skills4. Characteristics of today’s society

include multidisciplinary technological advancements, globalized markets, and emerging social

responsibility2. These societal characteristics are compelling engineering graduates to have a

broader range of skills rather than the highly focused technical repertoire demanded of engineers

in the past1, as well as an awareness of the effects of technologies on cultures, societies, and

economies2. Our approach is to provide students an opportunity to improve in a variety of areas,

including multidisciplinary teamwork, communication and global perspective, by working on

virtual teams to design assistive technology for disabled clients in a pre-capstone experience.

We believe our approach is different when compared to other design programs for

underprivileged clients because of our use of virtual teams and our collaboration between two

small engineering programs at liberal arts institutions.

Engineers in industry routinely work on teams where members are at different geographic

locations that meet and collaborate electronically. These virtual team members may have never

met in person. In order for these virtual teams to be successful, engineers need both strong

communication skills and teamwork skills. Even with video communication, engineers working

at different locations completing complex tasks can experience problems with communication

and teamwork. These issues can be made more difficult because of the potential lack of rapport

within virtual teams5; 6

. While industry makes use of virtual teams, it is not yet common for

engineering programs to give students opportunity to work in this setting.

The Program for Assistive Technologies for Underprivileged (PATU) was developed as a

collaborative effort between faculty at two small liberal arts institutions separated by more than

800 miles. Each institution offers an ABET accredited engineering degree (Engineering Science

and Industrial Engineering) and each graduates ~7-12 engineers per year. Sharing expertise,

capabilities, and faculty time are important considerations in developing the program because of

the very small size of each school’s departments.

The mission of the program is to allow students to practice engineering skills while they develop

strong communication and teamwork skills, gain global perspective, and learn social

responsibility through projects for persons with disabilities that otherwise could not afford

assistance, both locally and globally. At each institution the program is incorporated into

required sophomore and junior-level design-intensive courses. The course is offered every-other

year at one institution, enrolling 20-25 students in the course, and every year at the other

institution with 10-15 students. It has been suggested that early incorporation of

multidisciplinary teamwork into the curriculum is a more effective strategy than waiting for

senior design7, supporting the effectiveness of our inclusion of PATU into sophomore and

junior-level courses. In addition, the integration of engineering and non-engineering students in

collaborative virtual teams has proven to be an effective learning strategy in multidisciplinary

teamwork8. These projects provide students practice in the engineering design process and with

communication techniques.

One of the key components of the program, and one reason the co-institutional collaboration is

critical, is that it allows for the development of multidisciplinary teamwork. Because each

institution is small and offers only one or two programs in engineering, working across different

engineering disciplines is difficult or impossible without this collaboration. Design course teams

include Industrial Engineering (IE) and Mechanical Engineering (ME) students from one

institution, and Engineering Science (ES) and liberal arts students from another. Masters-level

occupational therapy students and faculty also collaborate on functionality of resultant designs.

Collaboration between health sciences, liberal arts, and engineering students during the design

and implementation process through cross-discipline virtual teams enhances students’

communication skills and is important for the development of a quality end-product. While

completing projects for persons with disabilities or the underprivileged is not unique to our

program, it is a critical component to the success of our virtual teams because it appeals to the

students’ desire to complete the design successfully and motivates them to work through any

difficulties encountered with team dynamics.

Program planning and management

Prior to the start of the semester, faculty from each institution carefully organize and discuss the

course organization, delivery, and management (Figure 1). Two main faculty, one at each

institution, have been involved each year to date, with an additional faculty at each institution

playing a smaller supporting role. Responsibilities of each faculty are negotiated, including

delivery of each lecture, development of assessments, grading, expectations of students and

timeline of deliverables. Courses at each institution meet at the same time, allowing materials to

be delivered synchronously to both student groups through the online course delivery systems

Elluminate and Blackboard. Although the majority of the effort by faculty is in the planning

immediately prior to the course implementation, work for the program is completed year-round.

Figure 1: Annual program implementation timeline. Detailed timeline of tasks completed by

faculty at both institutions for one year (July to June) to implement PATU. Length of bar under

each task denotes approximate length of time spent on the task. Timeline assumes

implementation of course in spring semester only.

Project ideas are collected throughout the year from several organizations near each institution

and clinics in Itabuna, BA, Brazil. Faculty select projects based on feasibility, student interests

and capabilities, as well as level of meaning and experience the project will provide for the

student and client. While each project represents a critical need for an individual client, many

projects may require more than a single semester or design iteration.

After being introduced to the projects and the students from the opposite institution, each student

ranks the projects in order of his or her preferences and lists two students with whom they would

prefer to work. Allowing students to choose at least one team member reduces the stress that

occurs when working with virtual teams. Course instructors from each institution, after

extensive discussion, assign teams and projects based on preferences, abilities, and learning

styles, to create teams that will work synergistically7. Because each program is small, instructors

are highly familiar with each student’s abilities and personalities, allowing for the construction of

teams designed for optimal performance7. As in many engineering design courses, a section early

in the semester is devoted to teamwork and collaboration skills. We know this is particularly

important because of the team dynamics as projects become more challenging and because of our

use of virtual teams. Difficulties experienced with virtual teamwork are particularly stressed.

For example, it is emphasized that tone can be difficult to assess through email and placing

blame for difficulties on team members from the opposite institution should be avoided.

Project teams include 2-3 students from each school, bringing complimentary abilities to the

group. Key knowledge and skills needed for successful device design are an understanding of

how devices affect the human body, effective manufacturing practices for both the individual

device and for future sustainable production, as well as technical skills in mechanical and

electrical design and computer programming. IE students have expertise in human factors and

manufacturing techniques and capabilities, ME and ES students bring skills in mechanical and

electrical device design, while all students have experience with problem solving and computer

programming. Students meet (virtually or in person) with a collaborator from occupational

therapy at least once a month to ensure that their devices properly satisfy the therapeutic needs of

the patient for whom it is being designed. Liberal arts students round out the multidisciplinary

teams by bringing a unique perspective to the project and its effects on the persons and societies

involved. The vast needs of these projects require a truly multidisciplinary team that is enriched

by this co-institutional collaboration.

Each student group determines the best modes and schedules of communication for the group

and delegates responsibilities for the completion of deliverables. A different team member leads

the composition of each of the design reports. The exercise of making such decisions is an

important educational experience. Faculty members provide guidance and assistance as

necessary to ensure smooth operation of the collaborative groups while remaining flexible in the

methodologies utilized by each group. Students most often communicate by email, phone, text,

and Skype, and share documentation through GoogleDocs and other file sharing sites. Constant

communication between faculty leaders is critical, especially during “storming” stages of team

development.

Product design and development

The course in which this program is implemented is typically the first time students experience

the engineering design process in great depth and work for a real client. Students are guided

through the steps of the design process by course instructors through course materials,

discussions, and the expectation that groups achieve certain milestones in a timely manner. The

course is co-taught as one cohesive course across the universities, with typical course

responsibilities shared among the faculty from each institution so that all students receive the

same instruction and grading. Once teams and projects are assigned, groups work to produce

specific goals the product must accomplish. In order to properly define these goals, teams are

expected to conduct interviews with the participants and extensive research into their living or

work conditions and maintain regular communications with support personnel at participating

organizations. For clients in Brazil, students communicate with therapists in the clinics through

email and other web-based video and voice conferencing systems.

Once customer needs are fully defined, students brainstorm ideas and produce prototypes of their

design. They are expected to complete several iterations of device fabrication until they create a

product suitable for delivery to the customer. Each institution has different fabrication facilities

and skill, and the sharing of fabrication resources between institutions is a critical aspect to the

collaboration. Students are given time during class to manufacture the projects but are expected

to work outside of class as well to complete their device. Each group decides which institution’s

facilities best fulfill their manufacturing needs. Different components of each project may be

manufactured at each institution, requiring students to send components between schools for

final assembly. Each group is given a budget of $500 to manage for supplies and other costs to

fabricate their devices. Currently, support for the projects comes directly from the operating

budget of each department (split evenly). Few projects have used the entire allotted budget.

Students gain global perspective and social responsibility through the integration of reading and

writing assignments throughout the course that explore the effects of technology on society as

well as Brazilian culture. Every other year, an optional short-term study abroad trip is offered.

While in Brazil, students experience area clinics, examine typical living conditions, meet with

clients and their families, and tour various manufacturing facilities. Information gathered is then

available to the next group of students completing projects for Brazilian patients. Students and

faculty from both institutions participate in the study-abroad program. Without this co-

institutional collaboration the cost of this program would be prohibitive.

Projects

More than ten projects have been completed through PATU over the course of two different

semesters, including projects that have required several iterations, projects that were completed

and delivered in one semester, and projects that require additional work before they can be

delivered. Examples of projects include a compression vest for a boy with sensory-seeking

behavior (Figure 2A), a blink-controlled communication board and power supply for a girl with

cerebral palsy (Figure 2B), a detachable desktop with communication icons for a boy with

muscular dystrophy (Figure 2C), and a volume feedback device for a girl with hearing

impairment. Projects have been completed for more than five different collaborating

organizations in two states and two countries.

Figure 2: Project examples. Photographs of projects completed for persons with disabilities.

A) Compression vest for Max, who has sensory-seeking behavior. B) Communication board

for Emanuelle, who has cerebral palsy. C) Wheelchair desktop with communication icons for

Manuel, who has muscular dystrophy.

Assessment of student learning

Despite additional challenges associated with virtual teams, quality of final devices produced for

terms with and without virtual teams were similar based on qualitative observation from faculty.

Both institutions have ABET accredited programs and, therefore, implement assessments

annually of the student outcomes (a-k). After completion of the course, students were asked to

complete an anonymous survey indicating their perception of their level of ability for each

outcome (a-k) by circling the appropriate number using the following scale:

5 = High level of competence - extensive ability

4 = Moderately high level of competence - good ability

3 = Average level of competence – some ability

2 = Low level of competence – little ability

1 = No level of competence – no ability

The course is offered at one institution every-other year and at the other institution every year.

Therefore, students participate in virtual teams only in alternating years. While unfortunate, the

situation allows for comparison of assessments. Student responses from this institution were

analyzed and one term when long-distance collaborations were completed (2013) were compared

with one term (2012) where long-distance collaboration was not available but similar projects

were undertaken. There were 11 student responses in 2012 and 10 in 2013.

Several observations can be made from the assessment results that support the advantages of this

long-distance collaboration between engineering programs at small liberal arts institutions.

Design (student outcome c) and utilization of engineering techniques (student outcome e) are

outcomes covered thoroughly throughout typical engineering curricula through highly technical

courses. While these outcomes are claimed to be important in this program, they did not

improve or improved very little in student ratings during the long-distance collaboration term

(Figure 3 A, J). This is not unexpected, as these concepts are easily covered in the course with or

without the collaboration. It is interesting to note that the use of virtual teams does not generally

reduce students’ sense of their design skills. However, the students’ perception of their ability to

solve engineering problems (student outcome e) was lower during the year of collaboration

(Figure 3C), perhaps because of the challenge of dealing with long-distance collaboration.

There are improvements in other student outcomes that can be observed when comparing the

term with virtual teams to the term without. The most notable differences occur in the outcomes

that address the softer skills in engineering, which are the key focus of this collaborative

program. The goal, “an ability to function on multidisciplinary teams” was rated at a level 5 by

63% more students during the long-distance collaboration term than the non-collaboration term

(Figure 3B). The outcome assessing understanding of the impact of engineering on society was

also greatly improved, rated at a level 5 by 60% of students during the collaboration term

compared to 18% without collaboration (Figure 3F). These results are supportive of our efforts to

enhance graduates’ skills in communication, multidisciplinary teamwork, lifelong learning, and

awareness of social and ethical considerations in addition to a firm grasp of science,

mathematics, and engineering fundamentals.

Future directions may include comparison of student performance in Senior Capstone

experiences with or without participating in this model of long-distance collaboration.

Figure 3: Student outcome assessment results. Percentage of students rating their perception

of their ability for each outcome for one year without collaboration (2012) and one year with

collaboration (2013). Each graph is one ABET Student Outcome or component of one ABET

Student Outcome assessed for the course. Eleven students were assessed in 2012 and ten in

2013.

Lessons learned about collaboration and virtual teams

The work required of faculty to coordinate this program is intense and critical to the success of

PATU. Faculty recruit projects and mentors, coordinate delivery of course material, manage

teams and project progress, and coordinate the short-term study abroad program (Figure 1). The

key to successfully completing the required work is regular and open communication. Faculty at

each institution communicate daily through email, online chat, and telephone. In addition, faculty

meet face-to-face at least once each year, which is critical for morale and understanding between

faculty at each institution. While today’s technology allows for clear visual communication

through the internet, there is something to be said for a good hand shake and sitting next to a

person for a serious discussion about lessons learned and future work that needs to be completed.

A key lesson learned during implementation of this program is the critical need for appropriate

video conferencing technology; a dedicated distance-learning classroom would be ideal. During

the program, classes at each institution meet at the same time and a faculty member lead both

groups of students via video through Elluminate and Blackboard. Elluminate is a video

communication module within Blackboard and allows for video communication while also

showing PowerPoint slides (or other type of writing/sketches). This was done to manage faculty

time as well as to give students from each school a different teaching perspective and experience

with telecommunication. We found that the technology for video conferencing (single camera at

each site with single microphones) that we used made teaching difficult. For example, without

multiple microphones, cameras, and a technician, engaging all students from both institutions in

conversation about a topic was quite difficult and often frustrating for both faculty and students.

Alternatively, the technology used by students for team collaboration (typically Skype and

simple integrated webcams) works very well because of the comparatively smaller number of

individuals utilizing the technology at a given time.

One of the greatest advantages for students at small liberal arts institutions is the relationship

developed with faculty. Students have faculty as an instructor for several courses throughout

their undergraduate careers and interact with them at many different levels (e.g. advising,

tutoring, and club mentoring). A lack of this history with collaborating faculty has, in the past,

caused some stress in students and impeded progress on design teams. We would like to find a

way to make it possible for each faculty to spend a few days at the beginning of each semester at

the collaborating school to enhance the understanding the students have of each of the faculty

involved in the program and their intentions.

In addition to strong relationships with faculty, students in small programs such as ours also

develop distinct relationships within their cohort. However, many of the usual challenges of

teamwork can be amplified in virtual teams. It is critical that faculty appeal to the goal of the

project (producing a life-changing device for a customer with a disability) and use difficulties as

an opportunity to talk about team dynamics. It is also essential to have at least two students from

each school on each team so that it is difficult for a single student to be marginalized. We also

found that categorizing projects and making final deliverable expectations clear helped to reduce

the level of anxiety that can occur with unrealistic expectations while providing a challenging

project with tangible results.

Finally, in the past faculty allowed teams to decide which campus would build their prototypes

and final devices. This meant that the non-building campus did not get the opportunity to take

part in that phase of the project where many times new challenges are faced. While the choice of

building seemed reasonable, especially given limited resources, in the future, we plan to have

both campuses work to build prototypes if at all possible. Final devices may still only be

fabricated at one location.

The study abroad trip was first piloted in Summer 2011. Students and faculty both felt the trip

was a unique and life-changing experience. The reaction of patients to the devices that we

delivered was also unforgettable. Emotions were high as they expressed their appreciation for

our projects, explaining how they never dreamed such assistance was even possible for them.

Our patients’ love for life, determination to find a way to improve their conditions, and sincere

appreciation of our work, sparked in us (both faculty and students) admiration and desire to

continue our projects for as many underprivileged persons with disabilities as possible.

This unique pre-capstone experience has helped shape the way we teach engineering design,

allowing us to observe students as they complete the entire design process, utilizing the classical

tools often published in engineering design process textbooks. By implementing the process

early, we see the advantages and disadvantages to the structured design process and the various

tools available to make design choices, and can make changes to those tools and processes we

use in later courses. Overall, this program and collaboration has improved our students’

learning, especially of the “softer skills” required of today’s engineering graduates, including

multidisciplinary teamwork, communication, social responsibility and global awareness (Figure

3). While these lessons are especially applicable to small liberal arts institutions with limited

resources, the improvement of critical skill such as communication and the ability to work on

interdisciplinary teams suggests that this pedagogy could also enhance these attributes in

students at large institutions. Collaborations between smaller and larger engineering programs

could provide benefits for both institutions. We know that the details of implementing this

program will continuously evolve, but will always remain rooted in the importance of

collaboration.

References

1. The essential skills and attributes of an engineer: A comparative study of academics, industry personnel and

engineering students. Nguyen, Duyen. 1, 1998, Global Journal of Engineering Education, Vol. 2, pp. 65-75.

2. The future of engineering education. I. A vision for a new century. Rugarcia, Armando, et al., et al. 1, 2000,

Chemical Engineering Education, Vol. 34, pp. 16-25.

3. Development and use of an engineer profile. Davis, Denny C, Beyerlein, Steven W and Davis, Isadore T. s.l. :

American Socity for Engineering Education, 2005. Proceedings of the 2005 American Society for Engineering

Education Annual Conference & Exposition.

4. Beyond technicalities: Expanding engineering thinking. Beder, Sharon. 1, 1999, Journal of Professional Issues in

Engineering Education and Practice, Vol. 125, pp. 12-18.

5. A Comparison of Group Processes, Performance, and Satisfaction in Face-to-Face Versus Computer-Mediated

Engineering Student Design Teams. Whitman, Lawrence E., et al., et al. 94, 2005, Journal of Engineering

Education, pp. 327-337.

6. Virtual Engineering Design Teams. Wyrick, D. and Cisse, A. 2009. ASEE Conference.

7. Using a Multidisciplinary Team Approach in Biomedical Engineering Senior Design. Enderle, John D.,

Pruehsner, William and Macione, James. 2000, Biomedical Sciences Instrumentation, pp. 63-8.

8. A Systematic Approach for Introducing Innovative Product Design in Courses with Engineering and

Nonengineering Students. Patterson, PE. 2007, Biomedical Sciences Instrumentation, pp. 91-4.

Incorporating Oral Presentations into Electrical and ... · Incorporating Oral Presentations into...

Documents

Transcript of Incorporating Oral Presentations into Electrical and ... · Incorporating Oral Presentations into...