Assessing Pedagogical Balance in a Simulated Classroom ...Assessing Pedagogical Balance in a...

Assessing Pedagogical Balance in a Simulated Classroom Environment

Gerald Knezek, Susan B. Hopper, RhondaChristensen, & Tandra Tyler-Wood

University of North Texas

David C. GibsonCurtin University

Abstract

simSchool, an online simulator thathas been used to enhance teacherpreparation since 2003, models dif-ferent types of students and providesvirtual practice sessions for teachersto assign tasks and interact with stu-dents. In this article the authors (a)examine changes in preserviceteacher perceptions of teaching confi-dence and teaching experience result-ing from simSchool use, and (b)report findings from recent studies ofa new proposed measure for sim-School data, pedagogical balance.Pedagogical balance is a differencescore that measures preservice teach-ers’ self-reported levels of confidenceminus experience, which indicates alevel of alignment in self-evaluationwhen balancing one’s perceptions ofcapabilities and experience. Findingsfrom two studies show that preserviceteachers significantly (p < .05)improve pedagogical balance andincrease awareness of effective teach-ing skills through simSchool training.

Teaching is the most important fac-tor of student achievement(National Commission on Teach-

ing and America’s Future, 1996), andeffective teaching can increase studentachievement outcomes (Darling-Ham-mond, Wei, Andree, Richardson, &Orphanos, 2009). The National Councilof Accreditation of Teacher Education(NCATE) describes teaching effective-ness as

teacher preparation/knowledge ofteaching and learning, subject mat-ter knowledge, experience, and thecombined set of qualificationsmeasured by teacher licensure asleading factors in teacher effective-ness. (Darling-Hammond, 2006)

In addition to having a positiveimpact on student learning, well-pre-pared teachers are also more likely toremain in teaching (according toNCATE). However, preparation pro-grams do not seem to be able to produceenough well-prepared teachers. Attritionamong beginning teachers has increasedsteadily over the past two decades(Ingersoll & Merrill, 2013), and morethan 42% of new teachers leave the fieldwithin 5 years of entry (Perda, 2013).Borman and Dowling (2008) found thatteachers with advanced degrees and withdegrees in mathematics or science aremore likely to leave teaching. In counter-point to these trends, a recent study byIngersoll, Merrill, and May (2014) exam-ined the relationship between beginningteachers’ education preparation andattrition. Findings showed that thoseteachers with more pedagogical prepara-tion were more likely to stay in teaching,whereas those with less pedagogicalpreparation were more likely to leave thefield after their first year of teaching.Mathematics and science teachers intheir study had more subject-area cour-sework but less pedagogical coursework(Ingersoll et al., 2014). Among the strat-egies that may be useful for increasingpedagogical practice is the use of simula-tions such as simSchool, which providesan Internet-based simulated classroomthat allows preservice teachers to experi-ence the outcomes of instructional deci-sions. simSchool has more than 13,000

registered users in more than 156 coun-tries (simSchool, 2015). The use of sim-School has been shown to demonstratebenefits to teacher preparation candi-dates in the areas of classroom manage-ment (Christensen et al., 2007), teachingskills (D. Gibson, Christensen, Tyler-Wood, & Knezek, 2011; Tyler-Wood,Knezek, & Christensen, 2007), motiva-tion (Tyler-Wood et al., 2007), andinstructional self-efficacy (Christensen,Knezek, Tyler-Wood, & Gibson, 2011;Knezek & Christensen, 2009). The studyreported here explores the impact ofsimSchool on a new measure—pedagogi-cal balance—as a proficiency indicator toenable students to understand the bal-ance between estimations of their confi-dence and experience. Pedagogicalbalance is defined as the differencebetween a teacher’s confidence and expe-rience ratings for teaching (Hopper,Knezek, & Christensen, 2013; Hopper,Knezek, Christensen, Tyler-Wood, &Gibson, 2014). The pedagogical balancescore may assist teacher educators byfacilitating an understanding of candi-dates’ perceptions of their teachingpreparedness.

Research QuestionsThe purposes of this article are to exam-ine changes in preservice teacher percep-tions of teaching confidence andteaching experience and report findingsfrom two studies of simSchool thatinclude the new measure of pedagogicalbalance. A measure of self-efficacy evalu-ates both an affirmation of a capabilitylevel and the strength of that belief (Ban-dura, 1994). Teaching self-efficacy asmeasured here uses self-reported percep-tions of teaching confidence and teach-ing experience. The teacher’s confidence

© 2015 ISTE j iste.org/jdlteColor versions of one or more of the figures inthe article can be found online at http://www.tandfonline.com/ujdl.

148 l Journal of Digital Learning in Teacher Education l Volume 31 Number 4

rating is taken to be an affirmation of acapability level and the strength of thatbelief is understood to be measured bythe teacher’s rating of experience. Expe-rience according to Bandura can begained through four sources:

� Successful repetition of a task.� Social modeling through the observa-tion of others completing a task.

� Social persuasion by competent othersinstilling confidence with encourage-ment to succeed.

� Situations for others to succeedthrough self-improvement (1994).

Research questions to be addressedare:

1. Does the use of simSchool influencepreservice educators’ self-perceptionsof confidence in their teaching skills?

2. Does the use of simSchool influencepreservice educators’ self-perceptionsof teaching experience?

3. Does the use of simSchool help bringinto balance preservice educators’perceptions of teaching experienceand confidence?

Conceptual FoundationsThe concepts of self-efficacy (Bandura,1994) and theory of action (Argryis &Schon, 1974) offer foundational ideasand rationales for the analysis of sim-School outcomes.

Self-EfficacyBandura (1994) indicates that studentsentering teacher education programscome with preconceived beliefs abouteducation based on their own schoolexperiences. Preservice teachers haveacquired knowledge about schools, class-rooms, and instructional practices fromtheir 13 years of formal education (Lor-tie, 1975). From these experiences, pre-service teachers have formed perceptionsabout their abilities to teach (Duffin,French, & Patrick, 2012 ;Woolfolk Hoy& Murphy, 2001). When preserviceteachers begin a teacher education pro-gram, they participate in many newlearning experiences (Cochran-Smith &Zeichner, 2005). The combination of

these preconceived beliefs, perceptions,and new learning experiences is differentfor every preservice teacher and formsthe foundation of teaching self-efficacy(Henson, 2001).

Bandura (1994) defines self-efficacyas one’s belief in one’s ability to suc-ceed in a particular situation. A teach-er’s sense of self-efficacy affects thatteacher’s attitudes and feelings towardthe educational process (Woolfolk Hoy& Hoy, 1990) and refers to a teacher’sability to carry out instructional prac-tices in the educational context thatresult in positive student outcomes(Bandura, 1997). Noncognitive varia-bles including persistence (S. Gibson &Dembo, 1984), motivation (Darling-Hammond, Chung, & Frelow, 2002),and organizational practices (Allinder,1994) influence the attitudes andbeliefs of a teacher’s sense of instruc-tional self-efficacy.

Previous studies using simSchool as anintervention for preservice teachers havereported large gains of self-reportedteaching skill level or instructional self-efficacy. In a study of 32 preserviceteacher candidates, from a reading/lan-guage arts methods course, at a largeSouthwestern university, students partici-pated in nine hours of simSchool train-ing. Findings in the area of instructionalself-efficacy resulted in pre–post gains forthe treatment group (effect size D .96)that were greater than the gain for thecomparison group (effect size D .40)(Christensen et al., 2011).

Gains in instructional self-efficacywere reported in a study of 104 preser-vice teachers who explored how toaccommodate the learning needs of asimulated student with disabilities in aninclusion classroom setting. The effectsize was large for the treatment group (dD .68, p D .03), whereas the comparisongroup made no significant gains in self-efficacy. Findings showed that simSchoolactivities resulted in gains in instruc-tional self-efficacy (Christensen et al.,2011).

Listed in Table 1 are Bandura’s fourskills related to gain in self-efficacy cor-related with corresponding activities per-formed when using simSchool.

Theory of ActionStudent teachers may underestimate thecomplexity of managing student behav-ior and student learning. The realities ofteaching may cause student teachers tobecome dismayed with the gap betweenthe expectations of their own abilitiesand their actual performance in theclassroom with students (Tschannen-Moran et al., 1998). Argyris and Schon(1974) proposed two aspects of a theoryof action—“espoused theory” and “the-ory in use”—that may explain why thegap between expectations and abilitiesoccurs with the student teachers. Anespoused theory is how people say theywould like to or believe they will behavebased on their personal values. A theoryin use, in contrast, is how individualsactually behave in spite of their person-ally espoused values and can be inferredfrom action. The espoused theory maybe the predominant theory under whichoptimistic student teachers are function-ing when they are challenged with diffi-cult classroom situations while studentteaching. Many individuals are likelyunaware that the behaviors in which theyactually engage may be different fromthe behaviors they espouse. Even fewerindividuals are aware of the actual theo-ries under which they operate, leading togaps between what student teachers saythey do (or intend to do) and what theyactually do (Argyris, 1980).

Literature ReviewThe literature relevant to the simSchoolinterventions studied for this article fallunder the categories of preparedness toteach and the foundations of the simula-tion’s model scenarios.

Preparedness to TeachPreservice teachers with teaching confi-dence can handle more difficult situa-tions in a classroom, reach variouslevels of learners, make a difference inlearning outcomes (Darling-Hammondet al., 2002), and have lower attritionrates (Ingersoll, Merrill et al., 2014).Preservice teachers who feel preparedto teach exhibit a higher level ofinstructional or teaching self-efficacy(Henson, 2001).

Assessing Pedagogical Balance in a Simulated Classroom

Volume 31 Number 4 l Journal of Digital Learning in Teacher Education l 149

The 2007–2008 Schools and StaffingSurvey (SASS) identified 24% of thenation’s teaching workforce as beginningteachers with 5 or fewer years of teachingexperience. Of the roughly 832,000beginning teachers identified, 61% expe-rienced 12 or more weeks of practiceteaching, while 20% had less than 12weeks of practice teaching and 19% hadno practice teaching. Support for the newteachers in their first year varied frominduction programs to common plan-ning time with peer teachers, seminarsfor beginning teachers, extra classroomassistance, guidance from a mentor, andregular communication with the princi-pal. Feedback from the new teachersreported their sense of preparedness insix areas:

� 59% felt well prepared to handle class-room management.

� 71% felt well prepared using a varietyof instructional methods.

� 83% felt well prepared in teaching sub-ject matter.

� 67% felt well prepared in usingcomputers.

� 70% felt well prepared in assessingstudents.

� 65% felt well prepared in adapting cur-riculum and instructional materials.

These data indicate that significantpercentages of teachers felt unprepared

to teach in various areas necessary foreffective teaching.

The “bright-person” myth (Darling-Hammond, 2000) supposes that any-one can teach knowledge to someoneelse, regardless of whether one hasreceived teacher training or is wellprepared. However, Darling-Hammondpresumes that when a teacher is tryingto convey a lesson to a learner, andthe learner does not understand thelesson, the teacher may become frus-trated and unable to proceed with thelesson because the teacher may lackthe training to reteach using a differ-ent pedagogical strategy. This lack ofability to engage students with a newstrategy may lead to resentment fromthe student, who feels the teacher isnot displaying sufficient effort. With-out adequate practice teaching thatprovides experiences in using variouseffective interventions during teachertraining, it may be difficult for a newteacher to select from a range of effec-tive strategies to teach students whoare experiencing initial difficulty withcomprehension of a lesson.

simSchool Model ScenariossimSchool promotes pedagogical exper-tise by re-creating the complexities ofclassroom decisions through mathemati-cal representations of how people learn

and what teachers do when teaching.The model includes research-based psy-chological, sensory, and cognitivedomains similar to Bloom’s Taxonomyof Educational Objectives (Bloom, Mesia,& Krathwohl, 1964). However, in sim-School these domains are defined withunderlying subcategory factors thatreflect modern psychological, cognitivescience, and neuroscience concepts. Forexample, the Five-Factor Model of psy-chology (McCrae & Costa, 1996) servesas the foundation of the student person-ality spectrum. This model includes thefollowing characteristics: extroversion,agreeableness, persistence, emotionalstability, and intellectual openness tonew experiences. For each of these fivefactors a continuum from –1 to C1 isused to situate the learner’s specific emo-tional processing propensities, which canshift as the context of the classroomchanges. A simplified sensory modelwith auditory, visual, and kinestheticperceptual preferences comprises thephysical domain. A flexible single factoris used to represent a specific academicdomain. Together, the physical, emo-tional, and academic factors are used torepresent salient elements of classroomteaching and learning (D. Gibson, 2007;Christensen et al., 2011).

Through the navigation of a technol-ogy-based platform, preservice teacherscomplete mini-scenarios that provide aholistic view of teaching and allow pre-service teachers to hone in on specificteaching strategies such as classroommanagement and differentiated instruc-tion. Grounded in educational theory,preservice teachers complete simSchoolmodules to make decisions about virtualstudents and practice teaching lessons tocritically challenge students using higherordered thinking skills. simSchool pro-vides feedback reports on teaching ses-sions for preservice teachers to analyzethe effectiveness of their teaching andmake adjustments to improve studentachievement. The adjusted lesson isrepeated in the simulator, and preserviceteachers compare and contrast studentoutcomes based on their instructionaldecisions. This type of feedback andimprovement cycle develops pedagogical

Table 1. Sources to Gain Self-Efficacy and simSchool Activities

Sources to gain a sense of self-efficacy simSchool activities

Successful repetition of task Preservice teachers practice how to:Repeat lessonsAdjust teachingAnalyze findings

Social modeling through the observation of others completing a task Preservice teachers observe:Trainer models effective teachingPeers model simSchool taskCompletions

Social persuasion by competent others instilling confidence withencouragement to succeed

Preservice teachers are encouraged by:Simulation feedbackPeer feedbackTrainer feedback

Situations for others to succeed through self-improvement Preservice teachers develop ways to:Make classroom decisionsAdjust mistakesRepeat lessonsConnect virtual teaching toauthentic teaching

Knezek et al.


knowledge about teaching using theoret-ical and practical experience (D. Gibson& Kruse, 2012).

Methods

Instrumentation for Measurementof Pedagogical BalancePedagogical balance is a new measurecreated with the support of grantsawarded by the U.S. Department of Edu-cation Fund for the Improvement ofPostsecondary Education (FIPSE), theGates/EDUCAUSE Foundation, and theNational Science Foundation to assessalignment of perceived confidence andexperience. The Survey of TeachingSkills (D. Gibson, Riedel, & Halverson,

2006) is a self-report of preservice teach-ers’ self-efficacy and divides the con-struct into two variables, teachingconfidence and teaching experience. Pre-service teachers assess their experienceand confidence levels using a five-itemscale that ranges from very low to veryhigh, as shown in Table 2 (see theappendix to view the entire survey).Using this instrument, self-efficacy ismeasured by the summation of confi-dence and experience, while pedagogicalbalance is measured by the difference

score between confidence andexperience.

The survey addresses eight teachingareas as described in Table 3.

Pedagogical balance is defined as thedifference between a person’s averageconfidence rating for teaching and aver-age experience rating for teaching (Hop-per et al., 2013; Hopper et al., 2014). Thelowest rating on the survey used forexamining pedagogical balance is 1.0 foreach measure, while the highest is 5.0, sothe greatest possible difference between

Table 2. Survey of Teaching Skills Experience and Confidence Level Measurement Scale

Experience level Very low Moderately low Medium Moderately high Very high

Confidence level Very low Moderately low Medium Moderately high Very high

Table 3. Eight Areas of Teaching Areas Measured in the Survey of Teaching Skills in simSchool

Teaching area Description

Knowledge of students Reading and using student records to make instructional decisions

Preplanning assessment and instructions to meet individual and group needs

Observing in-classroom behavior and making inferences about adaptations needed in instruction and assessments

Preplanning instruction Knowing what subject one is prepared to teach

Knowing how many and what kinds of tasks are suited and fit with a subject

Estimating the number of class sessions to teach a particular set of tasks

Making and using tasks Designing appropriate tasks

Sequencing tasks for best effect

Making and using assessments Assigning assessment items to a given objective

Estimating the number of and types of assessment items’ measures are suited to and fit for a particularset of objectives

Understanding the data produced by administration of pre-assessment

Replanning instruction Prior to instruction, choosing whole-class instructional strategies based on (aligned with) pre-assessment results

Prior to instruction, choosing individual instructional strategies based on (aligned with) student records and individualpre-assessment results

Classroom decision making Interpreting in-class performance (on-task vs. off-task behaviors) as academic vs. emotional issues

“Reading” students via participation clues and language

Speaking to students in effective and appropriate ways

Grouping students for differentiated instruction

Adjusting instructional strategies based on in-class performance

Individualizing tasks

Focusing talk and discussion on improved student performance

Making and using postassessment Designing appropriate and aligned test items to assess a given “unit of study”(objectives plus instructional strategies and adaptations that have occurred during a number of class sessions)

Estimating the number of and types of assessment items’ measures are suited to and fit for the unit of study

Understanding the data produced by administration of a postassessment

Reflections on teaching Making mental notes (and possibly written records such as grade-book notations) about the evolution of aparticular unit of study—the interactions of one’s plans with the realities of teaching

Abstracting and articulating lessons learned from the whole experience



confidence and experience is 4.0. Thebasic principle underlying the idea ofbalance is that a difference score shouldequal 0.0 when the confidence of a pre-service teacher is aligned with his or herexperience (Hopper et al., 2013).

For example, teacher A perceives alow level of experience and a high level ofconfidence in knowledge of students(Figure 1). If teacher A has little experi-ence in knowledge of students, then whyis that teacher’s confidence so high?While the teacher may perceive that heor she is prepared in an area of teaching,the self-report indicates that the teacherlacks experience in working in that area.This example indicates an “overconfi-dent” imbalance of confidence overexperience.

In the next example, Figure 2 illus-trates teacher B’s report of high experi-ence and low confidence, whichdemonstrates an “underconfident”imbalance. Although this teacher mayhave a moderately high level of experi-ence with students, he or she has not yetgained in confidence from thatexperience.

The third example in Figure 3 illus-trates a balance of experience and confi-dence. Teacher C perceives that theconfidence and experience levels inknowledge of students are equal. Theteacher may perceive pedagogical bal-ance in a teaching area when his or herconfidence level aligns with his or herlevel of experience.

AnalysisThis study executed a quasi-experimen-tal design to quantitatively measure andexamine preservice teachers’ perceivedlevels of teaching confidence, teachingexperience, and pedagogical balancebefore and after using simSchool. Reli-ability measures, descriptive statistics, apaired-samples t-test, and multiple anal-ysis of variance (MANOVA) were usedto analyze the data. Descriptive statisticsincluding mean and standard deviationsfor the Survey of Teaching Skills datawere computed for confidence and expe-rience at pretest and posttest for Study 1and Study 2. A paired-samples t-test wascomputed to determine gains in

confidence and experience from pre- toposttest. MANOVA was computed todetermine whether changes in preserviceteachers’ confidence, experience, andpedagogical balance were differentbetween preservice teachers who usedsimSchool as a training tool and thosewho did not. MANOVA was used inwhich treatment and comparison func-tioned as the attribute variables. Thefindings for confidence and experienceare examined in two different studies.Pedagogical balance is explored by com-paring the two studies side by side.

Study 1

SampleIn the fall of 2012, 58 preservice teachersfrom an undergraduate technology inte-gration course at a large Southwesternuniversity participated in Study 1. Thetreatment group consisted of 31 studentsand the comparison group included 27preservice teachers from a different sec-tion of the technology integration coursewith no experience in simSchool. Thetechnology integration course intro-duced preservice teachers to the field ofeducational technology. Topics coveredin the course were those that impactededucators working in the classroomenvironment.

Instrument Reliability and ValidityThe Survey of Teaching Skills wasadministered to the treatment and com-parison groups as a pre- and posttest.Cronbach’s alpha for Experience Levelwas .96, and for Confidence Level was.94. According to the guideline byDeVellis (1991), both Cronbach’s alphascores were excellent, indicating highinternal consistency reliability for eachmeasurement index.

InterventionThe simSchool treatment took placemidway through the semester during thescheduled course meeting times, 3 out of4 weeks. The pretest was administered tothe treatment group prior to the sim-School training.

Preservice teachers in the treatmentgroup participated in 8 hours of sim-School incorporating three training ses-sions. The training occurred withmodules on the Big Five Factor model ofpersonality (McCrae & Costa, 1996),Bloom’s Taxonomy (Bloom et al., 1964)of higher order thinking skills, and stu-dent-centered instruction. The goal inSession 1 was to introduce preserviceteachers to simSchool and to connectpersonality traits with teaching andlearning style. Learning objectives for themodule were for preservice teachers to

Figure 1. Example of reported low experience and high confidence from the survey of teaching skills.

Figure 2. Example of reported high experience and low confidence from the survey of teaching skills.


Knezek et al.

realize that every student learns differ-ently. Experiential interactions betweenthe teacher and the virtual students wereto provide inquiry-based activities toassist preservice teachers in the discoveryof new teaching strategies. Session 2 builtupon the skills that the preservice teach-ers learned in Session 1 to consider howstudent personality traits influence stu-dent academic outcomes of teacher-planned lessons. The learning objectivesin Session 2 were to structure activitiesthat engage and challenge the learnerthrough the use of curriculum resourcesto purposefully create lessons that guideand direct student learning and behaviorimpacting academic achievement.

In Session 3 of simSchool training,preservice teachers compared and con-trasted the learning outcomes of a stu-dent-centered lesson on endangeredeagles with a teacher-centered lessonon the same content. This modulemodeled examples of higher orderthinking activities using different typesof pedagogical practice with technol-ogy integration. Students worked ingroups to create a digital storytellingproject on a topic of their choosing toanalyze their group’s perception andtheir own perception of a student-cen-tered lesson compared to a teacher-centered lesson.

The comparison group continued toparticipate in previously established

classroom activities. They completedtheir pretest surveys at the same time asthe treatment group, midway through thesemester, with their posttest 1 month later.

Findings

Treatment classroom. The mean differen-ces pre- to posttest within the treatmentgroup were examined using a paired-sample t-test. Significant gains (p < .005)in experience (d D .97) and confidence(d D .76) from pre- to posttest (Cohen,1988) are shown in Table 4.

Comparison classroom. The comparisongroup demonstrated significant gains (p< .05) in experience (d D .33, p D .006)and confidence (dD .58, p< .0005) frompre- to posttest with educationally mean-ingful effect sizes (Bialo & Sivin-Katch-ala, 1996), as shown in Table 5.

The effect sizes were much smaller forthe comparison group than for the treat-ment group, warranting further analysisusing MANOVA to explore the consis-tency of the simSchool treatmentbetween all subjects. The MANOVA testresults showed the pre to post gains forthe treatment group to be higher than forthe comparison group, and the gains inexperience (p D .036) were significantlyhigher. These findings shown in Fig-ures 4 and 5 suggest that 8 hours of sim-

School intervention increased the ratingsof experience in preservice teachers andthe training can be considered to be edu-cationally meaningful (Bialo & Siven-Kachala, 1996).

Study 1 results found that confidencein both the treatment and comparisongroups increased nearly at the same rate,whereas experience of the treatmentgroup increased significantly (p D.036)more than the comparison group.

Study 2

SampleUndergraduate students from two differ-ent teaching preparation courses at alarge southwestern university partici-pated in Study 2 in the fall of 2013. Thetreatment group consisted of 36 partici-pants in the pretreatment group and 37participants in the posttreatment group.Preservice teachers enrolled in a technol-ogy integration course (described inStudy 1) participated in 6 hours of sim-School with the same instructor as partof the course curriculum. The compari-son group consisted of 80 participants atpretest time and 77 participants for post-test data collection. Participants in thecomparison group were enrolled in arequired education course on teachingexceptional learners. The treatment andcomparison groups completed a pretestof the Survey of Teaching Skills 2 weeksinto the semester. The online pretest wascompleted by the comparison groupfrom five sections of the teaching excep-tional learners course. The comparisonstudents were offered extra credit to par-ticipate in the study. The posttest wascompleted by the treatment and compar-ison groups approximately 1 monthfrom the completion of the pretest.

Instrument Reliability and ValidityCronbach’s alpha for Experience Levelwas .93, and for Confidence Level was.93. Both Cronbach’s alpha scores wereexcellent, indicating high internal consis-tency reliability for each measurementindex according to the guidelines byDeVellis (1991).

Figure 3. Example of equal experience and confidence levels from the survey of teaching skills.

Table 4. Paired-Sample t-Test for the Treatment Classroom Study 1 Using simSchool, Technology Integration Course, Fall2012

Measuring indices N Mean SD Significance Cohen’s d

Experience Pre 31 2.50 0.80 .00 0.97

Post 31 3.20 0.64

Confidence Pre 31 2.80 0.82 .00 0.76

Post 31 3.30 0.49



InterventionThe intervention for Study 2 consisted of6 hours of simSchool training. The train-ing was comprised of Session 1 and Ses-sion 2 described in Study 1. Session 3was not administered in Study 2. Thesame instructor taught the simSchooltraining in both studies.

Findings

Treatment classroom. The mean differen-ces pre- to posttest were examined usinga paired sample t-test. The treatmentgroup experience (p D .003) showed sig-nificant gains (p < .05) from the pre- toposttest with the simSchool intervention,whereas the treatment group confidencewas not significant, as shown in Table 6.A moderate effect size for gains (d D .62)in experience was found according to theguidelines provided by Cohen (1988).The effect of simSchool can be consid-ered to be educationally meaningfulaccording to guidelines published byBialo and Sivin-Kachala (1996).

Comparison classroom. The comparisongroup did not demonstrate significant

gains (p D.99) from pre- to posttest andthe effect sizes were insignificant, asshown in Table 7. Additional analysisusing MANOVA to examine the consis-tency of the simSchool treatmentbetween all subjects was computed. TheMANOVA test results showed the pre topost gains for the treatment group to behigher than for the comparison group,and the gains in experience were signifi-cantly higher (p D.044). These findingsshown in Figures 6 and 7 suggest that6 hours of simSchool interventionincreased the ratings of experience inpreservice teachers.

Comparison of Study 1 and Study 2Findings concerning experience weresignificant in the pre to post gains inStudy 1 for the treatment and compari-son groups. In Study 2, experience wasfound to be statistically significant(p < .05) for the treatment group. Thedifferences in these findings between thetwo studies may be attributed to severalreasons. First, the course selection of thecomparison groups in each study wasdifferent. In Study 1 both the treatmentand the comparison groups were froma technology integration course for pre-service teachers. Study 1 took placemidway through the semester, so thecomparison group could have gained inexperience and confidence from pre- toposttest from quality course instruc-tion. In Study 2, the comparison groupwas selected from a required educationcourse on teaching exceptional learn-ers. The results found that effectivelyno change took place in confidence orexperience for the comparison group inStudy 2 from pre- to posttest. Thesestudents had not taken the technologyintegration course so they did not gainfrom learning in technology integra-tion. Another difference between the

two studies was that the treatment groupin Study 1 participated in 8 hours ofsimSchool, whereas the treatment groupin Study 2 participated in 6 hours ofsimSchool. The two additional hours ofsimSchool training of the Study 1 partici-pants may have impacted confidence orexperience levels in the treatment group.Further research is needed to determineto what degree more simSchool trainingprovides increased results. Both thecourse selection of the participants in thestudies and the number of simSchoolhours participants trained should beconsidered in comparing Study 1 andStudy 2. The authors propose anotherpossibility for the findings. Further clar-ification of these gains may result fromexamination of the measure of pedagogi-cal balance because the significant gainsin experience become more relevant withconsideration of the results ofconfidence.

Pedagogical Balance for Study 1and Study 2The mean differences from pretest toposttest were examined for pedagogicalbalance in Study 1 using a paired-samplet-test with n D 31 in the treatment groupand n D 27 in the comparison group.The mean of pedagogical balance of thetreatment group significantly improvedby moving closer to zero (p < .05) fromthe pretest (x D .38) to the posttest (x D.10). Note that by becoming closer tozero, pedagogical balance increased as

Table 5. Paired-Sample t-Test Comparison Classroom Study 1 Using simSchool, Technology Integration Course, Fall 2012


Experience Pre 27 2.73 0.86 .0006 0.33

Post 27 3.00 0.78

Confidence Pre 27 3.03 0.71 .00 0.58

Post 27 3.44 0.71

Figure 4. Confidence for the treatment and comparisongroups increased at the same rate (not statistically signifi-cant, p D.95).

Figure 5. Experience for the simSchool treatment groupincreased at a greater rate than the comparison group (sta-tistically significant, p D.036).


Knezek et al.

confidence and experience became morealigned. In contrast, the mean of peda-gogical balance of the comparison groupsignificantly (p < .05) became less bal-anced from the pretest (x D .30) to theposttest (x D .44), signified by the scoremean migrating further from zero, indi-cating that confidence and experiencebecame less aligned. Both the treatment(d D .46) and comparison (d D .33)groups were found to have mediumeffect sizes in pedagogical balance(Cohen, 1988), as shown in Table 8.

Study 2 had pedagogical balanceresults similar to the treatment and com-parison groups found in Study 1. Themean differences as shown in Table 9 ofthe treatment group improved bydecreasing from pretest (x D .27) toposttest (x D .01), whereas the compari-son group worsened by showingincreases in the mean differences frompretest (x D.25) to posttest (x D .33).Pedagogical balance became closer tozero for the treatment group and furtheraway for the comparison group, whichsuggests that the simSchool interventionprovided training that improved the bal-ance of preservice teachers’ confidenceand experience. The treatment group’spedagogical balance showed significantimprovement (p < .005) and a moder-ately large effect size (d D .59), whereasthe comparison group showed a slighttrend toward less balance (p D .22) witha low effect size (d D .12).

The MANOVA test results illus-trated in Figures 8 and 9 showed thepre to post movement for the treat-ment groups to be toward greater ped-agogical balance, while for thecomparison groups the pre- to posttestmovement was toward being more outof balance. The difference was statisti-cally significant in Study 1 (p D .031)and Study 2 (p D .033). These findingssuggest that the differences betweenconfidence and experience becamemore aligned in both studies as aresult of the simSchool intervention.Pedagogical balance significantlyimproved for both treatment groupsby moving closer to zero; however,pedagogical balance for the two com-parison groups moved further awayfrom zero. The members of the treat-ment group became more aligned inthe difference between their confidenceratings and experience ratings duringtheir simSchool training. Overall, pre-service teachers using simSchool sig-nificantly increased in experience andpedagogical balance.

Summary of FindingsThe results of Study 1 and Study 2 hadsimilar findings. Both studies foundinsignificant pre- to posttest gains inconfidence, and significant pre- to post-test gains in experience and pedagogicalbalance.

Finding for Changes in ConfidenceThe first research question addressed theeffectiveness of simSchool training forthe treatment group and found that pre-service teachers trained in 6 to 8 hours ofsimSchool did not show significantlyhigher pre- to posttest gains in teachingconfidence than those without thetraining.

Finding for Changes in ExperienceThe second research question addressedthe effectiveness of simSchool trainingfor the treatment group and found thatpreservice teachers trained in 6 to8 hours of simSchool showed higher pre-to posttest gains in teaching experiencethan those without the training.

Table 6. Paired Sample t-Test for the Treatment Classroom Study 2 Using simSchool, Technology Integration Course, Fall2013


Experience Pre 36 2.88 0.78 .003 0.62

Post 37 3.32 0.64

Confidence Pre 36 3.18 0.64 .164 0.29

Post 37 3.36 0.61

Table 7. Paired Sample t-test for the Comparison Classroom Study 2 using simSchool, Technology Integration Course, Fall2013


Experience Pre 80 2.60 0.90 .99 0.01

Post 77 2.61 0.83

Confidence Pre 80 2.90 0.88 .951 0

Post 77 2.90 0.83

Figure 6. Pre- to post- gains in confidence for the sim-School treatment group increased at varied rates (not statis-tically significant, p D .63).

Figure 7. Pre- to post- gains in experience for the sim-School treatment group increased at a greater rate than thecomparison group (statistically significant, p D .044).



Finding for the Balance BetweenConfidence and ExperienceThe third research question addressedthe effectiveness of simSchool trainingfor the treatment group and found thatpreservice teachers trained in 6 to8 hours of simSchool showed higher pre-to posttest improvements in pedagogicalbalance than those without the training.

DiscussionIn Study 1 (p D .036) and Study 2(p D .044) preservice teachers signifi-cantly gained in experience from theirinvolvement with simSchool, while con-fidence ratings were not significant. Onereason the confidence ratings may nothave been significant was that preserviceteachers may have overrated their confi-dence levels when self-reporting theirperceptions on the pretest, as the theoryof action (Argyris & Schoen, 1974) sug-gests. Typically, with the espoused theoryone reports how one would like tobehave based on one’s personal values.After the simSchool intervention, pre-service teachers may have realized thatthey were not as confident aboutteaching as they believed they werebefore their experience in simSchool.Prior to the simSchool experience,during the completion of the pretest,preservice teachers appear to have uti-lized the espoused theory reportinghigher confidence in their teachingability. However, after the simSchool

training it appears the theory in usewas predominant. The preserviceteachers may have reported lower con-fidence at the posttest because thesimSchool treatment provided teachingexperience, perhaps allowing the pre-service teachers to realize they did notknow as much as they thought theyknew about teaching prior to theirsimSchool experience. simSchool train-ing may have provided awareness topreservice teachers of some of theskills needed to teach that they hadnot yet developed. Although confi-dence decreased from pre- to posttest,it appears that the gap between whatpreservice teachers believed they coulddo and what they actually were able todo lessened due to their gain in aware-ness of their own abilities (or lackthereof).

The findings for pedagogical balancein Study 1 (p D .031) were similar to theresults in Study 2 (p D .033). The graphs(Figures 8 and 9) illustrate consistentresults in that experience and confidencebecame more aligned in the treatmentgroup, whereas the comparison groupbecame more out of balance. Confidencemay have decreased, but as previouslystated the confidence self-report mayhave been somewhat inflated at the pre-test. To counteract the decrease in confi-dence, experience increased, causingimproved alignment in pedagogicalbalance.

Conclusions and Implications of FindingsThe results of these research studies sup-port the following conclusions:

� Preservice teachers may overrate theirconfidence levels at pretest time.

� Preservice teachers seem to gain teach-ing experience in simSchool.

� Experience and confidence measuresseemed to become more balanced as aresult of simSchool use.

� Preservice teachers seem to gainawareness of their teaching skills (orlack of) through the use of simSchool.

Study 1 and Study 2 provide evidencethat simSchool training offered the pre-service teachers additional paths to

Table 8. Pedagogical Balance Findings From Study 1 Treatment and Comparison Groups, Technology Integration Course,Fall 2012

Variable Test n x s p d

Treatment pedagogical balance Pre 31 .38 .73 .05 .46

Post 31 .10 .41

Comparison pedagogical balance Pre 27 .30 .34 .00 .33

Post 27 .44 .49

Table 9. Pedagogical Balance Findings From Study 2 Treatment Technology Integration Course and Comparison Excep-tional Learner’s Course, Fall 2013

Variable Test n x s p d

Treatmentpedagogical balance Pre 34 .27 .56 .00 .59

Post 34 .01 .25

Comparisonpedagogical balance Pre 75 .25 .66 .22 .12

Post 75 .33 .69

Figure 8. Pedagogical balance for the comparison groupbecame more out of balance (further away from 0); how-ever, the simSchool treatment group improved pedagogicalbalance (closer to 0) (statistically significant, p D.031).

Figure 9. Pedagogical balance for the comparison groupbecame more out of balance (further away from 0); how-ever, the simSchool treatment group improved pedagogicalbalance (closer to 0) (statistically significant, p D.033).


Knezek et al.

practice and improve teaching skills,connected learning theories in the class-room, and developed experience withoutthe ill impacts of practicing on real stu-dents. In addition, findings indicate that6 to 8 hours of purposeful activities inthe simulator may improve pedagogicalbalance through the alignment of confi-dence and experience.

Based on the findings of Study 1 andStudy 2, pedagogical balance holds poten-tial for future use as a proficiency indica-tor for preservice teachers, to enable themto understand their individual alignmentof confidence and experience. Thisknowledge could increase awareness ofthe skills that preservice teachers need todevelop to be effective teachers, and helpto bridge the gap between what preserviceteachers espouse to know and what theyactually know.

FundingsimSchool was created by Dr. David C.Gibson and colleagues through Prepar-ing Tomorrow’s Teachers to Use Tech-nology (PT3) funding from the U.S.Department of Education beginning in2003. Additional awards spanning 2006–2012 were secured by Drs. Christensen,Knezek, and Tyler-Wood at UNT (U.S.Department of Education Fund for theImprovement of Postsecondary Educa-tion grant P116B060398; EDUCAUSEModules Project; U.S. National ScienceFoundation Research and DisabilitiesEducation [RDE] grant 0726670).

Author NotesDr. Gerald Knezek is a Regents Professorof Learning Technologies at the Univer-sity of North Texas and lead principalinvestigator for the U.S. National ScienceFoundation Innovative TechnologiesProject Going Green! MSOSW (numbers0833706 and 1312168). He is a past presi-dent (2008–2011) of the Society for Infor-mation Technology & Teacher Education(SITE). He was a Fulbright Scholar at theTokyo Institute of Technology during1993–1994 and shared time betweenTexas and Ecuador on a Fulbright SeniorSpecialist appointment during 2006–2007. Dr. Knezek’s research interestsinclude technology integration, teacher

preparation, and mathematical modelsand simulations.

Susan B. Hopper, PhD, is an Innova-tive Learning Consultant and facilitatesteacher preparedness to effectively teachin a digital, connected world. She hasmore than 24 years of experience ineducation as a teacher, trainer, admin-istrator, and consultant. While earningher PhD she worked for the Institutefor the Integration of Technology intoTeaching and Learning in the Depart-ment of Learning Technologies at theUniversity of North Texas on NationalScience Foundation (NSF)-funded proj-ects and taught undergraduate/gradu-ate courses as a teaching fellow. Herresearch interests include preservice andin-service teacher training, simulations,technology integration, and globallearning. Please address correspondenceto Susan B. Hopper, University ofNorth Texas, Learning Technologies,Denton, TX, USA. E-mail: [email protected]

Rhonda Christensen, PhD, is a researchscientist in the Learning TechnologiesDepartment in the College of Informationat the University of North Texas. She isan associate director of the Institute forthe Integration of Technology into Teach-ing and Learning at UNT. She is a co-principal investigator for the GoingGreen! Middle Schoolers Out to Save theWorld (MSOSW) project funded by theU.S. National Science Foundation (NSF)Innovative Technologies Program. Herresearch interests are the impact of tech-nology integration in education, preser-vice and in-service technology integrationeducation, enhancing STEM education inmiddle schools, and mobile learning ineducation.

Dr. Tandra Tyler-Wood is a professor inlearning technologies at the University ofNorth Texas. Her primary area ofresearch interest is applying technology tofacilitate the learning of special popula-tions in STEM. She serves as a co-directorfor the Institute for the Integration ofTechnology into Teaching and Learning,along with Drs. Rhonda Christensen andGerald Knezek.

Associate Professor David C. Gibson isDirector of Learning Engagement at Cur-tin University in Perth, Australia. Heworks as a thought leader, researcher,learning scientist, professor, and innova-tor. Gibson’s research focuses on gamesand simulations in education, learninganalytics, complex systems analysis, andthe use of technology to personalize learn-ing. He has published 8 books, 14 chapters,and more than 60 articles and presenta-tions on these topics. He is the creator ofsimSchool, a classroom flight simulator forpreparing educators, and eFolio, an onlineperformance-based assessment system andprovides vision and sponsorship for CurtinUniversity’s Challenge, a mobile, game-based learning platform.

ReferencesAllinder, R. M. (1994). The relationship between

efficacy and the instructional practices of specialeducation teachers and consultants. TeacherEducation and Special Education, 17(2), 86–95.

Argyris, C. (1980). Inner contradictions of rigorousresearch. New York, NY: Academic Press.

Argyris, C., & Schon, D.A. (1974). Theory inpractice: Increasing professional effectiveness. SanFrancisco, CA: Jossey-Bass.

Bandura, A. (1994). Self-efficacy. In V. S.Ramachaudran (Ed.), Encyclopedia of humanbehavior (Vol. 4, pp. 71–81). New York, NY:Academic Press.

Bandura, A. (1997). Self-efficacy: toward a unifyingtheory of behavioral change. PsychologicalReview, 84, 191–215.

Bialo, E. R., & Sivin-Kachala, J. (1996). Theeffectiveness of technology in schools: Asummary of recent research. School LibraryMedia Quarterly, 25(1), 51–57.

Bloom, B., Mesia, B., & Krathwohl, D. (1964).Taxonomy of educational objectives, Theclassification of educational goals: Handbook II,Affective domain. New York, NY: David McKay.

Borman, G. D., & Dowling, N. M. (2008). Teacherattrition and retention: A meta-analytic andnarrative review of the research. Review ofEducational Research, 78(3), 367–409.

Christensen, R., Knezek, G., Patterson, L.,Wickstrom, C., Overall, T. & Hettler, L. (2007).Early Experiences with SimMentoring: FromVirtual to Real Teaching. In R. Carlsen et al.(Eds.), Proceedings of Society for InformationTechnology & Teacher Education InternationalConference 2007 (pp. 1186–1188). Chesapeake,VA: AACE. Retrieved from http://www.editlib.org/p/24719.

Christensen, R., Knezek, G., Tyler-Wood, T., &Gibson, D. (2011). SimSchool: An onlinedynamic simulator for enhancing teacher



http://www.editlib.org/p/24719


preparation. International Journal of LearningTechnology, 6(2), 201–220.

Conchran-Smith, M., & Zeichner, K. M. (2005).Studying teacher education: The report of AERApanel on research and teacher education.Mahwah, NJ: Erlbaum.

Cohen, J. (1988). Statistical power analysis for thebehavioral sciences (2nd ed.). Hillsdale, NJ:Erlbaum.

Darling-Hammond, L. (2000). Teacher quality andstudent achievement. Education Policy AnalysisArchives, 8(1).

Darling-Hammond, L. (2006). Powerful teachereducation: Lessons from exemplary programs(pp. 21–25). San Francisco, CA: John Wileyand Sons.

Darling-Hammond, L., Chung, R., & Frelow, F.(2002). Variation in teacher preparation: Howwelldo different pathways prepare teachers to teach?Journal of Teacher Education, 3(4), 286–302.

Darling-Hammond, L., Wei, R. C., Andree, A.,Richardson, N., & Orphanos, S. (2009).Professional learning in the learning profession.Washington, DC: National Staff DevelopmentCouncil.

DeVellis, R. F. (1991). Scale development. NewburyPark, CA: Sage.

Duffin, L. C., French, B. F., & Patrick, H. (2012).The teachers’ sense of efficacy scale: Confirmingthe factor structure with beginning pre-serviceteachers. Teaching and Teacher Education, 28(6),827–834.

Gibson, D. (2007, June). simSchool—A complexsystems framework for modeling teaching andlearning. Paper presented at the NationalEducation Computing Conference, Atlanta, GA.

Gibson, D., Christensen, R., Tyler-Wood, T., &Knezek, G. (2011). simSchool: Enhancingteacher preparation through simulatedclassrooms. In Proceedings of Society forInformation Technology & Teacher EducationInternational Conference 2011 (pp. 1504–1510).Chesapeake, VA: AACE.

Gibson, D., & Kruse, S. (2012). Learning to teachwith a classroom simulator. In P. Resta (Ed.),Society for Information Technology & TeacherEducation International Conference. Chesapeake,VA: Association for the Advancement ofComputing in Education (AACE). Retrievedfrom http://editlib.org/p/39733

Gibson, D., Riedel, E. & Halverson, B. (2006).Survey of teaching skills. Retrieved from http://simschool.org

Gibson, S., & Dembo, M. H. (1984). Teacherefficacy: A construct validation. Journal ofEducational Psychology, 76, 569–582.

Henson, R. K. (2001). The effects of participation inteacher research on teacher efficacy. Teachingand Teacher Education, 17(7), 819–836.

Hopper, S., Knezek, G., Christensen, R., Tyler-Wood, T., & Gibson, D. (2014). The relationshipof personality traits to perceptions of teachingeffectiveness for teacher preparation candidatesin a simulated classroom environment. In M.Searson & M. Ochoa (Eds.), Proceedings ofSociety for Information Technology & TeacherEducation International Conference 2014 (pp.1411–1416). Chesapeake, VA: Association forthe Advancement of Computing in Education(AACE).

Hopper, S., Knezek, G., & Christensen, R. (2013).Assessing alignment of pedagogical experienceand confidence in a simulated classroomenvironment. In R. McBride & M. Searson(Eds.), Proceedings of Society for InformationTechnology & Teacher Education InternationalConference 2013 (pp. 2870–2876). Chesapeake,VA: Association for the Advancement ofComputing in Education (AACE).

Ingersoll, R. M., & Merrill, L. (2013). Seven trends:The transformation of the teaching force.Philadelphia, PA: Consortium for PolicyResearch in Education.

Ingersoll, R., Merrill, L., & May, H. (2014).Whatare the effects of teacher education andpreparation on beginning teacher attrition?CPRE research report RR-82. Philadelphia, PA:Consortium for Policy Research in Education.doi:10.12698/cpre.2014.rr82

Knezek, G., & Christensen, R. (2009). Preserviceeducator learning in a simulated teachingenvironment. In I. Gibson, R. Weber, K.McFerrin, R. Carlsen & D. Willis (Eds.),Proceedings of Society for InformationTechnology & Teacher Education InternationalConference 2009 (pp. 938–946). Chesapeake, VA:Association for the Advancement of Computingin Education (AACE).

Lortie, D.C. (1975). School teacher: A sociologicalinquiry. Chicago, IL: University of Chicago Press.

McCrae, R.R., & Costa, P.T. (1996). Toward a newgeneration of personality theories. Theoreticalcontexts for the five-factor model. In J. S.Wiggins (Ed.), The five factor model ofpersonality: Theoretical perspectives (pp. 51–87).New York, NY: Guilford.

National Center for Education Statistics. (2008).Schools and Staffing Survey (SASS) and TeacherFollow-Up Survey (TFS) [Data file]. Washington,DC: U.S. Department of Education.

National Commission on Teaching andAmerica’sFuture. (1996).Whatmatter’s most: Teaching forAmerica’s future. Retrieved fromwww.namodemello.com.br/pdf/tendencias/whatmattersmost.pdf

National Council of Accreditation of TeacherEducation. (n.d.).What makes a teachereffective? Retrieved from http://www.ncate.org/Public/ResearchReports/TeacherPreparationResearch/WhatMakesaTeacherEffective/tabid/361/Default.aspx

Perda, D. (2013). Transitions into and out ofteaching: A longitudinal analysis of early careerteacher turnover (Unpublished doctoraldissertation). University of Pennsylvania,Philadelphia, PA.

Tschannen-Moran, M., Woolfolk Hoy, A., & Hoy,W. K. (1998). Teacher efficacy: Its meaning andmeasure. Review of Educational Research, 68,202–248.

Tyler-Wood, T., Knezek, G., & Christensen, R.(2007). Demographic, learning style, andattitude towards technology differences betweenonline and face-to-face students in anintroductory level special education course. In R.Carlsen, K. McFerrin, J. Price, R. Weber, & D.Willis (Eds.), Proceedings of Society forInformation Technology & Teacher EducationInternational Conference 2007 (pp. 526–529).Chesapeake, VA: AACE. Retrieved from http://www.editlib.org/p/24594

Woolfolk Hoy, A. E., & Hoy, W. K. (1990).Prospective teachers’ sense of efficacy and beliefsabout control. Journal of Educational Psychology,82(1), 81.

Woolfolk Hoy, A., & Murphy, P. (2001).Teaching educational psychology to theimplicit mind. In B. Torff & R. J. Sternberg(Eds.), Understanding and teaching theintuitive mind (pp. 145–185). Hillsdale, NJ:Lawrence Erlbaum Associates.


Knezek et al.

http://simschool.org

http://simschool.org

http://www.namodemello.com.br/pdf/tendencias/whatmattersmost.pdf

http://www.namodemello.com.br/pdf/tendencias/whatmattersmost.pdf

http://www.ncate.org/Public/ResearchReports/TeacherPreparationResearch/WhatMakesaTeacherEffective/tabid/361/Default.aspx






Appendix: Survey of Teaching Skills (Gibson, Riedel, & Halverson, 2006)



Assessing Pedagogical Balance in a Simulated Classroom ...Assessing Pedagogical Balance in a...

Documents

Transcript of Assessing Pedagogical Balance in a Simulated Classroom ...Assessing Pedagogical Balance in a...