Evaluation of the F-22 Raptor and G-Force e-Learning Module

Introduction

The e-learning module, The F-22 Raptor and G-Force, was designed for the Junior Test Pilot School project for the Air Force Flight Test Museum at Edwards Air Force Base, which is the Air Force’s flight test center. The purpose of the project is to fulfill the museum’s science, technology, engineering, and math (STEM) educational outreach mission, and the intended learners are elementary school children, specifically 3rd to 5th graders. By creating modules that teach STEM lessons using the history and the operations at Edwards Air Force Base and examples in flight and flight test, the museum can meet the Air Force’s commitment to promote interest in STEM subjects and the museum’s purpose to preserve the history of Edwards, the people that have been there, and the remarkable advancements in flight and aerospace that have been established there. As the learners work through each module, they will earn a patch. This mirrors the actual experience of a test pilot and the flight test engineer as he or she completes a special mission or project.

This module was designed to be effective as an independent learning module; however, the hope is that it will be used by teachers in a classroom setting or as homework as well. Using video, audio, and interactive activities, the module presents instruction on the F-22 fighter jet, test pilots, velocity, acceleration, force, Mach speed, the effects of high g-force, and the protection needed by test pilots to prevent g-induced loss of consciousness.

The e-learning module was not designed with motivation in mind; however it was designed using learning theory principles which include generating interest, making the learning meaningful and relevant, and using engaging activities to stimulate attention and retention. These learning theory principles -- found in constructivism, schema theory, and situated cognition – are necessary components in many theories of motivation. Yet, other elements such as social learning, modeling, intrinsic motivation, goal mastery, and attributional feedback are not directly addressed.

In order to improve the module, it was evaluated for motivation using a specific rubric designed to measure motivational elements in the instruction from the social cognition theoretical perspective (Appendix A). The elements of the rubric -- analysis, design, development, implementation, and evaluation (ADDIE) -- are addressed in their own section with references to examples of how the instruction meets the required criteria, and in instances where it is not met, how it may be improved.

ANALYSIS

Questions

Yes

No

Recommendation/Identification of the element in the instruction.

Learner Analysis

Does the learner analysis measure interest in the subject material?

X

The initial learner analysis includes the question: Would you like to learn more about the F-22 and fighter jets?

Does the learner analysis indicate what type of activities the learner prefers?

X

Include a question in the initial learner analysis which would ask the learner to identify what types of activities he or she would prefer with specific examples.

Does the learner analysis indicate the positive attitude of the learner to the subject?

X

The learner analysis includes the question: Do you think it would be fun to learn more about testing how fast airplanes go?

Does the learner indicate that he or she finds the subject relevant?

X

Include a question to ask if the learner believes that this subject matter can relate to the learner.

The initial survey for the learner analysis does not address two main components of the social cognition theory of motivation, but does address interest and attitude. Interest has been found by many researchers to be an intrinsic motivator and is a subject that is addressed in many learning theories (Schunk, Meece, and Pintrich, 2014). It is an important component to the instruction and can provide the motivation necessary to engage the learner and create self-efficacy or a “personal factor” (Schunk, Meece, and Pintrich, 2014, p. 128). The positive attitude of the learners is also measured. Research by Csikszentmihalyi and others show that positive regard and attitude toward a subject is a key factor in the ability for deep learner engagement (Schunk, Meece, and Pintrich, 2014). In social cognition specifically, valence, mood, and the psychological situation all affect goal achievement and expectancy value with positive attitude yielding higher efficacy.

Recommendations for redesign:

The two areas that need to be addressed in the learner analysis to maximize entering motivation are relevance of the material and the preferred activities of the learners. If the material is not perceived as relevant, the expectant value of the learning will be low. To stimulate the learner’s value of the material, a question should be included in the survey. An appropriate question for this age group could be one that would elicit a positive response, like: Have you ever been driving in the car and you go over a bump and feel weightless? That’s g-force. Do you think that learning more about what g-force is would help you to understand why that happens? The instructional material does have many examples that link the material to the learner’s personal experience (Appendix C, p. 41-44); however to promote motivation, the relevancy of the learning material should be illustrated at the beginning of the instruction.

Also, along these same lines, determining what type of activities the learner prefers should be determined as a part of the initial analysis. The learners do need to be asked. To re-design the instruction, the learner test group should be given a list of activities and asked to rank them. Instructional activities should be designed around the answers.

EX: What type of activities do you prefer? Rank the following in the order you like best. 1 is your favorite and 6 is your least favorite.

Evaluation of the F-22 Raptor and G-Force e-Learning Module

____ Watching videos

____ True/false questions

____ Word Problems

____ Matching exercises

____ Seeing comparisons

____ Group activities

DESIGN

Questions

Yes

No

Recommendation/Identification of the element in the instruction.

Instructional Goal

Does the leaner have the prerequisite skill set and/or knowledge to achieve the instructional goal?

X

The analysis revealed that the learners did not; however, the instructional material was designed to teach all of the pre-requisite material to reach the instructional goal.

If not, do the performance objectives adequately prepare the learner to be able to master the goal?

X

See Table 4 on page 6 of Appendix C.

Is there a clear and meaningful objective stated for the goal? EX: At the end of this module, you will evaluate an instructional design for efficacy.

X

Yes, the purpose of goal is stated at the beginning of the instructional module. (Appendix C, p. 16)

Recommendations for redesign:

The instructional goal is clearly stated and the instruction provides for the learner to be able to reach the goal. “From a motivational perspective, goals of moderate difficulty—those learners perceive as challenging but attainable—are most effective…” (Schunk, Meece, and Pintrich, 2014, p. 141). The objective for attaining the goal is also clearly defined. However, the objective stated for the goal could be amended to add for more relevancies. Including a statement about how the learning can be applied back to one’s life and physical responses to the force of gravity would be helpful in providing additional motivation.

Questions

Yes

No

Recommendation/Identification of the element in the instruction.

Performance and Site Context

Does the site where the instruction is taking place provide for the planned activities (computers, desks, tables for group work)?

X

This instruction is designed to be in an independent online environment. It does not currently provide for any social interaction or group work, therefore it provides adequately for the current context of the instruction.

Does the site provide for the ability to have a variety of learning activities?

X

The e-learning instructional environment includes a variety of learning activities.

Does the site have adequate lighting and seating?

X

The site is online and therefore provisions for lighting and seating would be provided by the learner. The assumption is that the learner would have adequate lighting and seating by choice.

Is the site’s environment comfortable?

X

The assumption is that the learner would choose a comfortable environment.

Currently, the instruction is completely independent and is designed for the learner’s individual, self-paced learning in a virtual environment. Because of this, the performance context currently meets the needs of the instruction. In the social cognition theory of motivation performance context is not addressed. However, there has been much written about the environment of the learner affecting the ability to learn well and retain the information.

Recommendation for redesign:

In order to improve motivation in the instruction, the recommendation is to provide additional activities that promote social learning, opportunities for reciprocal teaching, modeling, and mentoring. This would require that the instruction be conducted in a classroom which included the ability to move the desks or utilize tables. The room should be brightly lit, a comfortable temperature, and have adequate seating. The e-learning portion of the instruction can be used as a homework assignment if there are no computers in the classroom available.

DEVELOPMENT

Questions

Yes

No

Recommendation/Identification of the element in the instruction.

Instructional Activities

Do they provide for choice or learner control?

X

The module was designed to be worked through progressively; however it can be navigated by individual objectives.

Do they offer the opportunity for the learner to discover or construct some of his or her own learning using guidelines and a stated goal?

X

At the end of each lesson, there are open-ended critical thinking questions for the learners to answer which relate the material to a personal experience. Also, in the Mach speed lesson, there is an exploratory math problem.

Are they varied to provide for surprise and disequilibrium?

X

The module contains many videos, audio files, and varied instructional strategies to provide for surprise.

Are they activities challenging but attainable?

X

All of the instructional activities can be achieved. They are challenging, but the activities are designed to be mastered.

Do they include group work to problem-solve?

X

There are no group activities.

The instructional activities meet the criteria of the social cognition theory in all but the social aspects. The requirement of learner control is adequately met. The learner has multiple opportunities to construct some of his or her own learning, and there are varied activities to provide for surprise. However, disequilibrium is not provided for. This can be done further in the independent online environment, but it is not ideal and beyond the capacities of the instructional designer.

Recommendation for redesign:

The module provides a sound instructional base for social learning which can be achieved in a classroom environment. An instructor can incorporate fun group activities which will provide variety, disequilibrium, and opportunities for social learning. In the attached lesson plan, the instruction includes a jumping exercise which illustrates g-force, demonstration of a blood pressure cuff to give an example of what a g-suit feels like, and a group activity to calculate distances from one point to another going Mach speed (Appendix B).

IMPLEMENTATION

Questions

Yes

No

Recommendation/Identification of the element in the instruction.

Implementation

Is the instructor positive and enthusiastic?

X

The voice and the programmed feedback of the module are; however, there is no personal feedback.

Does the leaner have the opportunity to adequately absorb the material prior to the activities?

X

Appendix C, p. 18-80

Do the learners have the opportunity to observe the learning they should be modeling?

X

X

There is no modeling in the activities.

Does the instructor provide positive feedback while the learners progress through the instruction?

X

There is no instructor.

Does the instruction provide for rewards?

X

The learners earn patches for successful completion of the module.

Based in the research on imitation and social learning theory that asserts that behavior is learned in a social setting, social cognition theory in motivation asserts that learners achieve performance and learning goals through collaboration, modeling, reciprocal teaching, and collective self-efficacy (Schunk, Meece, and Pintrich, 2014). The e-learning module achieves many of the criteria of motivation like rewards, adequate time for learning, and an enthusiastic ‘teacher’ in the voice of the program and presentation of the materials; however it does not execute the cornerstone of social cognition theory as previously stated. The program fails in the provision for group learning and authentic attributional feedback.

Recommendation for redesign:

To improve the motivational strategies in the implementation of the instruction, an instructor and a group learning environment are necessary. To this end, an instructor’s manual or lesson plan is needed. The learners can complete the module as homework, and additional activities can be done in two one-hour classes which will allow for the instructor to model enthusiasm for the subject, have collaborative group work, individual teaching in the form of presentations, and provide feedback.

EVALUATION

Questions

Yes

No

Recommendation/Identification of the element in the instruction.

Evaluation

Are the students given a pre-test and post-test to measure learning?

X

At the beginning of the module is a required pre-test to measure entry knowledge. A post-test is given at the end of the instruction which has the same questions to measure learning. Neither can be retaken.

Are the students given a survey to measure attitude at the end of the instruction?

X

No attitude survey is given in the module.

The learners should be able to master all of the lessons. While the concepts are unfamiliar to the learners, they are relatively simple concepts. The students should achieve the goals, will be able to see that they have, and their self-efficacy will rise. The goal is that their intrinsic motivation will also rise and their affinity and motivation for the STEM subjects and engaging in future learning will rise as well. In social cognition theory research, goal accomplishment does raise self-efficacy (Schunk, Meece, and Pintrich, 2014) and the assumption that intrinsic motivation and value will increase is reasonable.

Recommendation for redesign:

Currently there is no measurement of attitude at the end of the instruction. This can be easily remedied with the inclusion of a survey at the end of the instructional module or by the instructor administering this in class. For the redesign proposal, the lesson plan is attached which reflects the inclusion of the quick survey (Appendix B).

CONCLUSION

The F-22 and G-Force e-learning module does not meet all of the requirements to provide the best strategies to stimulate motivation in the learners as an independent, online module. However, it does satisfy sixteen of the twenty-three strategies that maximize motivation (approximately 70%). In a virtual environment and for the requirement for this module to be independent of a classroom and a personal instructor, the motivational strategies imbedded in the instructional module are sufficient for the needs of the program.

There is no provision that the module cannot be adapted to be used by an instructor in a classroom environment. An instructor facilitated lesson can be developed to provide for the motivational strategies which are lacking. The proposal for an instructor’s manual (IM) is attached (Appendix B). This IM provides additional activities which employ the appropriate motivational strategies which will optimize the learning experience for the learners.

The purpose of this module is to make learning fun and engaging for students in the elementary grades so that their expectancy of success and positive attitude in STEM subjects is raised while teaching about flight test. Creating additional materials that can optimize motivation through instructor’s lessons supports the mission of the Air Force Flight Test Museum. It is my intent to create instructor’s manuals which use the strategies in motivation theory for all of the modules that are developed in the program.

REDESIGN

Appendix B is the proposed instructor’s manual to use the e-learning module in a classroom environment. This was created after the initial instruction was evaluated for motivation. It contains additional activities and strategies which successfully create instruction that meets the motivation strategy of social cognition theory. Below is the revised rubric to reflect the modifications made to the instruction. This indicates specifically how the additional activities meet the criteria for improving motivation to learn.

Questions

Yes

No

Evidence

Learner Analysis

Does the learner analysis measure interest in the subject material?

X

Appendix C, p. 4

Does the learner analysis indicate what type of activities the learner prefers?

X

Appendix B, p. 2

Does the learner analysis indicate the positive attitude of the learner to the subject?

X

Appendix C, p. 5.

Does the learner indicate that he or she finds the subject relevant?

X

Appendix B, p. 2.

Instructional Goal

Does the leaner have the prerequisite skill set and/or knowledge to achieve the instructional goal?

X

The analysis revealed that the learners did not (Appendix C, p. 4); however, the instructional material was designed to teach all of the pre-requisite material to reach the instructional goal.

If not, do the performance objectives adequately prepare the learner to be able to master the goal?

X

Appendix C, Table 4, p. 6.

Is there a clear and meaningful objective stated for the goal? EX: At the end of this module, you will evaluate an instructional design for efficacy.

X

Appendix C, p. 16.

Performance and Site Context

Does the site where the instruction is taking place provide for the planned activities (computers, desks, tables for group work)?

X

This instruction is designed to be in an independent online environment, Appendix C, p. 3.. In class, yes, Appendix B, p. 3.

Does the site provide for the ability to have a variety of learning activities?

X

Appendix B, p. 3.

Does the site have adequate lighting and seating?

X

The online portion does. It is assumed that the classroom will as well.

Is the site’s environment comfortable?

X

The assumption is that the learner would choose a comfortable environment while working through the module.

Instructional Activities

Do they provide for choice or learner control?

X

Appendix B, p. 4 & 5; Appendix C, p. 17.

Do they offer the opportunity for the learner to discover or construct some of his or her own learning using guidelines and a stated goal?

X

Appendix B, p. 4 & 5; Appendix C, p. 23, 41, 44, 45, 66, & 80.

Are they varied to provide for surprise and disequilibrium?

X

The module contains many videos, audio files, and varied instructional strategies to provide for surprise, Appendix C, p. 18-80. Appendix B, p. 5 & 6.

Are they activities challenging but attainable?

X

All of the instructional activities can be achieved. They are challenging, but the activities are designed to be mastered.

Do they include group work to problem-solve?

X

Appendix B, p. 3 & 4.

Implementation

Is the instructor positive and enthusiastic?

X

Appendix B, p. 2 & 4.

Does the leaner have the opportunity to adequately absorb the material prior to the activities?

X

Appendix C, p. 18-80

Do the learners have the opportunity to observe the learning they should be modeling?

X

Appendix B, p. 3, 4, 5, 6.

Does the instructor provide positive feedback while the learners progress through the instruction?

X

Appendix B, p. 3 and 5.

Does the instruction provide for rewards?

X

The learners earn patches for successful completion of the module. Appendix B, p. 2 & 6.

Evaluation

Are the students given a pre-test and post-test to measure learning?

X

At the beginning of the module is a required pre-test to measure entry knowledge. A post-test is given at the end of the instruction which has the same questions to measure learning. Neither can be retaken.

Are the students given a survey to measure attitude at the end of the instruction?

X

Appendix B, p. 6.

References:

Brown, L.S. (2013). The F-22 Raptor and G-Force e-Learning Module. https://dl.dropboxusercontent.com/u/51200747/story.html

Schunk, D.H., Meece, J.L., & Pintrich, P.R. (2014) . Motivation in Education: Theory, research, and applications, 4e. Boston, MA: Pearson.

Appendix A: Initial Analysis for Motivation

Appendix B: Instructor’s Manual for using The F-22 Raptor and G-Force e-Learning module in a classroom environment for 3rd to 5th grade class.

Appendix C: Paper prototype for the F-22 Raptor and G-Force e-Learning Module (final version of the module is online)