FORMATIVE EVALUATIONpatstrawser.weebly.com/.../41538517/formative_evaluation.pdf · 2019. 11....

FORMATIVE EVALUATION

Table of Contents

Introduction ......................................................................................... 3

Alpha Test .............................................................................................. 3

Beta Test ................................................................................................. 5

Assessment Questions ..................................................................... 7

Evaluation ............................................................................................. 9

Evaluation Results ............................................................................15

3

INTRODUCTIONThis Formative Evaluation report details the process by which the Band Orientation Program for Almost Elton John and the Rock Band was evaluated and tested. This review includes data on a the Alpha and Beta tests for the Program, detailing the Objectives of the tests, the audience, the criteria, the process, and results and feedback.

ALPHA TEST

Objective

The objective of the Alpha test was to test basic functionality of the course and the medium of the course. This included testing for usability of the slides, continuity of content, navigation, audio and video quality, and other technical issues.

Audience

Alpha testers were comprised of:• A Subject Matter Expert (SME) • An instructional design expert• Three instructional design graduate students

Criteria

All Alpha testers were asked to evaluate the program for clear statement of objectives and instructional goals, content quality and accuracy, content continuity, relevance, and scope. The testers were asked to pro-vide feedback in the form of a written report.

Process

The Alpha testers were instructed to take the program by following a weblink provided by email. The testers were provided with a password to access the program, and informed that running time for the program was approximately 1 hour.

4

Results and Feedback

Based on the comments from the Alpha testers, most were impressed with the overall look and layout of the slides and graphics. The text was consistent and easy to read, and the voiceover was clear and easy to under-stand.

The suggested revisions revolved largely around technical aspects of Captivate, grammatical and spelling errors, some content accuracy, and the functionality of the assessment section. Also, the videos required some editing to remove some noise and false starts from the footage. The overall assessment, though, was positive.

Below are excerpts from the reports of two of the Alpha testers:

Subject Matter ExpertWe need to edit some of the clips. Misspelling: ADVISE should be spelled with a C ADVICE. LEATHERHEADS is misspelled. ID Thief should be IDENTITY THIEF; DESERT SYMPHONY – Palm is not a part of their name. I need to get you a current Stage Plot. The do’s and don’ts are hysterical!!!Maybe trim the interviews? They go on a bit long. I was getting bored listening to myself! Brevity is soul of wit! Rehearsals are paid at $25 per rehearsal – SLIDE 20. Travel Tips: Some of the onscreen writing didn’t show on the second screen. Liquids, gels are allowed in 3 oz containers – SLIDE 25

Instructional Design ProfessionalOn the initial loading screen: you can speed things up by going to Preferences -> Project -> Start and End, checking Preloader, and setting Preloader % to something like 15. This tells Captivate to start running the project as soon as it has loaded 15% of the content, instead of waiting until 100% is loaded. For users with slow connections, this can save quite a bit of time. You can tweak other Project settings to improve perfor-mance as well.

I noticed at least one slide, I think the first Introduction slide, advanced to the next slide by itself when I had to leave the tutorial for a while. You can prevent this by including an invisible click box on each slide with the option “Pause project til user clicks.” You can see examples of this in the Arkansas files.

Some typos...Introduction slide -- replace “it’s” with “its”; Mission statement slide -- commas go inside quota-tion marks; Vision slide -- period goes inside quotation marks

Obviously, the videos need some edits to remove his flubs and false starts. It would be good to have the con-tract PDF open in a new window rather than the same window as Captivate. Some users might have trouble navigating back to the tutorial from the PDF. On the assessment questions, I was only able to attempt each question once as I went through -- probably intentional?

Question 3 -- response J is cut off. This seems like a weak question because there are too many options. Question 8 -- response D is cut off I answered 73% of the questions correctly and this was a failing score -- probably intentional. Clearly I’m not ready for the Rocket Band yet!

5

BETA TEST

Objective

The objective of the Beta test was to further check for functionality of the course, as well as to assess aspects of the program related to subject matter, auxiliary information, interface, navigation, and pedagogy.

Audience

The Beta testers were comprised of:• Three Instructional design graduate students• Two Current members of the Rocket Band (who are the intended audience for the program)

Asseessment Criteria and Instruments

All Beta testers – like the Alpha testers – were asked to evaluate the program for clear statement of objec-tives and instructional goals, content quality and accuracy, content continuity, relevance, and scope. Addi-tionally, they were to evaluate the program relevance of the subject matter, auxiliary information, interface design and functionality, navigation, and pedagogy.

The instrument used for assessment was a Google Form survey, which was linked to the program.

Procedures

Beta testers were instructed to go through the entire program. On the final slide, the testers were to follow a hyperlink to the Google Form. The survey asked the testers to rate the program on a scale of 1 to 5 in terms of it’s level of effectiveness in a number of areas (subject matter, auxiliary information, interface, navigation, and pedagogy). For each area, the testers was given the opportunity to give feedback in the form of written comments.

Limitations of Beta Test

There are a number of important limitations that must pointed out pertaining to the Beta test.

To start, we were unable to find any new band members to go through the program. While the feedback from existing band members is certainly useful, the other intended audience is band members who are not already acquainted with the normal operating procedures of the organization.

Another limitation was in the format of the survey. It was mentioned by a number of the Beta testers that the survey was too long, and that certain questions seemed irrelevant or confusing. While there were fields available for comment feedback, we did not receive as much comment feedback as we would have liked.

6

Results Feedback

Most of the feedback indicated that the testers found the program engaging, informative, functional, aes-thetically attractive, and entertaining. Testers indicated that the program appeared to be an interesting and unique way to understand the subject matter and to learn the various learning objectives.

Criticisms of the program included largely technical matters, such as slide duration, design anomolies, and issues with the assessment questions. There were also comments about the Google Form Evaluation Survey itself, which had two of the ratings headings switched around. Overall, the feedback again was very positive.

Comment Feedback on Subject MatterI am not really an expert on this subject. -intro slide was a bit quick -code of conduct video bits...there was a ghostly image and slide bar in the middle...a remnant of something that had been removed? -assessment section, contracts...”reresponsibility” instead of responsibility. I learned a lot from the basic info in your pre-sentation. Since I’m not a musician I can’t reliably comment on tech jargon. I think language and layout and design fit your target audience well. -did u switch places on the good/acceptable scale? from first set of?s to next sets of ?s? i consider good better than acceptable on the poor to excellent scale... It was absolutely great. The interviews were informative but I felt a sense of wanting them to be expedited.

Comment Feedback on Auxiliary Informationi didn’t check out the links but they looked to be related to the subject matter. the poor/needs work/good/acceptable/excellent arrangement still bothers me...think good/acceptable should be reversed, (isn’t good better than acceptable?) but that’s just me. I only put needs work because I am unsure where your help fea-tures were. Is cc considered a help feature, is the toc considered a help feature?

Comment Feedback on InterfaceN/A

Comment Feedback on Navigationi did not try to bookmark anything or restart the training and just went straight thru.... but i loved the simple straightforward navigation bar you used. it looked good and was easy to use.

Comment Feedback on PedagogyIf I was a musician, hired to tour with the band, this module would give me a good foundation to function appropriately in the band. Good tips and clear goals. Excellent use of photography without being distract-ing. Very professional layout. I answered the quiz at the end and it said that I failed showing that I didn’t even get one answer correct - I am sure that it was an error. Not sure why it would not let me start again. I did not hit “submit” after the last question because I did not see it clearly. I suppose that this is why I FAILED. GREAT presentation - makes me very proud to be a member of the Rocket Band!!!

7

ASSESSMENT QUESTIONSPlease answer the following questions. (The correct answers are in red)

1. The mission statement of Almost Elton John and the Rocket Band is: A. Be the best Elton John Tribute in the United States B. Celebrate the songs and career of Elton John C. Help introduce people to music of the 1970’s D. Play with symphony orchestras

2. Members of the Rocket Band: A. Are all “hired guns” and are there to just play another gig B. Have equal billing and prominence in the Almost Elton John show C. Aren’t very sociable because they work so hard D. Are like a family

3. Which of the following are acceptable behaviors for members of the band? Please select all that apply. • Get to the hotel and spend quiet time alone. • Show up at least 10 minutes early to all band engagements. • It’s okay to be difficult to deal with, as long as you’re a really good musician. • Show up to the performance in blue jeans and make faces at the audience. • Ask the band leader if you can have a solo number in the show. • Ask the band leader how you can help improve a certain part of the show. • Make silly jokes during a van ride. • Pass out personal business cards during the intermission at a performance. • While in flight to engagement, invite other people to a concert. • Demand that you have your own hotel room because you have 20 years of experience in the industry. • You have to miss a rehearsal, but you called two weeks ahead of time to let the band leader know. • You have to miss a concert that you are under contract to perform because a better paying offer came through, but you called two weeks ahead of time to let the band leader know.

4. True or False: Craig views his band as a group of highly-trained professionals who are in the group to simply do a job, nothing more.

5. Complete the following sentence:Craig is interested in creating ____________________ in the lives of those around him. A. Careers B. Opportunities C. Purpose and Meaning D. Cheesecake

6. Contracts are: A. Important because they obligate you to certain responsibilities with the band B. Important because all bands have contracts C. Negotiable and can be broken at any time with no fear of recompense or termination D. Optional

7. During air travel, it is recommended that you: A. Meet the band at the flight gate

8

B. Meet the band at the security gate C. Meet the band at curbside dropoff D. Meet the band at the Delta Sky Priority Desk

8. Showing up to soundcheck is: A. Highly recommended, but not mandatory. B. Completely optional and you can stay at the hotel if you like. C. Mandatory for all members of the band. D. Not necessary if you’re a backup singer because you don’t have equipment to setup.

9. Craig believes in __________________ the people he works with. A. Challenging B. Simply employing C. Scrutinizing D. Investing in

10. If you have special dietary needs, please make sure they are included in: A. The band schedule and itinerary B. Your per diem C. The production rider D. Your personal contract

11. If you get an email from Craig requiring a time-sensitive reply: A. It’s okay to call another band member B. Wait about a week to get back to him C. Reply as soon as possible. D. Don’t respond

12. True or False: the ideal employee will make the band look good, sound good, and feel good.

9

EVALUATIONA live form of the Evaluation Survey can be found here.

15

Poor 0 0%

Needs Work 0 0%

Acceptable 0 0%

Good 0 0%

Excellent 5 100%

5 responsesPublish analytics

Summary

Learning objectives were clear [1 – SUBJECT MATTER]

Content structure was easy to follow [1 – SUBJECT MATTER]

Edit this formPat Strawser

Band Orientation Evaluation Form - Google Forms https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

1 of 20 4/23/14 7:20 PM

EVALUATION RESULTSA live form of the Evaluation Survey can be found here.

16

Poor 0 0%

Needs Work 0 0%

Acceptable 0 0%

Good 2 40%

Excellent 3 60%

Poor 0 0%

Needs Work 0 0%

Acceptable 0 0%

Good 1 20%

Excellent 4 80%

Poor 0 0%

Needs Work 0 0%

Content was accurate [1 – SUBJECT MATTER]

Language, style, and grammar was correct [1 – SUBJECT MATTER]


2 of 20 4/23/14 7:20 PM

17

Acceptable 0 0%

Good 0 0%

Excellent 5 100%

Poor 0 0%

Needs Work 0 0%

Acceptable 0 0%

Good 0 0%

Excellent 5 100%

Poor 0 0%

Needs Work 0 0%

Acceptable 0 0%

Good 1 20%

Reading level was appropriate for the audience [1 – SUBJECT MATTER]

Cultural bias – language [1 – SUBJECT MATTER]


3 of 20 4/23/14 7:20 PM

18

Excellent 4 80%

Poor 0 0%

Needs Work 0 0%

Acceptable 0 0%

Good 1 20%

Excellent 4 80%

Poor 0 0%

Needs Work 0 0%

Acceptable 0 0%

Good 1 20%

Excellent 4 80%

Cultural bias – reference [1 – SUBJECT MATTER]

Correct use of technical terms and jargon [1 – SUBJECT MATTER]


4 of 20 4/23/14 7:20 PM

19

Poor 0 0%

Needs Work 0 0%

Acceptable 0 0%

Good 1 25%

Excellent 3 75%

Spelling and punctuation is correct [1 – SUBJECT MATTER]

Subject Matter Comments

I am not really an expert on this subject. -intro slide was a bit quick -code of conduct

video bits...there was a ghostly image and slide bar in the middle...a remnant of something

that had been removed? -assessment section, contracts..."reresponsibility" instead of

responsibility. I learned a lot from the basic info in your presentation. Since I'm not a

musician I can't reliably comment on tech jargon. I think language and layout and design fit

your target audience well. -did u switch places on the good/acceptable scale? from first set

of?s to next sets of ?s? i consider good better than acceptable on the poor to excellent

scale... It was absolutely great. The interviews were informative but I felt a sense of

wanting them to be expedited.

Introductory slides are clear, engaging, and informative [2 – AUXILIARYINFORMATION]


5 of 20 4/23/14 7:20 PM

20

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 0 0%

Excellent 5 100%

Poor 0 0%

Needs Work 0 0%

Good 1 20%

Acceptable 0 0%

Excellent 4 80%

Directions are clear [2 – AUXILIARY INFORMATION]

Help features are accessible and easy to use [2 – AUXILIARYINFORMATION]


6 of 20 4/23/14 7:20 PM

21

Poor 0 0%

Needs Work 1 20%

Good 1 20%

Acceptable 0 0%

Excellent 3 60%

Poor 0 0%

Needs Work 0 0%

Good 1 20%

Acceptable 0 0%

Excellent 4 80%

Assessment/quiz questions are reasonable and appropriate for scope ofthe training [2 – AUXILIARY INFORMATION]

Auxiliary Information Comments

i didn't check out the links but they looked to be related to the subject matter. the


7 of 20 4/23/14 7:20 PM

22

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 0 0%

Excellent 5 100%

Poor 0 0%

Needs Work 0 0%

Good 1 20%

poor/needs work/good/acceptable/excellent arrangement still bothers me...think

good/acceptable should be reversed, (isn't good better than acceptable?) but that's just

me. I only put needs work because I am unsure where your help features were. Is cc

considered a help feature, is the toc considered a help feature?

Slide displays are clear and easy to read [3 – INTERFACE]

Presentation modes are appropriate and engaging (i.e. video clips, audioclips, etc.) [3 – INTERFACE]


8 of 20 4/23/14 7:20 PM

23

Acceptable 1 20%

Excellent 3 60%

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 0 0%

Excellent 5 100%

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 0 0%

Text/fonts are clear and easy to understand [3 – INTERFACE]

Assessment/quiz questions are reasonable and appropriate for scope ofthe training [3 – INTERFACE]


9 of 20 4/23/14 7:20 PM

24

Excellent 5 100%

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 0 0%

Excellent 5 100%

Poor 0 0%

Needs Work 0 0%

Good 1 20%

Acceptable 0 0%

Excellent 4 80%

Animation and graphics are smooth and professional [3 – INTERFACE]

Audio is clear and of good quality [3 – INTERFACE]


10 of 20 4/23/14 7:20 PM

25

Poor 0 0%

Needs Work 0 0%

Good 1 20%

Acceptable 0 0%

Excellent 4 80%

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 0 0%

Excellent 5 100%

Video is clear and of good quality [3 – INTERFACE]

Interface Comments

It is easy to navigate the slides within the training [4 – NAVIGATION]


11 of 20 4/23/14 7:20 PM

26

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 0 0%

Excellent 5 100%

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 2 40%

Excellent 3 60%

Navigation icons are clear and consistent [4 – NAVIGATION]

The training can be restarted with no difficulty [4 – NAVIGATION]

Bookmarking within the training is clear and easy to implement [4 –NAVIGATION]


12 of 20 4/23/14 7:20 PM

27

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 1 20%

Excellent 4 80%

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 0 0%

Navigation Comments

i did not try to bookmark anything or restart the training and just went straight thru.... but i

loved the simple straightforward navigation bar you used. it looked good and was easy to

use.

The methods used to present the content are appropriate for theaudience [5 – PEDAGOGY]


13 of 20 4/23/14 7:20 PM

28

Excellent 5 100%

Poor 0 0%

Needs Work 0 0%

Good 1 20%

Acceptable 1 20%

Excellent 3 60%

Poor 0 0%

Needs Work 0 0%

Good 1 20%

Acceptable 0 0%

Excellent 4 80%

The training is interactive and engaging [5 – PEDAGOGY]

The training allows for learners to work together [5 – PEDAGOGY]


14 of 20 4/23/14 7:20 PM

29

Poor 0 0%

Needs Work 0 0%

Good 1 20%

Acceptable 0 0%

Excellent 4 80%

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 1 20%

Excellent 4 80%

The program provides other learning tools and learning strategies [5 –PEDAGOGY]

The program gives user sufficient control (i.e. volume, pause, continue,table of contents, etc.) [5 – PEDAGOGY]

Assessment questions are relevant to the learning objectives [5 –


15 of 20 4/23/14 7:20 PM

30

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 0 0%

Excellent 5 100%

Poor 0 0%

Needs Work 0 0%

Good 1 20%

Acceptable 0 0%

Excellent 4 80%

PEDAGOGY]

Assessment questions encourage the learner to think and process theinformation [5 – PEDAGOGY]

There is variety in the assessment questions (i.e. multiple choice, fill inthe blank, true/false, etc.) [5 – PEDAGOGY]


16 of 20 4/23/14 7:20 PM

31

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 0 0%

Excellent 5 100%

Poor 0 0%

Needs Work 0 0%

Good 1 20%

Acceptable 1 20%

Excellent 3 60%

Answering questions is clear and reasonable [5 – PEDAGOGY]

Questions are not misleading or ambiguous [5 – PEDAGOGY]


17 of 20 4/23/14 7:20 PM

32

Poor 0 0%

Needs Work 0 0%

Good 1 20%

Acceptable 1 20%

Excellent 3 60%

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 1 20%

Excellent 4 80%

Question feedback is constructive and appropriate [5 – PEDAGOGY]

The program teaches valuable information [5 – PEDAGOGY]


18 of 20 4/23/14 7:20 PM

33

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 0 0%

Excellent 5 100%

Poor 0 0%

Needs Work 0 0%

Good 0 0%

Acceptable 1 20%

Excellent 4 80%

Successful completion of the assessment illustrates mastery of theinformation taught [5 – PEDAGOGY]

Pedagogy Comments

If I was a musician, hired to tour with the band, this module would give me a good


19 of 20 4/23/14 7:20 PM

34

foundation to function appropriately in the band. Good tips and clear goals. Excellent use of

photography without being distracting. Very professional layout. I answered the quiz at the

end and it said that I failed showing that I didn't even get one answer correct - I am sure that

it was an error. Not sure why it would not let me start again. I did not hit "submit" after the

last question because I did not see it clearly. I suppose that this is why I FAILED. GREAT

presentation - makes me very proud to be a member of the Rocket Band!!!

Number of daily responses


20 of 20 4/23/14 7:20 PM

FORMATIVE EVALUATIONpatstrawser.weebly.com/.../41538517/formative_evaluation.pdf · 2019. 11....

Documents

Transcript of FORMATIVE EVALUATIONpatstrawser.weebly.com/.../41538517/formative_evaluation.pdf · 2019. 11....