The Online Academy(TOA)
John BaekMichelle Didier
Dan FeinbergLisa Herbert
Danalyn RobinsonHeather White
Overview
The project– An online high school– A partnership between GMU and Northern VA
county school systems
Overview
Review of last semester– Focus on performance analysis, design &
development
Mission Statement• Develop high quality courses, correlated to the
Virginia SOL and COPLS model• Support the expert mentors
Overview
• Redirection of focus
• Focus this semester– Framework for continued development: code library,
site structure, consistent design using style guidelines and templates
– Field testing– Evaluation & usability testing– Complete the Intro module/ Resource guides
Overview
• Our process– Team meetings – SME (teachers) meetings– Project management– Divide & conquer
Presentation Outline
• Evaluation– Evaluation needs & goals– Methods– Findings & recommendations
• Future of TOA– Framework for future development– Project recommendations
Evaluating TOA
Need for Evaluation
Methods of Evaluation
Findings and Recommendations
Need for Evaluation• Inform and direct current and future development
• Examine the usability of the checklist– Organization and self-regulation
• Determine effectiveness of navigation– Locating resources– Intuitiveness—do the learners know where to go next?
• Review the course modules– Clarity of instruction– Engagement and motivation of instruction – Adequacy of the resources, skills, and content for challenge– Pedagogical re-engineering (approach and format)
Evaluation Goals
• Analyze instructional strengths • Determine the effectiveness of the design
and transfer of learning • Evaluate the usability and the benefits of
the checklist to the learner • Locate ease of use problems (navigation)
Methods of Evaluation
• Quality assurance
• Field testing
• One-to-One evaluation
• Expert review
• Case study
Meeting the Evaluation Goals
Instructional strengths
•One-to-one evaluation•Expert review•Case study
Effectiveness of the design and transfer
•Quality assurance•Case study•Expert review
Checklist Usability
•Expert review•Field test
Locate ease of use problems
•Field test•One-to-one evaluation
Quality Assurance
For each course, at least two modules were field tested.
•Chemistry•Mars Matters•Sappy Solutions
•Earth Science•Passing Gas…For Effect Only•Like It…Weather or Not
•Algebra•Graphing – Slopes
•Geometry•Right Triangles/ Trigonometry•Points, Lines, Planes, Segments, and Angles
•English 11•The Two Macks: A Harlem Renaissance Club•Regionalism, Realism, and Nationalism
•World History I•India and China•African Civilizations
•World History II•Imperialism•Between the Wars
Before anything was field tested, the modules were evaluated for Quality Assurance.
• Forms that don’t work• Typing mistakes• JavaScript errors• Form elements not properly labeled• Pages that don’t print correctly• Consistent styles• Missing page titles or code comments
The result is a much more polished looking product.
Field Testing
Field Testing
• 14 Modules
• Technical problems
• Student response
• Self regulation
• Synchronous discussion (Digichat)
Digichat• Encourages peer
interaction• Encourages
mentor-learner interaction
• Mentor office hours
Intro Module
Intro Module• Need for Intro Module• Elements of most
modules• Use basic TOA
functions
Mentor Resources
• Overview• Objectives• Answer keys
Challenge• Problem based context for entire module• Background building knowledge• Culminating activity
The Checklist•Self regulation
•Mentor-student negotiation
•Links to activities
TOA Introductory Module
• Research for the module was gathered from several sources– Teachers’ feedback from their field testing– Results from one-to-one evaluation– Review of related literature
• Maintain theme of the story introduced in the Challenge
Working with a Mentor
Working with a MentorActivity
Being an Online Learner
Being an Online LearnerActivity
Online Research
Online Research
Online Research
One-To-One Evaluations
A face to face meeting with members of our target audience where we ask them to use TOA and give us their opinion on
the site.
• Are the instructions clear?
• Can the user locate resources?
• Can learners navigate?
• Are the resources, skills, and content adequate for completing the challenge?
•Is the instruction engaging and motivating?
•Will the student know the references?
•Is the content at the appropriate level?
•Does the checklist provide self-regulatory benefits?
One-To-One Evaluations
1. Recruit students.
2. Prepare evaluation questions.
3. Design data collection tools.
4. Write script.
5. Set up testing area.
6. Conduct evaluation.
7. Analyze results.
8. Write evaluation report.
One-To-One Evaluation Process
1. Recruit Students
• We were provided access to two students at a Washington, DC private school
• “Ralph” – A 16 year old white male currently enrolled in Chemistry
• “Kelly” – A 16 year old white female who had taken Chinese History
2.Prepare Evaluation Questions
3. Design Data Collection Tools• Tape recorded and
transcribed• Field notes were
taken by interviewer and observer
• Follow up questionnaire was given
4. Write Script
5. Set Up Testing Area6. Conduct Evaluation
7. Analyze Results
8. Write Evaluation Report
World History I Module
World History I Module
World History I Module
World History I Module
World History I Module
World History I Module
World History I Module• Summary of students’ impressions:
– Kelly:• Liked the story presented in the challenge and carried
through the activities• Liked the PowerPoint activities• Thought the activities were explained well• Liked self paced aspects of course
– Ralph:• Navigated using pull-down menus• Liked self paced aspects of course• Unsure of being motivated enough to complete activities• Liked the layout and color scheme of the site
Expert Reviews
Expert Review Steps
1. Recruit experts
2. Prepare questions
3. Design data collection tools
4. Set up testing area
5. Conduct evaluation
6. Analyze results
7. Write evaluation
Dr. Dabbagh’s Interview
• Earth Science Module on Gas
• Open questions based on likes and dislikes
• How learning theories are applied
• Does this address multiple learning styles?
Earth Science
Passing Gas…
The Checklist
An Activity
Dr. Dabbagh’s Comments
• Thought the content was very situated in real life scenarios
• Liked the sub activities under the challenge, how main challenge is umbrella
• Liked the graphics, thought they were appropriate to content
• Liked the embedded links to additional information
Dr. Dabbagh’s Recommendations
• Main module page needs to reflect flow of lessons (ex. If there is an order to a lesson, then having all the links available makes users think they can skip around.)
• Students need to have a better understanding of where they are in the order of things.
• Unclear how the module works, ex. the challenge is actually done last.
• Needs to closely match the checklist with the verbiage of the lessons.
• Include the amount of time available to complete module and the number of hours students are expected to devote to the module.
• Should provide opportunity for reflection.
TOA Recommendations
• We suggest that curriculum writers strive to write clear, specific instructions for activities.
• The titles of lessons and activities should also be more
consistent and simple. • Another recommendation is to use consistent wording in
the checklist, lesson, and activities so they match each other.
• Design opportunity for reflection into The Online Academy.
The Checklist– A self-regulation tool– List activities & due dates– Negotiated with mentor
The Checklist
Evaluating the Checklist• Expert review: Dr. Kitsantas &
Dr. Dabbagh
• One-to-one evaluation
The ChecklistThe Findings
Areas for improvement:• Have students set goals by adding columns for
planned, actual and revision dates• Keep track on progress • Self-reflect on how they performed• Provide scaffolding and modeling• Link all activities
The Checklist
Recommendations
• Support the checklist with a database to track:• Dates: planned, actual and revision• Record grades
• Link key activities
• Provide scaffolding for completing activities within lesson
• Provide opportunity for reflection: by mentor or in module
The Checklist
Suggested Checklist
Case Study
Background
• Research Methods course project
• Research partner, Kimberly McDuffie
• Quantitative & qualitative methods requirement
Purpose
• To examine the experiences of students with learning disabilities with TOA’s design (not academic achievement)
Research Questions
• How does the students’ relationship with the mentor impact the online learning experience?
• Do certain design elements of The Online Academy accommodate students with a learning disability?
Literature Review
• Use of computers with students with learning disabilities (LD)
• Use of hypertext with students with LD
• Multiple design elements in technology-based instruction
• Lack of research in specific study meant exploratory approach
Methods
• Participants– 2 high school Geometry students– Recruited because there was field testing in
inclusive classroom
• Measures– Survey– Interview
Survey Results• Attempt to measure the aspects of the
mentor-student relationship:
Time Spent 2.88 (3-5/week)
Quality of Communication 4.20 (Often)
Amount of Communication
3.13 (3-5/week)
Use of Feedback 3.00 (Sometimes)
Interview Codes
• Clarity of instruction• Chunking• Readability• Navigation• Asynchronous communication• Synchronous communication• Organization of their learning
Conclusions
• Overall – LD students would not do it again, but recommended it for others
• Preferred physically present teacher
• Expressed difficulty communicating their learning problems through text
• Nothing positive reported about design elements
Limitations
• Field test was artificial, done in a classroom
• Self-made survey instrument, not validated.
• Only two students, no external validity
Geometry Module
Geometry Challenge Page
Geometry Assignment
TOA Evaluation Summary
Evaluation informs design
• QA plan
• Field tests
• One-to-one
• Expert review
• Case study
The Future of TOA
The TOA Code Library
The Print Version• Printable version of Code Library to keep as a
reference• Code is color coded to find editable areas
Online Version• Online Code Library is available on the VHS
website• User can copy and paste code into their html
document
What’s Left To Do
• Find a Partner
• Login for site and Digichat
• Retroactive changes based on our evaluation findings
• Mentor resources
Thanks
• Priscilla Norton• Brenda Bannan-Ritland• Bill Warrick• All of the Virtual High School teachers• Nada Dabbagh and Anastasia Kitsantas• Graduate School of Education• Stafford, Loudon & Frederick school divisions
Top Related