DEVELOPMENT AND PRELIMINARY RESULTS OF AN ADAPTIVE GRADING/LEARNING SYSTEM (AGLS)

Kevin MatthewsThomas Janicki

University of North Carolina Wilmington

Background

Literature Review

Benefits and Need

System Development

Hypothesis

Limitations

Conclusions

Overview

Increased need for computer literacy skills

◦ National Science Foundation (NSF)◦ National Research Council (NRC)◦ College requirements for literacy

Background

Increased class size as budgets are cut, may lead to:

◦ Fewer assignments◦ Simpler assignments◦ Less personalized feedback◦ Longer delay in returning grading◦ Increase plagiarism

Problems

Question?How could technology assist the instructor to increase student learning in computer literacy courses?

Literature reviewNine elements that should be present in order for learning to occur (Gagne, Briggs, Wager)

Gain attention

Inform learner on the objective

Stimulation recall of prior learning

Presenting the stimulus

Provide learning guidance

Eliciting performance

Providing feedback

Assessing performance

Enhancing retention and transfer

*most related to success(Martin Klein, Sullivan)

Contiguity

Repetition

Feedback

(Gagne, Briggs, Wager)

Behavior Learning Theory

Traditional Learning Resource Based Learning

Teacher as expert model Teacher as facilitator / guide

Textbook as primary source Variety of sources / media

Facts as primary Questions as primary

Information is packaged Information is discovered

Emphasis on product Emphasis on process

Assessments is quantitative Assessment is qualitative / quantitative

Resource Based Learning

Rakes

Need for automation

Using course management software

◦ 30% of graded assignments had NO feedback Excluded perfect score assignments

◦ Average of 28 days to score an assignment!

◦ Various ‘grading keys’ when done manually

◦ These do not lead to learning

Feedback comments/response

To be effective, the student’s response to concepts should immediately follow instruction.

Multiple assignments of similar nature should be presented repetitively to reinforce new material presented during a lesson.

Quick and customized feedback allows a student to identify correct answers and see errors in incorrect answers.

The best way to reach this goal of student success is to adopt a resource-based approach to learning.

Key criteria to increase learning

Build or ‘buy’ a solution

Benefits/Features AGLS Case-based Procedural Test-BankChallenging, real-world problems ■ ■

Automated grading ■ ■ ■ ■

Consistent grading ■ ■ ■ ■

Instant feedback ■ ■ ■ ■

Customized feedback ■

Web interface/portal ■ ■ ■ ■

Multiple skills assessed concurrently ■ ■

Hands-on experience ■ ■ ■

Smaller one-skill problems ■ ■ ■

Question/assignment library ■ ■ ■

Reduced preparation/paperwork time for instructor ■ ■ ■ ■

Availability of student reporting ■ ■ ■ ■

Expandable answer banks ■

Repository for file submissions ■ ■

Plagiarism detection ■ ■Instructor created exercises ■ ■

LimitationsAGLS

Case-based

Procedural

Test-Bank

Answers must be exact matches

■ ■ ■

Limited number of cases ■ ■

Textbook/supplemental required

■ ■

Software must be installed ■

“Simulated” environment ■

Other purchases required ■ ■ ■

System design

AGLS(Grading Modules)

Grading Key

Student Files

Service Requested

Score

Correct Answers

Incorrect Answers

Feedback

INPUT OUTPUT

Key (ideal) correct answer given

Student answer differs

Instructor prompted with correct answer and student answer

Instructor deems an answer as correct or incorrect

An answer is only seen once

Excel – alternative answers

Key (ideal) names given for tables, fields, etc.

Students may have typos or misspellings

Instructor prompted with all table names, field names, etc.

Instructor deems a name acceptable.

A penalty may be given.

Access – naming alternative

Data Gathering

◦ Existing grade book of feedback comments and response time

◦ Expert panel to evaluate quality of ‘feedback comments’

Hypothesis and experiment

Hypotheses

H0 (μ1 – μ2 = 0): The use of the AGLS will not affect the amount of feedback provided to students.

H1 (μ1 – μ2 < 0): The use of the AGLS will increase the amount of feedback provided to students.

1. Affect on Quantity of Feedback

Control Experiment

Mean 45.10 71.36

Sample Size 628 3138

t statistic -6.57

Two tail p value 5.8 x 10-11

H0 :The use of the AGLS will not affect the quality of feedback provided to students.

H1 : The use of the AGLS will increase the quality of feedback provided to students.

2. Affect on Quality of Feedback

Control Experiment

Mean 3.27 3.45

Sample Size 49 51

t statistic -1.13

Two tail p value .26

H0 :The use of the AGLS will not affect the time for an assignment to be graded.

H1 (μ1 – μ2 < 0): The use of the AGLS will decrease the time for an assignment to be graded.

3. Affect on Response Time

Control Experiment

Mean 28.59 8.13

Sample Size 628 3138

t statistic 33.43

Two tail p value 9.7353 x 10-215

The use of the AGLS will increase the amount of feedback provided to students.

The use of the AGLS will NOT affect the quality of feedback provided to students.

The use of the AGLS will decrease the time for an assignment to be graded.

Conclusions

Only Access and Excel 2007

Only certain items were initially included to be graded

Limitations

Web Service

Refined library of assignments and correct/incorrect answers

Additional features to be graded

Testing outside of introductory computer course domain

Future work

Questions