Modeling Education with Work Domain Analysis: A Work in

Progress

Lee Nickles

September 11, 2002

Outline

• Defining Educational Evaluation

• Obstacles to Performing an Evaluation

• Work Domain Analysis as Solution

• Process of Building the Model

What is Educational Evaluation?

“Evaluation in the context of educational systems is briefly defined as examining the effectiveness of an educational system (or component of that system) in meeting learning and teaching goals.”

Based on:

(Bloom et al, 1971; Stevens, et al [NSF], 1993; Walker, 1997)

Formative vs. Summative Evaluation

• Formative – During development and implementation

– Examines system to reveal problems and improvements

• Summative– At the end of the system’s life

– Examines overall success of the system

• My focus: Formative

Formative Evaluation is a “Good Thing™”

• Similar to Quality Control, leads to continuous improvement

• Changes can be made in the current life cycle of the system

Evaluation Obstacles (1)

• Education systems have multiple, interacting components– Work together to produce learning

• Isolating which component produced what or (how much) learning is difficult– Testing each individually is too great a burden– Components build on each other...

Which of these activities produces learning?

Evaluation Obstacles (2)

• There is a lack of validated measurement instruments for evaluation– There may not be a test that analyzes whether

or not your specific web site produced learning– You may not have the time or resources to

generate a valid measurement

Validation Solution

• Measurement Triangulation– Use multiple measurements of different types

to measure the thing you want to evaluate• Subjective Opinion/Attitude

• Performance

• Process/Behavior

– If they give similar/confirming readings, you can have greater confidence in your result

Triangulation Problem

• How do we apply this practically?

• That is, how do you know your measurements are triangulating properly?– They are measuring slightly different things

• Need a structured way to apply triangulation

Proposed Solution: Work Domain Analysis

• Interacting components– Specifically examines tightly coupled systems,

with many relationships between the components

• Structure to apply triangulation– Provides the abstraction hierarchy framework

to guide measurement triangulation

Work Domain Analysis - Abstraction Hierarchy

• Rasmussen, 1985

• Modeling tool

• Broad and deep view of the system

• Way to represent the levels of abstraction of a system together and how components, functions, and goals connect to each other

Levels of System Abstraction

Can describe a system at levels of abstraction from the physical

Concrete Abstract

Physical Objects

System Functions

Goals and Objectives

Means-Ends Relationships

Items in one level of abstraction are related to others by means-ends relationships

1: Goals

2: Functions

3: Physical Form

Means: Level 3 tells HOW Level 2 is accomplished

Ends: Level 1 tells WHY Level 2 is performed

Example for Community of Scholars

Levels ofAbstraction

Components

GeneralPurpose

Develop an intellectually stimulating environment in thebasement of the ESM

AbstractPurpose

Increase awareness of everyone’s research in basement of ESM,Provide forum for “scholarly banter”

GeneralFunction

Share ongoing research, Learn about new research applications,Critique research, Find areas for collaboration

PhysicalFunction

Presentations, discussion of presentations

Physical Form Students (speakers, participants), Advisors, Meeting space (fortalks/discussion), PowerPoint presentations

Education and the AH

• Problem: the AH hasn’t really been applied thoroughly to intentional systems– Intentional systems: systems where actions are

based on the user’s intentions more than physical constraints

• Some say it can’t be done (Wong et al, 1998)• Same say it can (Hajdukiewicz et al, 1999)• No really good examples in the literature

Building an Abstraction Hierarchy to Describe Education

• I’ve gone through this process for some time (too long?)

• Many iterations, back and forth with Dr. Pritchett

• NOT Done!

• Process may be as informative as the product, so here is (approximately) the process...

Early Versions

• Considered various ways an AH has been represented– Separating very different domains that are part of

the same system

– Think about how cognitive functions, social activities, and information flow should be represented

– Systems based on user intention rather than physical constraints

Early Framework - Domains

DOMAINSLevels ofAbstraction

Social/Interaction

Cognition Education

Course Goals

“FunctionalGoals”EducationalFunctionsAtomicComponents

Difficult to Separate/Categorize things

I noticed something...

• Process Control– Ongoing process

– Goal: keep process stable

– Change is bad

• Education– Defined start/end of

educational system

– Goal: produce learning in the students

– Change is desired

AH has typically been applied to process control

Education as Change Over Time

STAGESLevels ofAbstraction

Acquire factsin coursematerial

Acquire skillsfor course

Apply facts& skills tonewproblems

Awareness ofinterconnec-tions betweentopics

Course Goals

“FunctionalGoals”EducationalFunctionsAtomicComponents

Education as Pragmatic Change Over Time

GeneralPurpose

Learn IE Knowledge/Skills

Freshman Sophomore Junior SeniorAbstractPurpose

Learn basicdesign, math,etc. (courselist)

(course list)

GeneralFunction

(courseobjectives)

PhysicalFunctionPhysical Form

Is it this simple? Not very general!

Education as Pragmatic Domains

Levels ofAbstraction

Domain of Student Domain of Instructor

GeneralPurpose

Acquire marketable skills Grow Intellectually Socialization

Grow as instructor Propagate domain Job requirement

AbstractPurpose

Student learns X Uphold Honor Code

GeneralFunction

Knowledge transmission Skill acquisition Grading

Low-levelFunctions

Assignments Tests Lecture Reading

Atomic Form Assignment problems Lecture notes Case Studies Exams SyllabusE-mail Textbook

Interesting, but is the domain only interesting at the top level? Did we lose the element of change?

Three Dimensional AH?

• Maybe this was going to take multiple dimensions of domains to represent

• About this time, I switched from pen and paper to Excel to create AHs and used the IT Web site for ISyE 4009 as a case

Multi-Dimensional, “Unit” Based

Unit 1 Course Unit (Lecture? Week?) TopicUnderstand how w e, as engineers, can design information systems to create effective w ork processesLearn how to identify and design for the needs of w orkers and organizationsCreate machine interface designs that establish and support good w ork practicesBe able to identify and communicate the properties of a machine's interface

General Purpose

Understand the limitations of human operators under a variety of situations

(Course Objectives)

Understand the principles of human-integrated systems evaluation - and apply them to your project design

Conceptual Purpose ?? Course Introduction, Learn Procedures

Introduction, Procedure Following Concepts, Preflight Inspection Experiment, Procedures and Roles of Humans

Cognitive Functions Assessment of learning

Acquire domain knowledge (AL), Practice for skill acquisition (AL, LFB) Acquire declarative knowledge (IT)

Low-Level Functions

Work exam, grade exam, review exam solution

Work homework, grade homework, review homework View powerpoint slides

Atomic Elements

IT Web website, syllabus.doc, Spring2002Exam1Solutions.doc homework1.doc, homework2-12.doc

Introduction.ppt, ProcedureFollowing.ppt, PreflightInspectionExperiment.ppt, ProceduresAndRolesOfHumans.ppt

What is a unit?

Multi-Dimensional, Week BasedWeek 4 Course Unit Topic

Understand how w e, as engineers, can design information systems to create effective w ork processesLearn how to identify and design for the needs of w orkers and organizationsCreate machine interface designs that establish and support good w ork practicesBe able to identify and communicate the properties of a machine's interface

General Purpose

Understand the limitations of human operators under a variety of situations

(Course Objectives)

Understand the principles of human-integrated systems evaluation - and apply them to your project design

Conceptual Purpose

Modeling Work I, Decision Making - Naturalistic Models

Work Models Overview, Physical Models, Artifact Models, Cultural Models, Sequence Models, Flow Models, Non B&H Models, Tactical and Strategic Decision Making

Cognitive/ Educational Functions

Practice (AL, LFB), Assessment of learning Practice (LFB) Acquire declarative knowledge (IT)

Low-Level Functions

Work Project (Part A), grade project, review project

Work homework 3, grade homework, review homework View powerpoint slides

Atomic Elements

IT Web website, syllabus.doc, DescriptionOfProject.ppt, DescriptionOfProjectPartA.ppt homework3.doc

ModelingWorkOverview.ppt, HelpfulModelsandStructurestoGuideContextualInquiry.ppt, FlowModels.ppt, SequenceModels.ppt, CulturalModels.ppt, ArtifactModels.ppt, PhysicalModels.ppt, TacticalStrategicDecisionMaking.ppt

This is getting somewhere! But weeks are not general.

Multi-Dimensional, Assessment Based

A2 Course Assessment Topic

Course Objectives

Learn how to identify and design for the needs of w orkers and organizations; Be able to identify and communicate the properties of a machine's interface

Bloom's Level

Knowledge, Comprehension, Application, Analysis Knowledge, Comprehension, Application

Cognitive/ Educational Functions

Research, reflective learning, problem-based learning

Acquire declarative knowledge, Reflective learning

Low-Level Functions

Research, Critical analysis, Applying principles View PowerPoint slides, reflective thinking

Atomic/IT Web Elements Homework 2

Procedure Following Concepts; Preflight Inspection Experiment; Procedures and Roles of Humans

An assessment (homework, exam, project, etc.) as an essential part of course structure

Today: Bring Back Domains, Widen Scope

StudentAssignment 1 Course Assessment Content

Course Objectives

Understand the principles of human-integrated systems evaluation - and apply them to your project design

Cog./Edu. Functions Discovery Learning (sort of) Acquire declarative knowledge

Low-Level Functions Research, Identify terms, submit results

View/Read PowerPoint slides, reflective thinking, ask questions of instructor, meet in study groups, attend lecture, participate in lecture

Atomic Elements

IT Web, instructor, students, classroom, presentation equipment, syllabus/calendar, informal student groups

Homework 1, student submissions, job listings

Introduction Lecture, Results of Employment Ad Keywords

InstructorAssignment 1 Course Assessment Content

Course Objectives

Understand the principles of human-integrated systems evaluation - and apply them to your project design

Cog./Edu. Functions Discovery Learning (sort of) Transmit declarative knowledgeLow-Level Functions Create assessment, compile results

Create PowerPoint slides, lecture, answer questions, lead discussion

Atomic Elements

IT Web, instructor, students, classroom, presentation equipment, syllabus/calendar Homework 1, student submissions

Introduction Lecture, Results of Employment Ad Keywords

Where do I go now?

• Maybe stop now and model an entire course with this method– Are there things we’re leaving out?– What can we leave out?

• Consider, how useful is this for evaluation?– Can we easily generate an AH for a course?– Is it general enough to apply broadly?– Is it specific enough to be useful?