OATS: Georgia Tech’s Online Assessment Tracking System SAIR 2005 Conference Charleston, SC October...
-
Upload
millicent-stevenson -
Category
Documents
-
view
223 -
download
0
Transcript of OATS: Georgia Tech’s Online Assessment Tracking System SAIR 2005 Conference Charleston, SC October...
OATS: Georgia Tech’s Online Assessment Tracking System
SAIR 2005 ConferenceCharleston, SC October 24, 2005
Joseph HoeyJon GordonOffice of Assessment, Georgia Tech
OATS Presentation Outline
Development of OATS to document Institutional EffectivenessDescription of OATS features OATS structure and elementsIssues encountered, solutions generatedPath forwardQuestions and discussion
Documenting Assessment With a Web-Based System
Web-based documentation of assessment for demonstrating progress towards Institutional Effectiveness is an option, not a requirement – we have found it very useful at Georgia Tech.Georgia Tech’s approach to documenting systematic assessment processes is our Online Assessment Tracking System (OATS).Similar systems have been developed by other institutions and commercial vendors.
OATS: Background Annual Assessment Update concept was generated by GT unit coordinators in 1998 to document Georgia Tech’s responsiveness to SACS recommendations re: assessment of student learning.Requests received to move to online environmentOnline process provides structure, formalizes best practices in assessment, serves as a cross-campus communication vehicle, and thus facilitates demonstration of compliance. Having an online documentation system proved to be a key feature in Georgia Tech’s compliance review with the Principles of Accreditation in 2004.
Deciding What to Document and At What Level
We document and track assessment at the degree program levelFor each degree program, we document: Faculty expectations for student learning How these expectations are defined in
practice How these expectations are assessed What results are gained (summary level) Based on results gained, what actions are
being taken
Old Paper-Based Annual Assessment Update Structure
Three open-ended sections: What did you look at? What did you find? What did you do about it?
Submission as MS Word or .pdf documents
Web-Based Annual Assessment Update Structure: OATS
Feature ComparisonPaper-Based Method
Many different formats Hard copy only Difficult to track progress
over time Flexibility (but no
consistency across Institute)
Difficult to provide feedback internally and to facilitate institutional sharing of good practices
Online Method Consistent format Database storage Ability to track progress
over time Flexibility maintained Process facilitates
accreditation e-review Easier to provide
feedback; facilitates institutional sharing
OATS: Integrates Assessment and Annual Reporting
Annual Assessment Update now done online via OATS.Becomes a longitudinal record of assessment and changes to each instructional program.Ideally, can be used to satisfy multiple internal and external information and accreditation needs.
OATS Application: Features
Includes user id/password login and Georgia Tech authentication Web accessible from any location; software plug-in required for editing capability Defined format structure—Objectives/Outcomes, Methods, Results, and Actions/ImpactAllows posting of formatted text (tables, charts, etc.)Default annual rollover feature (except for assessment results and actions taken) saves timeAssessment Updates due: November 1 each year
OATS Application: Features
Archiving of all past submissions by degree program, Print-preview feature that produces a laddered report of objectives, methods, results, and actions taken, Notes feature that permits the storage of submission notes and commentary that won’t be “seen” officially; could be notes to colleagues Feedback feature is built in that permits units to obtain formative feedback on their update prior to final submission. College and Institute-level views, whereby deans are able at a click to view the online assessment update submissions of all units within their colleges.
Description of OATS Elements
Main MenuProgram MenuUnit Summary SectionObjectives/OutcomesMethodsResultsActionsPrint Preview
OATS Main Menu
Includes links to instructions and changes in OATSLink to administrative page for those with administrative accessSelections for: editing current updates, viewing current updates (print preview format
only), and viewing updates from previous years (print
preview format for 2003; .pdf for earlier years)
Main Menu: Current Year and History
OATS Program Menu
Searchable by CollegeIncludes enumeration of objectives/outcomes, methods, results, actions/impacts and date last modifiedIndicates status/progress towards completionOptions to combine programs, view notes, edit individual update, and delete update (“Are you sure?” screen appears)Notes feature can be used as information storage area by user (“here’s why we did this last year…”) or for feedback from Office of Assessment
College Level: Ivan Allen College
Unit Summary Section
Summary section of each OATS update includes portions for program purpose, responsibility for implementation, and operational (programmatic) objectivesPrograms use this section to describe the program purpose, set the context, describe operational objectives for the program, and to provide a brief summary of overall direction.
Objectives/Outcomes - 1
We ask programs to limit OATS updates to 3-8 most important student learning objectives/outcomes faculty expect from program graduates. Engineering undergraduate programs:
prioritize and develop a rotational timeline for assessing ABET EC 2000 program outcomes – don’t try to assess all of them each year.
Objectives/Outcomes - 2
For each objective/outcome, we ask programs to operationally define the activities and competencies to be assessed. What will students actually demonstrate? Describe what is to be achieved.State in terms of expected behaviors.Measurable.Aggregate (group) level – not individual.
School Level: History, Technology & Society
Assessment Methods - 1
We ask programs to specify the method by which student competencies will be assessed (presentations, embedded exam questions, projects, etc.)Optional: Indicate location in curriculum.Optional: Specify performance standards to be achieved.
Assessment Methods - 2We remind programs that the same method may be used to cover more than one learning objective/outcome. e.g., rubrics or checklists developed to assess
senior design projects might be used to assess technical skills, teamwork, and communication skills.
Consider triangulation (multiple methods) to increase reliability of measurement.Consider the reliability, validity, and quality of instrumentation. Will we be willing to trust the results?
School Level: History, Technology & Society
Assessment Results
Succinctly summarize assessment results found e.g., “95% achieved passing scores
on relevant portion of Fundamentals of Engineering Exam.”
Provide brief analysis and interpretation of results.
School Level: History, Technology & Society
Action: Use of Assessment Results
Include a discussion of actions taken or pending in context of the instructional program based on consideration of results obtained.Refer to instrumentation used if URL is available.Give timetable for completion of actions. e.g., “The school has redesigned our
undergraduate structures sequence and hired an additional faculty to teach the additional course beginning next semester.”
Print Preview
OATS includes an option to create a formatted report for each assessment updateSummary section of update appears as distinct from learning outcomes sectionUpdate report is “laddered” such that methods, results, and actions are indentedPrint preview may be saved as an html document and email to others for review
Degree Program Level: BS in HTS
Issues Encountered
Who is really the customer?Multiple operating systems and levels of securityIssues with plug-in software, supported applicationsTraining needsOperational roles vary across unitsVarying levels of administrative enthusiasm and involvementTree vs. multidimensional structure
Other Issues Encountered
Multiple evaluations/multiple cultures problem: Many external reviews and
accountability demands, all with unique features: SACS, Board of Regents program review; specialized accreditation (especially ABET)
Culture of disciplines demands custom solutions (especially ABET)
Path Forward - 1
Process to move towards integrated, customizable, culturally acceptable solution Interviews with key players and users to
assess needs during Summer 2005 Task force assigned to develop conceptual
recommendations by November 2005 Technical working group will write
specifications by March 2006 Programming and testing completed by
August 2006 Next OATS version operational for November
2006 OATS
Path Forward - 2
Desirable features identified so far: Customized “shell” views selected at login for each
relevant specialized accreditation body (ABET, CAC, AACSB, NAAB, NASAD, etc.)
Ability to link to representative courses in curricula Ability to view OATS entries by objectives/outcomes,
by assessment methods used, and by courses in major
Ability to import IR data into unit summary Ability to import assessment data from GT online
assessment data query tool directly into updates Ability to link across disciplines for project or
program evaluations Ability to do all of the above by yesterday with no
faculty effort
Questions?
Contacts: Joseph Hoey,
[email protected] or telephone 404.894.0510
Jon Gordon, [email protected] or telephone 404.385. 1419