It Takes a Village: Collborating for success with high-volume assessments

Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

2013 Users Conference Baltimore | March 3rd – 6th

It Takes a Village Collaborating for Success with High-Volume

Assessments

2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark

is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Rio Salado College is part of the Maricopa Community College District

Rio Salado offers over 600 unique courses online Our model – one course, many sections Development/Production of courses is centralized Perception users since 2003

A Little About Us

Slide 2



Penny Freuler, Director of Instructional Technology and Course Production

Liz Burnett, Coordinator of Distance Learning Services

Heather Tyler, Testing Coordinator

Lance Cummins, Networking Manager

Introductions

Slide 3



How we use Perception with our LMS How we use Perception Data outside of the application How we work as a team to support assessment

delivery What we have learned to do/what we have learned not

to do

What We Will Cover

Slide 4



Building Assessments

Delivering Assessments

Scoring Results

Supporting Services

Course Production

Slide 5



Topics and questions Outcomes Generator

Assessment controls Password protection Time limits Randomization Open access for self-assessment

Outcomes Feedback on question, topic, or assessment level Branch to another assessment

Building Assessments

Slide 6



LMS Integration

Offline versions available for Testing Services

Schedule control Sequencing

Device support Perception 5.2 inclusion of device settings file

Delivering Assessments

Slide 7



Entry points from LMS to Scoring Tool

Scoring Tool updates have not kept pace with the rest

of the Perception application

Building, delivering, and scoring work together

Post Perception 5.4 Deployment Integrating scoring functions in LMS

Scoring Results

Slide 8



Work with helpdesks, testing services, students, faculty to troubleshoot issues Greatly improved in Perception 5.2, 5.4 Coordinate resolutions with Programming, Networking,

Technical Support, Instructional Design Evaluate coaching and transcript reports

Work with Questionmark staff on system issues

beyond our control and best practice advice

Supporting Services

Slide 9



Exam Administration

Proctoring Services

Academic Integrity

Data for Content Owners - Reports

Testing Services

Slide 10



In-Person Exams Midterm and Final exams delivered through QuestionMark Password protected Identification verification Monitoring during exam

Close working relationship with Course Production and Networking allows issues to be identified sooner

Testing Services- Exam Administration

Slide 11



Same level of testing security required for all out of area students. Proctoring staff dedicated to vetting and communicating with

proctors to assure same testing standards are met Creates the challenge for Networking and Course Production

that there is potentially a student taking a high stakes exam 24 hours a day 7 days a week.

Issues encountered Transcript report research Branching profile workaround

Testing Services – Proctoring Services

Slide 12



Working with Faculty Supply supporting evidence for a reported academic integrity issue

• Turn-It-In Reports • Peer-to-Peer Reports • Adjunct Suspected (similar course work submitted) • AI Team suspected (data pulled from reports)

Working with Financial Aid Supply supporting evidence for a suspected FA issue Working with the Student Academic Progress process AI team works to alert FA to potential issues before they happen

Testing Services – Academic Integrity

Slide 13




Slide 14

What actions are taken by the Judicial Affairs Department?

What does Academic Integrity

look to identify?

Who/What can instigate an inquiry?

Financial Aid SAP Process

Sufficient Evidence to

confirm less than honest behavior

New Students: Send Internal Review

Letter to gather further documentation

Current Students: Actions necessary to

include suspension and expulsion

Insufficient Evidence to

confirm wrong doings

Record Research This database will

house all confirmed, suspected and cleared

student research

Staff Referrals Advisement Counseling Admission and Records

Financial Aid

Academic Integrity Reports




Slide 15

•Questionable Documents submitted for SAP. FA staff marks student record with ‘Referral’ Service Indicator and emails [email protected] with student information, description and supporting documentation. •Examples of questionable documents •Falsified obituaries •Altered id cards

Financial Aid

•Reviews and researches all referrals. •Sufficient Evidence •Compile all documentation necessary to support recommendation for further action.

•Insufficient Evidence •Remove Referral service indicator •Alert FA that student is not considered a risk •Forward all research to Judicial Affairs for storage.

AI Team

•Review all records and recommendations •Confirmed Sufficient Evidence •Pre-Student Enrollment •Send Internal Review Letter •Current Student •Actions necessary

•Insufficient evidence •Return to AI for further research or clarification or file with other insufficient evidence files

Judicial Affairs

mailto:[email protected]



Item Analysis Allows Content Owners to review Assessments based on • Question Difficulty – P-Value Proportion Correct • How did students perform on the questions • What can this tell us about the question or the content

Testing Services – Reports

Slide 16



Testing Services – Reports

Slide 17



Testing Services – Reports coming in 5.4

Slide 18



Team approach Programming Networking Production Support Services

Work closely with Questionmark staff

Version Testing

Slide 19



Some lessons learned Previously patched items Greatly improved with Perception 5.4

Items created in previous versions Items created that employ irregular practices Test, retest, test again, and go back and test If you think users won’t use it that way, they will.

Version Testing

Slide 20



Server Builds Install and configure Perception Servers Test and Production environments Maintain multiple versions of Perception throughout testing and

development cycle Integrate into Rio Salado’s custom LMS Coordinate testing and development cycle with other

departments such as Course Production

Network Administration

Slide 21



New versions of Perception require testing and development by multiple teams We delivered over 2.3 million assessments in 2012 Networking test builds– Build and configure new servers,

copy database (~40-50GB). Course Production – Test features, look for bugs, ensure that

previously patched items are retained in the new version Programming – Ensure code works with potential QM

changes, modify as needed.

Version Testing

Slide 22



Course Production Takes initial reports of issues and troubleshoots to determine

if there is an issue with the Perception servers, the application itself, or some other issue that needs to involve Networking

Cuts down greatly on the amount of issues that come to Networking. Since we don’t use the application ourselves, it takes much longer for us to determine where the issue lies.

Now, if it comes to us we know there’s been a great deal of groundwork done and we know more about the sort of problem people are having. It also weeds out things like user error and problems with the course content.

Collaboration

Slide 23



Testing Services With students taking exams all day in our testing centers, and

proctored exams possible at any time, server downtime must be kept to a minimum

Perception servers are load balanced, so we can reboot them one at a time without affecting students.

SQL– On the rare occasion that there is an issue with our main SQL cluster, Perception and other applications may experience a few moments of downtime while SQL is moved over to another cluster node. This will cause problems for students currently taking exams.

We notify Testing as soon as possible before or during outages so that they can be aware of issues and take appropriate action in our testing centers to minimize lost work.

Collaboration

Slide 24



Questionmark – If/when issues are discovered, we work with QM support to see if it’s a configuration issue or a bug. If it’s a bug then they work on getting it fixed and either hotfixed or rolled into their next update, depending on severity.

For previous versions we’ve had weekly or semi-weekly meetings with multiple teams at QM (e.g. Customer Care, Tech Support, and QM developers) to go through our list. Some issues were bugs, and some were unique ways we were using Perception that didn’t work as expected, but weren’t necessarily bugs.

Testing/Support

Slide 25



Several years ago Questionmark began offering Enterprise support. It’s a paid service, but one I highly recommend

This has saved us a great deal of time Previously we’d have to bring multiple techs up to

speed on issues we were having, had to send the same files/logs, etc. repeatedly

Now we have one point of contact, and he deals with us often enough to have a grasp on our environment without having to start at square one each time we contact him.

Enterprise Support

Slide 26



For 5.4, QM has instituted many changes to their development process, and we’ve been very pleased with our testing

In fact, for the first time we’ve not needed anything hotfixed, and are preparing to deploy 5.4 to production at the end of the month. We would have gone sooner, but we had some code changes on our end that held it up, not directly QM related.

Perception 5.4

Slide 27



Best Practices – read them Review and understand server settings Use system as designed

Questions?

Remember

Slide 28

It Takes a Village: Collborating for success with high-volume assessments

Technology

Transcript of It Takes a Village: Collborating for success with high-volume assessments