Course Evaluation - €¦ · • The Course Evaluation V4 web-based app tool has been tested by 13...

Course Evaluation

- HCI group report -

FLSHASE\ECCS\COSA\HCI group

V4 - Winter Semester 2016-2017

• The Course Evaluation V4 web-based app tool has been tested by 13 students before the launch of the new online campaign (Winter semester 2016-2017).

• The main objectives of these usability tests were:

➡ Test the application with the latest versions of the OS and browsers to detect and fix any technical issues.

➡ Get a first understanding on how the Course Evaluation Report is perceived by the students in order to quickly address the main issues that could be easily and quickly changed.

➡ Collect students’ perception and thoughts about the “anonymity” in the Course Evaluation process with possible areas for improving the “trust” in the anonymity.

➡ Confirm the web-app is still perceived to be usable and enjoyable.

Summary

Course Evaluation - V4 - Winter Semester 2016-2017 2

Date & Place

• We conducted 13 usability tests from the 3rd to the 11th

November 2016, in the Usability Laboratory of the University.

The usability laboratory provides a controlled environment for

non-interfering observation and recording of user behaviour.

• The participants were all students at the FLSHASE.

They were recruited by posters displayed in the Faculty and by notification posted on Moodle.

➡ We selected amongst all the volunteers only those who had not participated to the previous usability test session.

• To thank them for agreeing to participate, they received an

Amazon voucher of 30 euros.

• Each session (debriefing included) lasted approximately 60 mnand was held in English or French depending on the participant’s

preferences.

Usability Testing - Methodology

Usability Laboratory

Recruiting Poster


Welcome &

Consent

Scenario &

Tasks

Debriefing

1. The student was welcomed in the laboratory, received information about the context

and purpose of the study and was familiarized with the equipment of the laboratory

(microphones, cameras).

The student agreed to participate to the study by giving his/her informed consent.

2. The student was asked to answer the Personal Data questionnaire and then evaluate

3 fake courses using the Course Evaluation web-based app on 3 different devices (computer / tablet / phone). The order of using the devices was counterbalanced between the participants so as to control for sequence effects.

3. When all the tasks were completed, a debriefing allowed to deepen the

understanding of the observed student’s activities and to learn more about their

impressions regarding the tool.

Usability Testing - Methodology

Test Protocol


• Task 1 to 10:

The students have to evaluate the courses on three different devices.

• Task 11 to 13

They have to open on the computer the different types of Result Report (“positive”, “mixed” and “programme”). They have to answer several questions aiming to investigate their level of understanding of the charts (own score, meaning of left and right percentages, calculation of the “Neutral” percentages...).

Usability Testing - Tasks

”Positive” Course report ”Mixed” Course report ”Programme” report

• Task 14

Before the debriefing session, they had to complete 5 sentences in order for us to gather their thoughts about some critical aspects of the Course Evaluation campaign.

What I like about evaluating my courses is …Like

What I don’t like about evaluating my courses is … Dislike

When evaluating my courses, it would be great if I could …Expectations

When evaluating my courses, I would be more confident regarding the anonymity of my answers if… Anonymity

I think that more students would participate and evaluate their courses if …Participation Level


Gender Study Programme Semester DisciplineAlready used CE web-based app?

P1 F Master 1 European Governance NO

P2 F Master 1 Enseignement Secondaire NO

P3 M Bachelor 5 Sciences de l'Education YESP4 F Master 3 Gerontology NO

P5 F Master 1 Learning and Communication NO

P6 M Bachelor 1 Psychology NO

P7 F Bachelor 1 Psychology NO

P8 M Master 1 Enseignement Secondaire NO

P9 F Master 1 Geography NO

P10 M Bachelor 3 Cultures Européennes - German NO

P11 M Master 3 Philosophy NO

P12 F Bachelor 1 Psychology NO

P13 M Master 1 European Governance NO

• Sample size: 13 students have participated into the test of V4 of the CE web-based app.

• Age: M = 25.07 (from 18 to 41)

Usability Testing - Demographics

1 - Not at all comfortable7 - Very comfortable

• Languages: - Luxembourgish: 4

- German: 2

- Italian:2- Other (French, Ukrainian...): 5

1 2 3 4 5 6 70

5

10

5.8

• Technology familiarity:

M = 5.8

The students were asked to indicate on a seven-point Likert scale “Applying new technologies you feel...”

Num

ber o

f stu

dent

s


Usability testing - Devices

• 10 different devices have been tested by the

students (mobile phone, tablet, and computer with

different screen sizes and OS, and running different

web browsers).

➡ Chrome, Firefox, Safari latest versions.

“Small” Mobile phone size version

Android Samsung S4 mini 4.3” 4.4.2

“Large” Mobile phone size version

iOS iPhone 6S 4.7” 10.1.1

Android

Samsung A5 5.0” 6.0.1

Sony Xperia C4 5.5” 5.1

Nexus 6P 5.7” 7.0

Tablet size version

iOS iPad 9.7” 9.3.1

Android Samsung Galaxy Tab 10.1” 5.0.2

Windows

Surface 12.0” 8.1

Surface 12.0” 10

Computer size version

Mac OS Computer 23” 10.7.5


V1

Summer 2015 Winter 2016

Usability testing - Versions

Winter 2015 Summer 2016

V2 V3 V4

Version tested now


Course Evaluation - General Findings


“Mi-décembre c’est toujours la période où on a beaucoup de choses à rendre avant les vacances, donc ce n’est pas une priorité !”P 2

“Just ignore it, it’s like a survey.And it’s boring, unless you are really enthusiasm about a course.”

P 11

Why they didn’t evaluate their courses last semester...

• Amongst the students already enrolled at the Uni last semester, only one participated in the CE campaign.

➡ For those who didn’t participate in the preceding CE, the main reasons expressed were related to:

- Lack of interest / No perceived usefulness

- Time consuming / Busy exam period


Email Campaign

• Amongst the students already enrolled at the Uni last semester, all remembered that they have received a call for participation by email,

whereas no one remembered an email for consulting the results.

➡ They clearly expressed their interest for this information.

“Avoir un mail où on dit : on peut évaluer maintenant… Et pas lundi prochain !”P 11

• One student complained about the email sent BEFORE the

evaluation effectively started and would prefer having an

email on the starting day only.

• Some students also suggest to enhance the visibility of the CE campaign on Moodle.


Sentence Completion – Like / Dislike

What I don’t like about evaluating my courses is…

➡ The time is the most frequently expressed concern.

What I like about evaluating my courses is…

➡ The concrete impact of the evaluation is mainly mentioned.

… that it gives me the power to improve the course for future generations.P 6

… that it takes a lot of time.P 8


Sentence Completion – Expectations / Participation Level

I think that more students would participate and evaluate their courses if…

When evaluating my courses, it would be great if I could…

➡ The students are very interested in the results. ➡ Here again, they wish to know that their answers

could effectively change the courses.

… they knew their answers will effectively have a future impact on the concerned courseP 13

… compare the results of different yearsP 12


Sentence Completion – Anonymity

When evaluating my courses, I would be more confident regarding the anonymity of my answers if…

… I didn’t have to login using my student ID number (which can lead back to me).P 2

➡ 2 students trust the system.

➡ The most frequent suggestion is spontaneously: replace the student ID with a specific code per student or common to a class.

However, some students understand this option is not realistic, the student ID is required to avoid unfair feedback and replacing their ID with some other unique identifier does not directly imply a different level of anonymity.

And during the debriefing, it appears that the main concern about giving their ID is that they are not clearly informed on how it will be used afterwards.


Anonymous – Increase the trust

Ensure the anonymous treatment of the data: be transparent and provide more information.

- One student suggested to provide a printable official declaration from the University that guarantees the respect of the anonymity.

Students tend to trust that the data are analysed anonymously, however they would like some “proofs”.

- The principal request is about the exact nature of the feedback given to the teacher / course director. The

students don’t know if the student ID or personal information are used any further and would like to have

concrete examples of the results.

“Je me demande toujours est-ce que mon âge est communiqué au prof ? Je suis le seul étudiant de cet âge. Et aussi avec le sexe… Je sais pas dans quel sens les profs ont ces informations-là. Est-ce que les profs ont les codes étudiants ? (…) Montrer avec un exemple de cours fictif ce que les profs vont avoir comme réponse, parce que là on saura vraiment… les profs vont avoir ça, ils ne vont pas voir l’âge... On ne voit jamais derrière, on ne voit pas ce qui se passe.“

P 3


➡ The main level of improvements seems to regard the communication on the treatment of the results.

➡ One additional improvement: make the “anonymity” more official (for instance in the wording, or/and by providing in the app a link to the website and the “Rules and Regulation” document).

Course Evaluation – Web-based app


• The students are very positive about the CE app once again.

Web-based app

“Simple” “Easy”

“Good” “J’aime bien !”

• The minor issues encountered are quite similar to those already observed in our previous test sessions:

- A few participants would prefer to access the evaluation directly without clicking on the button at the

bottom. One participant was blocked for a few seconds and required some help for starting.

- Some students suggested to improve the layout on the computer and centre the scale.

• The design and the ”Overview” at the end are very appreciated.

“Clear” “Intuitif”

“C’est vraiment simple, ça motive les gens pour participer !”

P 11


Questions

• Globally, the students find the questions to be clear and understandable and covering the main topics.

“It’s difficult to evaluate, and depends on the classmates, how motivated they are and not the quality of the teacher!”P 12

- Again, 3 students find the “Work climate” question

difficult to answer and also point out that it does

not necessary depend on the teacher.

- They again express some doubt about the “missed course” question.

• The suggestions made are similar to those previously found:

- They suggest to include more questions about the availability of the teacher or the language of the course.

- They also point out the difficulty to evaluate when courses are given by more than one teacher.


Chart

• Globally, the results are perceived to be

complicated, with a lot of information to understand. “Confusing”

“Not clear”

“Compliqué”

“Needs time”

“Difficult to understand”

- Some students express that statistical knowledge is needed.

- It takes some time to understand.

- And they would like to have an explanation at the beginning. “On est censé avoir une explication avant ? Comment le lire ?”

P 4


“You need to study statistics to understand”

P 11

“Si je prends mon temps, ca va… Il faut beaucoup de temps pour comprendre”

P 13

Chart

“On peut voir si ce qu'on a ressenti, c'est le ressenti de la plupart des étudiants ou si c'est juste que, personnellement, on n’aime pas un prof...”

P 13

• The students like the possibility to compare the score they have given to that of their classmates.


Main findings - “Positive” Course report

Q: According to the chart, what was your response to the question 6?

• 85% of correct answers.

• Errors mainly due to confusion between questions AND misreading of the legend (grey colour associated with “3”).

Q: According to the following table, what is “7”?


• Error: One student associated the score “7” with the number of students who have answered “completely agree”.

Q: What percentages of students have answered “neutral” for the question 8?

• Only 23% of correct answers!

• Most of the students do not immediately understand if “neutral” is part of the 56%.


Q: According to the chart, what was your response to the question 6?


• Errors mainly due to confusion between questions.

Q: Which question has the highest percentage of “positive” scores overall?

• 54% of students have answered at least 2 or 3 questions.

• The other students just look at the “highest” bar.

Q: What percentage of students have answered “Completely agree” for the question 10?

• 61% of correct answers

• Errors mainly due to confusion between questions (estimate the % for Q9).

Main findings - “Mixed” Course report


Q: According to the chart, what was your mean response for “structure and form”?

• 100% of correct answers. Q: What does this chart represent?

• Most of the students say that this chart represents the evaluation of a singlecourse.Q: Can you briefly explain

the “Overall mean score” in the chart?

• For non English native speakers, the word “mean” is sometimes confusing: one student understands it as “unkind”, one other as the “means” used in the course.

Main findings - “Programme” report


• PROPOSAL 1: Make the legend easier to read.

- Make it clearer which number is associated to which colour: for instance, add some space between the numbers or place the numbers at the top.

- And add the labels to the scale and remove the table at the bottom (advantages: no need to scroll anymore -avoid misunderstanding about the score -> “7” for “number of students”).

Design suggestion 1

Design suggestion 2

Report - Proposal

“Having the label with the colours”

P 1

• PROPOSAL 2: Add a sentence at the top that would clearly explain what the different percentages are.

• PROPOSAL 3: Use the same text on each chart for the confidentiality (currently, on the Programme chart, only “Confidential Information” is written). Moreover, make it more explicit that the students are not supposed to share the results.

• PROPOSAL 4: Provide a translated version of the results in FR and DE.

• PROPOSAL 5: For the “Programme” chart, display as a title the name of the programme (like the Course Name for the other charts).


Students’ suggestions

• Percentage of participation: how many students have answered?• Most frequent answers for the open questions.• More details about the different percentages of the scale (no need to calculate “neutral” for instance).

• See the previous evaluations, in order to see the progress (if any).

• One student points out that this kind of chart is not adapted for mobile devices, as there is too many information displayed.

• The questions are summarized, and long after the evaluation, it could be difficult to remember their exact signification.

“I don’t remember the question... It’s a sum up...”

P 1


Design inspirations...

- ODD SCALE: In this chart, the “Neutral” percentages are displayed

apart. Therefore, it’s easier to view the “positive” and “negative” trends

at a glance.


- DETAILED SCORES: The percentages are displayed for each

level of the scale AND question by question.

- NICE DESIGN: Alternative to the

classical “line” chart.

Design inspirations...

http://www.lemonde.fr/les-decodeurs/article/2016/11/21/les-resultats-de-la-primaire-soulignent-encore-les-limites-des-sondages_5035269_4355770.html

http://thenextweb.com/apps/2014/02/20/surveymonkeys-new-app-lets-you-create-surveys-and-monitor-results-on-the-move/


- MOBILE VIEW: The level of information on the screen is

reduced by giving the results for a single question.

And here again, all the detailed scores are provided.

- INTERACTICVE VIEW: Here, you can view the exact

score of an element by clicking on the chart.

➡ The V4 Course Evaluation web-based application has been very positively perceived by the students once again.

• 2 technical issues have been found and solved on the new devices (Windows 10, Edge).

• Regarding the opinions gathered during the interviews, it seems that the main potential points for increasing the

participation and the trust in anonymity are:

- Reassure on the anonymity by making the feedback given to the teacher more transparent and concrete.

➡ Provide the students with an example of a report.

- Improve the “Moodle” communication.

- Improve the information about the course evaluation process. Even if all the explanations are already

available on the CE website, they are difficult to find (link to the website on the app.?).

- Illustrate by concrete examples how the courses have been improved thanks to the evaluation.

Conclusion


“On ne voit pas toujours les changements, on ne voit pas ce qui a été fait… Parfois, on a l'impression de faire les évaluations et il n'y a rien qui change ! On se demande vraiment pourquoi il faut faire ces évaluations. (…) Avoir quelque chose qui nous dit : Vous voyez, il y a des changements qui ont été faits ! "P 3

Course Evaluation - €¦ · • The Course Evaluation V4 web-based app tool has been tested by 13...

Documents

Transcript of Course Evaluation - €¦ · • The Course Evaluation V4 web-based app tool has been tested by 13...