A Case-Study of a Web-Based Method for Repeated-Measures and Multi-Source Research Michael J. Walk,...
-
Upload
claribel-elliott -
Category
Documents
-
view
223 -
download
1
Transcript of A Case-Study of a Web-Based Method for Repeated-Measures and Multi-Source Research Michael J. Walk,...
A Case-Study of a Web-Based Method for Repeated-Measures and
Multi-Source Research
Michael J. Walk, M.S.University of Baltimore
[email protected] SCiP—Chicago, IL, Nov. 2008
Web-Based Research Valuable research tool for Psychology Dominated by cross-sectional, between-
subject designs
Speaking Abstractly WebRTS:
Web-Based Research Task System A web-based system for poly-task online
research designs. Adhered to methodological and ethical
recommendations for online research. Tested WebRTS in a multi-source research
design (N = 28).
The Task Page – Purpose To control the ordering of stimulus
presentation To prevent repeating or premature
completion of tasks (Reips, 2000)
To separate the research design into short, distinct tasks (Reips, 2000)
To allow participants to return to complete at a later time
The Task Page - Function Find out who the user is Find out what tasks are done so far
Put a check mark by those tasks
Find out what the next task is Make this task an active hyperlink
Find out what tasks are in queue Make these tasks inactive text
WebRTS – Basic Page Structure Header
Same on every page = coherence (Nosek, Banaji, & Greenwald, 2002)
Logout Button (if logged in) University Logo = trustworthiness (Reips, 2000)
Content Footer
Dynamic (page links to important pages) = navigability Email the researcher link = experimenter presence Quit the study link = debriefing (Nosek, Banaji, & Greenwald, 2002)
Website FlowHomepage
Informed Consent
Login Page
Processing script
Task update script
Task 1 Task 2 Task k
Task Page
Debriefing
Yes
Last Task?
New Participant
No
Current Participant
Quit?
Internet
Give co
nsent
Pre-Consent Exit Interview
Refuse consent
WebRTS – Auxiliary Pages Forgotten password retrieval Administrator’s page
Send reminder emails to participants Add fields to database
WebRTS – Technology Specs MySQL database (hosting and database
provided FREE by www.agilityhoster.com) PHP server-side scripts JavaScript for form validation Pilot version
www.ubpsychportal.org/ssa Updated version (WebRTS 2.0)
www.ubcareerlab.org/cip
Case Study Used in a multi-source, poly-task research design
testing relationships between self-monitoring (Gangestad & Snyder, 2000) and accuracy of predicting personality ratings (e.g., Walk, Mitchell, & Yun, 2008)
Anecdotal usability evidence all positive Brief follow-up survey administered online in Aug.
2008 (3 months after Case Study) N = 6 (21%) Green & Pearson’s (2006) Web-Site Usability Instrument
16 items, (1) strongly disagree to (7) strongly agree 4 Open-ended questions
Notable Usability Results I completed the task on the Web site
without much effort. (M = 6.33, SD = .82) After learning to use part of the Web site, I
easily learned to use another part. (M = 6.17, SD = 1.17)
The Web site interface was consistent throughout the site. (M = 6.17, SD = 1.17)
All items M > 5
Open-Ended Questions What part or page of the website did you like
the best? I liked it all. It was user friendly. I liked the fact that the survey was multiple choice.
This meant that it took less time to complete the tasks.
I liked the fact that the website was easy to navigate when completing the tasks.
Easy to readfollow The set-up of the whole thing was easy to use. i dont know
Open-Ended Responses (cont.) What part or page of the website did you like
the least? No complaints. Unfortunately, I did not like the part of the survey that
had someone else fill out a survey… The part of the website that I did not like was the
portion in which you had to have a supervisor or co-worker complete…
lacks color, kind of boring NA I dont know
WebRTS 2.0 System pages are “easily” configurable
Titles, pictures, colors, researcher names, host institution, etc.
Can set open / close dates Can set max participants Can randomly assign to ordering conditions Prevents repeat submissions by the same
user.
WebRTS – Uses and Applications Repeated measures designs Longitudinal designs Poly-task cross-sectional designs Multi-source (e.g., participant & rater)
designs Diary designs
ReferencesGangestad, S. W., & Snyder, M. (2000). Self-monitoring: Appraisal and
reappraisal. Psychological Bulletin, 126(4), 530-555.Green, D., & Pearson, J. M. (2006). Development of a web site usability
instrument based on ISO 9241-11. Journal of Computer Information Systems, 66-72.
Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002). E-research: Ethics, security, design, and control in psychological research on the internet. Journal of Social Issues, 58(1), 161-176.
Reips, U. (2000). The web experiment method: Advantages, disadvantages, and solutions. In Birnbaum, M. H., ed. Psychological Experiments on the Internet. Academic Press: San Diego, CA.
Walk, M. J., Mitchell, T., & Yun, G. (2008). Know thy social self? Self-monitoring predicts accuracy in rating one’s reputation. Poster session presented at the 20th Annual Meeting of the Association for Psychological Science, Chicago, IL, May 2008.
Questions?