Effect of Computer-based Testing on Candidate Perception of Selection

Justice Aylin Dincer, Levent Sevinç, Merve Şuşut, İpek Altıntaş

8th ITC Conference, Amsterdam 2012

IntroductionIntroduction

• Recent advancements in technology have led organizations to make use of technology widely in their recruitment process.

• As a result, there is an increasing adoption of online test applications in employee selection procedures.

IntroductionIntroduction

• Online recruitment is a reality across Europe with the more technologically advanced European community countries leading the way in these developments. (Lievens et al., 2002)

• Parallel to these developments in Europe, online recruitment has been much more utilized in Turkey as well. – Nevertheless, online testing has started being used more

recently, yet the demands of organizations for online test applications are increasing rapidly.

IntroductionIntroduction

• In online testing, human resources professionals mostly focus on the reliability of the results of applicants. – Although validity and reliability of the test results are

crucial, only focusing on psychometric issues results in neglection of non-psychometric factors.

• In order to acquire a holistic view, it is also important to examine applicant reactions to online testing.

AimAim

• To this end, this study aimed at examining the effect of computer-based testing on candidates’ perception of fairness of selection process.– For this purpose, this study was carried out

in the selection process of a bank in which paper-and-pencil and online tests were administered to candidates.

METHOD

MethodMethod

• Research was conducted on a group of candidates of a private bank’s credit card sales department (n=235) attended a general ability test.– Candidates were separated into two groups: One group

(n=162) were administered a computerized version of the test and the other group (n=73) were administered a paper-and-pencil version.

• After completing the test, each candidate group filled out a scale evaluating selection justice.

MethodMethod

• Candidates were selected by the human resources department through CV screening among the job applicants.

• The bank carries out the test application online and proctored in the office environment (fully managed). – For this research, test was administered to some of the

candidates in paper-and-pencil version. The candidates were assigned to each group according to the bank’s decision.

Measures:Measures:Ability TestAbility Test

• In the study, all candidates were assessed with the same cognitive ability test. – FVAT Cognitive Ability Test used in this process was

developed by Assessment Systems.• Test-retest reliability .89 • Split-half reliability .87• Predict job performance (manager rating) r=.51 p<.01 • Correlation coefficients of online and paper-pencil versions .91

– The test consists of 40 face valid questions, with a duration of 60 minutes. It measures 4 different abilities: Numerical, verbal, analytical thinking and attention.

• Average difficulty index .55

Measures:Measures:Ability TestAbility Test

• Psychometric properties and results of the test did not indicate a significant difference between the two groups.– The split-half reliability of the test did not significantly

differ between the two groups.• Computer-based = .87; Paper-and-pencil =.86

– The difficulty index did not indicate a significant difference between the two groups.

• Computer-based = .57; Paper-and-pencil =.56

– There was no significant difference between the number of correct answers in each group.

• Computer-based : M=22.66 (SD=7.2); Paper-and-pencil: M= 22.41 (SD=8.1); t=-.24 p=.81 >.05

Selection Justice Scale Selection Justice Scale

• In this study, Gilliland’s organizational justice model (1993) was used to examine the reactions of the applicants in terms of selection justice.

• Studies of applicant perceptions of selection processes mostly use an organizational justice point of view (Ryan & Wessel, 2008).– Gilliland (1993) pioneered in applying organizational

justice theory to understand how applicants react to the selection process.

Gilliland’s Gilliland’s Selection Justice ModelSelection Justice Model

• Distributive justice, concerns applicant reactions of whether a specific selection decision (hiring or rejection) is fair.

• Procedural justice, concerns applicant reactions to– the tests and techniques used in evaluating their

qualifications. – the organizational policies implemented for processing

applicants, and – the interactions that applicants have with various

members of the organization during the hiring process.

Gilliland’s Gilliland’s Selection Justice ModelSelection Justice Model

• Gilliland mentioned 10 rules, can be grouped in 3 categories, concerning procedural justice– The formal characteristics of the selection process

which includes job relatedness, opportunity to perform. consistency

– Information given to applicants about the hiring process and decision, it includes rules like feedback and honesty in communications with candidates.

– Interpersonal treatment which includes treatment during the selection process, two-way communication, and propriety of questions.

Selection Justice ScaleSelection Justice Scale

• The Selection Justice Questionnaire developed by Gilliland ve Honig (1994) was adapted for this study. – Procedural justice was considered within 3

categories rather than 10 sub-scales.– Accordingly, the scale consists of 4

dimensions together with distributive justice.

Selection JusticeSelection Justice Scale Scale

• The scale is composed of 14 items arranged along a 5-point Likert-type Scale.– Distributive Justice, 4 items.

• “I think assessment based on this test will be fair.”

– Procedural Justice, 10 items.• “This test gave me the opportunity to demonstrate my abilities

to the authorities of the organization.” • “The candidates were sufficiently informed about the

announcement date of results.” • “During the test application, the authorities answered the

questions sincerely.”

Selection JusticeSelection Justice Scale Scale

• Factor Analysis revealed that the scale consists of 3 dimensions.– Two categories of Procedural Justice (interpersonal and

informational) were combined into one dimension and named as «interactional».

– KMO=.71; Bartlett’s Test chi square 693.47 p=.00<.01; total variance explained=57%

• The Cronbach’s Alpha values calculated for each dimension indicate that the scale has high internal consistency. – Distributive= .88; Procedural=.82; Interactional=.87

Participants Participants

• The distribution of the participants (n=235) in terms of their demographic characteristics is as follows:– 100% university graduate– 52% males– Mean age 22.4 (SD=1.8)

• The distribution of each group in terms of their demographic characteristics is similar.

RESULTS

Results Results

Pen-and-Paper(n=73)

Computer-based(n=162)

t Test

M SD M SD t p

Distributive Justice 4.23 .75 4.62 .44 -4.96 .00**

Procedural Justice 4.10 .92 4.56 .49 -4.95 .00**

Interactional Justice

4.70 .41 4.68 .44 .39 .70

• In both groups, candidates perceived the process positive (>4.00). In candidates who were administered the computerized version, perceptions of procedural and distributive justice were significantly higher (<.01).

**p<.01

ResultsResults

• According to the results, candidates who were administered the computerized version mentioned that; – Assessment based on the test would be much more

fair. (distributive justice) – The test used gave them the opportunity to highly

demonstrate their abilities and test was strongly job related. (procedural justice)

ResultsResults

• On the other hand, interactional justice perception was not significantly (p>.05) different between two groups.– In both groups candidates stated that proctors

permitted them to ask questions and answered the questions sincerely.

• Insignificance may be explained by employing proctor in both of the groups.– Because of the organizational policies, proctors’

behaviors would have probably been similar to each other.

ComparisonComparison

• When results are compared to a similar study on perception of justice about selection process (Sevinç & Yılmaz, 2005);– Researchers examined selection justice perceptions of

a bank candidates (n=97) according to Gilliland’s model as well, but the test had no online version, it was administered in paper-pencil form.

– In this study, it is revealed that the perception justice scores of candidates who were administered paper-pencil test was parallel to 2005 study results.

– Thus, the increase in the perception of justice scores may be explained by online test administration.

CONCLUSION

ConclusionConclusion

• When the candidates take the test online, they evaluate the selection process and the content of the assessment tools positively. – Philo and Green (2007) showed that the candidates of a

company which used online test for the first time judged the content to be more fair when compared to traditional testing methods. Thus, it can be said that applicants found the online test experience positive.

– Online testing is relatively a new concept in Turkey. Candidates would have taken online test for the very first time and this situation would have affected them to evaluate the process positively.

ConclusionConclusion

• The perception of fairness is much more important than whether the test is fair or not.– Today, it is not sufficient for the assessment tools to

be valid and reliable. The perception of candidates towards the tools eliminates the importance of psychometric properties and influences perceptions about the organization.

– Therefore, even if we have considerably strong tools as developers and practitioners, it may lead adverse perceptions if implementation of these tools is not combined with modern technology.

ConclusionConclusion

• Each new generation makes use of technology more widely compared with the previous generation. – Online testing may be considered as new and

different. However it is obvious that it will become second nature for next generations in the near future.

– Therefore it can be foreseen that the positive impact of using technology in testing will disappear in the future.

SuggestionsSuggestions

• It is suggested that new methods should be implemented in online testing in order to manage the perceptions of candidates.– In the near future, it will be a must for the tests to be

available on social networks like Facebook and mobile devices which are indispensable for the new generation.

Thank you!