8/7/2019 Effects On

1/20

Effects on employeesinformation security abilities

by e-learningJanne Merete Hagen

Gjvik University College, Gjvik, Norway, and

Eirik AlbrechtsenSINTEF Technology and Society, Safety Research, Trondheim, Norway

Abstract

Purpose The purpose of this paper is to measure and discuss the effects of an e-learning tool

aiming at improving the information security knowledge, awareness, and behaviour of employees.Design/methodology/approach The intervention study has a pre- and post-assessment ofknowledge and attitudes among employees. In total, 1,897 employees responded to a survey before andafter the intervention. The population is divided into an intervention group and a control group, wherethe only thing that separates the groups is participation in the intervention (i.e. the e-learning tool).

Findings The study documents significant short-time improvements in security knowledge,awareness, and behavior of members of the intervention group.

Research limitations/implications The study looks at short-time effects of the intervention.The paper has done a follow-up study of the long-term effects, which is also submitted to Information

Management & Computer Security.

Practical implications The study can document that software that support InformationSecurity Awareness programs have a short-time effect on employees knowledge, behaviour, andawareness; more interventions studies, following the same principles as presented in this

paper, of other user-directed measures are needed, to test and document the effects of differentmeasures.

Originality/value The paper is innovative in the area of information security research as it showshow the effects of an information security intervention can be measured.

Keywords Data security, E-learning, Individual behaviour, Employees, Training

Paper type Research paper

1. IntroductionMaintaining information security in an organization requires the commitment ofemployees at all levels. Without full employee commitment, security mechanisms may

be diminished or bypassed entirely (Ward and Smith, 2002; Schneier, 2004; Schultz,2005). Thomson and von Solms (2006) argue that employees compliance with anorganizational security policy can best be achieved through education and awarenesscampaigns. Though training and education are generally considered more effective thanmore formalistic measures such as procedures and controls (Hagen et al., 2008a), studiesshow that many organizations neglect to provide adequate training (Hagen et al., 2008a,b). A European Network and Information Security Agency (ENISA) report gives anoverview of training practices in 69 companies in nine European countries.

The current issue and full text archive of this journal is available at

www.emeraldinsight.com/0968-5227.htm

IMCS17,5

388

Received 26 June 2009Revised 3 August 2009Accepted 4 August 2009

Information Management &

Computer Security

Vol. 17 No. 5, 2009

pp. 388-407

q Emerald Group Publishing Limited

0968-5227

DOI 10.1108/09685220911006687

8/7/2019 Effects On

2/20

Approximately, 50 percent of the companies used computer-based training (ENISA,2007).

While the most common way to evaluate the effect of Information SecurityAwareness training is quizzes and/or before and after surveys (ENISA, 2007), our

study evaluates and discusses the effect of awareness training using a research designthat includes the use of a control group. Our experiment is carried out in the Wilh.Wilhelmsen (WW) Group, a leading maritime industry group that delivers logisticssolutions and maritime services worldwide. The effect of their computer-based securitytraining program, named Individual Security Awareness (ISA), was tested. A largepart of the program is dedicated to the various aspects of information security. Ourstudy addressed three research questions:

RQ1. Did the computer-based program result in improved employee information security knowledge, awareness, and behavior?

RQ2. Did the extent of the training (i.e. the number of modules performed in the

program) contribute to the improvement of employees information securityknowledge, awareness, and behavior?

RQ3. Why did the program either result in changes or fail to create changes in theemployees security knowledge, awareness, and behavior?

Section 2 of this paper presents an introduction to ISA; Section 3 outlines the paperstheoretical framework; Section 4 explains our applied methodology; and Section 5describes the findings. In Section 6, the findings are discussed according to thetheoretical framework. Section 7 discusses the conclusions reached.

2. The intervention project: Individual Security Awareness (ISA)In the WW Group security (including information security) has recently become ahigh priority. A corporate security policy was developed and signed by the chiefexecutive managers, user guidelines were developed, and a Corporate SecurityForum established. There are no technical restrictions on web surfing, butsome restrictions on downloading executable files. In addition, filters are used toreduce spam and malware from e-mail. To raise individual awareness, in March2008, the WW Groups own academy launched the e-learning program, ISA. ISAconsists of six modules, which introduce the following aspects of security:

. Module 1: introduction. This module introduces various security and risk issues,defines the risks, and describes the security organization and security

responsibilities.. Module 2: information security. This module focuses solely on information

security. It defines information security as confidentiality, integrity, andavailability, explains the threats that may exist to information security, andshows how employees should handle different classes of information.

. Module 3: travel security. This module explains how to deal with the risks thatmay occur while traveling, such as mugging, street robbery, kidnapping, hotelfires, diseases, accidents, etc.

Informationsecurity abilities

by e-learning

389

8/7/2019 Effects On

3/20

. Module 4: personal security. This module is about being able to take care ofyourself, your colleagues, and your family, and discusses ways to deal with riskssuch as fire, burglary, kidnapping, loss of sensitive information, etc.

. Module 5: security of facilities. This module is concerned with the workplace. It isabout protecting premises and detecting and preventing unauthorized entry tothe premises.

. Module 6: internal/external communication. This module is about being aware ofwhat and how you communicate, both internally and externally.

Even though only one module focuses solely on information security, informationsecurity is included in the other modules, too.

ISA gives an overall introduction to security and information security, but does notteach the details on how to for instance separate lure web pages from real web pages,or how to encrypt e-mail, or detect social engineering attacks. It teaches the employeesabout the risks connected to these issues and other issues. Each employee can access

and quit ISA through his/her computer, whenever he or she has time to do so. The ISAe-learning software begins with a vocal introduction, uses pictures, music, and texts toillustrate the risks, and then provides exercises that motivate reflection. There are alsomultiple-choice tests with immediate feedback, including the correct answers. Usersare free to choose whichever module they want to start with. The estimated time tocomplete one module is 10-15 minutes. Figure 1 shows a screen-shot from Module 2,information security. Here, the employee must choose the appropriate security action totake when leaving documents or a computer in a public place. If the employee makesan incorrect choice, he or she receives corresponding feedback, but can then continue tothe subsequent question.

Figure 1.ISA

IMCS17,5

390

8/7/2019 Effects On

4/20

Our intervention study includes two evaluations of the ISA e-learning software:

(1) Evaluation of ISAs effect on employees information security knowledge,awareness, and behavior.

(2) Evaluation of the training extents effect: is there any difference between theemployees who completed only Module 1, Modules 1 and 2, and those whocompleted all six modules?

The next section will discuss theories of safety and information security managementand the ways that a variety of useful measures may change employees behavior.

3. Theoretical framework: information security measures that influenceuser performanceIndividuals performance with regard to information security may be influenced by awide range of formal and informal factors: security technologies; formal organizationalstructures; awareness, values, and norms; and social relations and interactions

(Albrechtsen, 2007, 2008; Lund and Aar, 2004). One potential categorization ofinformation security measures directed at users is shown in Figure 2.Figure 2 shows what is potentially the most effective sequential ordering for use

of the categories of intervention measures (based on Rundmo, 1990; Hovden et al.,1992). First, conditions in the working environment should be changed as needed tobe appropriate for and conducive to good security-observant behavior. If thisimprovement proves insufficient, it indicates that the workers need further education.If additional education proves insufficient, it indicates that employees need the kind ofinformation that will influence their attitudes toward information security. If the effectof awareness activities proves unsatisfactory, employers should adopt sanctions and

Figure 2.Information securitymeasures directed at

IT users

Are technical and organizational preconditions for

safe and secure behaviour satisfactory?

Are the working methods safe and secure?

Are employees positive to make safe and secure

actions?

Are employees qualified to perform

safe and secure actions?

Is employees knowledge on safe and secure

working routines satisfactory?

YES

YES

YES

YES

YES OK!

Source:Albrechtsen (2008)

NO

NO

NO

NO

NO

MEASURE:

Measures improving working conditions:

- Technological security measures (e.g. access control)

- Physical measures (e.g. door locks)

- Formal administrative measures (e.g. policies and instructions)

Measures improving skills and knowledge:

- Experience-based learning (performed work activities;

experienced incidents; simulators)

- Training and education (e.g. tutorials, e-learning programs)

Measures improving attitudes:

- Information, e.g. newsletters, e-mails, web-pages, posters,

screen-savers; mouse pads; direct communication; dialogue

Selection of personel:

- Positive: engage qualified personel; security clearance

- Negative: remove persons with unacceptable behaviour

Measures improving behaviour:

- Rewards: praise; competitions; gifts; wage scale

- Sanctions: cautions; threats; punishment; financial sanctions/compensation

Informationsecurity abilities

by e-learning

391

8/7/2019 Effects On

5/20

rewards that may modify the employees behavior. In dealing with employees whomay be undesired security risks, a more careful selection of employees is indicated as afinal solution. Unqualified employees should be transferred or dismissed, and worktasks should be assigned according to individuals qualifications. The measures in

Figure 2 should be regarded as complementary. Once, the more technological andadministrative means have been chosen and changes have been implemented,additional softer resources can be used if necessary to modify individualsperformance. The higher up a chosen measure is shown in Figure 2, the more likelyimprovement will occur at the individual level. The following sections provide asynopsis of how a variety of information security measures may influence employeesbehavior.

3.1 Improving working conditionsThe working conditions or environment, within which employees perform their jobs,includes typically technological and formal administrative measures and also cultural

aspects, the norms, relations, and interactions that exist between individuals.Computer security systems should be installed that both preserve security and areusable by employees. However, since users may take short-cuts or lose their motivationdue to a poor user-interface, it is often difficult to take into consideration all therequirements that might reduce the security threat or risk level (Furnell, 2005).

In addition to technological measures, there are technical-administrative means policies, instructions, and plans that document and specify a required level ofbehavior such documents may provide the basic parameters of desired individualand organizational behavior. To date, the main emphasis in regard to nontechnicalinformation security has been to use technical-administrative measures (Dhillon andBackhouse, 2001). However, we know from organizational theory that a planned ordesired behavior may differ considerably from the behavior that actually occurs

(Braverman, 1974; Brunsson, 1989). It is, therefore, likely that security routines andtechnology may not be sufficient as single measures to influence individual awarenessand behavior (Albrechtsen, 2008). Nonetheless, it is necessary for control and forstrategic, systematic information security management (Albrechtsen and Hagen, 2008).

3.2 Improving skills and knowledgeMeasures that are designed for the purpose of improving skills and knowledge consistof either experience-based learning activities or systematic training and education. Theformer involve learning by personal on-the-job experience; the latter involve learningby participating in formal education (Hale and Glendon, 1987). Research shows thatmost people learn better by actually being involved and doing a particular job than bysitting in a classroom listening to a lecturer (Wang and Yestko, 2005; Albrechtsen,

2008).This paper focuses on both the planned training and education courses initiated by

management and the teaching of security expertise through interactive courseware.Interactive courseware is computer software and associated materials, usuallymultimedia in nature, designed for educational and training purposes that teach alesson, many times accompanied by a test or quiz. Well-designed interactivecourseware has proven to be an effective teaching mechanism and one that encouragesinteractive learning (Wang and Yestko, 2005).

IMCS17,5

392

8/7/2019 Effects On

6/20

8/7/2019 Effects On

7/20

3.5 Selection of personnelThe categories of information security-awareness measures listed above are designed,intended, and presumed to improve employees qualifications for achieving adequatesecurity performance. But the process of selecting personnel is the opposite: people are

selected to do jobs based on their qualifications (i.e. positive selection). Traditionally,this strategy has long been enforced in the security field by the requirement forpersonnel security clearances. In Norway, security clearances are legally permittedunder the Norwegian Security Act (Forsvarsdepartementet, 1998). As opposed topositive selection, negative selection means removing personnel from jobs theycannot handle in a safe and secure way. It typically means relocation or dismissal. If anemployee loses his security clearance, he will automatically be dismissed.

3.6 Combination of measuresTo ensure that an organizations information system is fully secure, a combinedapproach including a wide range of measures is needed (Albrechtsen and Hagen, 2008).

The best approach is to combine and implement all the measures discussed in previoussections to create synergies. For example, a technical administrative system must be inplace before a system of training and education is adopted, because the formal systemprovides a framework for the content of the training program. Technology must alsobe in place and is an important contributor to the overall effectiveness (Hagen et al.,2008a). The same is true in regard to our focus here: ISA should be a part ofmanagements overall information security efforts.

4. MethodologyThis section describes and discusses the research method we used for the interventionexperiment at WW Group.

4.1 Research design, data collection, and statistical analysisAlthough there have been few attempts to systematically evaluate the effects ofdifferent ISA programs (Albrechtsen, 2008), intervention studies have long been usedas part of occupational health and safety systems (Goldenhar and Schulte, 1994;Kristensen, 2005; Robson et al., 2001). The existing studies explore the effects ofplanned activities at business worksites that aim at improving the working conditionsand/or the health of workers (Kristensen, 2005). The research design of our studywas inspired by the intervention study literature that focuses on the safety researcharea.

To analyze the effects on employees security awareness and behavior, we designedan experiment to measure individual awareness and behavior both before and after the

intervention (shown in Figure 3). In cooperation with WW Groups securitymanagement, web-based survey questions were developed. The 3,994 employees weredivided randomly into an intervention group that would use the ISA e-learningsoftware and a control group that would not use the e-learning software.

First, the WW Group published a newsletter on the intranet, signed by the topmanager, announcing ISAs and the scientific experiment that would measure ISAseffectiveness. All 3,994 employees of the WW Group were encouraged to follow theinstructions given in upcoming e-mails and to participate in the experiment.

IMCS17,5

394

8/7/2019 Effects On

8/20

A week before ISA was launched; an initial survey (t1 ) was distributed to allemployees. Three weeks after the launch of ISA to the intervention group, a secondsurvey (t2) was launched. Both surveys included the same questions about knowledge,awareness, and behavior. The knowledge questions were in the form of amultiple-choice test, with three possible answers per question. The awarenessquestions were answered at five-point Likert (1932) scales; behavior questions were

answered at a five-point scale measuring frequencies of actions (from always toseldom). The second survey included additional questions about which of the ISAmodules the respondents had completed and about any changes in their awareness orbehavior that had occurred since the launching of the first survey.

The response rate was 68 percent (2,709 answers) for the first survey and 65 percent(2,587 answers) for the second survey, which is considered a good response. A total of2,456 respondents answered both surveys. This sample size was reduced to 1,897respondents after screening both the intervention group and the control group andexcluding those who did not follow the recommendations for participating in theexperiment. The final sample of 1,897 employees constituted the study population, ofwhich, 1,208 were in the intervention group, and 689 were in the control group. All of

the 1,208 respondents in the intervention group had completed module 1. From those1,208 employees, three distinct subgroups were formed (Figure 4):

(1) Subgroup A. About 631 respondents who completed Module 1.

(2) Subgroup B. Almost 115 respondents who completed Modules 1 and 2.

(3) Subgroup C. About 356 respondents who completed all modules.

A residual of 106 respondents who completed all or some of Modules 3-5 are notincluded in Step 2 analyses.

Figure 3.The research design anduse of the control group

A

B

C

A

B

C

Control

group

Control

group

t2t1

Study

pop.

Test group Test group

I

S

A

Informationsecurity abilities

by e-learning

395

8/7/2019 Effects On

9/20

In answering two basic research questions, this research design enabled us to testseveral corresponding hypotheses.

RQ1. Did implementation of ISA result in improved employee information securityknowledge, awareness, and behavior?

H01. There was no improvement of information security knowledge, awareness,and behavior at t2 compared with t1 among members of the interventiongroup.

H02. There was no change of information security knowledge, awareness, andbehavior at t2 compared with t1 among members of the control group.

RQ2. Did the extent of the training (i.e. the number of modules performed in the program)contribute to the improvement of employees information security knowledge,awareness, and behavior?

H03. There was no improvement of information security knowledge, awareness,and behavior at t2 compared with t1 among members of Subgroup A.

H04. There was no improvement of information security knowledge, awareness,and behavior at t2 compared with t1 among members of Subgroup B.

H05. There was no improvement of information security knowledge, awareness,and behavior at t2 compared with t1 among members of Subgroup C.

First, an independent-samples t-test procedure was used to compare means for both theintervention group and the control group. Since we rely on large randomized numbers(Ayres, 2007), the respondents were randomly assigned to the two groups, and only theintervention group got the training (Robson et al., 2001), any difference in responsebetween the groups can be assumed to be due to training, or a lack of training, not otherfactors.

Figure 4.

The three subgroups

Completed

module 1

Completed

module 2

Completed module

3, 4, 5 and 6

Subgroup A

(n = 631)

Subgroup B

(n = 115)

Subgroup C

(n = 356)

IMCS17,5

396

8/7/2019 Effects On

10/20

Anonymous, unique identification numbers were assigned to each respondent.This made it possible to perform time-series analyses and strengthens the validity ofthe analysis. A paired-samples t-test procedure computed the differences in valuesbetween the two variables for each group and tested whether the average differed

from 0. A paired-sample t-test was applied to test H01-H05. Cases were excludedlist-wise.

4.2 IndexesThe following single items measuring security knowledge were responded to ona three-point multiple-choice scale. They measured employee knowledge accordingto the learning objectives in the e-learning software. To make the scale equal tothe scales used to measure awareness and behavior, the scale was transformed toa binary scale of 1 and 5 so that a wrong answer equals 1 while a correct answerequals 5.

Knowledge indexes. definition of risk;

. definition of security policy;

. definition of integrity; and

. definition of physical security.

Based on factor analyses, the following awareness indexes were constructed to analyzethe intervention outcomes for employee information security knowledge, awareness,and behavior. Cronbachs a is used to measure the reliability of the indexes, and avalue above 0.7 is usually considered as satisfactorily (Ringdal, 2001).

Awareness indexes.

Security versus functionality. Information security perceived as not being anobstacle and not only being a technological challenge, consisting of five items,Cronbachs a 0.67.

. Reporting. Willingness to report a colleague or a superior who breached securityto the security management, consisting of two items, Cronbachs a 0.76.

. Importance of generic security and safety means. Perceived importance offollowing security guidelines, of health, environment, and safety management,and of fire protection, consisting of four items, Cronbachs a 0.80. Thereliability of all awareness indexes for the first data set was satisfactory.

It is a concern that people can claim that they understand the vulnerability of writingdown passwords and locking PCs, but not actually do it. Therefore, the questionnaire

also had several questions regarding the frequency of certain security behavior.We examined how often the respondents performed different information securitytasks such as keeping passwords secret. The items were responded to on a five-pointscale from always to seldom. Some items had skewed distributions; for instance,for the questions on how to treat sensitive information, where more than 90 percentagreed with the correct statements. As it would be difficult to get significant changesfor these items, they were excluded from the analyses. Only the following itemsmeasuring security behavior were included in the analysis.

Informationsecurity abilities

by e-learning

397

8/7/2019 Effects On

11/20

Behavior indexes

. write down passwords on paper;

. lock the PC whenever leaving it; and

.

reporting incidents when detected.

4.3 Limitations of the studyOne limitation of the study was that, due to time constraints, a pre-test was notconducted outside the project group. A trade-off had to be made to meet the WW Groupmanagements objective in launching ISA on time. Some of the employees who

participated had difficulty interpreting some of the questions in the survey. That kindof feedback could have been sorted out in a pre-test.

The major weakness of the study was the short-time distance between the ISAintervention and the two surveys, the first survey at t1, a week before ISA was launchedand the second survey at t2 three weeks after ISA was launched. Because of certain

practical arrangements, no longer time schedule was possible at this time. Therefore,only short-term effects were measured. It would be interesting, however, to determine

the long-term effects as well, so as to study how knowledge, awareness, and behaviorchange over time. The research design worked for the measurement of short-timeeffects, and it will probably work for the measurement for long-term effects as well.

Another weakness with the study is the meaning of the test scores obtained. Theyare a matter of discussion because the ability to answer the questions does notnecessarily correlate with critical security-related changes in job behavior (Schultz,2004). An independent experiment on, for example, social engineering could have

strengthened the internal validity, but was not possible to arrange within the limitedtime constraints.

There are, of course, differences between a training context and real-life

decision-making and actions. A security awareness training program cannot coverall possible security problems and dilemmas an employee faces in his/her working day.However, any training program serves as a preparation for employees decisions and

actions by training the ability to anticipate monitor and respond to any securitychallenges, independent of the topics covered by the training. Our research design,based on pre- and post-surveys, also address a few concrete security issues. We must

thus assume, if the quantities surveys indicate changes, that the training programinfluence the way employees think and act regarding information security on a general

basis. Furthermore, we cannot validate whether the employees actually behave in adifferent way when it comes to practical problems in the real world. This is a generalproblem for all quantitative research on individual attitudes and behavior what

people state regarding their attitudes and behavior may differ from how they actuallythink and behave. Nevertheless, we must assume that indications of changes inattitudes, knowledge or behavior provided by quantitative surveys, also reflect changes

in real-life. In the long-term run, it is possible to study whether the training program hasmaterialized into changed awareness and behavior in the organization, by studying ifthere has been a decrease in the number of security incidents, which could be explained

by the training program.The next section presents the results of the experiment.

IMCS17,5

398

8/7/2019 Effects On

12/20

5. ResultsThis section first describes the demographic characteristics of the participants andthen presents the results of the experiment.

5.1 ParticipantsTable I gives details about the demographic characteristics of the two groups and theirbackground variables. The WW Group consists of several companies, but the majorityof the employees (80 percent) that participated in the experiment came from WMS ShipServices and WMS Ship Management. A majority of the participants werewell-educated: half had more than 15 years of education; an additional one-thirdhad 13-15 years of education. Every fourth participant was also a manager. x2-tests atthe 5 percent level confirm that there were no significant differences between the testgroup and the control group in regard to gender, age, education, experience, orresponsibility. Significant differences were identified, however, between the groups

VariableControl group in percent

(n 689)Intervention group in percent

(n 1,208)

GenderMale 67.9 65.0Female 32.1 35.0

Age18-25 7.7 7.926-30 13.8 15.131-35 12.8 15.636-40 16.3 18.841-50 28.3 25.551-60 17.1 17.461 4.1 3.8

Formal education (years)Up to 7 0.6 0.78-12 15.2 16.213-15 32.8 33.3More than 15 49.9 47.4Other 1.5 2.4

Employment period (years)0-1 19.2 20.22-5 25.1 28.16-10 19.8 20.011-24 28.9 23.325 7.8 8.4

Level of positionTop manager 7.4 5.3Middle manager 33.8 36.8Employee 58.8 57.9

Dedicated security responsibilityYes 32.2 40.6No 67.8 59.4

Note: n 1,897

Table I.Demographic

characteristics ofparticipants in the

intervention and controlgroups at t1

Informationsecurity abilities

by e-learning

399

8/7/2019 Effects On

13/20

with respect to which of the two companies they worked for and whether they had anydedicated security responsibilities.

The next section presents the result of the pre- and post-tests.

5.2 Pre- and post-testsIndependent-sample t-tests of the intervention group and the control group show nosignificant differences in knowledge, awareness, or behavior between them at t1.Together with the results of the x2-tests, these results show that, except for thedifference between the companies the respondents represented, and their sense ofsecurity responsibility, there was no initial skewing between the two groups. Table IIshows the results of the pre- and post-survey paired-sample t-tests.

Index t1 (mean (SD)) t2 (mean (SD)) t (df)

Knowledge

Definition of riskIntervention group 2.15 (1.81) 2.01 (1.74) 1.99 (1,016) * *

Control group 2.14 (1.80) 1.98 (1.72) 21.66 (517)Definition of security policy

Intervention group 1.78 (1.58) 1.90 (1.67) 2.07 (1,016) * *

Control group 2.03 (1.75) 2.06 (1.77) 0.37 (517) *

Definition of integrityIntervention group 1.93 (1.69) 2.27 (1.85) 4.67 (1,016) * *

Control group 2.03 (1.75) 2.03 (1.75) 0.00 (517)Definition of physical security

Intervention group 3.21 (1.99) 3.48 (1.94) 3.72 (1,016) * * *

Control group 3.27 (1.98) 3.29 (1.98) 0.25 (517)Awareness

Security versus functionalityIntervention group 3.46 (0.61) 3.56 (0.61) 6.12 (1,016) * * *

Control group 3.40 (0.61) 3.49 (0.57) 3.92 (517) * * *

ReportingIntervention group 3.80 (0.71) 3.85 (0.72) 2.32 (1,016) * *

Control group 3.76 (0.71) 3.81 (0.67) 2.08 (517) * *

Importance of generic security and safety meansIntervention group 4.04 (0.58) 4.06 (0.62) 1.50 (1,016)Control group 3.99 (0.61) 3.98 (0.58) 0.72 (517)

BehaviorWrite down passwords on paper

Intervention group 4.19 (1.07) 4.29 (0.95) 3.64 (1,016) * * *

Control group 4.14 (1.06) 4.16 (0.99) 0.30 (517)Lock the PC

Intervention group 3.84 (1.22) 3.99 (1.14) 4.24 (1,016) * * *Control group 3.85 (1.17) 3.82 (1.13) 0.75 (517)

Report incidentsIntervention group 3.90 (1.29) 4.09 (1.12) 4.67 (1,016) * * *

Control group 3.87 (1.23) 3.82 (1.21) 21.14 (517)

Notes: *p , 0.10, * *p , 0.05, * * *p , 0.001; SD standard deviation; t t-value; df degrees offreedom; scales: the indexes ranges from 5 best to 1 poorest; the test for the intervention group andcontrol group are two tailed

Table II.Results of thepaired-sample t-tests ofthe pre- and post-surveyfor the intervention groupand the control group

IMCS17,5

400

8/7/2019 Effects On

14/20

The paired-sample t-test results indicate an improvement in the information securityknowledge, awareness, and behavior of the employees in the intervention group ascompared with those in the control group, although the awareness of members of thecontrol group also changed to some extent. The knowledge indexes show significant

improvements for the intervention group for all indexes except risk, revealing a greaterunderstanding of security policy, physical security, and integrity of information.The results of the paired-sample t-tests show that both the intervention group andthe control group improved their awareness of the necessity to reporting incidentsand their view on security versus functionality. After the intervention took place,employees in the intervention group showed improved behavior in protecting access totheir computers. They reported security violations and incidents more often, andlocked their PCs more often whenever they left them. In addition, they did not writedown passwords as often as they had before. All these aspects are a focus of the ISAprogram.

To validate our findings, we asked the participants in the intervention group whythey thought their attitudes had changed. Of the 736 employees who answered thisquestion, 66.3 percent reported that it was due to their use of ISA. We also asked therespondents in the intervention group their opinion of the learning effects of ISA. Ofthe 1,206 employees in the intervention group, 49 percent reported that they hadchanged in their use of the internet, 45.8 percent had changed the way they kept theiruser names and passwords secret, and 55 percent noted a change in their awareness ofhow to treat internal and sensitive information. About 55 percent reported an increasein their attention to security incidents, 31.4 percent reported a change in how theymanage visitors to the site, 20 percent noted a change in their willingness to reportsecurity incidents and weaknesses, while 40.4 percent reported they had had a changein attitude toward the importance of information security versus productivity.

These results correspond to the findings of the paired-sample t-tests and confirm

that the short-term effect of ISA in the WW Group was improvement in the employeesinformation security knowledge, awareness, and behavior even though far from allemployees completed the program.

Nonetheless, when 23 top managers representing different parts of the WW Groupwere questioned five months after ISA was launched, they reported a diversified viewof the observed effects of ISA. None of them had noticed any increase in reportedsecurity violations, but some of the managers found that they personally had becomemore aware of their own security behavior and also noticed more discussions aboutsecurity in their organizations.

Altogether, given these findings, we reject H02 of no effect of ISA. The controlgroup showed three significant changes in knowledge and awareness regardingsecurity policy, reporting, and their attitudes towards security versus functionality.

These findings may be explained by the effect that participating in the experimentsinfluenced their awareness. Therefore, we can partly reject H03 that there were nochange in the control group. In the next section, we continue with our analysis of theeffects of extent of training variations on employees.

5.3 Post-test results: RQ2The intervention group was divided into three subgroups according to how muchof ISA they had completed. Table III shows that completing all six modules of ISA

Informationsecurity abilities

by e-learning

401

8/7/2019 Effects On

15/20

proved to have the greatest effect on employee information security knowledge,awareness, and behavior. In Subgroup C, nine indexes showed significantimprovements, while in Subgroups A and B, respectively, six and three indexesimproved significantly. At t2, the mean values of the nine indexes of Subgroup C

Index t1 (mean (SD)) t2 (mean (SD)) t (df)

KnowledgeDefinition of risk

Subgroup A 2.18 (1.83) 2.03 (1.75) 1.57 (542)Subgroup B 2.08 (1.78) 2.42 (1.92) 1.38 (92)Subgroup C 2.08 (1.78) 1.86 (1.64) 21.83 (298) *

Definition of security policySubgroup A 1.72 (1.54) 1.81 (1.61) 1.21 (542) *

Subgroup B 1.77 (1.59) 1.82 (1.62) 0.23 (92)Subgroup C 1.90 (1.67) 2.10 (1.68) 1.74 (298) *

Definition of integritySubgroup A 1.86 (1.64) 2.04 (1.76) 1.98 (542) *

Subgroup B 1.90 (1.68) 2.33 (1.89) 1.99 (92) * *

Subgroup C 2.08 (1.78) 2.66 (1.97) 4.75 (298) * * * *

Definition of physical securitySubgroup A 3.14 (2.00) 3.48 (1.95) 3.29 (542) * *

Subgroup B 3.32 (1.99) 3.15 (2.00) 0.67 (92)

Subgroup C 3.29 (1.98) 3.52 (1.94) 1.81 (298) *AwarenessSecurity versus functionality

Subgroup A 3.43 (0.58) 3.53 (0.58) 4.43 (542) * * * *

Subgroup B 3.34 (0.61) 3.46 (0.59) 1.98 (92) *

Subgroup C 3.53 (0.62) 3.63 (0.67) 4.00 (298) * * *

ReportingSubgroup A 3.80 (0.70) 3.79 (0.71) 20.25 (542)Subgroup B 3.72 (0.82) 3.81 (0.76) 1.32 (92)Subgroup C 3.78 (0.70) 3.93 (0.74) 4.04 (298) * * * *

Importance of generic security and safety meansSubgroup A 4.04 (0.58) 4.05 (0.62) 0.37 (542)Subgroup B 3.99 (0.58) 3.95 (0.72) 20.52 (92)Subgroup C 4.04 (0.59) 4.12 (0.59) 2.23 (298) * *

BehaviorWrite down passwords on paper

Subgroup A 4.12 (1.07) 4.24 (0.95) 2.98 (542) * * *

Subgroup B 4.17 (1.16) 4.28 (0.93) 1.23 (92)Subgroup C 4.28 (1.00) 4.36 (1.90) 1.74 (298) *

Lock the PCSubgroup A 3.85 (1.20) 3.92 (0.16) 1.45 (542)Subgroup B 3.81 (1.34) 4.05 (1.15) 2.49 (92) * *

Subgroup C 3.91 (1.17) 4.07 (1.11) 22.56 (298) * *

Report incidentsSubgroup A 3.91 (1.28) 4.06 (1.13) 22.59 (542) *

Subgroup B 3.98 (1.26) 4.06 (1.09) 20.70 (92)Subgroup C 3.92 (1.32) 4.19 (1.11) 24.02 (298) * * * *

Notes: *p , 0.10, * *p , 0.05, * * *p , 0.005, * * * *p , 0.001; SD standard deviation; t t-value;df degrees of freedom; scales: the indexes ranges from 5 best to 1 poorest; the tests for theintervention group and control group are two tailed

Table III.Independent samplet-tests when two ISAmodules were completedversus more

IMCS17,5

402

8/7/2019 Effects On

16/20

are higher than the mean values of Subgroups A and B. Subgroup B members showonly three significant improvements, which is a bit surprising because their modulesfocused on information risk and security, and we had expected that responses to thetypical computer security questions would show employee development. Based on

these results, H03-H05 can be rejected.

6. Discussion6.1 Did ISA change employees information security knowledge, awareness, andbehavior in the short-term?The theoretical model discussed in Section 2.1 describes how security measures can bedirected at employees so as to influence their behavior. The WW Group had thetechnical and organizational measures in place before the ISA experiment started, butaccording to the security management, there was room for improvement to achievecompliance with the security policy and guidelines. Also, the use of sanctions andrewards (Step 4) and selection of personnel for security reasons (Step 5), were not

applied.The intervention study documents that ISA managed to significantly change thesecurity knowledge, awareness, and behavior of employees in the intervention group.These findings are well in line with Wang and Yestko (2005) who found thatwell-designed interactive courseware show effects in improving teaching effectivenessand encouraging active learning. The statistical results were confirmed by answersgiven by the participants. However, the top managers reported diversified views of theeffects of the ISA experiment and did not notice any change in reported incidents. Onehypothesis for this finding is that reported incidents are filtered out on their way upthrough the hierarchical structure of the organization, so top management does notperceive any change.

There were significant changes in both the intervention and the control group in

respect to improved attitudes toward security versus functionality and reporting ofsecurity violations. This may be explained by the Hawthorne effect (Olson et al., 2004),that the employees were influenced by the experiment itself, and adjusting theirbehavior towards what was expected. The Hawthorne effect may be caused by thepromoting activities of the ISA program before and under the experiment.

The many skewed answers in our study indicate that many employees initiallyalready had a high level of awareness of some information security issues. The skewedanswers correspond well with the finding that many of the employees who participatedin the study were not only well-educated, but many also working with security issues.This fits well with the findings of Albrechtsen (2008) that user involvement is the bestmethod to get employees security conscious. The relatively limited potential forimprovement may explain why the measured improvements themselves, though

significant, were not extreme.One main effect of ISA is recognition of the necessity for employees to report every

security incident that is detected, whether it involves a superior or a colleague at thesame level. This finding is important for three reasons. First, it is expected thatemployees will confront a superior or a colleague regarding security and report anylapse to the security manager. Second, according to the findings of Wiant (2005) andHagen and Spilling (2009), an increase in reporting will have a deterrent effect,preventing future incidents. Third, it will also give the security management a more

Informationsecurity abilities

by e-learning

403

8/7/2019 Effects On

17/20

up-to-date picture of the companys overall security status, providing them with bettersecurity management information. Hagen (2009) found in her study a correlationcoefficient of 0.6 between detecting and reporting of security incidents. Employeeswere reluctant to report a colleague or a superior, they lacked sufficient security

knowledge, or considered an incident insignificant, or they did not consider securitytheir responsibility. The results of our study show that computer-based training, likeISA, where the aim is to create greater security knowledge and awareness amongemployees, can influence some of these undesired attitudes among employees, at leastin the short term. A study of long-term effect can provide more advice regarding thenecessity to frequently repeat training.

6.2 Were there differences between the employees that completed Modules 1 and 2 andthose that completed all modules?Employees are required and expected to comply with their organizations securitypolicies. However, while the security management works as a counter-balance to move

away from the boundaries of unacceptable risk, employees work under pressure frommanagement to move toward optimum efficiency and their own goal of exerting theleast effort (Rasmussen, 1997). We observed exactly this phenomenon in ourexperiment when the managers asked: what is all this security issue about anyway?Besides, while most of the people in the test group completed Module 1, fewercompleted Module 2, and there was a significant drop with Module 3, and remainingmodules. Moreover, the qualitative answers given by the respondents in the twosurveys indicate a conflict in goals participating in the security training versusdoing their daily jobs as one of their reasons for not completing ISA.

The results show varying significant improvements in employee informationsecurity knowledge, awareness, and behavior among those in Subgroups A-C, asdocumented in Section 5.3. One possible explanation for this variation is that the people

participating in the experiment had a relatively high level of competence and thus therewas only limited room for improvement, from good to even better, if just a part ofISA was completed. Another possible explanation may lay in the way the questionswere raised: many of them focused on general security and risk management issues.The findings confirm that, in comparison to completing only parts of ISA, completingthe entire program results in an increase in employee security knowledge, awareness,and behavior. Employees should therefore be encouraged to complete all the ISAmodules.

6.3 Why did the intervention result in changes in employee awareness and behavior?The results indicate that ISA was shown to be an effective method for trainingemployees, which is well in line with existing theory on interactive courseware (Wang

and Yestko, 2005). However, according to the experiment indicators, the control group,which was exposed to the promotion and questionnaires, also showed a significantchange over time in three indexes, but not in knowledge or behavior. The followingdiscussion aims to clarify how this might happen.

While the intervention consisted basically of the ISA e-learning software, newsabout the launching of the program and the scientific experiment was first publishedon the intranet. In an effort to promote employee participation, before the experimentbegan, a trial module test was also made available on the intranet. Therefore, everyone

IMCS17,5

404

8/7/2019 Effects On

18/20

in the organization received some information about what was going to happen andthat the focus was on security. This activity may have had two impacts: a largeresponse rate and engagement in the experiment, but also a psychological side effectsimilar to the Hawthorne effect, influencing the awareness also in the control group.

Through the given answers in the surveys it became clear that the participants learnednot just only from the ISA e-learning software, but also from the surveys theyparticipated in and their interactions and contacts with colleagues.

7. ConclusionsThe implementation of ISA in the WW Group provides a large-scale, computer-basedand standardized security training that can facilitate employee compliance with theorganizations security policies by raising individual security knowledge andawareness. We conducted an experiment to evaluate ISAs effectiveness in those areas.

Our results show that the program had a significant short-term effect on employeesecurity knowledge, awareness and behavior. There were significant differences

between the intervention subgroups and that, in order to get the full benefit of thetraining, all employees should be encouraged to complete the entire program. ISA alonewas shown to have a significant effect on improving employee security knowledge andbehavior. The combination of the ISA e-learning software with surrounding activities,ISA promotion, and surveys may all have contributed to the observed change in allemployees security awareness, as seen in the changes in the awareness of the controlgroup. Good promotion contributed probably to a high-response rate at the cost of theHawthorne effect. One lesson learned from the experiment is to discuss research designand questions with physiologists to eliminate any psychological side effects that mightoccur during such experiments.

Finally, it should be noted that the long-term effect has not yet been analyzed, andthat individual learning is not the same as organizational learning, where the latter

results in a change in common understanding, relations, and interactions. Ourintervention study did not use a group-based approach in which employees could shareknowledge and experience. Rather, ISA is a tool for raising individual employeessecurity awareness and, as such, is a good starting point for building a corporatesecurity culture based on common values and attitudes. The experiment showed thatISA itself started some knowledge-sharing processes in the organization.

This study has focused on the short-term effects of ISA. We intend to continue witha follow-up study on the long-term effects of the program. In this follow-up study, wewill discuss computer-based training compared with human intervention and actionresearch and their effects on organizational learning.

References

Albrechtsen, E. (2007), A qualitative study of users view on information security, Computers& Security, Vol. 26 No. 4, pp. 276-89.

Albrechtsen, E. (2008), Friend or foe? Information security management of employees,Thesis No. 2008:101, Norwegian University of Science and Technology, Trondheim.

Albrechtsen, E. and Hagen, J. (2008), Information security measures influencing userperformance, in Martorell, S., Soares, C.G. and Barnett, J. (Eds), Proceedings of Safety,

Reliability, and Risk Analysis: Theory, Methods, and Applications, CRC Press, London,pp. 2649-56.

Informationsecurity abilities

by e-learning

405

8/7/2019 Effects On

19/20

Ayres, I. (2007), Super Crunches: How Thinking by Numbers is the New Way to be Smart, BentamBooks, New York, NY.

Braverman, H. (1974), Labor and Monopoly Capital: The Degradation of Work in the TwentiethCentury, Monthly Review Press, New York, NY.

Brunsson, N. (1989), The Organization of Hypocrisy: Talk, Decisions, and Actions in Organizations,Wiley, Chichester.

Dhillon, G. and Backhouse, J. (2001), Current directions in IS security research: towardssocio-organizational perspectives, Information Systems Journal, Vol. 11 No. 2, pp. 127-53.

ENISA (2007), Information Security Awareness Initiatives: Current Practice and theMeasurement of Success, European Network and Information Security Agency, Heraklion.

Forsvarsdepartementet (1998), Lov om forebyggende sikkerhetstjeneste (Sikkerhetsloven). TheNorwegian Security Act, Forsvarsdepartementet, Oslo.

Furnell, S. (2005), Why users cannot use security, Computers & Security, Vol.24 No. 4,pp. 274-9.

Goldenhar, L.M. and Schulte, P.A. (1994), Intervention research in occupational health andsafety, Journal of Occupational Medicine, Vol. 36 No. 7, pp. 763-75.

Hagen, J.M. (2009), How do employees comply with security policy? A comparative case studyof four organizations under the Norwegian Security Act, The Human Factor behind theSecurity Perimeter. Evaluating the Effectiveness of Organizational Information Security

Measures and Employees Contributions to Security, doctoral dissertation, The Faculty ofMathematics and Natural Sciences, University of Oslo, Oslo.

Hagen, J.M. and Spilling, P. (2009), Do organizational security measures contribute to thedetection and deterrence of IT-system abuses?, Proceedings of the 3rd InternationalConference on Human Aspects of Information Security and Assurance (HAISA 2009).

Hagen, J.M., Albrechtsen, E. and Hovden, J. (2008a), Implementation and effectiveness oforganizational information security measures, Information Management & ComputerSecurity, Vol. 16 No. 4, pp. 377-97.

Hagen, J.M., Kalberg-Sivertsen, T. and Rong, C. (2008b), Protection against unauthorized accessand computer crime in Norwegian enterprises, Journal of Computer Security, Vol. 16,pp. 341-66.

Hale, A.I. and Glendon, A.I. (1987), Individual Behavior in the Control of Danger, Elsevier,Amsterdam.

Hovden, J., Ingstad, O., Mostue, B.A., Rosness, R., Rundmo, T. and Tinnmansvik, R.K. (1992),Ulykkesforebyggende arbeid (Accident Prevention), Yrkeslitteratur, Oslo (in Norwegian).

Hubbard, W. (2002), Methods and techniques of implementing a security awareness program,SANS Institute White Paper, SANS Institute, Bethesda, MD.

Iversen, H., Rundmo, T. and Klempe, H. (2005), Risk attitudes and behavior among Norwegianadolescents: the effects of a behavior modification program and a traffic safety campaign,

European Psychologist, Vol. 10 No. 1, pp. 25-38.

Klinke, A. and Renn, O. (2002), A new approach to risk evaluation and management: risk-based,precaution-based, and discourse-based strategies,Risk Analysis, Vol. 22 No. 6, pp. 1071-94.

Kristensen, T.S. (2005), Intervention studies in occupational epidemiology, Occupational and Environmental Medicine, Vol. 62 No. 3, pp. 205-10.

Likert, R. (1932), A technique for the measurement of attitudes,Archives of Psychology, Vol. 140,pp. 1-55.

Lund, J. and Aar, L.E. (2004), Accident prevention: presentation of a model placing emphasison human, structural, and cultural factors, Safety Science, Vol. 42 No. 4, pp. 271-324.

IMCS17,5

406

8/7/2019 Effects On

20/20

Olson, R., Verley, J., Santos, L. and Salas, C. (2004), What we teach students about theHawthorne studies: a review of content within a sample of introductory I-O and OBtextbooks, The Industrial-organizational Psychologist, Vol. 41 No. 3.

Rasmussen, J. (1997), Risk management in a dynamic society: a modeling problem, Safety

Science, Vol. 17 Nos 2/3, pp. 183-213.Ringdal, K. (2001), Enhet og mangfold: samfunnsvitenskapelig forskning og kvantitativ metode

(Unity and Diversity: Social Science and Quantitative Methods ), Fagbokforlaget, Bergen(in Norwegian).

Robson, L.S., Shannon, H.S., Goldenhar, L.M. and Hale, A.R. (2001), Guide to evaluating theeffectiveness of strategies for preventing work injuries: how to show whether a safetyintervention really works, NIOSH Publication No. 2001-119, NIOSH, Cincinnati, OH.

Rundmo, T. (1990), Atferdsvitenskaplig sikkerhetsforskning (Safety Research on Behavior),SINTEF Report STF75A9007, SINTEF, Trondheim (in Norwegian).

Schneier, B. (2004), Secrets and Lies: Digital Security in a Networked World, Wiley, Indianapolis, IN.

Schultz, E. (2004), Security training and awareness: fitting a square peg in a round hole,Computers & Security, Vol. 23 No. 1, pp. 1-2.

Schultz, E. (2005), The human factor in security, Computers & Security, Vol. 24 No. 6, pp. 425-6.

Thomson, K.-L. and von Solms, R. (2006), Towards an information security competencematurity model, Computer Fraud & Security, No. 5, pp. 11-15.

Voss, B.D. (2001), The ultimate defense of depth: security awareness in your company,SANS Institute White Paper, SANS Institute, Bethesda, MD.

Wang, A.J.A. and Yestko, K. (2005), Building reusable information security courseware,paper presented at the 2005 Information Security Curriculum Development Conference.

Ward, P. and Smith, C.L. (2002), The development of access control policies for informationtechnology systems, Computers & Security, Vol. 21 No. 4, pp. 365-71.

Wiant, T.L. (2005), Information security policys impact on reporting security incidents,Computers & Security, Vol. 24 No. 6, pp. 448-59.

Corresponding authorJanne Merete Hagen can be contacted at: janne.hagen@hig.no

Informationsecurity abilities

by e-learning

407

To purchase reprints of this article please e-mail: reprints@emeraldinsight.comOr visit our web site for further details: www.emeraldinsight.com/reprints