A Comparison of Web-Based and Paper-Based Survey Methods

http://erx.sagepub.com/Evaluation Review

http://erx.sagepub.com/content/33/5/464The online version of this article can be found at:

DOI: 10.1177/0193841X09340214

2009 33: 464 originally published online 15 July 2009Eval RevCorey Greenlaw and Sharon Brown-Welty

Testing Assumptions of Survey Mode and Response CostA Comparison of Web-Based and Paper-Based Survey Methods:

Published by:

http://www.sagepublications.com

can be found at:Evaluation ReviewAdditional services and information for

http://erx.sagepub.com/cgi/alertsEmail Alerts:

http://erx.sagepub.com/subscriptionsSubscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

http://erx.sagepub.com/content/33/5/464.refs.htmlCitations:

What is This?

- Jul 15, 2009 OnlineFirst Version of Record

- Sep 3, 2009Version of Record >>

at U.A.E University on October 26, 2014erx.sagepub.comDownloaded from at U.A.E University on October 26, 2014erx.sagepub.comDownloaded from

http://erx.sagepub.com/

http://erx.sagepub.com/content/33/5/464

http://www.sagepublications.com

http://erx.sagepub.com/cgi/alerts

http://erx.sagepub.com/subscriptions

http://www.sagepub.com/journalsReprints.nav

http://www.sagepub.com/journalsPermissions.nav

http://erx.sagepub.com/content/33/5/464.refs.html

http://erx.sagepub.com/content/33/5/464.full.pdf

http://erx.sagepub.com/content/early/2009/07/15/0193841X09340214.full.pdf

http://online.sagepub.com/site/sphelp/vorhelp.xhtml



A Comparison ofWeb-Based andPaper-Based SurveyMethods

Testing Assumptions of SurveyMode and Response Cost

Corey GreenlawFresno County Office of Education

Sharon Brown-WeltyCalifornia State University, Fresno

Web-based surveys have become more prevalent in areas such as evaluation,

research, and marketing research to name a few. The proliferation of these

online surveys raises the question, how do response rates compare with tra-

ditional surveys and at what cost? This research explored response rates and

costs for Web-based surveys, paper surveys, and mixed-mode surveys. The

participants included evaluators from the American Evaluation Association

(AEA). Results included that mixed-mode, while more expensive, had higher

response rates.

Keywords: Web-based surveys; response rates; online surveys; mixed-

mode surveys; survey costs

Surveys are an integral part of an evaluator’s toolkit. They can be an

effective means of collecting subjects’ opinions, demographics, or

feedback in a straightforward and potentially low-cost manner. Although

the data gathered through surveys can be useful, there are certain considera-

tions (such as the validity of the data analysis due to low response rates and

the cost of producing and administering the survey), which may affect the

decision about the method of data collection chosen.

Authors’ Note: Please address correspondence to Sharon Brown-Welty, California State Uni-

versity, Fresno, 5005 N. Maple Ave., MS ED 117, Fresno, CA 93740; e-mail:

[email protected].

Evaluation Review

Volume 33 Number 5

October 2009 464-480

# 2009 SAGE Publications

10.1177/0193841X09340214

http://erx.sagepub.com

hosted at

http://online.sagepub.com

464

at U.A.E University on October 26, 2014erx.sagepub.comDownloaded from


Another consideration important to evaluators is the validity of the sur-

vey results as a result of response rates (Krosnick 1999; Cook, Heath, and

Thompson 2000; Paolo et al. 2000; Sheehan 2001; Idleman 2003; Mertler

2003; McCabe 2004; Kiernan et al. 2005). If one aspect of survey metho-

dology consistently produces significantly more responses from partici-

pants, there is a greater likelihood questions can be answered and error

can be minimized (Dillman 2000). Response rates, particularly low

response rates, were the subject of a study by Cook, Heath, and Thompson

(2000). Specifically, a mean response rate of 39.6% was found in a meta-

analysis of response rates where the researchers reviewed response rates for

68 surveys included in 49 different studies. When Cook, Heath, and

Thompson only looked at surveys where there were no missing data, the

mean response rate was even lower, 34.6%. In contrast, Converse et al.

(2008) conducted a study that explored response rates using a mixed-

mode of e-mail/mail and mail/e-mail administrations that resulted in a

higher response rate. The subject of the survey was salient to participants

in the study as it was related to the National Board for Professional Teach-

ing Standards assessment for recognizing accomplished educators.

The survey by Converse et al. (2008) was sent to 1,500 PreK-12 teachers

in two states and yielded an overall response rate of 76.3%, a higher

response rate than is normally seen in survey response rates (Cook, Heath,

and Thompson 2000; Sax, Gillmartin, and Bryant 2003). For the initial con-

tact in the study of Converse et al., 61.3% responded and an additional

15.0% responded to the second contact. Disaggregated further, there was

an 80.7% response rate for the first contact of the mail-survey mode and

a 41.8% response rate for the first contact of the e-mail-mode administra-

tion. The authors calculate the costs for the mail/e-mail survey to be higher

than the e-mail/mail administration (US$5.32 for the former administration

as compared to US$4.95 for the latter; p. 105).

Web-based surveys have been purported to be a means of collecting data

from large sample groups quickly and with minimal cost (Schonlau,

Fricker, and Elliott 2002). The design, dissemination, data storage, and data

analysis of Web-based surveys is efficient and is becoming more

user-friendly with the introduction of multiple survey Web sites (e.g.,

Zoomerang.com). Furthermore, using Web-based surveys and a mixed-

mode approach can increase the response rate that may result in a more

valid analysis of the data collected. The effectiveness of this mode of sur-

vey data collection has been researched using many different populations

and in many different settings (Heberlein and Baumgartner 1978;

Cobanoglu, Warde, and Moreo 2001; Shannon et al. 2002; Archer 2003;

Greenlaw, Brown-Welty / A Comparison of Web-Based and Paper-Based Survey Methods 465



Carini et al. 2003; Idleman 2003; Mertler 2003; Sax, Gillmartin, and Bryant

2003; Sun and McClanahan 2003; Kaplowitz, Hadlock, and Levine 2004;

McCabe 2004; Kiernan et al. 2005). The results of these studies, however,

are mixed. With the greater acceptance of an online environment and

increased connectivity, continued research into this mode of survey

administration is warranted.

One single administration mode can serve to provide ample data in some

situations but may not be the best approach. Dillman (2000) suggested that

the mixed-mode administration is a more dynamic approach to surveying.

He noted that the different approaches ‘‘provide an opportunity to compen-

sate for the weaknesses of each method’’ (p. 218). The ability to provide

multiple means of survey completion has been shown to have an impact

on response rate in many studies (e.g., McCabe 2004; Kiernan et al.

2005), and as increased response rate is a prime factor in survey validity,

serious consideration should be given to this method.

Although there have been mixed results from studies measuring

response rates between paper-based and Web-based surveys in the past,

recent studies have demonstrated an increase in Web-based response rates

as compared to paper-based response rates (Sax, Gillmartin, and Bryant

2003; McCabe 2004; Kiernan et al. 2005). These studies also suggest that

in similar populations where participants are provided multiple options,

they are more often choosing the Web-based survey method. Even so, pro-

viding response method choice has a direct impact on response rates, which

may then have an impact on the validity of the data analyses and results.

When conducting a survey, response rate is not the only consideration.

Researchers and evaluators must function within the constraints of budgets

and must consider which survey mode will meet the needs of the study

while not depleting too many resources. The costs associated with each sur-

vey mode become an important variable; one that must be measured.

Whether those costs consist of an overall total or are a part of an evaluation

plan, the survey administration cost need to be taken into account.

The costs associated with survey administration are many and varied

and depend on the administration mode chosen. The primary costs for most

surveys include the consumables (i.e., paper and postage) and labor (i.e.,

stuffing envelopes, inputting data, and creating the Web survey). The con-

sumables required to complete a paper-based survey are considerable and

can expend considerable resources. Postage for the prenotification, produc-

tion of the survey, and the reminder and possible follow-up survey can

quickly become costly (Shannon and Bradshaw 2002). Time spent attach-

ing addresses, stuffing and sealing envelopes can also amount to a

466 Evaluation Review



significant time and cost investment. In addition, once the survey is

returned, the data are usually entered into a database or spreadsheet, which

can be laborious as well as time-consuming (Cobanoglu, Warde, and

Moreo 2001). At times, the costs associated with the paper-based adminis-

tration mode can be minimized by using graduate students or volunteers.

For an evaluator or researcher in private practice or within a company,

however, these costs must be accounted for and included in their budgets.

The costs associated with the Web-based survey administration are dif-

ferent than those associated with the paper-based survey and tend to be less

costly. One cost for Web-based surveys that may be higher than for paper-

based surveys are the costs associated with labor; the survey must be

entered into a Web site designed for survey administration. Authors have

reported that creating the survey online was a significant cost and required

specialized knowledge (Shannon and Bradshaw 2002). However, as tech-

nology has improved, the user interface for these types of Web sites has

made this input process much less time-consuming, and therefore, less

costly. It is now possible to create a survey, upload e-mail addresses, and

distribute the survey in a few hours, at relatively little expense as compared

with the mailing expense of paper-based surveys. In addition, when

participants reply to the survey, the results are immediately recorded on the

Web site for later download. The electronic reply eliminates the data input

process, which further reduces the time element as well as decreases the

potential for transcription errors.

Comparing costs related to different administration modes has proven to

be somewhat difficult because of the way in which survey costs are calcu-

lated. Furthermore, the data reported for survey costs associated with admin-

istration mode were varied within the literature. In studies by Cobanoglu,

Warde, and Moreo (2001) and Shannon and Bradshaw (2002), only the total

cost of each administration type was used in contrast to Archer (2003), who

noted each individual cost related to each survey administration. In a com-

pletely different fashion, Kaplowitz, Hadlock, and Levine (2004) reported

the cost for each survey administration by cost per response. This final

method, cost per response, provided a compelling blend of both response rate

data and the calculation of the costs required to obtain each response.

Purpose

The purpose of this study was twofold; first, it explored the possibility of

a significant difference in response rates between a Web-based survey




administration, a traditional paper-based survey administration, and a

mixed-mode method that used both Web and paper-based methods. The

second issue examined in the study was which survey mode was the most

cost-effective mode of survey administration.

Study Process

To study the difference between Web-based and paper-based survey

response rates, collaboration was formed with the American Evaluation

Association (AEA) to create a survey that would collect data regarding sal-

ary and work-related information from its membership. The approximately

4,000 members are generally very well educated and technically literate.

The employment survey served as the vehicle for a study to assess the

effectiveness of Web-based surveys as well as the associated cost of admin-

istration. The AEA was chosen primarily for two reasons; employment

information was needed and useful to the organization and the evaluative

focus of the association would increase the saliency of the data, which

could increase response rates (Groves, Singer, and Corning 2000; Groves,

Presser, and Dipko 2004). All members were randomly assigned to one of

three groups, so variables such as computer use, age, and issue saliency

were not factors in the final analysis. Other groups of professionals (e.g.,

lawyers, educators, etc.) would be closely related to the general member-

ship of the AEA and would therefore benefit from the results of this study.

The study drew survey responses from all of the general membership of

the AEA. Only residents of the United States were included in the study to

reduce the complication of monetary exchange rates. Participants also had a

listed address and a listed e-mail address to allow for random placement in

treatment and control groups. There were 1,986 total participants in this

study with 96% of the respondents holding a master or doctoral degree.

Data Collection

The data collected and analyzed were response rates and total monetary

costs for the survey administration. Response rates were calculated using

the method described in the article Standard Definitions by The American

Association of Public Opinion Research. Researchers have noted that there

are multiple ways of calculating a response rate, and as a result, it is diffi-

cult to build on previous research as their computations do not always




match. To alleviate this obstacle, AAOPR suggested using a calculation of

response rate that is the industry standard. Using this calculation allows

future studies to use the collected data in a valid manner as well as attempt

to replicate the study.

The AAOPR (2006) recommendation of using the calculations that com-

pose Response Rate 2 (RR2) best addressed the factors involved in the

study. The calculation of this formula is the number of completed surveys

(completed and partially completed) divided by the number of completed

surveys (completed and partially completed) plus the number of all non-

completed surveys (sent to a potential participant but not completed as a

result of incorrect e-mail address, incorrect mailing address, or other rea-

sons). The use of this formula standardizes response rates so that response

rate comparisons can be undertaken within the literature.

Cost Analysis

Costs for each survey administration were calculated assuming the costs

associated with the creation of the survey were the same across all admin-

istration modes. An average cost-per-hour for the labor related to the survey

production was calculated using the average of the median annual income

of a secretary and the median annual income of an office clerk as was listed

in the United States Department of Labor’s Occupational Outlook Hand-

book (2006). Using this method, the average hourly rate was calculated

to be US$14.44. This hourly rate was multiplied by the total number of

hours spent on survey production.

The costs for the paper-based mode were calculated using the cost of

production and materials including the actual printing of the survey, post-

age, and the labor required to process the survey and results. Labor calcula-

tions for the paper-based administration included sorting pages, stuffing

envelopes, affixing postage, sealing envelopes, opening and sorting

responses, and entering data into an Excel spreadsheet.

Costs for the Web-based administration included the cost of the sub-

scription to Zoomerang (the Web-based survey creation tool) and time

spent creating and sending the Web-based surveys. The annual subscription

cost for Zoomerang was US$400, which is the price of a nonprofit

subscription.

The costs for the mixed-mode administration included all of the costs

associated with both the paper-based administration and the Web-based

administration. In addition, the mixed-mode administration required




completing the tasks associated with both the previous survey administra-

tion modes.

Survey

The survey created was an employment survey that collected informa-

tion about employment status and compensation as an evaluator. Once the

general framework of the survey was completed, and to establish content

validity, a draft was sent via e-mail to four AEA members who are

professionals in survey research. Each of the professionals was asked to

take the survey. A meeting was held at the AEA national conference in

Toronto, to which all of the four professionals noted above were invited

to discuss the survey instrument and their experience while taking it.

During the meeting, suggestions were made and the survey was refined

to better meet the needs of the AEA. In addition, suggestions were made

to increase readability and to change some of the survey format to

increase response rates. A final draft of the survey was then sent to the

committee members for final feedback.

Study Design

The study consisted of a random assignment design in which partici-

pants were randomly assigned to one of three survey administration groups.

Group 1 (Web-based administration) consisted of members who

received the survey instrument in Web-based form. These members

received the survey invitation and follow-up reminders about the survey

only through e-mail. They were able to access the survey through a link

embedded in the e-mail message. This link appeared in the primary survey

invitation message and the reminder message sent to the participant.

When respondents clicked on the embedded link, a Web page opened

and they had access to the introduction to the survey. Once the participant

began the survey, he or she was able to return to complete the survey if

needed by clicking on the link provided in the e-mail. A unique identifica-

tion was created for that survey by the e-mailed link. When the respondent

completed and submitted the survey, he or she was not able to modify or

change answers.

Group 2 (paper-based administration) received their notification and the

actual survey instrument through the United States Postal Service. All




reminders took place through regular United States Postal Services.

Reminders and invitations were sent to the address listed in the AEA in

their membership roster.

Group 3 (mixed-mode administration) received both the above treat-

ment conditions. They received notification through the Internet and

through the United States Postal Service. They had the opportunity to com-

plete the survey online or complete the paper and pencil instrument that

was sent to them. Communications were conducted through the mail and

through e-mail.

A reminder postcard was sent to all members of the paper-based group

2 weeks following the initial administration of the survey. The postcard

reminded the potential respondent to complete the survey they received and

to return it in the envelope provided. The postcard also contained an e-mail

address where the respondent could e-mail the author for an additional copy

of the survey.

A unique number was assigned to each of the members included in the

mixed-mode administration group and was placed on the survey mailed to

that individual. Web-based responses were matched with the individuals in

the mixed-mode group through the use of a query in a database. The com-

pleted paper-based surveys were compared to the resulting database to

identify and eliminate any duplicates.

During the pilot study, the expected amount of time it would take a

respondent to complete the survey was established to be below 10 minutes.

This respondent burden was included in the cover letter for both the paper-

based and Web-based survey. The AEA requested that the data be collected

because it did not exist in any databases and the study collected two types of

useable data; data to explore survey response rates and data about evalua-

tor’s salaries and workloads, both of which are included under the concept

of managing respondent burden.

Results

The results of the current study demonstrated a significant difference in

response rates and costs between the paper-based administration, the

Web-based administration, and the mixed-mode administration. The

mixed-mode survey administration produced the greatest response rate but

at a considerably greater cost. The Web-based administration produced

greater results than did the paper-based administration overall and was

substantially less costly to administer.




Response Rate

The total response rate for the AEA Employment survey was 51.58%,

which was a higher response rate than many response rates reported in the

literature (e.g., Sax, Gillmartin, and Bryant 2003; Converse et al. 2008).

When comparing response rates, the mixed-mode survey administration

received the largest response rate (60.27%) with the Web-based administra-

tion following at a 52.46% response rate (see Table 1). Table 2 includes the

breakdown of response rates (RR2) by response type (paper-based and

Web-based) regardless of administration mode. When examining the data

solely by paper-based or Web-based administration (in this analysis, the

mixed-mode was separated into responses received through either the

paper-based survey or the Web-based survey), the response rate for the

Web-based administration was 61.7% as compared to the paper-based

administration with a response rate of 39.1%.

Table 1

Response Rate for AEA Employment Survey Administration

Groups

Mode N # Responses RR2

Paper-based 1,280 538 42.03%

Web-based 1,281 672 52.46%Mixed-mode 1,281 772 60.27%

Total 3,842 1,982 51.58%

Note: AEA ¼ American Evaluation Association; RR2 ¼ Response Rate 2.

Table 2

Response Rate by Response Type

Mode n # Responses RR2

Paper-based 2561a 774 39.1%Web-based 2561a 1223 61.7%

Total 1997b

Note: RR2 ¼ Response Rate 2.a Includes total number of potential respondents for each mode of survey completion. Mixed-

mode group members appear in each mode.b Includes 15 duplicate responses that were pulled before the final analysis.




To test whether there was a significant difference between the

paper-based administration, Web-based administration, or mixed-mode

administration, a one-way analysis of variance was performed. The results

indicated there was a significant difference between methods of survey

administration (F(2, 3,840) ¼ 44.799, p < .001). The results of a Tukey’s

honestly significant difference (HSD) analysis revealed that all response

rate differences were significant at the p < .01 level. Specifically, the

mixed-mode administration produced the greatest number of responses.

Finally, related to response rates, there were no significant differences in

response rates for either education level or gender.

Cost

A cost-per-response figure was calculated for each administration mode

based on the method used by Kaplowitz, Hadlock, and Levine (2004).

The Web-based survey was by far the least costly administration

(US$0.64 per response) followed by the mixed-mode administration

(US$3.61 per response). The most costly survey administration in this study

was the paper-based administration at US$4.78 per response. The cost

breakdown for each administration mode is outlined in Table 3. A graphical

representation of the response rate and costs is provided in Figure 1.

Discussion

The results of this study show that the use of Web-based survey admin-

istration produced higher response rates when administered to an educated

population with access to computers. The overall cost per response was

notably less expensive than the paper-based administration and the effort

Table 3

Cost Breakdown by Administration Group

Hours Total Cost Cost Per Response

Paper-based (RR2 ¼ 42.03%) 57 US$2,573 US$4.78

Web-based (RR2 ¼ 52.46%) 2 US$429 US$0.64

Mixed-mode (RR2 ¼ 60.27%) 44 US$2,785 US$3.61

Note: RR2 ¼ Response Rate 2.




necessary to produce and distribute the Web-based survey was markedly

reduced.

This finding is in contradiction with the findings of the research under-

taken by Converse et al. (2008). The reason for this difference may be in the

benefit the survey rendered to the respondents. In the case of Converse

et al., respondents were completing an evaluation instrument in which there

was a possibility that they would benefit personally; recognition as an

accomplished educator. In the survey administered in this study, the data

Figure 1

Response Rates and Response Cost

Res

pons

e ra

tes

30%

40%

50%

60%

70%

80%

Web−based (N = 623, RR2 = 52.46%, US$0.64/response)*

Paper−based (N = 538, RR2 = 42.03%, US$4.78/response)*

Mixed−mode (N = 772, RR2 = 60.27%, US$3.61/response)*

*Difference significant p < .01

Adapted from Kaplowitz, Hadlock, and Levine 2004.




would be valuable indirectly to the individual (e.g., Were they making

competitive salaries as compared to peers in the field?). Although the find-

ings of Converse et al. of the actual cost incurred for each survey-mode

administration was different than the costs calculated for this study, they

did find that the paper-based survey mode was more costly than the

Web-based survey mode, as was also found in this study.

While the mixed-mode group produced the greatest response rates, more

than two thirds of the responses in the mixed-mode group came from the

Web-based survey and more than 60% of the total survey responses came

via the Web-based survey. With the advent and greater use of Internet

surveys to collect multiple types of data, the current study demonstrates

how response rates can be improved with the use of Web-based surveys.

Dillman (2000) noted that ‘‘Mixed-mode surveys provide an opportunity

to compensate for the weaknesses of each method’’ (p. 218). Survey

response rates from this study demonstrate that Dillman’s suggestion was

accurate. Respondents in the mixed-mode group had the option to choose

either the paper-based format or the Web-based format to complete the

survey. The significant difference in response rate for the mixed-mode sug-

gests that people are more likely to respond when they can choose the

response format. Some respondents choose not to respond because they

were unfamiliar with technology and did not feel they have the technologi-

cal capabilities to complete an online survey. Although these numbers will

continue to diminish as computers become even more prevalent in society,

for now it is important to consider how the format of a survey administra-

tion may affect response rates.

This study found that the mixed-mode survey administration produced

the greatest response rate as compared to the other three groups. However,

there was also a significant difference in response rates when the three

groups were collapsed into paper-based response and Web-based response.

These response rates confirmed other studies that found significant differ-

ences in response rates between Web-based and paper-based administra-

tions. Kiernan et al. (2005), McCabe (2004), and Sax, Gillmartin, and

Bryant (2003) had similar results when offering respondents the option to

take the survey in either format. Specifically, people selected the online

survey significantly more often. All of the above studies had very homoge-

nous samples in that two of the studies included only college students who

tend to be more computer literate and the other study only offered the

survey to those individuals who stated that they were computer literate. The

results of the current study demonstrate a greater degree of external validity

in that respondents were selected solely on their membership in the AEA




and their computer skills (perceived or reported) were not a factor in group

placement.

The results of the study provided overwhelming support for the cost-

effectiveness of Web-based survey administrations. In addition, tasks

involved with the paper-based administration were time-consuming

(Archer 2003) and included standing in line for postage, meeting with the

printer, and deciphering the costs associated with bulk mail. With the

Web-based administration, once the subscription is paid and the survey is

created, it can be easily sent to additional respondents without cost.

Furthermore, reminders can be sent and tracked easily within the Web site.

An additional factor that becomes very important in comparing the cost

associated with survey administrations is the time required for data entry.

The paper-based results of the current survey were hand entered into an

Excel spreadsheet for use in data analysis. The data input process required

a total of 39 hours to complete. The Web-based data were already in a

digital format and only needed to be downloaded for analysis. The accuracy

of the data is also a factor that must be noted with Web-based survey

administrations. The process of inputting responses onto a spreadsheet can

lend itself to error. These errors might be avoided by allowing the partici-

pant to directly record their responses.

Limitations

The sample used in the current study may limit the ability of the results

to be generalized to other populations. The respondents in this study

appeared to have a particular interest in the results of the survey and might

have felt a professional obligation toward completing the survey. Further-

more, these participants may be more computer literate and have greater

computer accessibility due to the nature of their work. A word of caution

in generalizing these results may be that Web-based surveys should be

considered for working professionals or for responses desired from individ-

uals who are known to have access to and be users of the Internet. Some

examples of populations who might be targeted for Web-based surveys

might include postsecondary education employees, PreK-12 employees,

employees in technology fields, medical field employees, or surveys that

are targeted at issues related to employment.

The bias that may be present in the response rate may be due to the fact

that the respondents in this study regularly use surveys as tools of their

trade. Other professionals (e.g., psychologists, researchers, and marketing




professionals) use surveys and data collection instruments on a regular

basis as well and would therefore be more likely to have similar results

as were found in this study. In addition, any group or organization that is

collecting data that are of high interest (high saliency), like salary informa-

tion, might also be similar enough to the present sample to generalize the

results of this study. However, if the sample receiving the survey is not one

of the professional groups noted above, the results of the study may not be

generalizable to that sample or population.

Another possible limitation of the current study is the context in which

the data are being gathered. As noted previously, survey costs vary between

studies (e.g., Archer 2003; Kaplowitz, Hadlock, and Levine 2004). The var-

iation in cost might be due less to actual costs and more to context. An

example of context might be the difference between a small business that

needs to hire clerical staff to input survey data and universities who use

graduate assistants or student assistants. The amount of work required

could be the same, but the cost might be significantly different. Another

example would be the improvement and increase in online survey creation

tools. Previously, these types of tools were very expensive and required

specialized knowledge to use. Now anyone can use Web sites like Survey-

Monkey.com for free. As competition increases and Web sites improve,

those cost differences may become accentuated. As future researchers

attempt to replicate the findings from this study, they may find it difficult

to match the precise context in which the data were collected.

Recommendations for Further Research

The current study used a sample of potential respondents that were

deemed to have an interest in the results and therefore the subject matter

within the survey was salient to their profession. The issue of saliency was

not tested in any experimental fashion in the current study but a presump-

tion can be made that this issue may have had an impact on response.

Response rates were considerably higher in the current study than in many

other studies (e.g., Sax, Gillmartin, and Bryant 2003 24%; Kaplowitz,

Hadlock, and Levine 2004; highest group 31.5%). In the current study, the

overall response was 50%, with the highest group achieving a 60%response rate.

Groves, Singer, and Corning (2000) and Groves, Presser, and Dipko

(2004) have begun the foray into the issue of whether saliency has an

impact on response rates. The response rates in these studies indicated that




the manipulation of the variable saliency can be difficult but may lead to

promising results. The comparison of response rates from a survey directly

related to the goals and needs of an association and a comparison group

consisting of nonmembers might be a logical next step for this research.

Concluding Discussion

Overall, the mixed-mode design appeared to be the best source for

response to the AEA Employment Survey. When individuals were given

a choice of either survey administration mode, they were significantly more

likely to respond. Although the difference in response was significant, the

cost associated with the additional responses was considerable. In this

study, the additional 149 responses obtained in the mixed-mode group as

compared to the Web-based group cost a total of US$2,356 in postage,

duplication, labor, and data entry. That total cost broken down by response

resulted in US$15.81 per response.

Evaluators must balance the need for increased response rates with the

real cost associated with data collection. In the larger business community,

the monetary expense necessary to produce greater response rates may be

justified. In smaller business settings and in contracting economies,

increased expenses to increase response rates may be deemed excessive and

not an appropriate use of funds. The application of these results must also

be viewed from the perspective of the sample chosen for future surveys. If

the population from which the sample is drawn does not have access to the

Internet, then the above discussion becomes moot. If responses to surveys

are important and necessary to the outcome of an evaluation, then surveys

must be disseminated in a manner in which they will be easily accessed,

responded to, thereby increasing response rates.

References

Archer, Thomas M. 2003. Web-based surveys. Journal of Extension 41:1-5.

Carini, Robert M., John Hayek, George Kuh, John Kennedy, and Judith Ouimet. 2003. College

student responses to web and paper surveys: Does mode matter? Research in Higher Edu-

cation 44:1-19.

Cobanoglu, Cihan, Bill Warde, and Patrick Moreo. 2001. A comparison of mail, fax, and web-

based survey methods. International Journal of Market Research 43:441-52.

Converse, Patrick D., Edward Wolfe, Xiaoting Huang, and Fredrick Oswald. 2008. Response

rates for mixed-mode surveys using mail and e-mail/web. American Journal of Evaluation

29:99-107.




Cook, Colleen, Fred Heath, and Russel L. Thompson. 2000. A meta-analysis of response rates

in web- or internet-based surveys. Educational and Psychological Measurement 60:821-

36.

Dillman, Donald A. 2000. Mail and internet surveys: The Tailored Design method. 2nd ed.

New York: John Wiley & Sons, Inc.

Groves, Robert M., Eleanor Singer, and Amy Corning. 2000. Leverage-saliency theory of sur-

vey participation: Description and an illustration. Public Opinion Quarterly 64:299-308.

Groves, Robert M., Stanley Presser, and Sarah Dipko. 2004. The role of topic interest in sur-

vey participation decisions. Public Opinion Quarterly 68:2-31.

Heberlein, Thomas, and Robert Baumgartner. 1978. Factors affecting response rates to mailed

questionnaires: A quantitative analysis of the published literature. American Sociological

Review 43:447-62.

Idleman, Lynda. 2003. Comparing responses to mail and web-based surveys. Paper presented

at the annual meeting of the American Educational Research Association, April 21–25 in

Chicago, IL.

Kaplowitz, Michael, Timothy Hadlock, and Ralph Levine. 2004. A comparison of web and

mail survey response rates. Public Opinion Quarterly 68:94-101.

Kiernan, Nancy Ellen, Michaela Kiernan, Mary Ann Oyler, and Carolyn Gilles. 2005. Is a web

survey as effective as a mail survey? A field experiment among computer users. American

Journal of Evaluation 26:245-52.

Krosnick, Jon A. 1999. Survey research. Annual Review of Psychology 50:537-67.

McCabe, Sean Esteban. 2004. Comparison of web and mail surveys in collecting illicit drug

use data: A randomized experiment. Journal of Drug Education 34:61-72.

Mertler, Craig. 2003. What . . . another survey? Patterns of response and nonresponse from

teachers to traditional and web surveys. Paper presented at the annual meeting of the

Mid-Western Educational Research Association, October 15–18 in Columbus, OH.

Occupational Outlook Handbook. 2006. Bureau of Labor Statistics http://www.bls.gov/oco/

home.htm (accessed April 2, 2006).

Paolo, Anthony M., Giulia Bonaminio, Cheryl Gibson, Ty Partridge, and Ken Kallail. 2000.

Response rate comparisons of e-mail and mail-distributed student evaluations. Teaching

and Learning in Medicine 12:81-4.

Sax, Linda J., Shannon Gillmartin, and Alyssa Bryant. 2003. Assessing response rates and

nonresponse bias in web and paper surveys. Research in Higher Education 44:409-32.

Schonlau, Mathias, Ronald Fricker, and Marc Elliott. 2002. Conducting research surveys via

e-mail and the web. Santa Monica, CA: Rand.

Shannon, David, and Carol Bradshaw. 2002. A comparison of response rate, response time,

and costs of mail and electronic surveys. The Journal of Experimental Education

70:179-92.

Shannon, David, Todd Johnson, Shelby Searcy, and Alan Lott. 2002. Using electronic sur-

veys: Advice from survey professionals. Practical Assessment, Records and Evaluation.

http://pareonline.net/htm/v8n/htm (accessed June 19, 2009).

Sheehan, Kim. 2001. E-mail survey response rates: A review. Journal of Computer Mediated

Communication 6:1-20.

Sun, Anji, and Randy McClanahan. 2003. Is newer better: A comparison of web and paper-

pencil survey administration modes. Paper presented at the annual meeting of the Amer-

ican Research Association, April 21–25, in Chicago, IL.




Corey Greenlaw, EdD, is the evaluation and assessment coordinator for the Fresno County

Office of Education. His work centers on helping teachers and administrators improve the

impact of instruction on student learning through the use of assessment data. In addition, he

is a part-time faculty member teaching statistical and research methods in the Kremen School

of Education and Human Development at California State University, Fresno.

Sharon Brown-Welty, EdD, is a professor of education administration and the director of the

doctoral program in Educational Leadership at California State University, Fresno. She has

conducted research in the areas of program evaluation, educational leadership, and student

achievement.




A Comparison of Web-Based and Paper-Based Survey Methods

Documents

Transcript of A Comparison of Web-Based and Paper-Based Survey Methods