© Marshall School of Business – Wayne Wilmeth Qualtrics_Randomization_Nov2012.docx 11/14/12 Page 1 of 5

USC Marshall School of Business QUALTRICS SURVEY SYSTEM Marshall Information Services Answer, Question, and Block Randomization This guide covers how to randomize the order answers to a question are presented, the order questions are presented, and the order blocks of questions are presented.

ANSWER RANDOMIZATION

Answer Randomization allows you to randomize the order answers are presented for a specific question. It is often used to prevent any bias created by the user simply selecting the first answer they see in a question either intentionally or unintentionally. Note that answer randomization must be done on a question-by-question basis.

1. Click the “Edit Survey” tab. 2. Click the question whose answers you

wish to randomize. 3. Click the purple Advanced Question

Options down arrow and select “Randomization”.

4. Select the method of randomization you wish to apply (see below) and click “Close”.

Remove Randomization

Randomize all of the answer choices for the question.

Randomly displays the number of answer choices you specify. For example, if there are four possible answers and you type a 3 in the box, Qualtrics will uniquely present any three of the 4 possible answers

Advanced Randomization is covered on the next page.

Note that the symbol shown to the right will appear on any question whose answers have been randomized.

© Marshall School of Business – Wayne Wilmeth Qualtrics_Randomization_Nov2012.docx 11/14/12 Page 2 of 5

Advanced Answer Randomization can be used in several different ways. Leave “Other” at the Bottom In this first example, we wish to randomize our first four answers but we always want “Other” to be the last choice. To accomplish this, you would setup randomization as shown in the image below. Note that “Other” remains in the “Fixed Display Order” column while all of the remaining choices were moved into the “Randomize Choices” box using the right arrow button.

Using Subsets Subsets allow you to randomly hide choices. In this next example, “Other” is still always at the bottom. “Biking” and “Tennis” will always be presented but in a random order. “Golf” and “Basketball” have been moved into “Random Subset” and only one of these will be randomly presented. In other words, the user will always see Other, Biking, and Tennis but will see either Golf or Basketball but not both.

© Marshall School of Business – Wayne Wilmeth Qualtrics_Randomization_Nov2012.docx 11/14/12 Page 3 of 5

RANDOMIZE QUESTIONS WITHIN A BLOCK

This technique allows you to randomize the order questions are presented with in the same block. Note the following:

Any Skip Logic, Display Logic, and Page Breaks you are using on the block will be ignored.

This randomizes the order the questions are presented, not their answers.

Branching may or may not function as desired depending upon whether or not the question(s) that the branch keys off of is presented or not.

1. Create your questions you wish to randomize within the same

block. (Any block is fine). 2. Click the “Block Options” down arrow located in the upper right

corner of the block whose questions you wish to randomize. 3. Select “Question Randomization”.

You will see the same choices as shown on the previous page. 4. Make you choices and click “Close”.

Remove Randomization

Randomize the order of all of your questions.

Randomly displays the number of questions you specify.

All questions placed in this box will be displayed but in a random order for each user.

Questions placed in this box may or may not display. In this example, only 1 of the two questions will get displayed for each user.

Questions placed in this box never get displayed.

This allows you to control the number of questions displayed per page in the block.

© Marshall School of Business – Wayne Wilmeth Qualtrics_Randomization_Nov2012.docx 11/14/12 Page 4 of 5

RANDOMIZE BLOCKS

This allows you to randomize the order that blocks of questions are presented in. Note that it does not randomized the order of questions on the blocks, but the order of the blocks themselves. If you are survey also involves Branching, use the Block Randomization technique shown on the next page rather than the one shown below. 1. Create your block and questions. (“Add block” can be found at the bottom of every bock and

under “Advanced Options”.) 2. Click “Advanced Options” located in the upper right. 3. Click “Block Randomization…”.

To randomly present some of the blocks but not all: a. Check the blocks you wish to randomize (see step 5 above). b. Click “Advanced Randomization”.

4. Click “Randomize Blocks”.

5. Check the blocks you wish to randomize.

6. Click “Save”.

c. Specify the number of blocks you wish to randomly present & click “Save Flow”.

© Marshall School of Business – Wayne Wilmeth Qualtrics_Randomization_Nov2012.docx 11/14/12 Page 5 of 5

About Using Branching with Block Randomization

If your survey contains branching and you also wish to randomize some of your blocks, you must be careful that the two are compatible. Use the Survey Flow environment when using both Branching and Randomization. If you are unfamiliar with branching, you read the Qualtrics – Branching guide. 1. Be on the “Edit Survey” tab. 2. Click the “Survey Flow” button located at the top of the screen.

The trick to randomizing blocks is to add the “Randomizer” element to your flow and then place the blocks which are to be randomized on its branch without interfering with any of your conditional branches.

Example In the example below we have the 4 question blocks:

The Default Question Block will always be the first block.

The Interests and the Work blocks will appear in a random order after the Default Question Block.

The Education block will appear after the Default Question Block but only if the user answered “>100K” to a question on our Default Question Block.

The finished survey flow for this scenario is shown to the right.