[IxDA Discuss] Asking questions to participants in a positiveor negative way ?

[Chauncey Wilson]

Caroline's suggestion about doing a think-aloud study of the
questionnaire is excellent.  I routinely do this as part of the
questionnaire design process.  I ask them to read it aloud and give me
feedback about meaning, bias, instructions, wording, terminology, and
anything else that comes to mind.  For items with un-ordered response
categories (for example job title), you might find that you are
missing a key item. These think-aloud sessions can be short, say 15-30
minutes.  Though it is a bit harder, you can do this over the phone or
though remote collaboration software like GoToMeeting or LiveMeeting
or other systems where you can display a question and hear the person.

If you are using an electronic survey, you might get feedback about
navigation, going back, required fields, etc.  For example, I recently
reviewed a survey where a single question listed about 10 fields for
address,phone, etc. It turned out that all fields were required, even
the one that was Addresss 2 which many people would not need to fill
out.  When the participant ignored that field and got a warning, it
was confusing so he eventually typed some junk into Address 2 and
could proceed.  The software allowed you to make individual fields
within a single question required or not which is good, but that
feature is buried.

Your test of the questionnaire might reveal missing categories or the
wrong time reference or frequency responses.  If you were asking about
a CRM or financial system and your last response was "I use this a few
times a day", your interview might reveal something like "Wow, I use
this features 50-100 times a day".  If you find out that a number of
respondents make REALLY heavy use of a system, that is critical design
input and may affect features for expert, high-frequency users.  A few
times a day is much different than 50-100 times a day.

When you test the questionnaire (with people who are as close to those
you will be sampling), watch for pauses and facial expressions and ask
what people where thinking or what caused "that smile" or "frown".

Having people read through a survey line by line and give you feedback
is a variation on usability testing called the user edit or usability
edit which is not too well known, but a powerful way to get feedback
on procedural documentation (and questionnaires).  Here are some
references to the user edit method:

Atlas, M. (1981). The user edit: Making manuals easier to use. IEEE
Transactions on Professional Communication, 24:1 (March): 28-29.

Atlas, M. (1998). The user edit revisited, or "if we're so smart, why
ain't we rich?". Journal of Computer Documentation. 22:3 (August). ACM
Press: New York, NY. 21-24.

Schriver, K. A. (1991). Plain Language for Expert or lay audiences:
Designing text using User Edit. (Technical Report Number 46)
Pittsburgh, PA: Carnegie Mellon University, Communications Design
Center.

Soderston, C. (1985). The user edit: A new level. Technical
Communication, 1st Quarter, 16-18.


Chauncey


>