Articles: Collecting Data From Users
What to Consider
:
Understanding and Comparing Techniques
for Gathering Data From Users
Links to Related Articles To design a
site that works for you and your intended
audience, you have to know a lot about
those people. They may be "customers,"
"consumers," "researchers,"
"the public." Let's call them
"users" because what you want
them to do is use your site.
What to Consider
Verify or challenge your
assumptions about users. Thinking about
users only gets you so far in designing
a successful site. Your thinking brings
out your assumptions about the users.
To learn about users' reality, you need
to get out and meet them, work with them,
involve them in helping you to understand
their:
needs for information
ways of thinking about, grouping, and
organizing information
expectations about your site
levels of knowledge about the subject
matter
levels of experience with the Web and
similar types of sites
By working with users, you can gather
many realistic scenarios and learn what
makes a Web site work or not work for
them.
Let users help you build
the site for them. Many useful techniques
have been developed to get useful information
from users and about users before you
design a site.
Understanding and Comparing
Techniques for Gathering Data From Users
The following is an overview
of data-gathering techniques, what they
are, and how they differ.
Technique Characteristics
Early usability tests Users usually come
to you
You usually develop the scenarios
Small numbers: one or two users at a time
Total numbers: five to 12 users
You observe and listen to actual behaviors
May be formal or informal, quantitative
and/or qualitative results
Tester and user need not be at same location
Contextual interviews You go to the user's
home or work site
Users do their own work (different scenarios
with different users)
Small numbers: one or two users at a time
Total numbers: five to 12 users
You observe and listen to actual behaviors
You see users' environments and the technology
users have
Usually informal dialogue with user, qualitative
results
Interviewer and user are physically at
same location
Online surveys May have large number of
responses
Get users' self-report
Good for wish lists, attitudes, experiences;
not for actual behaviors
Usually mostly closed questions (yes/no,
multiple choice, short answer)
May include open-ended questions, but
they require more analysis
Users may be located anywhere
May be single-survey or iterative series
Individual interviews Face to face, by
telephone, through instant messaging or
other computer-aided techniques
Small numbers: one user at a time
Total numbers: usually five to 15 users
Rich data — you can follow up on
questions
Can include both closed and open-ended
questions
Self-report; good for attitudes, experiences,
wish lists
Not good for actual behaviors
Focus groups Small group discussion Moderated
by trained facilitator
Usually everyone is in same location Self-report;
good for attitudes, experiences, wish
lists
Not usually good for actual behaviors,
but can combine with some aspects of behavioral
usability testing Discussion influenced
by group dynamics (for good or bad)
Can be done as an electronic meeting,
which allows for anonymity and reduces
the effect of group dynamics
Card sorting Usually used after gathering
information with one or more of the other
techniques
Each card represents a possible topic
on the site
Need a start on content topics —
so have some cards to sort
Usually small numbers: one or two users
at a time
Typical total numbers: five to 12 users
You usually observe and take notes as
users talk about what they are doing
Can be done remotely with a Web-based
tool — so can be large numbers
Early usability tests
Consider starting your project
with a usability test.
If you already have a Web
site, you can find out what works well
for your users and what does not. If you
do not yet have a site, use a competitor's
site or one that has similar purposes.
You can learn a great deal
that will help you build a new site —
what to keep, what to expand on, what
to change, how to avoid others' mistakes.
A usability test can be
done quickly and inexpensively. What a
usability test reveals about what users
actually do is usually more valuable than
what you learn in interviews and focus
groups where you ask users about themselves
and their work.
What users say they do and
what they actually do are often different
— because people aren't always aware
of how they work. When talking about our
work, we all skip steps because we do
them automatically. We often cannot remember
exactly how we do or did something. Watching
and listening as users work is the most
informative way to see what people do
— and to get what you need to build
a successful site. For more on how to
plan, carry out, and analyze data from
a usability test, see Conducting and Using
Usability Tests.
• For information on
doing remote usability testing, see http://research.cs.vt.edu/usability/projects/
remote%20evaln/remote%20evaln.htm
Contextual interviews
Contextual interviews are
like usability tests because you watch
and listen as users work. They differ
from usability tests in location . . .
you go to the user. That way, you see
the user's environment and the actual
technology the user has to work with.
Seeing the user's environment
can be very useful. What is the social
environment like? Are there people around
to help the user? What is the physical
environment like? Is the user on a slow
modem? Does being online tie up a phone
line so the user wants to be on and off
the Web quickly?
Contextual interviews are
more natural and realistic than usability
testing. In a contextual interview, you
watch and listen as the user does his
or her own work. You don't usually impose
tasks or scenarios on the user. In a usability
test, on the other hand, you usually have
all users do the same scenarios, which
gives you comparative data from several
people trying the same thing.
You can, however, combine
aspects of both: Contextual interview:
Take scenarios along and combine watching
the user do his own work in his environment
with asking the user to try a few of your
tasks.
Usability test: Interview
the user to find out the sorts of questions,
issues, tasks he or she would do with
the site. Let the user do his or her own
task. Also have the user do some of your
tasks to get data on tasks from all the
users.
A contextual interview is usually informal.
The observer listens to the user but may
also ask clarifying questions and probe
to gain greater understanding of what
the user is doing and thinking. The results
are usually qualitative rather than quantitative.
Usability testing in Web
site development today is also often informal
and is often conducted much like a contextual
interview. However, usability testing
can range from informal and qualitative
to quite formal and quantitative.
Online surveys
Use an online survey to
gather information from all of your Web
audiences.
Some hints for effective
online surveys:
Decide why you are surveying
your users.
• At this early stage of Web site
development, you probably want to learn
more about who they are, what experiences
they have had with your site or similar
sites, and what they want — that's
all!
Decide where you will find
the people you want to survey.
• If you already have a site, post
it there.
• Also consider other
sites where users for your site might
go and obtain permission to announce your
survey there.
• Consider getting
permission to send broadcast email about
the survey to lists of potential users
through professional societies, listservs,
discussion groups, etc. Be sure to get
permission from the list owners before
broadcasting email.
Keep the survey short. Under
10 items is preferable.
Make the survey easy for
users to do. It should take no more than
five or 10 minutes to complete. Consider
a mix of closed questions (multiple choice)
and open-ended questions (users write
down what they want to tell you).
• Closed questions are easier to
analyze.
• Open-ended questions
may give you richer data and offer a glimpse
at the terminology your users use.
Consider a mix of questions
about demographics, users' prior experiences,
and what users want.
Consider using a series of surveys.
• Gather information with a short
survey that you make available where many
people might see it and respond.
• On the survey, ask
if people are willing to answer more in-depth
questions — and ask those people
to give you an email address.
•Send follow-up electronic
questionnaires to willing respondents
who meet your criteria for the type of
user you want to know more about.
Consider combining an online
survey with individual interviews.
• With an online survey, you reach
many people in many places, but you don't
get the depth of data that you get from
individual interviews.
• If you do individual
interviews first, you get good ideas for
survey questions — and for the items
on multiple-choice survey questions.
• If you do individual
interviews after gathering some survey
data, you get to follow up with some people
on issues and ideas that come from the
survey answers.
Individual interviews
What do we mean by individual
interviews?
Why conduct individual interviews?
When should you conduct individual interviews?
Individual interviews and focus groups:
What's the difference?
What makes an interview successful?
• What do we mean by
individual interviews?
We are using "individual
interviews" to refer to talking with
one user at a time (for 30 minutes to
an hour) face to face or by telephone
or with instant messaging or other computer-aided
means. These interviews do not involve
watching a user work. Thus, this is different
from interviewing users in a usability
testing session or conducting contextual
interviews.
• Why conduct individual
interviews?
Individual interviews can
give you a deep understanding of people
who come to your site. You can probe their
attitudes, beliefs, desires, experiences.
You can ask them to rate or rank choices
for the Web site content.
• When should you conduct
individual interviews?
Use individual interviews
to supplement online surveys. You can
do interviews first to refine questions
for the survey. Or you can do interviews
after a survey to probe for details and
reasons behind answers that users give
on a survey.
• Individual interviews
and focus groups: What's the difference?
Individual interviews resemble
focus groups because they involve talking
with users. You do not see them perform
work/tasks as you do in usability tests
and contextual interviews. The obvious
difference between an individual interview
and a focus group is that you are talking
to one person at a time. In an individual
interview,
you have more time to discuss
topics in detail
you do not have to worry about the group
dynamics that inevitably occur in focus
groups
you can give the interviewee your full
attention and you can adjust your interviewing
style to draw out shy users and keep others
on topic
• What makes an interview
successful?
Select participants to represent
the types of users you want to come to
the Web site. (This is true of all the
data-gathering techniques.)
Decide what you want to learn. (This is
also true for the other data-gathering
techniques.)
Write an "interview protocol"
for the interviewer to follow. (In focus
groups, the comparable document is called
a "script." An interview protocol
includes questions and probes to use to
follow up on questions.)
Hire a skilled interviewer who will make
interviewees feel comfortable, ask questions
in a neutral manner, listen well, know
when and how to probe for more details,
and keep track of time unobtrusively.
Allow the interviewer flexibility in using
the protocol. (Although you want all the
questions answered, this is not a survey
but can be an opportunity to get a deep
understanding of users.)
Get permission to tape the sessions and
have one or more people take good notes.
(You are looking for answers to the questions
and for insights about users that will
help you build a Web site that meets their
needs.)
Focus groups
What is a focus group?
What do you get from a focus group?
What do you not get from a typical focus
group?
What makes a focus group work well?
• What is a focus group?
A focus group is a moderated
discussion among eight to 12 users or
potential users of your site. A typical
focus group lasts about two hours and
covers a range of topics that you decide
on beforehand.
Focus groups are a traditional
market research technique, so marketing
departments are often more familiar with
focus groups than with usability testing
or contextual interviews. However, the
techniques produce different kinds of
information. In a typical focus group,
participants talk; you hear them tell
you about their work. In a typical usability
test or contextual interview, users act;
you watch (and listen to) them doing their
work.
• What do you get from
a focus group?
users' attitudes, beliefs,
desires
users' reactions to ideas or to prototypes
• What do you not get from a typical
focus group?
how users really work with
Web sites
what problems users really have with sites
• What makes a focus group work well?
Select participants to represent
the types of users you want to come to
the Web site. (This is true of all the
data-gathering techniques.)
Decide what you want to learn. (This is
also true for the other data-gathering
techniques.)
Write a "script" for the moderator
to follow. (In usability testing, you
write scenarios — tasks — for
users to perform. In contextual interviewing,
you let the context and the user's work
shape the dialogue.)
Hire a skilled moderator to facilitate
the discussion so that everyone participates
and the group stays on track. (In the
other techniques, you need skilled observers
and listeners.)
Allow the moderator flexibility in using
the script. The script usually gives the
moderator questions to ask and topics
to cover. The moderator may change the
order of questions and topics to keep
the discussion flowing smoothly. The moderator
has to be a good judge of time to decide
when to encourage more discussion on a
topic and when to move on.
Tape the sessions and have one or more
people take good notes. (This is also
true of other data gathering techniques.
Good notes are critical to making sense
of what you see and hear in all these
techniques.)
Card sorting
What is card sorting?
What happens in a card sorting session?
Why use index cards with one topic per
card?
How does card sorting work?
What about doing card sorting remotely
with many users?
• What is card sorting?
This is a technique that
involves users in grouping information
for a Web site. Card sorting helps you
build the site's structure and site map,
decide what to put on the home page, and
label the home page categories.
• What happens in a
card sorting session?
You give a user (or two
users working together) a set of index
cards on which you have put likely content
topics for the site (one topic per card).
The user takes the first card and puts
it on the table. The user takes the second
card and decides whether it belongs to
the same group of information as the first
card or if it deserves its own category,
etc. — and so on through the set
of cards. The user "thinks aloud"
during the session and describes his organizing
strategy.
• Why use index cards
with one topic per card?
To allow users to group
— and regroup — the cards.
To have users build hierarchies that reflect
the categories they want on the home page
and how they would group information in
those categories on second-level and lower-level
pages.
• How Does Card Sorting Work?
Getting the cards ready
Arranging for card sorting sessions
Conducting a card sorting session
Analyzing data
• Getting the cards
ready
1. List the content topics
or types of information that you are likely
to have on the site. (These may come from
your objectives for the site, from scenarios
you wrote, from what you learned from
users with the other data gathering techniques.)
2. Write each content topic
or information type on a separate index
card. (Hint: Use slef-adhesive labels
and a word processor. These cards will
be neat, legible, and consistent. Also,
you'll have the list of topics in the
computer for easy analysis later.)
3. Limit yourself to less
than 100 cards. (About 50 is a good number.)
4. Have blank cards available
for users to add topics and to name the
groups they make when they sort the cards.
(Hint: Consider using a different colored
card for naming groups.)
5. Number the cards in the
bottom corner or on the back.
• Arranging for card
sorting sessions
1. Select participants to
represent the range of users — pull
from different user groups with different
levels of experience.
2. Plan about one hour for
each session — longer if you have
many cards.
3. Arrange for a space where
the user has enough room to spread the
cards out on a table. A conference room
works well.
4. Plan to have someone
take notes as the user works and thinks
aloud.
5. As with other techniques,
arrange for payment or other incentives
to thank the user for spending the time
and effort helping you.
• Conducting a card-sorting
session
1. Show the user the set
of cards and explain what you want the
user to do. Explain that you are asking
for help to find what categories of information
should be on the site's home page and
what those catagories should be called.
Explain that you want to see what groupings
of cards make sense to the user and that
when the user has grouped the cards, you
will ask that they name the groups.
2. Ask the user to talk
out loud while working. (This is the same
technique employed use in usability testing.)
You want to understand the user's thoughts
and rationale.
3. Let the user work. Also,
let the user add cards — for example,
to indicate lateral hyperlinks or additional
topics. Let the user put cards aside to
indicate topics the user would not want
on the site. Mimimize interruptions, but
encourage the user to think aloud.
4. At the end, if the user
has too many groups for the home page,
ask him or her to create hierarchies of
the groups.
5. Give the user a different
colored card for each group and ask the
user to name the group. What words would
the user expect to see on the home page
or second-level page that would lead the
user to that particular group of cards?
6. At the end, thank the
user and give the user the payment or
other gift as promised.
• Analyzing data 1.
Use the numbers on the cards to quickly
record what that user has done. Write
down the names that user gave to each
grouping and the numbers of the cards
the user included under that name. Then
you can reshuffle the cards for the next
session.
2. If you want a complete
picture of the detailed site map each
user has created, create a computer file
for each session. Working from your original
list of topics, move topics around to
recreate each user's groupings and enter
that user's names for the groupings.
3. For a less detailed analysis,
use your notes and recordings of the users'
names and card numbers under each person's
name to find commonalities from different
sessions.
• What about doing
card sorting remotely with many users?
The National Institute of
Standards and Technology (NIST) has developed
a tool for card orting. You set up the
cards, and you name the categories. Users
drag and drop cards into the categories.
For more information on
the NIST tool for card-sorting, see
http://zing.ncsl.nist.gov/WebTools/
|
