[IxDA Discuss] advice on usability testing for complex sites

[Katie Albers]

You may or may not be at liberty to do this, but my first action in a 
case like this would be to examine how much of that complexity was 
actually necessary and how much resulted from the development 
process. I've seen more cases than I care to think about in which 
that kind of examination resulted in a massively simplified 
interaction. Questions like "Are we asking users to perform tasks 
that could easily be performed by the system or vice versa?" "Are we 
dividing up tasks/uses appropriately?" and other such points often 
reveal places where you can eliminate several layers of complexity.

But as far as being stuck with what you've got...forget the 120 
minute test. The user may or may not fade (probably will, but might 
not), however your researcher is definitely going to go nuts during 
the course of the 2nd or 3rd test. Look at how users will be 
employing the app in real life: will it be something they always go 
straight through? Will they be likely to leave it and return? Will 
they use it daily, weekly, annually?

This is one of the few cases where I would recommend re-using a 
particular set of testers. It looks like you're going to relying on 
their memories in order to get any kind of actionable results.

Are there sub-tasks within the overall task? Test those as separate 
entities and incorporate what you learn in the app that you test all 
the way through.

[It's important to note here that I'm assuming one-on-one testing here]

If they're always going to go straight through then you really need 
to test it that way. Make sure you schedule plenty of time between 
for your researcher to decompress.

A possible alternative (and sort of a version of the sub-task 
testing) is to test A on day 1, save the data or whatever you can or 
need to have to start B, and then on day 2 test B. or you can divide 
it up as morning session and afternoon session. Just make sure the 
testers have been through A as recently as possible so that it can be 
relatively fresh in their minds. Ask *them* to walk *you* through 
what happened in the previous session. You can also build a mock-up 
screen-by-screen review for the testers that they can walk you 
through before they begin Section B.

On the whole, though, something where the *testing* is this complex 
and difficult to design is likely to be hell to actually use. I wish 
you luck.

Katie

At 6:06 PM -0400 3/19/08, Meredith Noble wrote:
>Can anyone recommend methods for performing usability tests on large,
>complex applications with lots of conceptual dependencies?
>
>We're running into issues in our design of a test. We want to test
>"Section B" of our application, but "Section B" doesn't make a lot of
>sense unless you've already been exposed to "Section A". The trouble is,
>Section A is pretty complicated in itself. They're definitely too big to
>test together in a single 60 minute test.
>
>What to do, what to do...! So far I've thought of (with drawbacks in
>parentheses):
>
>a) Have a facilitator walk them through Section A for 15 minutes before
>they do the 60 minute Section B test (perhaps a bit overwhelming, hard
>to digest)
>
>b) Ensure the participants who test Section A come back and test Section
>B (good in theory, but difficult to schedule)
>
>
>
>c) Test the two back-to-back in a 120-minute-long test (participants
>might fade)
>
>d) Pretend the dependencies don't exist and have them test Section B
>with no background knowledge (not realistic, but hey, maybe the others
>are too ambitious)
>
>Surely other people have had experience with this sort of thing - any
>recommendations on what has and hasn't worked well? Am I approaching it
>all wrong?
>
>
>
>Thanks so much,
>
>Meredith

-- 

----------------
Katie Albers
katie at firstthought.com