[IxDA Discuss] A List Apart web design survey results

[Caroline Jarrett]

From: "Matthew Nish-Lapidus" <mattnl at gmail.com>


:I agree that the sample set is skewed towards ALA readers.. but they
: got over 32,000 responses.  That's a pretty good number.  And 
really,
: most people working on the web read ALA.  I, and most people in my
: office, fill out the survey every year.

Hmm. ALA says "close to 33,000 responses", but there's no hint about 
how the survey was conducted or what the sampling method was. I'm 
assuming that ALA posted a link and anyone who chose could fill it in.

So, we don't know what proportion of ALA readers chose to respond. 
Therefore, it's quite likely that there is some non-response bias: 
that is, people who chose to respond are different to people who chose 
not to respond.

It's probably fair enough to treat ALA readers as a proxy for web 
designers in general. Or is it? What do people on this list think? And 
would we draw a distinction between people who occasionally read an 
ALA article (and therefore are fairly unlikely to have seen this 
survey) and regular readers who had enough time to respond to the 
survey?

I'm guessing that regular ALA readers with enough time to respond are 
likely to be people with a bit more time on their hands in general. 
That might well affect some of the reported results e.g. hours worked 
and having a personal web site/blog.

If we don't know how many people there are in the population and we 
don't know whether those who responded are different to those who 
didn't and we don't know whether the people in the population who had 
a chance to respond are an accurate reflection of the overall 
population (sorry if I seem to be drowning a bit here)... then it 
doesn't really matter how many responses. The stats are shot anyway.

Having said all that, there's some 'face validity' in anything that 
gathers together a lot of data. Meaning, it seems good if you don't 
look too closely. If you're planning to use this data in a salary 
discussion with a boss, then I'd suggest you also consider the 
statistical knowledge (or lack of it) of the boss.

Aside: slightly longer article by me on the topic of response rates:
http://www.usabilitynews.com/news/article4209.asp

Best

Caroline Jarrett
caroline.jarrett at effortmark.co.uk
01525 370379

Effortmark Ltd
Usability - Forms - Content