3-20-14 What is a statistically valid survey?
Wednesday, March 19, 2014 at 3:11PM
Carolyn Tamler

Self-selected surveys, as we have discussed in previous blogs, are valuable tools for learning more about the people who are most enthusiastic about sharing their thoughts - most often the most positive and the suggestions for improvements.

A lot of non-reliable information is being passed off as a “survey.”  With Survey Monkey and other simple ways of sending out questionnaires available on line, it’s easy to believe that if you just send out a survey on the internet, it will produce reliable information.

However, to be a “random sample” survey with a measurable degree of accuracy, the sample must be pre-selected using a random number system.  If someone chooses to respond to an on-line survey (like Survey Monkey), by definition this is not a statistically valid process.

Telephone interview surveys are still considered to be the most cost-effective way of getting statistically valid information.  There is a margin of error, based upon the size of the sample (generally, a minimum sample of 200 is the industry standard for reliable data about any population segment).  Overall, random sample telephone interview surveys provide reasonably accurate information about the population from which the sample is drawn.

With the advent of cell phones, I became concerned about what this means for the traditional method of doing random sample surveys.  I have discussed this with Don Morgan, the President of GMA Research, the company that has done most of the data collection for my surveys for several years.  It turns out that a survey firm, like GMA Research, can now purchase cell phone lists in the same way they have purchased land-line telephone lists, from which the random samples are drawn.  Although some do have cell phones with a prefix that is different from where they reside, most people have cell phones that have been issued in their region.

So, be aware when you look at the results of a any "survey" if the study used a process for collecting statistically valid and projectable data, or if the results are from a self-selected survey where participants choose to respond.

Article originally appeared on Carolyn Browne Tamler (http://carolynbrownetamler.squarespace.com/).
See website for complete article licensing information.