Telling Good Research from Bad

By Ron Sellers
Originally published in The NonProfit Times, November 1997

You see public opinion research everywhere: quoted by politicians to make a point, promoted by salespeople as proving their product is best, used by newsmakers and the media to explain public sentiment. How much of it can you really believe?

There will be times when you run across marketing research data in an article or an advertisement that could be very useful to your organization. Not having conducted the research yourself, how can you be sure whether it's an accurate look at public sentiment or a biased view of non-reality?

Start asking questions. Research can be accidentally or intentionally biased when it is conducted. More commonly, even if the study itself is entirely trustworthy, the media generally report only the most surprising and interesting findings. This can lead to an incomplete understanding of the data, or even total inaccuracies.

Before you rely on a figure you read in an article or see quoted in a book, do some research of your own. A few simple questions can save embarrassment down the road

Is it reliable?
The most important step is to be willing to ask questions. When you're putting together a proposal, strategic plan, or advertisement, and you find a piece of data that supports your case, it's mighty tempting to plug it in without first questioning its accuracy. Avoid this temptation. If your supporting data can be called into question, the rest of your content will soon follow.

Who conducted the research?
Marketing research that is conducted by a legitimate research company is more likely to be unbiased than a study which is conducted by a company or organization with a point to prove. Some organizations have experienced, unbiased research professionals on staff, and do reliable work in-house. But others assign the "study" to a marketing assistant with no research savvy, telling him to "go get us some data that shows people agree with our agenda."

Whenever possible, try to access the original study from the research firm, rather than relying on quotes from the sponsoring organization or from an article. That way you know you're getting the whole picture. For example, in the book Virtual America, author George Barna quotes his own survey which shows that 35% of all Americans watch Christian television in a typical week. He continues by saying he doubts the validity of this figure, because people often have a very broad, incorrect definition of what "Christian television" is. Without this caveat, Barna's figures could easily be misused in an article or a press release. With the original study in hand, you have the full story.

When was it conducted?
Some national attitudes change very slowly, while others can change almost overnight. Because a study is three years old doesn't necessarily mean it's inaccurate. For example, the proportion of Americans who own a VCR won't change much from last year to this, because it's a mature technology. But the proportion of people who access the Internet will change drastically in a year's time, because the technology is new. Religious attitudes don't tend to change radically over a year or two, unless something unusual happens (church attendance increased during the Gulf War, for instance). But political attitudes can change month to month. Look closely at when the study was conducted, and what might have changed in the interim.

What was the sample?
The sample is the small group of people drawn from the overall population for the research. A sample that is representative of the population being studied is a key element of reliable research, but this is often overlooked.

Internet users are not representative of all Americans, for instance. Women who read Redbook or Woman's Day are not representative of all women. Yet magazines often sponsor or conduct research of their own subscriber base and claim this represents a broader population. One memorable "research" study found that senior adults are actually heavy computer users. Problem: the study was conducted among active, younger, wealthier seniors who were attending a travel convention, rather than among a representative sample of seniors. Alfred Kinsey's groundbreaking studies on human sexuality are now coming under fire due to accusations that his research sample used large numbers of prison inmates and prostitutes to represent typical Americans, badly skewing his findings.

Phone-in polls (where people proactively call in, often on a 900-number, to express their opinions) have no relationship to reality. Neither does "research" collected in a fundraising mailing. Don't be fooled by large samples, either. A recent article in a marketing trade publication described a mail survey of 50,000 people. Upon review, it was clearly bad research: the response rate to the survey was only 5% (which is unacceptably poor), and the findings claimed that 65% of all Americans own a computer (which is about double the proportion found in any other legitimate study).

It is easy to bias a sample of 10,000 people. On the other hand, a sample of as few as 200 people can accurately represent the feelings of an entire nation if the research is conducted properly. Only if the sample of people interviewed is truly representative of the overall population is the study reliable. This would mean that (within reason) every member of the population being studied had an equal chance of being selected for the survey. Surveys conducted during conventions, for instance, are only representative of the people attending the convention.

How were the questions worded?
If you know the sample was reliable, and you know how the questions were phrased, you're most of the way toward determining if the research is valid. Biased questions provide biased data. "Should women have control of their own bodies, or should the government take away their right to reproductive choice?" is not a question which will accurately reflect Americans' views on abortion, for example. Yet surveys much more biased than this have received media coverage.

Far more insidious is the study that is biased not by what it asks, but by what it doesn't ask. For instance, it's not uncommon for liberals to promote a survey which shows Americans want more government spending, while conservatives have a competing survey proving Americans want lower taxes. The funny thing is, often they're both right – because of how the questions were asked.

Paradoxically, Americans often desire both higher government spending and lower taxes. They want the best of both worlds. This will come through as seemingly contradictory survey findings if the research is incomplete. Some survey respondents think we can have both, just by eliminating government waste. Others want lower overall taxes, but increased spending on certain issues. Others just give knee-jerk reactions to whatever sounds good during a telephone poll; providing more realistic options (i.e. you can have one or the other, but not both) would force them to reveal which is actually more important to them. Knowing how the questions were asked can help you discern if the study is reliable.

Is it logical?
This is a simple test, but an important one. Claiming that seniors are heavy computer users flies in the face of what is already known of the senior market, which should lead the reader to start questioning whether the study is accurate. If a study claims the average gift of an American donor is $100, and your organization has an average gift of $12, take a second look at the data.

Because survey findings are surprising doesn't mean they're wrong, but if they contradict what you already think to be true, that's a good reason to take a closer look.

This all may seem like a lot of work, and frankly, it can be. It's probably not worth all this effort if your favorite trade publication columnist quotes some research, and you're just curious how accurate the data is. However, research figures are commonly used in advertising, direct marketing, financial forecasts, proposals, strategic plans, and other things which can have a huge impact on your organization.

It's wise to spend the necessary time to confirm the accuracy of research statistics you dig up, rather than just assuming it must be accurate because "it's research." Much of the data available to you will be accurate and reliable, but it's essential to eliminate dubious research efforts before you rely on the findings. By asking these few questions, you can be confident you have a picture of reality before you use the data to help shape the direction of your organization.