Research Soapbox: Here’s a quick stats lesson to help you better understand polls – ALWAYS LOOK AT THE “MARGIN OF ERROR.” It’s typically written as “MOE +/-“at the bottom of the reported results.

I’m on a tirade this morning because I just saw a poll on TV with a +/-8% MOE…this means if the survey was given to a random sample, the answers would all be either plus or minus eight percentage points from the current answer. For example, if 50% of the sample supports a candidate, if the pollsters administered the same survey to another equal-sized random sample in the same population…they should expect support between 42% and 58%.

Oh, and I guess I should also mention that a +/-8% MOE means they only surveyed 150 people. Here was a major news source reporting the likely results of a Republican Primary outcome as if Moses himself walked down from the mountain with the information chiseled on stone tablets.

Now, here’s the problem with the whole thing. First, true random samples are tough to get…that’s what the whole “MOE” measure is built on. The standard definition of a random sample is, “Each individual is chosen entirely by chance and each member of the population has an equal chance of being included in the sample.” Most of the polls are not truly random…just think of how many times you’ve been called to give your opinion, and you chose to hang up. Congrats, you screwed up a random sample. Are you happy with yourself now?

Second, you need to understand the numbers behind the margins of error. If the polls are reporting MOE, here are the sample sizes they used – basically the number of people the pollsters spoke to:

• +/-10% – 96 person random sample
• +/-9% – 119 person random sample
• +/-8% – 150 person random sample
• +/-7% – 196 person random sample
• +/-6% – 267 person random sample
• +/-5% – 384 person random sample
• +/-4% – 600 person random sample
• +/-3% – 1067 person random sample
• +/-2% – 2401 person random sample
• +/-1% – 9604 person random sample
• 0% MOE – you have to interview the whole population

Big red flag: If they don’t report a margin of error, question the validity of the poll.

Most political polls presented in the media fall in the margins of 3% to 5%…yep, all of the hoopla is being based on the opinions of approximately 400 to 1,100 people. And just think, you hung up on them when they were trying to get you to be part of this group.

Come on people, let’s be smart about our use of research…especially when we’re trying to decide who’s possibly going to lead our country.

Educate yourself beyond polling. As a researcher, I think if polls are done well, they can be a great tool but ultimately, nothing should substitute your own education on a candidate or issue.

Tait Martin, Ph.D.
President & CEO
The iNSiGHT Cooperative
www.iNSiGHTCoop.com
www.facebook.com/insightcoop
twitter.com/insightcoop

This post serves a few purposes:

1. It is a fan letter to Dr. Annie Pettit, Chief Research Officer of Conversition Strategies and author of one of my favorite blogs, LoveStats;
2. It is a perfect response for those agencies who think they can skimp on research or treat it as a trivial function;
3. It provides questions that you should ALWAYS ask your potential research partners.

First, Annie does an incredible job setting the stage to combat crappy insight in her March 31^st post, “This is why your research sucks.” The following words are Annie’s taken directly from the LoveStats blog:

“You might not want to admit it, but at one time or another, you were probably on a team responsible for some research that sucked. Wonder why? Let me help you out.

1) You didn’t have a trained, experienced researcher at the helm.

Researchers are not a luxury component of research projects. Researchers know what makes a quality, unbiased, nonleading, useful questionnaire and focus group. They know what the most appropriate sample sizes are and WHY those are the most appropriate sample sizes. They know which statistics are the right ones and WHY those are the right ones. Researchers know how to take a problem and funnel it into a measurable, valid, and generalizable project.

2) You failed to identify and follow through on specific objectives.

There are two places where it is essential to focus on your objectives. First, when designing your research, you need to have a problem to solve or a reason to do the research. Without a problem, you could write a 400 question survey and still be trying to add more. Second, you need to focus once you get your data. Most surveys result in 300 page data-tables which are completely overwhelming, even for great researchers. Without focus and silo-vision, you will never find an answer. You can search but you will not find.

3) You focused on price and speed rather than quality and quantity.

Sure, you can choose the research with the best price and speed. But validity and reliability depend on sample sizes, and data cleaning, and appropriate statistical testing, and quality research design. These things are not quick nor cheap but when you need to accurately predict future sales or which TV show will be canceled or which product test will succeed, this is how you must do it.

4) You don’t follow through on the results.

Lots of really great research actually does get done, in large and small companies, via surveys and focus groups and social media research. But research is just crap if you don’t follow through on the results. If you KNOW you aren’t going to follow through on a set of findings, don’t bloat your survey and fatigue your respondents with it. If you KNOW you won’t have the time to follow through on the results for six months, DON’T do the research for six months. Research in a drawer is money in the toilet. Or, you could just give that money directly to me. Paypal accepted.”

Thanks, Annie, I’ll take it from here…

Strong research should not be a project extravagance.  It should be the foundation of every strategy – no matter if you’re launching a new product; developing an outreach campaign; or implementing a behavior-change approach.

The most important word in the previous paragraph is “strong.” Just as Annie states, research should be done by people trained to do it.  So many times, organizations try to make vital decisions based on the work of the inexperienced.

Here are a few more signs that your research probably sucks:

• Your research department consists of an intern doing Google searches;
• Your research is overruled by an owner of the company because he/she “knows the right answers”;
• Your analysis is dependent only on crosstab “banners” and prayer;
• Your focus group recruiting plan is calling friends to come by for a conversation and a free sandwich;
• Your creative department doesn’t use your research be they think you and “your numbers” are confusing;
• You try to do trendy things like develop “games” or “fun scenarios” without understanding the underlying psychological principles behind the methods;
• You automatically revert to the methodology that you know: “We always do focus groups!” or “We have to do a survey!”;
• Your idea of random sampling is “whoever walks in the office”;
• You are more worried about “not upsetting the client” than finding out what’s really going on.

If any of these signs describe the way you collect data, it’s time to find a credible, seasoned research partner. Here are the questions you should ask a potential research partner:

• Will the research be led by a Ph.D. (or someone with proper credentials and experience) who is well versed in the methodologies and analyses pertinent to your project?  (The answer should be “YES”)
• How comfortable are you with challenging a client’s ideas? (The answer should be “VERY”…if they say, “The client is always right.”…Run!)
• What professional organizations are you active in? (They should be active in at least three with one being specifically focused in the practice of research.)
• What methodologies have you used to understand _____________(insert your issue here)? Can you give examples? (If they can’t give examples, go somewhere else.)
• Would you provide references for your research capabilities? (Quality researchers have at least five references they can rattle off immediately who would vouch for their work.)
• How many people are on your research team? What are their backgrounds? (You want to be able to work with people who understand your field.)
• What experience do you have working with other teams like creative, human resources, management, etc.? (Strong researchers should be able to make the data meaningful for all of your organization’s teams.)

Be smart about your research approach…the success of your project depends on it.

Tait Martin, Ph.D.
President & CEO, The iNSiGHT Cooperative

WHO WE ARE: The iNSiGHT Cooperative is an international network of researchers and industry professionals that helps clients focus their outreach messages, build stronger relationships and develop programs that better define the way they reach the people who matter to them.

Specifically focusing on innovative research techniques, the Co-Op primarily works in the fields of health, education, safety, economic development/entrepreneurship, environmental sustainability and human behavior change.

WHY WE EXIST: The iNSiGHT Cooperative was created to fill a specific need in the social marketing, advertising, consulting, and public relations fields. Too many times, agencies conduct research with no one on staff qualified to do so.  Or, worse yet, someone on staff “thinks” they know how to conduct solid research and comes to false conclusions or indefensible results.

We are not an advertising agency or public relations firm.  Rather, the Co-Op partners with agencies as their research and strategy arm. We provide insights on their issues and help to develop creative solutions to their marketing, advertising and communication campaigns.

EXPLORE OUR SITE TO SEE OUR TEAM AND RESEARCH CAPABILITIES.  THANK YOU FOR VISITING.