TraQline Is Now Open brand

We are proud to announce that OpenBrand, a market intelligence platform for brands and retailers to leverage AI-driven insights and data, has completed its acquisition of TraQline! Read more here.

Durable IQ™, Durables Digest Podcast, Grills & Smokers

Durables Digest Podcast Episode 9: Can’t Avoid Survey Bias: Talking Bias Types, Challenges & Solutions — Plus Data Trends in Grills

June 17, 2024
Ashley Jefferson Author 39 mins read
  • Facebook
  • Twitter
  • Email
Durables pricing floor team

Watch Now!

Episode 9 Transcript

Brian Lamar (00:12.859)

Okay. Three, two, one. Hey everybody. Welcome to Durables Digest, episode nine. My name is Brian Lamar. My lovely coworker, Eric Voyer. How are you, sir?

Eric Voyer (00:25.73)

I feel lovely. Thanks for asking. How are you doing?

Brian Lamar (00:27.771)

Yeah, I’m doing all right. You know, I haven’t seen you. I feel like you’ve talked off air in a while. We’ve passed each other in the hallway, kind of glanced at each other, needed each other. We never got to connect, I don’t think.

Eric Voyer (00:39.618)

That’s right. Well, I did call you on your way home on a Friday at like, you know, 9 PM or something like that. So we did talk then it has been busy and that’s good.

Brian Lamar (00:44.923)

Yep, that’s right. Very busy. That’s very good. Today, I think is a great episode. For me, at least I get to nerd out a little bit today. Get back to my roots, review my graduate school syllabus. This is always good, like foundational, what is bias kind of topic. You’re a great researcher, but you’re going to ask me some questions around bias and try to stump me. And that should be fun.

Eric Voyer (00:58.818)


Brian Lamar (01:15.099)

but before we get there, data bytes, man, this is a fun data bytes little segment, I think.

Eric Voyer (01:20.674)

It is because I know I’m grilling out on Sunday. So I don’t know about you, but you know, father’s day is a great day to grill out. I’ve just got my patio finished. so I’m pretty stoked about that. I got this beautiful pergola put up, so I’m really excited. So let’s talk about grills and grilling and what do you got for us on data bites, Brian?

Brian Lamar (01:25.627)


Brian Lamar (01:41.467)

Yeah, so first of all, we published an infographic on our blog around barbecue grills and we used March 2024 data and wanted to share a bit. First of all, this really comes from your team, Eric. This is an amazing infographic. You don’t have to be a grilling expert to really find value in this. I could not believe how much I learned and just from the infographic and the kind of a little blog that went to compliment it. And so I’m really excited to talk about.

Eric Voyer (01:57.602)

Yes, they are awesome.

Brian Lamar (02:10.651)

grilling today and kind of the insights that we get from it.

Eric Voyer (02:12.898)

Can I tell you a secret about that infographic? So our infographics are actually three pages and that infographic that you’re seeing out there is really just one page. So our subscribers get access to a whole bunch of other information, infographics, market overview reports. We’ve got head -to -head analysis. We’ve got opportunity funnels, which is like a leakage tree.

Brian Lamar (02:15.323)

Yes, behind the scenes.

Eric Voyer (02:35.49)

basket reports, like just we’ve talked before about the wealth of consumer data that you get from TraQline. So there’s just a ton of that stuff out there for our subscribers. But if you want to know what any of that stuff is, I did not intend to pitch TraQline here. But if you want to know what any of that stuff is, feel free to reach out to us at TraQline .com [email protected] or [email protected]. I’m more than happy to talk through it or put you in touch with somebody. What kind of grill do you use? Brian? Do you use a do you have a grill? Let’s start there.

Brian Lamar (03:05.435)

I’ve always had a grill until end of last fall. My grill broke. I am a free agent. This topic could not be more timely or near and dear to my heart. I’ve heard about the Blackstone trend, but I’ve been kind of out of the market for about 10 years. So this is really fascinating to me. By the way, what are you grilling on Sunday?

Eric Voyer (03:07.362)

Okay. Yeah. Well, I don’t know yet. It’s, it has been so crazy. I haven’t had a chance to go to the store, but you know, the best bet for me is probably going to be chicken. and I have a penchant for a good beer boiled bratwurst. So I will probably, yeah, I’ll probably boil some, some brats and beer and then throw them on the grill. And I’ll tell you my secret to grilling is a instant read thermometer, ‘an’ instant read thermometer. If you don’t have one of those.

Brian Lamar (03:41.994)

my gosh.

Eric Voyer (03:55.042)

you know, you pop it open, you stick it in your meat, and then it tells you within seconds what the temperature is. And it’s better than any other way of figuring out whether your stuff is done. So especially when you eat chicken, like the last thing you want to do is have your father -in -law over and be like, have your father’s day, give him chicken. And next thing you know, he’s got chicken, you know, it’s raw. I guess you don’t get trichinosis from chicken. That’s pork, but you get some bad stuff. So you get sick.

Brian Lamar (04:07.451)

Yeah. Right. It’s raw. Right. Get sick, yeah. That’s the term I use.

Eric Voyer (04:23.298)

Yeah, yeah. So I have a regular grill. So mine is a hand me down from my father. It’s a DCS, which is made by Thermador, and it’s got a lifetime warranty on it. So I will probably literally never get rid of that grill. Their customer service is phenomenal. I’ve called them and said, hey, these greats aren’t are they’re separating and they’ll just send me new ones in the mail. It’s phenomenal customer service. But these griddles, from what I’m hearing, are I don’t know anybody that has a griddle that doesn’t say,

Brian Lamar (04:44.443)


Eric Voyer (04:53.09)

I love it. I want two of them. They all absolutely love it. And the last person I asked, I said, what do you make on them? They said, breakfast. Like we make smash burgers, like all kinds of stuff on their griddles. So apparently this is the newest versatile trend.

Brian Lamar (05:09.435)

Well, see, I’ve always learned a little bit of information about me is that I love diners. I love diner burgers. And the beauty of diners is that they have a flat top grill. They cook everything on it. They cook eggs, they cook pancakes, they cook burgers, they cook steak. And over time, probably like your grill over 20, 30 years, it just absorbs this taste and spiciness and you can’t replicate it. I think I’m going to go flat top.

By the way, we’ve teased biases and we’ve teased data. All we’re doing is talking about Father’s Day so far.

Eric Voyer (05:41.506)

We… And grills. So how… Tell me what the data says. Are we seeing an increase in griddle grills?

Brian Lamar (05:52.635)

Well, we are seeing in barbecue grill brands, there’s a kind of a decline in popularity. And so Weber and Charbroil are historically led in brand share, but kind of a steady decline over the past few years. Now you look at this data a lot, I’m relatively newer to it. Weber lost a couple 2 .4 points over a year ago, Charbroil 1 .4, and we’ve seen Blackstone as really the winner as they’ve emerged. And I’ve heard that just anecdotally from my friends. I hear the word Blackstone.

a lot. They’re up 3 .4 points year over year. Yes, that’s the highlight of the data.

Eric Voyer (06:25.81)

Yeah. They really, they really pioneered the whole flat top griddle and now you can get a grill, but you can remove a grate and set the griddle portion on it. And I don’t know much about that. David Garcia, we should have him on. He is like the grill master. I don’t know if you know that, but you know, he would probably know better, but anyway, I know you can do that, but Blackstone is doing phenomenal in the marketplace as it relates to this type of thing.

Brian Lamar (06:45.787)

Did not know that.

Eric Voyer (06:54.914)

category. So very cool.

Brian Lamar (06:56.379)

And when I think about this, I’m sure you’ll agree when you just kind of look at this data, it’s just data and you can interpret how you want. The key to me is the context around the data and the insights around the data. And that is challenging in some ways because I don’t know how many categories we work in. It’s well over 100, if not well over 200. And to become an expert in all of these categories and have the context around it and look at trends across categories, that’s something you’ve done for…

I don’t know, 20, 25 years, and I’m trying to just catch up.

Eric Voyer (07:24.77)


Eric Voyer (07:28.258)

Well, let me tell you what I would do with that information. Okay, so first off, if you’re a retailer, like you have to think about Blackstone as a brand that you might want to carry. Second, as a manufacturer, you have to look at why is Blackstone winning, right? Well, the only thing they make are these griddles, right? So then you can go into our data and you can actually look and see what percentage of grills are smokers, kettles, comatos, grills, griddles, smokers. How is that changing over time?

And if there’s a trend towards griddles and you’re not picking up on it, you don’t have the insert for your grill that has the griddle on it, or you’re not making a standalone griddle, you could be left behind. Unless this is a fad, which it may well be, but if you’re going to make the sales right now, you need to change your product lineup. So that’s how I would interpret this information.

Brian Lamar (08:10.107)


Brian Lamar (08:16.699)

Yeah, that’s excellent. Last time I bought a grill, I think George Foreman was the trend. That’s how long it’s been. That was a great little trend. Nothing back in 1998, Brian Lamaramar world, nothing being a George Foreman grill at the time. That’s right. Anything else before we move on?

Eric Voyer (08:21.698)


Eric Voyer (08:29.09)


Knocks out fat.

When we know I want to move, I’m going to transition. I’m going to use the whole grill conversation to transition into the bias conversation. Are you ready?

Brian Lamar (08:44.475)


Eric Voyer (08:46.146)

So what do you think is that data accurate? Is it a possibility that there’s any biases in that data, Brian? And before we get into that, what is bias? How’d I do? Maybe I asked the last question first, but anyway, yeah. What is bias?

Brian Lamar (08:52.89)

Brian Lamar (08:57.179)

That was a great –

 question. Okay, so the answer to your question, yes, there is bias. You cannot avoid it. All you can do is reduce it. You can reduce it and you can get really good at reducing it. The beauty of not to sell our TraQline Durable IQ product, but the beauty of it is that most of the questions we’ve been asking for a very long time for trend analysis and it’s kind of been vetted and we’ve removed a lot of the bias and it’s kind of standardized and that is an easy way to remove.

Eric Voyer (09:04.578)

Okay. that’s not what I thought you’d say. Okay.

Brian Lamar (09:29.947)

reduced greatly reduced bias. Most people don’t have that luxury. Most brands are asking ad hoc surveys every week or month or day. And bias is a huge challenge that I wish we had accreditation in our industry where people had to take a bias course almost to understand it. It’s everywhere. There’s a million different types of bias. While I don’t think you have a master’s degree in understanding

survey methodology design, I think you understand bias at the ideal level from a practical level. Cause a lot of this is kind of common sense, honestly.

Eric Voyer (10:07.234)

Okay, awesome. Is bias, we talk about bias in surveys, is bias only, does it only exist in surveys or are there other ways that bias can creep into other types of research? And I know we can’t cover all the different types of research, but.

Brian Lamar (10:22.779)

Yeah, there’s absolute interviewer bias and you know, I’ve done in -person interviews and when I was new to the industry, I’ll tell a little quick story, brand new and I was trying to just build up my resume to get into grad school. So I would do anything they asked. I was a focus group assistant. I would enter data. I would code data. I was a telephone interviewer. They asked me, they were short people on a focus group for, I think it might’ve been some sort of cooking research.

Eric Voyer (10:24.962)


Brian Lamar (10:51.195)

They were short African -Americans. And so they said, hey, Brian, it is Martin Luther King Junior Day tomorrow. There’s a parade going on. Can you go down there, recruit some people to the focus group? That could be biasing, right? Is that representative of African -Americans? If you go to a parade, are you different than other types of African -Americans? I don’t know. So there is certainly an inherent bias in.

Eric Voyer (11:03.17)

Mm -hmm.

Brian Lamar (11:17.147)

interviewers, a lot of it’s kind of conscious, a lot of it’s unconscious, a lot of it’s intentional, a lot of it’s unintentional. And so that’s certainly an example, not just in in -person research, on telephone research, on online research like we primarily do, the interviewer could insert its own bias to questions and not even realize it. So that’s actually a big one.

Eric Voyer (11:38.306)

So that’s a great story. Thank you. That’s an awesome story. I guess my follow up on that is, how would you know? What do you have to do? I want to talk about the different types of bias, but what do you have to do to make sure that you’re introducing as little bias as possible? Because I heard you say you can’t not introduce bias. So in that situation where it’s the parade,

Do you just have to be aware of the different types and then check the boxes? What do you do?

Brian Lamar (12:10.043)

Well, it’s all about trade -offs. And a lot of times, unfortunately, we have to trade off with a price and timing. And so as certain things get more expensive, you have to do a little trade -off. As the timing gets closer to a due date, you might have to do trade -offs. That’s a big challenge in our industry. What we do, ideally, in that situation is in kind of in -person intercepts is what I would call it, you do site screening. And so if I need only females,

Eric Voyer (12:16.418)


Brian Lamar (12:39.099)

for a focus group, I’m Brian Lamaramar looking for just females, right? I’m trying to find someone that I think fits the criteria. There could be bias in that, especially in 2024, there’s probably significant bias in that. I might be biased towards looking to people that I feel more comfortable talking to. And so to avoid that, you really have to be well -trained and understand it and really try to get down to basics with survey methodology and try to do some sort of randomization.

It’s nearly impossible from an in -person standpoint. It’s really hard.

Eric Voyer (13:11.874)

So if I hear you correctly, you’re saying just the experience of having done multiple types of research activities helps you realize, wow, this could introduce this type of bias or that type of bias. So experience helps. And then also being aware of the different types of bias can help as well. What are some of those types of biases that a researcher needs to look out for? Or hang on, actually not just a researcher.

Brian Lamar (13:26.843)


Eric Voyer (13:39.586)

right, because it could be very important to recognize the types of bias as a receiver of research as well to say, hey, wait, who did you do this with? So what are some of those types of biases?

Brian Lamar (13:45.755)

just as important.


One you kind of alluded to is, I think it’s so overlooked, is non -response bias. And non -response bias means that basically the people that are in the research are different than the people that aren’t taking the research. And so back in 2000, I worked at a company that was kind of newer to online research as most of the industry was. I think TraQline was doing the same thing. Stevenson Company at the time was trying to move from phone to online, right? And…

There certainly at the time was a debate in the industry about is online representative. Are the people that are taking a survey online different than the people that are taking the survey online? And so at the time it was, what’s penetration of online and if it’s over a certain percentage, you feel relatively comfortable about it, right? So it’s 70, 80 % of people are online. Okay, that’s probably close enough. Even today, we still have that debate depending on the population. There are certainly people that are not online.

Eric Voyer (14:25.922)

Mm -hmm.

Brian Lamar (14:50.651)

that they’re probably very different at a two million behaviorally than the people that are online. And that’s just one example. are the people see we’re fortunate when you talk about online purchasing is one of our topics, right? So you re in theory, you kind of have to be online. there’s lots of non -response bias. It exists everywhere and it’s really hard to account for. You just kind of have to think it through.

Eric Voyer (15:18.274)

Yeah, and I guess it’s okay to have some types of biases depending on the question that you’re trying to answer. It’s okay to trade off rather a certain type of bias for another, depending on the answer that you’re trying to get. Correct? So yeah. Okay.

Brian Lamar (15:18.715)

That’s one.

Brian Lamar (15:32.123)

Yeah, it’s depending on the risk of the business decision. That’s going to be an impact to it. Maybe like on the Martin Luther King Jr. Day example, if that’s not a super risky decision, let’s let some 22 -year -old kid go down to a parade and find a few more people because we want their opinions. It’s more important that we have a full focus group because we are not going to make a billion -dollar decision based upon these four people. Rather than an online survey.

where you’re going to make a decision where to create a product or to build a factory. There’s a lot of risk in that potentially. And then you have to really weigh all the different types of biases. So it kind of goes from risky to non -risky on how much bias you can accept.

Eric Voyer (16:13.154)

Okay, okay. Well, as always, I’m going to say this is the mantra in every podcast, we’re gonna run out of time. So tell me about some other types of bias.

Brian Lamar (16:22.299)

confirmation bias is a big one, I think, is when, and I think this is really prevalent because we’re human beings. And I think researchers are very curious people. And, you know, you kind of go into a survey with a premonition of how the results might look, right? Like, hey, you know what? I know grills, I grill every week. I know what’s going to happen with grills. I look at the data and you’re kind of biased in terms of confirming what you feel. You’re not looking at it.

Eric Voyer (16:38.722)

Okay, yeah.

Brian Lamar (16:51.067)

objectively. That is extremely common. We’re human beings. It’s hard to not think through. And that’s when someone is analyzing the data. You got to be really careful to not cherry pick data because, you know, we know this in all types of research and we see it in like political polling all the time or public opinion research that we can look at the same data set and have two different views on what’s happening. It’s easy to do with data. You can cherry pick it. And so confirmation bias is certainly something that’s hard to avoid.

Eric Voyer (16:58.53)

Mm -hmm.

Eric Voyer (17:19.618)

Yeah, there’s two, I want to make the distinction between cherry picking and confirmation bias though, because cherry picking is an intentional, at least the way I interpret it, it’s an intentional decision, whereas confirmation bias tends to be a little more subconscious. Would you agree?

Brian Lamar (17:34.523)

It could be subconscious, it could also be conscious, but yes. Yeah, probably most researchers are not intentionally being biased in most cases, but I think there are cases when people, you know, they’re trying to prove their worth or their job is at stake. You know, think about R &D with a lot of our categories. Yeah, it could be, absolutely.

Eric Voyer (17:36.674)

Okay. Okay.

Eric Voyer (17:50.69)


But again, that’s going to be cherry picking, right? That’s where you’re cherry picking for some. But no, I hear what you’re saying actually. No, because my job’s at stake. thank God the data says this. I got you. OK, that makes sense.

Brian Lamar (18:03.487)

Yeah. We had a situation my last role where if the data set a certain thing and a certain score hit that some guy in China got a bonus. And I thought about this unknown, I have no idea who this guy is in China. And I was hoping to get his bonus. I’m analyzing the data and it was really in the gray area. I’m like, I could, I could personally give this guy in China a bonus. Like that’s a lot of pressure on someone as a researcher to try to be unbiased and, you know, objective with the research.

Eric Voyer (18:12.802)


Eric Voyer (18:26.146)

Mm -hmm.

Eric Voyer (18:33.026)

Yeah. Is that different from social desirability bias?

Brian Lamar (18:37.755)

I would distinguish the two, yes. So social desirability bias, which I would label that as you are going to answer certain questions certain ways because you don’t want society to judge you. Right? Yeah, exactly. A lot of sensitive topics. And that’s really, really hard. And I know we’ll talk about solution for this one here in a second, I think we should. Some sort of psychological need to be

Eric Voyer (18:41.09)


Eric Voyer (18:54.882)

Do you do drugs?

Eric Voyer (19:04.738)


Brian Lamar (19:07.739)

perceived positively or strongly agreeing to lots of questions because you don’t want the person conducting the research to think that you’re very negative. So it could be things like, do you smoke cigarettes? Do you do drugs? It could be, I just don’t want someone judging me that I hate everything. I want them to think that I like it.

Eric Voyer (19:26.658)

Right. I would imagine there’s a ton of academic research here on how to reduce social desirability bias. What are some of the different things you said? We’ll talk about solutions and I would love to get back to some of these other bias types, but what is this? What’s a good solution? Would you say for social desirability bias?

Brian Lamar (19:44.091)

Well, one of our, this is a solution for just about any type of bias that exists is to really work with sample companies that you know and trust. And if, if you work with a company that you know and trust and they have their respondent database, their panel, which is what we call them, can trust them. If you’re sending a survey that says, Hey, I’m Brian Lamaramar. I want to know if you take drugs. Now.

If you don’t know who Brian Lamaramar is, you probably might have some acquiescence or social desire bias. But if you do know me, hey, you know what? Brian’s asked me surveys before. He’s been honest and trustworthy. He hasn’t shared my data with the world. I know him and trust him. I’m going to answer a little bit more accurately and honestly. That’s a key that is often overlooked is the relationship that we have with our respondents.

Eric Voyer (20:37.57)

What’s it called when you’re trying to answer the question in a way that you think they want you to answer? You know what I mean?

Brian Lamar (20:44.603)

Yeah, I would, that’s, it’s probably called acquiescence bias. That’s how I would label it. And sometimes I confuse acquiescence bias and social desirability bias because they’re kind of the same thing. I think your question though lends itself to be more acquiescence bias.

Eric Voyer (20:48.354)


Eric Voyer (20:54.274)

Okay, yeah.

Right. So, you know, and part of this is how you phrase the question, but, Hey, we’re doing some research on, you know, people’s, I’m making this up, but people’s ability to resist certain cancer fighting agents, you know, how do you feel, how do you feel when you smoke cigarettes? Right. I mean, you know, you’re gonna probably think that they’re asking for bad, you know? So, yeah.

Brian Lamar (21:02.555)


Brian Lamar (21:16.027)


Brian Lamar (21:21.467)

Right. Yeah, absolutely. If you say, Hey, we want to talk to people that smoke cigarettes and you’ll get paid $5, you’re going to get a different response than do you do any of the following? Do you smoke cigarettes? Do you shop at amazon .com? And by the way, I’m going to promote what we do here a little bit in that when I joined and I knew this from the Durble IQ team beforehand, because I was pretty aware of this study and the practices that it leverages.

Eric Voyer (21:31.01)

Yeah, yeah.

Brian Lamar (21:51.387)

I’ll promote Steve Weasley. He has built relationships with online panels that have supported this research for 20 plus years. And he is very reluctant to switch that up. And I am too. I think that’s somewhat of best practices. You want to build a strong relationship with your simple provider. It could be online. For us, it’s online. It could be offline. And build trust with them and be kind of somewhat loyal to them. These panels are not going to be perfect. We all know this. They’re going to mess up.

Eric Voyer (22:04.13)


Brian Lamar (22:19.323)

So to kind of give them a little leash and figure out what happened when things go wrong, don’t just fire them and bring them a new sample company, but building that trust over time is so important and key.

Eric Voyer (22:30.754)

little plug for the industry there. That’s good. So when you think about that too, again, we’re going to go off topic here. We’re just going to drift just a little bit. Is it fair to say that certain panels will have certain biases?

Brian Lamar (22:33.147)


Brian Lamar (22:43.131)

Well, now you’re getting into my previous role. Absolutely. And so I think I am coined to the term of sample selection bias. Now, most people think that’s a commodity that if you buy a sample from this one company that saw sample, if you buy a sample from another company by sample, you’ll get the same result. That is not true at all. And you can look at my former employer. They’ve written blogs and white papers and reports on it and proven this over time.

Eric Voyer (22:53.09)

Okay, love it.

Eric Voyer (23:05.218)

Mm -hmm.

Brian Lamar (23:12.859)

And the reason is that panel, here’s a little secret by the way, Eric and our listeners, panels are not built to be representative. It’s just us here, right? And our dozens of listeners, panels are about dozens of thousands. Yeah. Across the world. Panels are not built to be representative. They’re not built to match census. And once you realize that it helps you to a lot of things with research, they’re designed to answer clients’ business questions.

Eric Voyer (23:22.786)


Dozens of thousands. Yeah, dozens of thousands of listens. Yes. Yeah. All right. Give it time. Okay.

Eric Voyer (23:39.65)

I see.

Brian Lamar (23:43.099)

And so if there’s a client that says, I want a bunch of smokers. Well, you know what, what’s the panel going to do, but going to be rep? No, they’re going to go out and find a bunch of smokers and give them their panel. If they want a bunch of people that fly in United airlines, they’re going to try to figure out a way to find people that fly in United airlines. And so that creates a lot of biases right there in terms of sample source. Some of the largest sample companies have built exclusive relationships to build panels with certain types of people. So some are more affluent.

Some have more travelers, some have more people that might use an iPhone versus a different type of phone. And so that exists. And once you kind of acknowledge that exists, that helps you get half of the way there, I think.

Eric Voyer (24:22.658)

And let’s recognize that that’s the value, right? That’s one of the values that a panel provides because depending on your research, you may not want to be geographically and demographically representative of the US population. Your target audience is the sample that you want to go after. For Durable IQ, it is the total nation. It’s the US population. So that’s the audience that we get. We balance on US census statistics and geography.

Brian Lamar (24:28.923)


Eric Voyer (24:50.946)

geographics and things like that. So it actually is the fact that these panels aren’t equal is important, valuable, but also noteworthy, be aware of it as well. And so those are the questions that you should be asking when you’re building your sample.

Brian Lamar (25:05.147)

Yep. And we use a lot of different sample providers because we buy a lot of sample. These are giant studies. We buy close to a million responses a year in the U S and Canada on the, our various types of work. And so we use close to 20 different sample providers at all times. And it is important that we have a strategic blend of them and that let’s just use Dynata for example, if Dynata is 20 % of your blend, we try to keep Dynata to be 20 % of the blend for

Eric Voyer (25:14.274)

Mm -hmm.

Brian Lamar (25:34.715)

ever and ever and ever. You know, the joke that Steve always says, it’s kind of hard to get on our work. But once you get on a TraQline study, it’s really hard to get off of it. It’s really hard. We don’t want you to go away. We need you. And that’s because of the consistency. It’s because of we understand the inherent bias a panel might have. And we can kind of manage that if we know it. Once we know it and measure it, we can manage it. And that’s what Steve and I have done for a long time.

Eric Voyer (26:02.05)

Yeah. So you’re making me think of all these different pieces, you know, different panels fitting into the mix. Like one of the things that I think about is questions and how questions fit into the mix is the fact that you ask certain questions or ask them in a certain order. Is that introduce any bias as well?

Brian Lamar (26:20.475)

Great question. You’re a good, you know, by the way, Eric, we know, we all know, you know, the answer to some of these questions. You’re just kind of leading me through this. So.

Eric Voyer (26:28.77)

Cut, cut. Actually, I’ll be honest with you, Brian, the way you’re explaining this is a way that’s very, very valuable to me even. So yes, I’m a researcher, but you’re really putting it in terms that I think our listeners and certainly I’m finding very valuable. So yeah, some of it’s true, but some of it isn’t. Some of these are totally new to me. Yeah.

Brian Lamar (26:49.755)

Well, I’m academically trained, thank you, but I’m academically trained for a lot of this and I’m on a lot of different industry -wide committees and I understand best practices. You have been put in the fire of this by answering and defending research for 20 years with tough clients. And so you can argue what is the better way to become a better researcher. But to kind of answer your question, yes, there is question order bias, there is question or design bias. That is another big one.

Eric Voyer (27:00.226)


Brian Lamar (27:17.275)

So the order you ask questions is so important. You can really bias people later on in the survey. You know, we want to be careful of when we ask about brands, when we ask what they purchased and the order that we do it in, where they purchased it, if they purchased it online, because after they see those questions, they’ve got something in their head that’s going to affect the next line of questions. So you have to be very, very, very careful. Things like double -barreled questions. This is so common. I see this all the time where a client…

things that’s asking one question, but it’s really two questions. And within one question, that is horrible because which one are you asking or answering? If there’s not an option for certain questions, like, Hey, where did you buy this? Walmart or Lowe’s? And those are the only only options. There’s not an other, another, I don’t know. That’s very common. Leading questions are common. There are tons of things around questioner design that I would suggest that, by the way, we do really well at this and I’m not just selling TraQline.

Eric Voyer (27:49.89)

Mm -hmm.

Brian Lamar (28:16.251)

I’ve seen this in action that if I want to ask a question, ask a new question, this is very common for me. I’m new to TraQline and I want to come in and I want to do some cool stuff and make myself look credible and smart. Like, Hey, I’ll go to our team. I want to ask this question. They will, they will argue and reframe it and talk about how it’s biasing and it can be frustrating. Then I go to somebody else and it’s a whole different set of questions around it. And.

some point you’re like, do I really want to include this question at this point? Is it worth my time? You’re laughing because you know this is true. Yeah, you don’t even go they’re too big of a fight probably but

Eric Voyer (28:48.642)


I don’t go to these meetings anymore. I don’t go anymore. About six months ago, I told Stacey, can you go for me?

Brian Lamar (29:01.659)

Yeah. And that’s a good thing and a bad thing. You have to have thick skin. And what they’re doing is protecting us as a company and as a survey design methodology, those roles. And I appreciate those people, even though I might disagree with them, even though I might argue with them, even though it’s frustrating and time consuming. To add one question on our survey can be really, really challenging because people are really protective of the data, reducing that kind of questionnaire design bias and…

Eric Voyer (29:04.898)

Yeah, it’s a good thing.

Brian Lamar (29:31.067)

and question order bias and excellent work here.

Eric Voyer (29:34.85)

I’m proud of that process, honestly. I mean, it really is. I mean, it’s a quarterly process, but I’m proud of the fact that it is so rigorous. All right. So tell me, do you want to rapid fire any other different types of bias before we kind of wrap up here? Anything else that you’re thinking of?

Brian Lamar (29:36.891)


Brian Lamar (29:42.363)


Brian Lamar (29:48.475)

Yes, let’s see, we went through social desirability bias, we went through interviewer bias, we went through sampling bias, there was cultural bias. If you don’t understand this, it can be really challenging. For an example, this Hispanics are more likely to use extreme points of the scale. They’re going to say they love something or they hate something. And that is something about that subgroup of people. Whereas Asians, they avoid those extremes. They don’t want to offend anyone. They use the middle of the scale. And…

Eric Voyer (29:55.138)


Eric Voyer (30:02.914)


Brian Lamar (30:16.635)

You know, as researchers, we can talk about this. I think other people talking about Asians and Hispanics and Caucasians are different, can kind of get in a little bit of trouble. This has been proven by academia and researchers for a long time. You know, Caucasians are kind of in the middle, but you know, Hispanics ends of the scale, Asians avoid the extremes. And if you’re making a decision, you kind of have to put that into account. That’s another one.

Eric Voyer (30:38.274)

Do you? Can you quantitatively put that into account? Do you weight that data according to some sort of research anchors or anything like that?

Brian Lamar (30:47.515)

Yeah, that’s a really tough one. And I worked on some global tracking studies, for example. We’re doing the same research in Mexico as we are in Japan. Well, wait a minute, it’s a little bit different. Why is it different? Well, this is why. As long as you go in with an understanding that you can’t have the same kind of benchmark or success criteria by market or by country, because people are just different. Just naturally different.

Eric Voyer (31:12.482)

Well, that might be wise in that case. Let’s say you’re benchmarking your satisfaction score and you’re getting an average of five, but you’ve got wild extremes. You probably need to figure out what the competitor is as well in that case, because if you’re comparing your US score to your Mexico score independently, you might be in trouble. But if it’s your net increase versus your competitors in both markets, you might find that they’re the same.

Brian Lamar (31:21.115)


Brian Lamar (31:25.915)

Yep, absolutely.

Brian Lamar (31:40.379)

By the way, Eric, we should redo this podcast and I ask you the same questions and see what we’ll put them both out.

Eric Voyer (31:44.514)

wow. That would be interesting. Yeah, yeah. Well, I would just listen. I’m already going to listen to this like 30 times, so I’ll give you the exact same answer. Yeah.

Brian Lamar (31:53.243)

I’ve got one last bias and that is sponsor bias. And I ran into this before too. And that is, if you know who the sponsor of the survey is. And I was asked years ago to build a community of moms that bought diapers. And, you know, that’s, that’s an easy category to recruit to. There’s lots of moms. Moms are passionate about buying diapers and, but I couldn’t tell who the client was.

So I was joined this generic group of people to talk about diapers. And it was really hard to kind of get people, recruit them, to maintain them, because we’re just generically talking about diapers. Well, if I had done that and said, hey, we’re Pampers. We want you to join the Pampers community. People will sign up in droves because they love Pampers. They trust that brand name. The same thing happened with Starbucks. Would you rather talk about coffee in general, or would you rather kind of influence what Starbucks is doing? And that’s a

Eric Voyer (32:38.146)

Mm -hmm.

Brian Lamar (32:49.659)

big bias that, you know, do you blind it? And this is this is conversations people have all the time. Do you blind it? Do you tell people what it is? And that’s a big debate. A lot of times it’s tough. It’s a really tough question.

Eric Voyer (33:01.186)

Yeah. So we know about some of the problems, like we’ve talked through as we go through these things about these biases, but is there one problem that this creates or is there one simple bias that like, if you can only, you can only control for one bias, eliminate one bias. Is there one bias to rule them all?

Brian Lamar (33:23.099)

I think the best practice to reduce as many as possible is to have people review your questionnaires. And this is a lost art. In telephone research, we used to do this all the time. You’d have a group, all the interviewers would review a questionnaire and offer improvements and suggestions and problems. We don’t do that as much with online research, and we should. And not only, so if I’m designing a questionnaire, I’m going to run it by Tom Cofield, I’m going to run it by Teresa, I’m going to run it by Steve.

Eric Voyer (33:44.674)

Right. Yeah.

Brian Lamar (33:52.123)

They’re all going to have an opinion. They’re all going to see the blind spots that I might have as a questionnaire design person. Not only should you do that, you should ask someone that has no idea what we do for a living. You should ask your wife or neighbor or friend, someone to review something because that’s ultimately who has to take the survey. And that will, that will save so much bias reduction that will help you so much in terms of reducing it. It’s just having someone review it, review it, review it.

Eric Voyer (34:11.394)


Eric Voyer (34:21.666)

That reminds me of when back in my old days, I was doing some research and had to do a survey on something and I was working with an engineer. You know where this goes, right? So the engineer comes in and says, well, these are the questions that we want to ask. What do you think of the average power delivery for system XYZ 3297 XY in relation to its aspect ratio? And it’s like, okay, so what do you want to know?

Brian Lamar (34:31.003)


Brian Lamar (34:46.555)


Eric Voyer (34:51.574)

And so, you know, that’s the other thing you have to be careful of is that too technical or not technical enough, right? So there’s always that balance that you’re trying to…

Brian Lamar (34:52.155)


Brian Lamar (34:57.499)


Brian Lamar (35:01.371)

Yeah, they say, you know, ask the questions to like an eighth grader or fourth grader. And, you know, the clients, they’re in their own head. This is the business question they want to ask. And they want to know those questions, those very technical questions. So we as researchers, that’s our job is to say, Hey, hey, you got to be careful with how we ask. That’s just a different way.

Eric Voyer (35:18.306)

Yeah. So that’s the, that’s the argument against survey monkey, right? I mean, seriously. Yeah. There’s so many risks to it. Now that doesn’t mean it’s bad, but there’s so many risks to do in your own surveys because especially of all these bias types that you’ve entered, you’ve mentioned. And so this is a good primer. If you’re going to do your own survey, this is a great primer for people to read and listen to about, Hey, I gotta be aware because I don’t have budget. I’m going to do my own survey.

Brian Lamar (35:21.691)

Yeah, absolutely. That’s a whole other podcast.

Brian Lamar (35:46.747)


Eric Voyer (35:47.138)

Let me learn about some of these different types of bias and, and Brian, I know that they could text you or call you because you do nerd out on this stuff and you’d just be happy to just chat with them about it.

Brian Lamar (35:53.787)

Love it.

Would love to reach out to me. By the way, this is why we have error in surveys. All of these biases exist. We’ve measured it over time. We have error, standard error in every research study. And that accounts for a lot of these biases. So yeah, reach out. Reach out to Eric too. Ask Eric some questions.

Eric Voyer (36:10.978)


Eric Voyer (36:15.106)

Right. Yeah. I’ll be happy to forward it to Brian. I’m kidding. I feel like I’m responsive. I’m more than happy to. Anything else that we want to address in this podcast? I know we try and keep it under 30 minutes. We may have not done that this time. Is there anything else that’s important to you?

Brian Lamar (36:17.659)

Ha ha.

Brian Lamar (36:22.427)


Brian Lamar (36:31.931)

all that’s important to me is to thank people for listening. And if you can challenge me, I would love feedback on this, 502 -785 -3608 or [email protected]. I’m active on LinkedIn. You can reach us lots of different ways, but love feedback.

Eric Voyer (36:35.97)

Love it.

Eric Voyer (36:45.73) I’m also active on LinkedIn. I know it’s also important for people to realize, you know, a lot of these biases we’re controlling for in the research that we do every day. And so it’s hard to list every single thing that we do, but if you do have questions about it, if you have questions about best practices, Brian and I are here to help. We love this stuff. Brian, like you said, I’ve been doing this for 27 years. so I would love to have the conversation. how’s she doing?

Brian Lamar (37:14.555)

That’s our first Bonnie interruption in the podcast. The old podcast, it was constant and she snored, but that’s our first one. Of course, the trash goes right by during the end of this podcast and my dog is barking, so I apologize.

Eric Voyer (37:20.45)


Eric Voyer (37:27.97)

Now that’s good. We’re going to, we’re going to have to post a picture of her with this podcast. There she is. Hey sweetie. Hey, this has been so fun. You, I don’t even think you scratched the surface. I feel like this probably could have been two hours. I feel like you probably had some other biases in your head that we didn’t get to. Maybe that’s another one. If you liked it, if you want to hear more on biases, we want to hear your feedback. Brian gave you the number 502 -785 -3608. We’ve enjoyed this one. I’ve enjoyed it a lot.

Brian Lamar (37:31.226)

Yeah, we.

Brian Lamar (37:35.067)


Brian Lamar (37:40.283)


Eric Voyer (37:56.93)

Brian, you’re super smart, dude. Really appreciate chatting with you.

Brian Lamar (38:01.019)

You too, man. This is awesome. And I appreciate all the kind words and body says bye bye. Thanks everybody.

Eric Voyer (38:07.298)

Bye Bonnie, bye.

That’s a wrap.

Brian Lamar (38:12.507)

That was good, man. Did you like that?