The documents that were used in this webinar can be viewed here and here.
Do you think AI is at the peak of the hype cycle? If so, you’re already running late.
While in some industries, AI is seen as being at the peak of the hype cycle, in market research the changes it has wrought and will continue to make are both fast and dramatic.
Check out this insightful journey through a standard market research study, where we’ll highlight how AI is streamlining the process and accelerating time to value.
In this webinar we’ll look at both the easy wins and the areas to be cautious.
We'll cover
- Project management
- Study design
- Data cleaning
- Text categorization/coding
- Analysis
- Storytelling
Along the way we'll explore
- The things that AI can do, from understanding natural language inputs through to writing code
- Our roadmap for the future
Transcript
Today is all about AI in market research.
In the last few webinars I’ve run I polled market researchers on how often they use AI. I got the same distribution of responses in Asia, Europe, and the US, and when doing the survey among customers and non-customers.
Some customers are experiencing big productivity gains.
But two third are not yet using it.
Here’s the thing. For most market researchers, the productivity gain is well over 5%, so if you’re not getting a big productivity boost, I’ve got some good wins for you today.
Case study
A few years ago I conducted a concept test on the iLock. I’ll give you a moment to read it.
My focus today is on showing how AI has changed how I would go about doing this concept test if doing it again.
But first, let’s touch on some general principles of using AI.
What we want
The jargon for what we all want is fully autonomous AI agents. We want to be able to tell the AI to do something, and have it go away and just get it done. That is, we want the AI to be an awesome direct report.
We’d like to be able to tell the AI to do the whole study and come back with a decision for us.
They said we’d have flying cars
One of the great tropes of tech is they said we’d have flying cars, and all we got was iPhones.
Or, more recently, elon promised as self-driving cars by 2018, and we’ve still not got them.
Going back to our jargon, a self driving car is a fully autonomous AI agent. And, despite billions of dollars of investments and more than 10 years of work by the best minds in the world, we still don’t have them.
Why don’t we have them?
When to use Fully Autonomous AI Agents
There are two situations where we should use fully autonomous AI agents.
The first is when the AI is error-free. The tech titans haven’t yet got that working for cars.
The second situation is where it’s easy to spot errors in the work of AI, and it’s OK if there are errors.
The classic example of this is the chat bot, where if it gives a bad answer, the user can re-ask the question.
Now, as market researchers how many of you have clients that are OK if there are errors in the reporting? I think there aren’t so many of you out there…
Today, it’s the norm that AI makes mistakes
Now, as of today, it’s the norm that AI makes mistakes. Or, hallucinations are they’re more commonly known.
I asked ChatGPT to create an image of a cat drinking a milkshake with chips. It’s done well, but not well enough. Can you see the errors?
It’s written chips on the sugar shaker and the wall.
And, it’s topped the cat with cream and a cherry!
What we can’t (yet) have
I’m sorry to say that at the time of writing, market research AI technology isn’t error free, and, as our clients aren’t OK with errors, we just shouldn’t yet be using fully autonomous AI bots.
We’re of course working on the error rates and there are lots of useful technologies that are improving accuracy, so we’ll get there. But, we’re not there today.
When to use generative AI in market research
So, if we can’t have the fully autonomous AI agent, what do we get instead?
We need to have three things in place to deal with the errors in generative AI.
It needs to be easy to spot errors
It needs to be easy to correct errors
And it needs to save time.
When we have all of that, we’ve got the right application for AI.
Research process
So, let’s return to the study of the iLock and work through the time savings. I’ve broken the research process into stages, to identify where the big wins are located. We will start with study design.
So far I’ve been showing you everthing in Displayr, but I’m now switching to ChatGPT.
I devoted about 20 years of my life to being the most knowledgable researcher I could be. I was the guy that people would come to and work through the options of study design.
No more.
Anybody can do this now.
"Please play the role of an expert market researcher. Design a study for assessing the appeal of a product called the Apple iLock.
It's a device that you attach to your front door lock and it allows you to lock or unlock your door using your iPhone. The plan is to sell it for $199."
Hopefully you’re a quick reader. The key thing here is that ChatGPT gets it basically right. Even if you’re an experienced market researcher, it saves time as it’s laid out the basics and you can tweak it and move on.
"If I do my concept test as an online sample, what’s the smallest sample size that is acceptable? Please give an answer in less than 10 words."
I must stress that each time I’ve checked this I get a different answer. I’ve seen 200, 300, 400, and 1,000.
But, that shouldn’t surprise us. We’d get similar inconsistencies if we talked to multiple researchers.
We can take what we are getting here and quickly polish it. It’s a clear win.
When I wrote the iLock study, I had to go back into my backups and find all the old concept testing questionnaires I used to write in my youth. It took a few hours to do that. I wouldn’t do that now.
"Please write the questionnaire for me."
Chat GPT does a really good job. Again, it saves time by giving a good place to start.
So, there’s time to be saved for study design and writing a questionnaire. And, for a new researcher, it’s a lot of time.
I’ve used a agreen AI logo to flag that we get time savings using AI for these. And, I get these as a pretty experienced researcher. For a novice, the productivity dividend is much bigger.
Programming
The data collection platforms I use don’t have good AI for rogramming the questionnaire, so I wouldn’t be using it. But, this technology is easy to build and I think will become widespread in the next few years if it’s not already.
Fieldwork
I wouldn’t be using AI to do my fieldwork today. I’m not interested in a hallucination that leads to me having no men in the sample or having to pay for a sample size of 1 million.
But, this will change in the future.
Creating an analysis plan
How about the analysis plan?
"Please create an analysis plan"
Now, please appreciate I’m doing this live.
So, help me here. What can you see that’s wrong? Please type it into the Questions window in go to webinar.
For my own sense of self-worth, I’m delight to have found an area where AI isn’t so useful for me. This is all too generic and unhelpful for me. But I imagine a novice would get a lot of value.
So, I’ve not given it a score for creating an analysis plan.
Cleaning/tidying numeric data
I won’t waste your time on this one. Cleaning and tidying of numeric data is easy to automate with pre-AI technologies and AI’s miles behind in terms of permitting QA, so I wouldn’t use it.
Clearning/tidying text data
But, it’s great for text data.
Here’s the data from the iLock study loaded into Displayr
I’ve got two text variabels, and will ask the AI to find non-clean data.
Select Concept Likes and Concept Dislikes
+> AI Clean Text Data
Select all three variables > View in data editor
Sort descending by numeric
As you can see, it’s done a good job. Everything with a 1 next to it is junk.
But,AI is most useful when you can tweak it.
Let’s have a look at the prompt
" If you're unsure, give a score between 0 and 1 to indicate your uncertainty, with higher scores indicating that you are more likely to believe the data is bad"
Data translation is also great.
Select Concept likes
+> AI > Translate
"Translate the text into polite french, with a maximum of 10 words"
And, we can tell it to tidy as well as translate.
Select translated variable > View in data editor
Text Categorization > AKA Coding
The single biggest win, by far, is in terms of text categorization, or, as we call it in market research, coding.
Concept likes > + > Text Categorization > Multi >New
20
I find it most useful to start with about 20 categories
Now, this is going to take the longest of anything we’ve seen so far. There’s a lot of heavy lifting going on, and, we’ve also got to get some optimizations into this!
Many of my colleagues and clients have given me the same bit of feedback about what I’m about to show you. They all say that they just want to click a button. The don’t want to ahve to choose 20 categories and think that the AI should figure this out.
I agree it’d be great if it did. But, what we know from extensive testing is that it always makes mistakes. So, our user interface is desgined to make it easy for you spot and correct the mistakes.
Go through cycle of:
1. For each category check
2. Combine
3. Delete
Save categories
Weighting
Weighting is, in my opinion, a bit past the reach of AI today, but it’s not far away.
Data analysis
We already saw that AI didn’t create a great analysis plan. So, as a result, it’s not able to be given the job of doing all the data analysis.
But, a lot of people give it parts of data analysis. Let’s have a look
What’s the relationship … AI
AI > Data Analysis
Prompt: "How is purchase intent related to age?
Crosstab interpreted by AI
Doing the crosstabs yourself is generally much better.
But, what if you aren’t good at interpretation? AI can help.
AI > Interpret
Hook up to the table
Increase the font size
But, I’ve got to say, as an experienced researcher I bet you can do a better jon than this. The key pattern is in the first row.
Find the story > Use AI to summarize
What I’m about to show you is one of our most requested AI features. Personally, I don’t get the point, enough people have asked for it that I’m aware I’m the odd one out and it provides value for others.
As is well known, AI is great at summarizing documents, and this feature uses AI to automatically summarize any comments that you’ve created in your docuiment.
AI > Summary
The summary is longer than the text!
Prompt: "Summarize in 20 words or less."
Craft the story: Using AI to generate images
We have an in house illustrator, Claudia, and she’s much better than the AI still for some more bespoke visualizations.
This is the tab that I’ve used her to create. In terms of my time, I gave her a the list of 12 stages of the research process and asked her to create the 12 visualizations you can see here, and she did it as a fully autonomous human agent, whereas I’m doubtful I could have gotten something with this simplicity or design consistency from AI even if spending a whole day on it.
However, all the other illustrations I’ve used today were created via ChatGPT. I’m told the more specialized designer tools, like Adobe and Canva are even better.
We’ve also implemented an AI image generation, both as a time saver and because it allows you to hook up data and use it as an input to the image.
AI > Image: "Please create a Pixar style image showing how technology is more popular among younger people than older people."
