From Chat to Agents: The New Phase of AI in Market Research

I created these images using AI for a webinar in mid twenty twenty four. We had a choice, I said back then. Be smashed by the wave of AI or surf it like the cool cat.

In 2026, the image quality has improved a little bit.

The pundits say that AI is going to crush market research by software companies and the entire data analysis category is just going to disappear with everybody just chatting with AI instead.

So you may be a bit nervous about what's going to happen with market research, but I've got three separate forces making me anxious. And that's why it's an anxious rather than a cool cap 2026.

Needless to say, I spent a lot of time thinking about and using AI.

I don't think the game's over yet. I don't really agree with the pundits in terms of their long term view. We've all still got a role.

If you use AI today, you will know it still regularly makes dumb mistakes, and nobody in the know thinks this is really about to change.

Two years ago, if you asked AI how many r's went strawberry, it could only find two.

There are three.

This year, if you say, my car wash is 50 meters away. Should I drive or walk?

It says walk.

Now today, that might not be true, but it won't be true because they have got smarter. It'll just be true because they've hacked it in because so many people are testing the car wash thing.

Automated reporting analysis and simulated response are very much at the cutting edge of AI and market research. They're both big time savers. But how many poisonous mushrooms are being eaten today?

As promised with the webinar today, we're focusing on what's new, and new means not perfect. You're going to likely see quite a few bugs today. And you'll get a bit of an understanding of why we have these bugs and how to think about bugs now that you hopefully can become something you can build yourself with AI in the last bit of the webinar.

Agents keep getting smarter. AI in general keeps getting smarter, and there's lots of technical things going on under the hood which lead to this enhanced intelligence over time.

AI just knows more.

The third big change is relating to context window sizes. That is the amount of information that you can give to the AI when it does its thinking. They keep growing. The speed and cost economics keep improving. It's getting cheaper to get AI to do really clever things.

And most importantly, I think, is a growth in the ingenuity that we have, and I don't just mean despair, I mean, all of us have, in how we use AI.

Some of the pundits think that even if the rest of AI doesn't improve, we've got twenty years of innovation ahead of us just due to figuring out how to reuse what's already been built. And to give you a feel for this, these are all kind of the technical things that have been invented over the last three years. The key kind of technologies, I think, that are relevant in how we use AI. And all things that actually could have been invented three years ago, but many of them have only recently been invented just because it took us a while to get used to AI and understand what it could do.

If you stay up with the latest trends in AI, you've probably seen this chart already.

It's a chart which talks about how good the AI is, or AI in general is what's stated, the art AI, I should say, at thinking like a human. And how long it can think and when I say like a human, I mean, doing stuff that a human thinks is valuable. How long it can kinda keep going trying to solve a particular problem? And at the beginning of twenty twenty six, the answer was about four hours, and it's already all the way up to more than sixteen hours. And so going back here, each of these big five factors is coming along, but they're all improving things at the same time, which is why we have this exponential curve where AI is just getting better and better and better over time.

Some people think so good that's going to lead to the end of the world with the singularity, but I foresee a happier future. If we reflect on just our last three years, these are some of the things that have revolutionized a lot of how we do our jobs.

From summarizing reports to full traceable analysis

The one that I'm going to talk a little bit more about, because we're going to look at how it improves, is one of the big changes that's happened in a lot of insights teams. But now they tend to have their PowerPoint load up to SharePoint. They use Copilot, and they'll ask questions like Copilot. Can you create a report showing me the key trends related to snacking in the last month using my existing reports? And what Copilot does is it goes in and loads up all the old PowerPoints, extracts the key information, creates a new presentation. And often these presentations, I'm told, are pretty good.

And this is because these kind of technologies that harness the fact that AI is really good at interpreting text, really good at extracting the PowerPoint information, really good at formatting. So what's happened in twenty twenty six? Well, the first big build in twenty twenty six is not just going over PowerPoint, but being able to go over any type of analysis with traceability, which I'm going to explain. So what does that mean?

One of the classic deliverables in market research is a deck of crosstabs. A whole lot of tables that people read through trying to figure out what is interesting.

Skilled clients, particularly those with a background in market research, who in a rush still get these decks of tables sent to them, and they comb through them trying to find what's interesting.

But more often than not, the insights team will ask their research agency to prepare a top line report, an initial scan before the main report comes in, because we're in a rush, we want to find out what's just happened.

But AI means that the top line report should probably disappear. It's a skippable effect, and this is a part of a more general trend of AI, which is the disappearance of artifact creation and the disappearance of executional roles, which I talked about in the previous webinar.

Instead, the trick is let the clients use the AI themselves. So what does this mean in practice? Well, we can ask a question of the tables. We don't have to even read the tables.

Out of perceptions of Diet Coke vary by age.

Now let's raise the AI just to appreciate the benefit here. Okay. Age by preferred cola could be relevant.

And if I look here, it tells me that Diet Coke people are more likely to be thirty five to thirty nine, forty four. But then I gotta remember, this table's really not a clever one from the point of view of figuring out SKUs. It's really back front. Should we look at the right percentages? Let's go to the next oh, AI's already finished. Let's see what it worked out.

So in less than the time it took me to digest one crosstab, the AI read through the hundreds of crosstabs and figured out the core theme. So that's a really big time saver, and that's why I don't think top line reports have much of a role anymore.

But we talked about the mushroom before. The problem, has AI got it wrong? This is where we can introduce some jargon. There's a concept in engineering which I think becomes very relevant to all of us today, which is traceability, the ability to trace back any conclusion to the source data.

And what you'll see is with the chat that I just showed you, it wasn't simply analyzing the tables. It tells us which table it's relied on, so we can click on that and see the source data and get an understanding of what caused what. Now the tables aren't calculated using AI at all. They're just traditional crosstabs.

So AI has read them like we can, and we can go back and check the crosstabs.

Watching AI do the full analysis, not just summarize it

Now reading an analysis is a step beyond reading a PowerPoint, but the real place we all wanna get to be at is the world where the AI actually does the analysis. So it's not summarizing what people thought to analyze. It's doing everything. Let's do it.

We're going to pull in the dataset looking at color usage and attitudes.

Now me, we've got many years of market research background can look at this, and I can see to myself there's a little bit of a data integrity issue, which is that I've got a single question five. What is that? No label. It's a badly set up data file.

In the olden days, I'd get open my questionnaire, look it up, or I even remember. Do you remember some of you how you could used to be able to remember every question, but maybe you can still do that anymore. But I'm just going to get the AI to do the work. Check the names of each variable set, and if they're not clear, rename them. Joe, my little AI friend.

A few things to call out here. Think about the really big implication.

Display can do a lot. That makes it kinda hard to use, or it used to make it hard to use. Now the AI figures out how to use it, so we can use advanced software without having to figure out all the details of where to find each feature and how to click them.

Notice it's given us some feedback. It's tall. It's showing us the data it selected, and it's also telling us the changes it's made. It's made a number of changes, and what was previously called q five is now shown as brand attribute associations.

Reflect on this for just a moment. The AI is making judgments, the type of judgments that only humans used to make. Are the labels clear or not? Very subjective if you think AI has managed to do it.

Create a better label. Again, something a human used to do. We can get it to do more.

Create create a table showing the brand associations.

Now some of this stuff is a bit slow. Like, if I want to drag and drop this table and then you pass it, that would clearly be faster today. But as I showed you, everything is improving in AI, And that change is going to come pretty quickly, whether the AI will actually get that type of stuff stuff done faster. So but today, definitely worth if you're a busy person and you've got the time, learning just how to drag and drop. That's clearly a faster workflow today.

It won't be for long.

Let's get rid of that one.

Alright.

Let's get to a bit more. Please create a new page and add to it a correspondence analysis of the table currently shown.

Now anybody who's been in market research for a while will will know that back in the days, this was a really, really hard bit of work to do. So when I started market research, the way that I'll have to go about doing this was I would have to go through and get the data, run tables of means of each of these variables in SPSS, copy paste them into Excel, rearrange them in these columns, set up a new data file, put them back in, write some code to run the corresponding analysis. I got it down to about half an hour at the end. I was pretty good.

Here, it's actually just done it all for me magically. Now, display or queue made it easy to do it by point and click, but I didn't even need to learn how to point and click. A big, big time saver there. Let me give it a much more general question.

Please create cross tabs of each variable set by each of the demographics and by preferred column.

Now think about this as a question. I could learn how to do it again once you know it's pretty fast to create in this plan.

Most people traditionally, though, they would get the data processing person to do it. And often, the data processing person would make that kinda hard. They'd go, yep. Happy to do it.

I need you to fill in this spreadsheet. I need you to refer to the column numbers. Yes. Some people still get asked to do that.

Or the variable names, specifically, tell me what categories you wanna merge. How do you want me to change it all? And so you could spend half an hour filling in your request. But now we're just going to ask the AI, and it's going to figure it out for us.

And so we've got all of our crosstabs in no time at all. Now, again, it's a really big time saver to do this kind of stuff because we don't have to learn how to do it. Now I promised you new stuff.

What we've just done here, you couldn't do this even a week ago, the steps that I just went through. These are all very cutting edge changes that are live in display today.

And this is what's coming soon. I'll give you a chance to have a look.

When I say soon, one of our major engineering projects at the moment is the ability to not just ask questions of one dataset, but what if I've loaded up five hundred datasets, got my last five years worth of research studies, and I wanna see what trend it's got. The AI is that we're building at the moment will go into all of those datasets, figure out how to do the relevant analysis, and collect them all together. So similar to what people do in the PowerPoint, but because we're using the raw data, it can do any analysis, even things that people hadn't thought of when they created PowerPoints.

Always on. We're talk about the next two. One second.

Chatbots vs. agents — what's actually different

Chatbots versus agents. You can find a lot of things written about agents, but most of it is well out of date. Today, the distinction really comes down to this. Agents are given goals, and we were just doing that.

So the chat panel that we looked at in display, we were giving it goals. Create these tables. That was the goal. And they use trial and error to work out how to meet them.

And so if you look at the thing, you can actually it's actually showing us each of those steps. So sometimes it'll have little error messages. Stuff is just trial and error.

Agents act on external systems, so it means they can execute full workflows rather than ask questions. So some people today use court code to build workflows, and then they will run Excel, the data collection software, even good old despite from court code.

The classic example, though, is people use agents to write software where they give it the goal of what they want to build.

The cutting edge or the bleeding edge at this stage, and it's not ready for my for prime time yet, is the always on agent, which keeps the wheels of your oil spinning without you having to give it regular instructions. If you wanna learn more about this technology, just do a bit of research on OpenCore DLIW.

Before AI, if you wanted to automate, you needed to work through the detailed logic you wanted to follow, and then work at how to express this code.

We can go from goals straight to software, skipping to an extent the middle bits. And I would say to an extent, as it's still quite a bit harder than it sounds.

But pause for a moment and think about this.

A few years ago, writing software meant mastering some arcane computer language and understanding the entire logic of how everything fits together. Today, success can occur with rambling instructions written poorly or spoken poorly.

It's an easier world in many ways.

Skills: turning expertise into a reusable building block

Skills have been kicking around as an idea and technology for about a year, but they're having their breakthrough moment right now. You have them included lots of other products. I'm I'm going to very shortly show you the first iteration of this in display. Expect some errors, but we'll do a webinar in a few weeks when main kinks are out, and it's ready for prime time. We'll give you a much more detailed overview of the technology.

What is a skill? It's the definition. Let's go look at an example.

I'll give you a moment just to read this skill.

Skin grade, anyway.

So a few things to call out. It is written in English. There's no code here at all. It's a set of instructions.

Special things.

These instructions in the displayer, for the time being, will have to be provided as a text file of the older people and ASCII file.

The name of it tells us the name of the skill, and so words with dashes, and the file name is to end with dot q skill so that we can detect this as skill.

The description so the first three lines have a very clear formatting you gotta follow. Three dashes, description, and quotation marks. The description is used by the agent to know when and how to when when to use the skill or not using the skill. And then underneath, you have your description your instructions. Sorry. And the key thing about the instructions is they are just written in plain language. And the clearer the instruction, the better the chance the AI will have the following, just like a human being.

Here's the kind of workflow for how you use it, and let me show you that in practice. So we go back into display.

Give ourselves a bit more room, and we are going to go to add a cloud drive. Now go to the cloud drive, open up as a new tab. For those of you that haven't used the cloud drive before, it's where you can load up any auxiliary files that you wanna use in your work. So I'm going to upload the skill here.

And I refresh that display. Can check that it's got this new information around.

I'll just tell her. And, you know, get rid of the existing stuff just to make it a bit easy for everyone to see what's going on.

Alright. So back to the brand association table.

Run the moon plot skill.

Now, hopefully, you can see the exciting thing that these skills represent. But let me tell you a little bit more about what I always experienced in every market research company I worked at, which was that everybody talked about the need to productize.

Rather than just provide consulting services, we would need to encode our expertise as products, which we can then sell.

And we'd always start on these, and these projects would fall over because it just turned out to be too hard.

But now AI is making two big changes for us. The first of the big changes is we really need to find a way to differentiate ourselves, and so productizing seems to be a good way forward. The second thing is skills make the building of products vastly easier than ever before. Don't have to train the person.

Don't even have to write the code. We can write out our products as this concept of a skill. And look what we've got. My very special way of creating correspondence analysis as a moon plot is now entirely automated as the skill.

We're going to return to this topic shortly.

What market researchers should do now

I spent sixteen years of my life as a pretty much full time market researcher. I spent nine years studying at university focused on the algorithms market research and about twenty years building software. And for those of you trying to guess my age, yeah, I did a lot of these things at the same time fairly often. A bit of an overlap.

Here's my attempt to distill everything that I've learned in my journey in terms of a set of kind of principles about what you should be doing now as you start to figure out how to redesign your business with the benefits of AI.

I talked about the five factors that are all changing rapidly, with the net effect being that everything is changing rapidly.

It has a consequence. Each six months, things that were just impossible, even inconceivable six months ago, now become possible, which means every six months, have to start again. What can we do with AI now?

Lots of people pride themselves in their ability to figure things out, and there are lots of things you can learn that way, like gardening. But AI is moving too fast. It's no longer smart to be the guy that girl, she's the guy, actually, who feels that they figure everything out.

It's a loser play.

Reading, listening to podcasts, following Twitter, these are actually great ways to keep up with the latest innovations, but it's not enough. Most of what's there is out of date.

For many, many years, finding the smartest person and asking them someone with expertise was a winning play. It is a losing play with AI. The person you're talking to is almost certainly out of date, and they don't really understand your context and probably don't have the interest in listening to all of the detail.

The winning play is to ask AI. And when I say ask, I really mean debate. So what are my standard ways of operating and understanding AI, and indeed, how much of this presentation was written, was done in concert with AI. And the workflow that I had goes like this.

Put the harness on the dog, take the phone out of my pocket, go into ChatGee, Pitu, or Claude, and click the voice button, and tell it what I'm thinking or ask what I want to understand. And then it writes back to me, and often it gets it a little bit wrong or it misses my point or something. And so I correct it or I argue it. Those of you who know me would appreciate exactly what I'm talking about here because I do with humans too.

And I'll argue with it, and over time, my knowledge grows as the argument turns into something of an agreement by the end, and I learn something. And it is the way to use this technology.

There are no experts anymore. The AI knows the most. Work with it.

You need to understand things before you automate them. It sounds obvious, but few people do it.

If you don't know how linear regression works, how can you automate it? How will you ever know if the automation is correct?

I keep coming across people saying they have migrated their analysis to ChatGPT or Claude, and by and large, they're people who don't know how to do it. And so they're potentially eating some mushrooms, and they've got no way to spot.

Traceability is really important. Let let's go back to what we got Claude to do before. Claude to display it to do before. So I applied the skill. If I wanna trace this back, I can. There's a little trace button here, go to inputs and outputs. It tells me this was created from the table.

The table shows me now that this is created from a variable set called, meaning, associations. I can click on that, and I can see everything about it. I can see the individual variables and variable names, and it comes from this dataset. So even though the AI did the work for me, the traceability was built in. And this is the thing that you really need to be building in your own workflows. It's fine to use AI to automate something, but you need to make sure that you can trace it back. And you can only build that workflow if you kinda understand what good looks like.

Much of what you should be proactively doing these days is actually creating checklists. You wanna make the implicit explicit. Doctors and pilots crash this often when they follow checklists. It's the same for you and I when we build software or do our research. Checklist everything.

Here's the checklist that I use when I analyze brand association tables, and this is what you saw when we build into the skill before.

Standardize. Winning isn't each person has a checklist. Winning is has a company standard checklist. This next point is really important, kinda hard to do in practice.

But it's the idea that when you build out your new processes, you don't wanna lock them down. You you wanna build processes that have feedback loops in them so that you can continually improve those processes. Create what's called improvement loop.

These processes and checklist need to be digital. It's cool if you wanna write them on the board in your little meeting, but you really need to get into the practice of digitizing so a machine can read them like I showed you before. You need to make it easy for the AI to find this content. So the skills are an example.

I put them in that special file. I uploaded them to display it. That's what caused the connection. There's lots of other ways to connect, but the key principle is this.

You need to be connecting the content so that AI can use it.

And you need to be understanding reliability.

As I talked about in my last webinar and earlier today, AI makes mistakes, as do humans. And if you can automate a human process, you need to understand how reliable it is today and how reliable it becomes when you automate it. The naive view is to say you're looking for a hundred percent reliability, but humans were never that. AI won't be that. I'll talk about a bit more in a second. It's often okay as you migrate to AI for things to become a little less reliable.

I promised you some bugs, and we actually haven't had any yet. But we made a very conscious decision that we keep adding functionality via our chat, even though we know sometimes the AI gets it wrong, because the time saving is worth it for the user. And it's the same with any process that you automate. You will get have to start making conscious decisions about acceptable levels of reliability, but you can't do that if you don't know what your reliability or your mistake rate is. And you can't do that if you're not even aware or understanding whether something is reliable.

This next one has proven to be extremely hard for market researchers to wrap their heads around, and it's one of these things that's because of the strength.

Good market researchers are great project managers. Great project managers come up with plans, good plans that they then follow. Sure, a few little things go wrong, but basically, your skill of market researcher is largely related to your ability to plan through the entire process and follow it ahead of time. If you're bad at that, you tend to lose a lot of money and lose your job.

So market researchers are great project managers. And the way we approach is like this plan for building a car. We take each of the elements that we need to build up, we assemble them, and it all comes together magically at the end. This is an absolute loser play when building software. Horrible, horrible way to build.

Why? Well, building software is actually more complicated and more complex than market research.

And no one's really, for any interesting problem, able to think through everything upfront. And so instead, you need a different workflow, one where you get feedback along the way and keep iterating and improving. You can't come up with a great plan. And those of you who will ignore me will, until you get this point, fail in your automation projects.

I've learned this through a lot of pain. I was a good project manager once too.

The approach that you wanna take is much more like this. Even though you know you wanna get to a car, you don't build a car. You build something small and workable, a skateboard. Once you've mastered the skills of building your skateboard, you can then progress to scooters. And if you know how to build a motorbike, you're in a really good position to build a car.

I explain this again and again to software engineers, to market researchers, and everybody ignores me and goes here, and they don't finish. This is the way to build the software.

We internally use the language skateboard as a reminder, because smart people like to think they can do this, but then they're wrong. As I showed you earlier today, we should use skills as a new way of automating. It's a great building block for everything that we're doing.

Your first skill won't work. By the time you spend some time on it, maybe it works ninety percent of the time. But you'll find to get to ninety nine percent takes twice as long. To get to ninety nine point nine percent takes twice as long.

Again, each nine costs a lot. To use a bit of jargon, hardening is the process of making your software more robust, stronger, so it doesn't break down. But it's an expensive thing to do. And often the smart play, as I talked about before, is just to be happy at ninety percent to ninety nine percent, because the hardening cost is where things start to become very uneconomic.

What does hardening look like? Well, today, hardening with skills actually looks like intermingling a bit of technical know how into the skill. So here's a skill that I wrote for doing the moon plot, where I've actually said to it some quite precise stuff. It's kinda hard to guess to harden up.

I'm just doing this as an illustrative example. Hopefully, when we launch the skills in a week or two, we're not going to need there's not going to be any benefit to this hardening, but there always will be some. There will always be situations where actually writing some code in the skill will make the skill a bit better. And you'll be thinking, oh, I don't wanna write this code.

This defeats the whole point, and this is the trade off you make. And this is why hardening is hard.

And the last tip, this is a really important one, is don't try and build long skills. Build short skills which refer to other skills.

So if you look at my display at Moonplot skill here, it actually says at the bottom, use the brand Moon plus skill. Right? So I've got, like, nested skills if you like. Let's give it a go. Fingers crossed.

Run. I will delete this one here.

So this one here, just get a mental picture of what this looks like.

You see it's got Coke and Pepsi on the outside.

From the display let's check the name of it.

Moon plot skill focusing on Diet Coke.

So this is a very proprietary workflow. This combines a few things. I'm wanting to create the moon plot, wanting to tell it to focus on Diet Coke, which means it's going to do a special rotation of the correspondence analysis. It's going to do a whole lot of quite technical things that are built into what I regard as best practice within brand association data. And and the key point I'm illustrating here is that skills allow us to do what we think is correct, allow us to lock down our company standard processes. We can call them best practice if you're a narcissist like me, but it's really just about getting it done the way your company expects it to be done, or your clients expects it to be done the way you want to do. So we'll come back to this in a second.

Live audience Q&A

These are the topics we've gone through today. What questions or comments do you have? Please type them into the questions field in GoToWebinar. And while I await your questions, we'll go and see how the skill's coming along.

So you can hear it's thinking, it's iterating, it's trying to achieve its goal that I set it, which was to follow my set of instructions or my skill.

Alright.

And so it's told me a whole lot of things to do. Let's just click on little properties. And so, again, traceable is important. It's telling me some quite advanced things that is create the main plot, particular form of correspondence analysis that I'm very partial to.

It's telling me it switched the rows and the columns.

It's telling me it set the normalization, something called row principal scale, it set the focus to Diet Coke. So it's made a whole lot of changes.

Yeah. So that was skills. Hopefully, you found a compelling idea. No questions of it at all today.

So hopefully, that was good. Was so exciting. Jared's there. Thank you, Jared. It's always nervous when no one asks the questions.

There's the fear that the audio dropped out along the way.

Jared says, what I'm most interested in understanding is AI opening up new waves of research. A big one that we've been experimenting with is thematic and semantic coding of large amounts of qualitative response. Do you have any thoughts on other avenues that AI is making possible?

Well, I'm not sure if you've seen the text analysis capabilities, but we do that's something we launched a few years ago, and so there is the ability to certainly do that from within the app. I didn't kinda focus on it because it's not such a new thing.

So we can get we can tell it to create themes, and it'll automatically find the themes.

And then you can tell to classify the data into the themes. So it's kind of a kind of application. I I think a lot of these things at the moment and so this was built with a whole lot of handcraft software to optimize it. Stuff like this will, I think, in general, start to migrate to skills over time, and this creates new data with the with the categorizations. The the the big innovations that are kind of going on at the moment really relate to in micro searches, so to automate reporting and analysis, that whole side of things.

Qualitative was very greatly affected in the first couple of years, and that's obviously still improving along.

Thing that we're starting to play around with is the whole UX side of research. So get the AI to test the software directly by interacting with it, pretending to be human.

The simulated respondents are another area which is obviously very big at the moment.

There's lots out there. Jason says, the more of these webinars attend, the more I'm starting to like this. Well, that's lovely to hear. I'm a traditional SPSS user.

Well, that's a bit sad. Normally, meant to go SPSS. You use q for twenty years because q was designed to replace SPSS. And then after twenty years, you finally gradually move on to display because you love q so much.

Hopefully, there'll be display at one and on one videos. I can access there are lots of videos in display. You wanna go to display dot com, you can, of course, purchase.

But if you go into our resources, you'll find heaps and heaps of videos showing how to do stuff or book a demo or take out a trial that you've seen today. You can get it to do a lot. Laura says, using the AI functionality, does data stay in Australia if you sign up to an AI based offer? To sorry.

To an AI. Yeah. So if you look at pricing, if you want the data to stay in Australia, you need to be choosing an enterprise pricing model. And when you do that, all of the AI is done in Australia.

James. James says oops.

James says oh, too distracted by a phone call.

Are there any simple research skills you see as a good starting point? Well, the simple things we're actually building into the back end. Right? So a lot of the the first as as I was showing you before, I was illustrating the skateboard principle in action.

I showed you our very first go with building skills into Displayr. In a few weeks, we're going to have another webinar where we're actually going to launch skills. And there are two sets of skills. It'll be the ones that we create, and so a whole lot of standard workflows we're just going to build in, and then there's the ones you create where you create your own custom workflows.

So I'll talk much more about that in the forthcoming webinar on skills.

Lillian says all teams have their own acceptable level of reliability. How does your own team define the acceptable level of reliability for AI processes? And is there anything done from the human end afterwards to improve the reliability?

The I mean, the way that we kind of think about it is very much through the traceability lens. Right? So so the take these questions that we're asking.

Make it so that you can look at any visualization or anything that AI creates and always trace your way back to the raw data. That's our focus. So for us, we think the focus in what we build needs to be that it's a hundred percent ability for people to self work out whether something is accurate or not. The thing and I kind of didn't go into this at all before, which is kind of interesting but challenging is the biggest challenge with reliability is that people write things vaguely. And so whose fault is it if you write something vague and the AI does something different to what you had in your head? Is that reliability or something else? So I've kind of dodged the question a little bit there, haven't I?

The so the the our kind of internal standards, so we require that when we calculate anything, it's a hundred percent accurate. So we're always you know, we we set the reliability of any calculation. What I mean by this is these percentages, the way we calculate significance, they need to be a hundred percent accurate. How you interpret it, we push that to the client. We build tools so that if I want to I'm just putting some page to make it a bit easier.

So when a user chooses to use, say, AI to automate its interpretation, So it's going to give us a lot of suggested questions. But we then tell the user what they need to be telling the AI. And a sample description so it is a nationally representative sample of Australians. So we're encouraging you to provide as much context as you can to help the AI, but I'll go.

Are there any data integrity issues in the selected data, which is just the aged data.

So we we do all these things to try and encourage the user to give the relevant context in, and it's found a data integrity issue here.

But it's really for you to own this. We we see this as we're drafting it rather than finalizing the report, and then we provide the traceability.

I'm glad you liked that our pricing is transparent, Jason. Thank you. Dale says, in the data preparation agent, we haven't looked at this today, but but you can actually say to this player, please clean and tidy my data.

Do the general instruction like that. And it's actually going to fire up a special agent, hopefully fire it up. I promise bugs, and we haven't had any yet, which we'll call the data preparation agent, which is a type of agent where we've given a very detailed set of steps to really complex kind of workflow, replicating how a human might traditionally clean and tidy the data. So when it does that, it goes through and does a hundred and one things to check and clean the data, which we'll hopefully see very soon.

So back to what? So this is a data preparation agent. It gives us you choose all of your options and it tells you do everything. So back to Dial's question.

In the data preparation agent, is there a way to check through the entire data set for inconsistencies, responses instead of selecting specific survey variables? Yeah. You could you can certainly do that. I don't recommend you do it, though, because it'll just be super slow.

And once it's gone through this, I'll show you how to do that.

The the there's a kind of a the issue of everybody has this kind of idea in their head that they're worried that they've got bad quality data, and so they want to get the AI to find data that's inconsistent or generated by AI and exclude that data. And it's I think I understand why you want it. I'm not sure you really want So in my early research career, was doing this really important strategic study. There was a it's back in the nineteen nineties.

There was multiple billions of dollars at stake on this government project. And so I took my job very seriously. And I went through and I removed every respondent who had any contradictory data. And we ended up with seven respondents at the end.

And my CEO at the time, a lovely man called David Adams, he explained to me that I kinda didn't understand market research as well as I thought I did. And what I had to appreciate is it wasn't just that my humans in my survey had said contradictory things, so I shouldn't believe them. It was the wrong way to think about it. Humans are inherently error prone. And so when you listen, if you carefully listen to everything I've said, there's little mistakes throughout my conversation, and there's little mistakes and contradictions throughout surveys. And if you're too brutal and you exclude everybody, you just end up with no data at all.

And there's another quite famous in Australia story. This gentleman, and his ethnicity is important. He was a he was a Sikh gentleman. He didn't drink much.

And he had the job of cleaning the data for a a survey done by a large brewing company. And the brewing company had this problem. They never believed the results of the survey because it kept showing that in the Northern Territory, which is kind of our version of Montana, for those of you who aren't Australians. They kept showing that these Australians' consumption in the surveys was much lower than what was known to be consumption in the Northern Territory.

And the reason was this gentleman just couldn't believe the answer he was saying, and so he kept just deleting, changing the responses data down. When in reality, the true consumption that these people said that they were consuming in the survey did actually align with the consumption that brought was seeing.

And so data cleaning is a difficult thing. If you give AI the job of doing it, well, can the AI even detect whether somebody else has done it? It's kind of a hard problem, which is why there are many traditional data collection tools which have been used by the data preparation agent. You should check out some of the early webinars on that agent if you wanna learn more about that.

So, again, kinda dodged your question a little bit. I hope you understand where I've come from in that long answer, Dale.

Lillian says, it's great. Cool. Are these also available with Q? No. They'll never be available in Q.

You see, Q is built for a world where people have laptops and software sits on the laptops. AI doesn't work that way. It needs to go in the cloud, which one of the reasons we built this plan.

Akshat says, hi, Tim. How do agents handle requests that fall outside this defined scope? For example, if I have an analysis type, it doesn't natively support a question type. It can't confidently answer.

Does it flag uncertainty falling back gracefully, or will it attempt the task anyway? It will attempt the task anyway or try and write some code to solve the problem. And that's where traceability your expertise come into play. Agents aren't good at knowing what they don't know, just like most humans, in fact.

Jason says, I found that ChatGPT is not the greatest to help me work through display help me questions. However, your contact team are amazing. That's wonderful to hear, responsive and willing to get resolution. I loathe to send support for help, but your crew are brilliant.

They are brilliant. Thank you for that feedback. I'll pass it on to them.

And there are all the questions people have asked. Thank you so much for all our users out there. Thank you for being our users. If you're not one of our user, well, you can see that you can buy now at any stage. These are showing prices in Australian dollars. Thank you, everybody, who is our client, and have great days, everybody. Bye now.