Cluster Analysis: Classic, Powerful, and Still Worth Knowing

What is cluster analysis?

The top half is all about cluster analysis. The bottom half is all about the main application of cluster analysis, which is really market segmentation.

The oldest approach to clustering is almost one hundred years old, and we're going to understand how this approach, hierarchical cluster analysis, works with a really simple example. We've only got six observations on two variables.

The first step is to calculate the distance between each observation. There are lots of ways of doing this, but the easiest way to understand it is to imagine that we draw them, and then we measure the distance between each of those points. And if you remember your math from school this is the pythagorean or euclidean distance, and that's the by far the most common way that we measure the distance between points, and we then assemble them into something called a distance matrix, and this just shows us how far apart each of the points are from each of the other points.

Once we've done that, we move on to aggregation, which is merging clusters. So these are the two most similar points, we merge them together to create a bigger point or cluster to use the jargon.

Then these become the next most similar, so we merge them.

These are the next most similar, so we merge them.

These become the next most similar, so we merge them, and we keep proceeding until everything is merged together.

The next step is to extract or define the clusters, and the first bit of this is we create something called a dendrogram,  which just shows us our history of how we've merged things together.

And so you remember f and e were the two most similar points, and we merged them together first. And the way that we communicate, they're merged together first, is the line that connects them is drawn at the bottom of the dendrogram. A and B and X, so their lines are a little bit higher. D then got merged in with e and f, so its lines a little bit higher. C got merged in, so its lines a little bit higher. And finally, if you merge everything together, this line's quite a lot way higher because merging this cluster with this cluster is really merging some quite different or far apart points.

Having created the dendrogram, you then choose the number of clusters. Now you have to make a decision, like, do I want two clusters, four clusters, or six clusters? There is nothing in the dendrogram that tells us the correct decision. Now some people look at this and they go Oh, look, you can see the gap here is really big, which means two clusters. No, it doesn't mean that at all.

You're kind of guaranteed by the math of how these things work that the distances just get bigger and bigger the higher up you get. If you choose two clusters, you just draw a horizontal line through them, and that tells you that everything that's connected below that horizontal line will be in one cluster. So I've got one cluster of A and B, and another cluster of C through F. If we drew the line down here, then that would say that we'd have three clusters, A and B, a cluster just of C, and a third cluster of D, E, and F. So that is hierarchical cluster analysis.

Our prehistoric forebears used to drill holes in their heads.

Maybe they were releasing demons.

Perhaps they were curing brain cancer. We just don't know. But what we do know is that people survived that trepanation, but we don't do it at home anymore.

Now this is how I feel about hierarchical cluster analysis. A lot of people are like, cool, I understand it, now I want to use it, but it's actually just not a very useful technique. It gets you the wrong answer more than the right answer.

And I tell people this and explain it, and they go, yeah, sure, I like it though. Show me how to do it. So this next slide I'm just trying to affect those people who like it and I will show an application of it later, don't worry.

So excuse the candour of the title here why hierarchical cluster analysis and dendrogram suck'. So here is the dendrogram we created, we built this up from scratch, it showed us different similarities between different things. Now the question I want you to ask yourself is, is c more similar to b or d?

Now the way I've explained to read the dendrogram, the answer is very clear. C is clearly more similar to d because they're joined together at a much lower level. That's what the dendrogram shows us. A and b, and thus C verb B, could go up a long way, so they're clearly very dissimilar in the context of the dentrogram.

Obviously I'm showing you this because it's not true. When we look at the raw data that we started with, it tells us a completely different story. It tells us that in fact B is closer to C and the reason that we conclude the wrong thing from the dentagram is just because of the way it hierarchically clusters. So P and F got merged and then D got merged because it was similar to F and C got merged because it was similar to E and it was never really true that C and D were that similar. So this step by step merging of clusters causes this incorrect interpretation of the result.

Now using a bit of jargon, you can only accurately have a dendrogram or dendrogram will only accurately describe your data if you have something called ultra metric data, which is a fancy way of saying if there are clear hierarchies and when you merge them you don't get this distortion that I just showed you. In truth it never ever happens. I've never seen it.

So that's why people invented better methods and the the main method that replaced hierarchical cluster analysis, amongst the more numerous people anyway, was something called k means. It's almost fifty years old and it still works.

I use it from time to time.

I never use hierarchical cluster analysis. Anyway, k means.

Another very simple example, again with two variables, and we're doing this just so that we can plot it and so that you can visualize it. And if you look at this, it's clearly showing you that if you're going to say two clusters, you can see that this would be cluster one and this would be cluster two. And now we're going to use the algorithm to show you that it can identify those two clusters, but I would emphasize, in reality, you don't use k means for simple things that you can visualize. If you can visualize it, you just visualize it and use your brain.

K means is useful when you've got say three or one hundred variables and you can't see the pattern so you need the algorithm to do it. You know, as k means do it. Well as with the hierarchical cluster analysis, we start by specifying the number of clusters and so in this example I'm going to choose two.

Then we allocate the observations to those clusters. Now I favor an approach which is randomly doing it. Some people like to use hierarchical clustering.

There are other rules out there. SPSS has  a quite clever little algorithm. The thing that we do know is that if you randomly allocate and then you repeat that process hundreds of thousands of times, it tends to be the best way. And so that's what our software does.

It randomly allocates. It does it lots of times and chooses the best solution. Now yeah. So we randomly allocate it.

Now the nature when you randomly do stuff is you inadvertently create patterns. And you can see here, the pattern we have is that the darker, which is my first cluster, tends to, just by chance alone, be more common on the left-hand side and the lighter, therefore, on the right-hand side. The next step is we calculate the average of the points in each cluster. And so this darker cross is showing us the average of all of the people, or of the observations I should say, which are darker, and because more of them are on the left, that average point is closer to the left.

Similarly the light across is closer to the right.

Then we reassign the observations, moving them to the cluster main that is is closest to them and now you can see we've exactly replicated the true structure in the data, what our eyes tell us the structure is, and we've grouped together all of the darker observations into the first cluster and the lighter into the second. We then update the means and we would if they hadn't finished, we'd keep doing this multiple times until the means no longer changed, which to use a bit of jargon is we would take the approach of waiting till the analysis has converged. Now for those of you using SPSS out there, I would make sure I'd remind you not mind you. I'd tell you you want to always go into the options and increase the number of iterations because it's set at a very low level and often hasn't converged, but easy to fix.

The next technique that we're going to look at is latent class analysis.

It's really very similar to how k means works, but I say similar it's not able to be shown in a diagram, it's really only able to be shown with math, but it's actually better than k means and it's really an improvement, a straight improvement on k means. It's an improvement in that it's better for working with missing data. It works really well with non numeric data such as ordinal, which I can see Ben's asking about and come to shortly. It works with weights. It can be proven to be correct. It's got some pretty beautiful maths but it's much slower.

The newest generation of clustering models are the large language models or generative AI and tools like chat gpt.

If you've got purely text data these are now state of the art. They're much better than the historic methods that people used for category for for clustering with text data. I'll talk a little bit about how you can cluster use traditional methods with text data later, but if you've got purely text data you should use tools designed for it and we certainly have them in our app or particularly the Despyr app. The QAps are a little bit crude at this, where we use AI in the text categorization. I'll show that later.

I'm pretty confident in the long term, AI-driven models are going to replace these traditional models for many problems, not all. I don't think they'll ever replace them for conjure and not make stiff, but they'll replace them for many problems.

But at the moment, if you use a chat g p t or or Claude or something and you throw in a mixture of text and numeric data, it actually does a pretty bad job. And in the background, it just uses these old fashioned algorithms like hierarchical or k means doing some pre preparation of the data steps, which I'll show you.

There are lots of other clustering approaches.

They're really just variants of the others above and pretty trivial.

So what's best? Well, the best general purpose approach is certainly latent class analysis unless we've got text data and then we want to use the large language models or generative AI.

Why we cluster

Okay, so we've now gone through what is cluster analysis and then a little tour of the main algorithms.

Well, why do we cluster? There's three broad use cases and one of them is super important.

The first use case is using dendrograms to show data in a hierarchical structure.

The second is to find natural clusters in the data.

Now most of academic research in recent years has focused on this problem, trying to find clusters of any shape. And so you'd say with this data here there are three clusters. We've got this very long, weird, elongated cluster in the middle.

In market research, however, we tend to have a slightly different goal. We're trying to find diverse groups. We would like them to be natural. We don't need them to be natural. And so consider this long cluster in the middle. In market research, we would usually want to split it up.

We'll talk a little bit more about that later.

Okay. So I'll start with an example of hierarchical cluster analysis. When I started in market research, what I'm about to show you was a really common form of analysis.

You'd find some data. In this case, I've got data showing attitude to a series of color brands, and you try and understand the market structure by using a technique like hierarchical cluster analysis. Now if you don't know how to do something in Displayr, there's a similar search box and queue, just type, so I've got cluster, and it will show me through the menus, cluster, hierarchical cluster analysis.

Now if you look at this, it actually seems to have done a pretty good job. Coke and Pepsi have been grouped together. That makes sense. They're the nonsugar brands.

Diet Coke and Coke Zero have been grouped together. That makes sense. They're Coca Cola companies are diet brands. Pepsi Light, Pepsi Max grouped together, and these two clusters of the diet brands have been grouped together into a big diet brand cluster. So on face value, seems to have done a good job.

But as explained before, these dendrograms are usually wrong. Very simple example, they can look good, it's easy to create a good story.

I don't see these used anymore very often and that's good news. You're much better off using more sophisticated techniques such as principal components analysis for data like this, factor analysis, or alternatively correspondence analysis. And we've got webinars on both of those. They're always better than doing this approach even though it looks cool to do it this way.

The second use case is finding natural clusters. Now I think this is a hilarious hilarious wrong word, a fun topic, and I've devoted a lot of my life to these kind of algorithms. They're really cool, but they're complicated. What makes them complicated is to use approaches to finding natural clusters, you have to upfront identify what shapes of clusters you think are acceptable and probable to exist in the data and you also have to have some view about how likely they are to occur. Now here you can see we've got lots of weird and wacky shapes. So here we've got little horseshoe shapes being permitted.

Here we've got little circles within circles.

Now when you spend a lot of time thinking about it in market research, you tend to work out that actually there's only one type of shape we probably care about which is spherical data. We want groups of people who are broadly the same as each other but different to the other groups.

So that's the only shape we care about market research and the good news is the algorithms for finding such shapes, the good algorithms, are called k means and latent class analysis and all of the more exotic algorithms for clustering out there, they've really been developed for these more exotic shapes which tend to be be pretty relevant in market research.

Now we expect spherical clusters, no we don't, I don't, many of you won't, but most people new to cluster analysis and new to segmentation, which is the main application of cluster analysis in market research, they expect spherical shapes, but expectations never match reality. The reality is in all of our markets there are a continuum.

I have never seen a single example where analysis has identified clear natural groups.

And I should say natural groups are interesting. So I've certainly seen analyses that have recovered one group called men and another group called women. I've seen lots of analyses that have recovered people that buy a brand versus people that don't buy a brand, but none that recovered truly distinct groups of people that weren't known before the analysis was done.

Now the implication of this is really quite important because if there are no natural clusters we shouldn't try and look for them, But you can still form clusters.

So with this big sphere or continuum of data, these are four groups, and they are quite different to each other. Yes. The people on the boundary, it's ambiguous where they go. But treating people as these four groups is more useful than pretending they're all the same. And a good example for this is thinking of gender versus height. Gender, natural clusters.

Height, there's just a continuum of people. There's no line or gap between the short people and the tall people. We just draw a line and say, depending on the context, this person's tall, this person's short. That's the same approach to what we need to do in any situation where we don't have natural clusters. And as I'll explain, that's pretty much all situations. Now, the interesting consequence of that is it becomes possible to draw the lines in lots of places. We could have four clusters and notice I've called them segments to differentiate this idea of the natural grouping of the cluster perhaps from the segment, the more arbitrary thing or structure that we impose on the data.

And you could have four, you could have six, you could have five. There is no correct answer anymore. This is important because a lot of people new to cluster analysis, they're looking for a mathematical formula. They want to not have to do anything but find the correct formula and have the software spit out going, the correct answer is five.

And I get that. I love to have software that can do that, and I've written software that can do that if there are these natural clusters. But I said, aren't those natural clusters? It just gets you the wrong answer hundred percent of the time. Fortunately, there's a way of solving this problem. It's just not to rely on math.

The main analogy that I find helpful to think about when I'm doing clustering or segmentation in general is there is many different types of people as there are colors in pretty much any market.

Yet it's still very helpful to differentiate the different types of colors. So depending on the context, if I'm a graphic designer, I might know the name of a thousand colors in my head.

If I'm a preschooler, it's sufficient to go green, red, or blue.

We need to use our judgment to work out the appropriate granularity, number of clusters, or segments in our data.

Data Preparation

So we'll now move on to how we prepare data for cluster analysis.

There's a standard set of things to do. We want to set don't knows and not answered categories to be missing. We wanna convert ordinal data to numeric. And Ben earlier posted a question going, how do you treat ordinal?

And I made the point before that there are algorithms developed for ordinal data, specifically for ordinal data, and I've spent a lot of time developing one of them. If you use Despero queue and you change your ratings to something called a ranking question type or structure. It will give an ordinal algorithm in the background when you use the late class analysis, but all the work I can do, had to have done, suggests that you're actually better off treating it as numeric. You convert text data to numeric, you merge small categories, you can scale, and I'll come back and have an example of that.

And if you're not using latent class analysis, it tends to be an extra set of things you might have to do depending on how messy your data is, and we have a workshop which will talk about that a little bit later.

Now let's do another workshop. I have an example, which I'll talk about that a bit later. Okay.

So as I said, the first thing you need to do is set values as missing, if they're like, I don't know and convert ratings to be numeric. So here I have three ordinal variables in the dataset.

And if I click on them, you can see the structure here is shown as ordinal. In queue, these would be showing as a pick one multi question, so slightly different different. This is ordinal. I just need to change it to numeric.

Oh, so now it's showing the numbers. The second thing I need to do is I need to make sure that the don't know isn't included in the analysis, so I'm going to go exclude from analysis. Cool.

The next step we need to do is we need to convert text to numeric data, if we have text. Now I'll stress, if I've only got text, we shouldn't be doing this. We should just be using more modern text categorization tools. But if we've got a blend of text and non text data and we want to chuck it all into the k means or any other algorithm like a latent class, we need to convert that text into numbers.

Now I've got a predone one of these, which is turned at the background. I'll show you the workflow for this. I'm going to do this a little bit quickly because it's kinda technical, and most people don't need to do it most of the time. I'll go to advanced analysis, text analysis, and choose something called principal component analysis of text.

And I hook up the data to look at what people liked about their phone company. This takes a bit of a while to do, which is why I've pre done the analysis here.

Okay. Now we have webinars on principal components analysis, and there's a more detailed webinar text analysis. I'm going to go through this one a little bit quickly, but what we've done, we've converted the text data into two numeric variables, and we interpret them or we need to name them just like when we would name a component principal component analysis. Now the technical name for what we're doing here is something called an embedding in the world of AI, but the principal component analysis is kind of more market research language. Now the first component are people that liked nothing. So the main thing they have in common is there's a big correlation between whether they said nothing and this first component. So I'm going to name my first component, and first, I have to actually save these components out as variables.

Alright. So I'm going to name my first component likes nothing.

My second component, just scroll down to see this, these are people that liked the service. It's a very strong correlation for service.

Okay. So I've named my components. We'll come back and use them in a second.

We then move on to the next step. So we're now into the practical step, and this very closely overlaps our webinar on segmentation. The webinar on segmentation has a lot more detail about these steps if you want to do it, but we've also got a great ebook on it.

Create four clusters

Now we create four clusters is where we start.

You're thinking, why four? Well, it tends to be by far the most common number that market researchers end up with, but it's not just us philosophers and scientists have been doing this forever. There are four elements, four humors, four seasons, four temperaments. Four is just a really good place when you're trying to simplify something, to break it up and go, well, rather than everybody being the same, they're just these four types of people.

So how do we create these four clusters?

So for most problems, the best approach is to use latent class analysis. If you've got really neat data, K means does a fine job as well, and text categorization if you've got text data. There's these other things you can do, they're much more involved. When I say more involved, you can do them badly and then they're quick, but to do them so you get sensible answers, very very difficult.

So we're going to kick off and we're going to look at latent class analysis. So again, it's sitting here in the cluster menu.

Now we're going to use the data that contains the scores from the text PCA, which is where we turn the text into numbers, the satisfaction data, and I've got another question from or question set of variables coming from a conjoint study where I've worked out the importance of various things to you people.

Now all of our software has automatic tools in it to group the number of clusters, to find the correct number of clusters, but they all assume that there is natural clusters in the data or are natural clusters, which is a cluster is never really true. So if you want to, you can use the Bayesian information criteria. In q, you've got additional options to customize to choose other information criteria, but they don't really work. Not that we've programmed them badly just because you need to use judgment instead. So we're going to set this to four.

Now I will confess this output is a little bit hard going. I thought it was really clever when we designed it, but I don't anymore.

So we're going to return to this output, make it slightly easier to digest output shortly.

Evaluate the quality of the clusters

So we've now moved on to the stage of evaluating the quality of our clusters, and we're starting with these four clusters that we created.

We've got a special output of reporting the tool which we call, my colleague Andrew Kelly's response, this is a segment comparison table. Well sorry, for my colleague Andrew Kelly said we need it, my colleague Carmen programmed it. Now the way this works is when the latent class analysis, it automatically added this variable called latent class. I'm going to drag it up to this box, and then we're going to look at it initially by profiling variables.

We're not profiling. We're initially going to look at it by the text data, the satisfaction data, and the importance data, and we could also choose other things. I'll choose some demographics to check across as well.

Okay. Now I have here the criteria, and I'm first going to talk about them and then we'll illustrate some of them. So the first criteria when you're evaluating the quality is, do you have three to eight clusters when none are huge or tiny? Actually, we can quickly see here.

Smallest, nine. Large is forty nine. That's not great. Half the people on cluster is not ideal.

So I'm going to score us here as there may be a problem. I'm not sure.

Responses don't seem easy to don't seem to have a response bias. We'll return to that a bit later. Clusters are easy to name.

Clusters are inspirational. Let's start focusing on that. So let's look at cluster or segment number one here.

I might just increase the font size a little bit here because I imagine that's a bit hard to read.

Okay.

So cluster one, they have now the way we read this, sorry, is this the width of the bars show us how divergent the cluster is from the average. The color shows us if it's above or below. We can also see that from when they got positive or negative signs usually.

So we can see that cluster one have got a very high score for liking nothing. K? So that was the first thing that we we did, and they've got a low score for liking service. So they seem a bit unhappy with the phone companies.

We can see that cluster one have also got much lower average satisfaction with network coverage, internet speed, and value for money. So cluster one are clearly pretty unhappy people. So we could if we found similarly easy things for the other things that would tell us the clusters are pretty easy to name.

But notice something really important here, we know they're unhappy but the next set of variables are showing us the differences between these clusters in terms of their average importance for various attributes. And what we can see here is none of them are very different from the average. Now for a segmentation to be useful we need to know what we could do with that segment. We know these guys aren't happy but we don't know what we could do to make them happy. So I would say this is what I call a non inspirational segmentation.

You there's a problem. You can't find out what to do based on this.

You can also look at relationship to other data. And if you look here, you can see that this segment one, they're no different in gender, nothing significant there. There's no relationship to state. Well, there's some tiny little ones here, but there's some very small sample size, so I probably should have done state.

So I'm going to say relationship to other data. No. Not good ones. Sometimes, in some applications, we wanna build a predictive model, such as if we need to use the clusters to implement into database or some kind of model.

Anyway, so cluster's easy to name. Well, kind of, but this is a bad cluster. So we need to come up with a better one. There are various ways we could do it, which we'll talk about shortly, but the main thing I would probably do now is let's say, okay.

I care most about the importance data. Let's remove the satisfaction data and see if a new story comes about. So the basic way we wanna work is to be super iterative when we do these things rather than just assuming that we know the correct data to begin with. I'll talk more about that later.

Case study 2 — k-means and response bias

I'm going to bring in a data set and this data set has been provided by the kind people at Cranebook and it's very relevant data to us all. It shows the main things that clients are looking for when they hire market research consultancies.

Now you can see this is showing categories, so it's ordinal data here. So we're going to need to change into numeric data which we talked about doing before.

So we'll set exit still as nominal.

We'll check the values.

The missing data already excluded, that's good.

Now this is very neat data, it's all from a single set of scales, so I can use a technique like k means if I want to, and I am indeed going to use k means because despite the fact that late class is generally better, I still quite like k means, and it's got a better output. So let's do it.

Oops. Now we're defaulting to two clusters here.

We could, if we wanted an algorithm, use the Klinsky Haribas and choose the cluster with the lowest solution. And if we had spherical clusters, it would do an okay job, but we're not going to have spherical clusters because they never exist. So, again, we're going to have four clusters.

It's iterating along.

Cool. So now I've got my four clusters.

Now as before, we're going to go through and create a segment comparison table.

Now I need to with the k means, I've actually gotta go back and explicitly save that segmentation.

So I click on the output and click save cluster membership, and it adds it in here.

And now I'm going to hook it up to the satisfaction not satisfaction, the importance data that we used when we formed this, which I briefly can't find. Focused team, here we are.

And we could also drag across some other data if we wanted, which I'm not going to at this stage. I'll show you how to do that before. Okay. So three to eight clusters were not a huge or tiny. So you can see here actually, let's zoom in a little bit more.

So we can see here that we do have three clusters four clusters and they're all almost the same size, so cool. This is good.

Is there a response bias? This is a question for you audience. Can anybody spot a response bias? And hint, there's obviously a response bias because I'm asking it, but what is it? So please type into the questions field in what is now this interactive webinar and see what you can see. Can anyone spot the response bias in this data?

Now, what is the response bias? A response bias is where we believe the patterns in the data reflect something other than what we're trying to measure. Now, Now what we've tried to measure with importance is what's important to people. So how could their ratings differ if it didn't truly reflect what was important to them, but it also reflected something else? That would be response bias. Can we see it in the data?

Come on, guys. Tap into the chat field. Don't leave me hanging. I will drink some tea, which I've heard is referred to as dirty water in bits of America, but it's a wonderful beverage.

I'm basing that solidly on titlers, so sorry if that's a cultural stereotype of some kind.

And, Mark, I don't know how you managed to put a smiley face onto my screen. That's an impressive use of technology. Nice work. You're making me feel very old here.

Cool. Okay. Cluster four has a lot less responses, says Nicola. And that's right. Now let let me kinda rephrase that a little bit.

The average response the average importance score in cluster four is almost always lower than the other segments, often a lot lower. Cluster three, they tend to give a very high average. Now there are two explanations for that.

Explanation one is that the people in cluster four are just dead people. They don't care much. The people in cluster three, they're people who care a lot. They're really involved perhaps. That's possible.

But the more likely explanation is it's a response bias. There are some people who just they'd like to click write ten or whatever whenever they put on the importance. Or or in the case of conjoint, they just have very definitive answers.

And there are people who are more ambivalent. Maybe that's the difference between them. So I think we have a difference here or a clear difference likely to be on response bias. There's a standard playbook for solving this, and it goes like this. You take the actual input data and you want to scale or standardize that data.

And so the way we way we do this is and let me go a bit slower because it's actually a new change in the user interface and Displayr. You select your data and you click this plus to create new variables, and you say what you want them to do, and I want them to be standardized. And we choose this option called standardized within case. And what that does is it takes each person's actual data and it calculates each person's average and it subtracts that average from each person's data, so the average becomes zero. And this means no longer can we have differences between two people just because one person gave consistently higher ratings, but we still retain the relativity of the data.

The second thing we do is we divide each person's responses after the mean subtraction by their standard deviation, which means that each person has a similar level of variation, which is another type of response bias. And within case just means we do it for each response or each case. So I've created some new variables. I then go back to my original table, my original analysis, and I remove the original input data and I swap in the standardized data and it runs again. And so remember we're looking to see that there's a smattering of red and blue across all of the different clusters. Now the more red and the blue the stronger the relationship but we don't want to have a situation where one class has got a whole lot of blue and another a whole lot of red.

Excitement.

And the great news is we've achieved it. So we've got a really good distribution of blue and red across all of the clusters. So we've removed the response bias. Now I when I do cluster analysis or when I used to do cluster analysis because I'm retired as a researcher, I pretty much never show the standardized scores to the client because it's just another thing they have to get across, another barrier to them of understanding things.

I still would always go and show the segment comparison table using the original data. Now some of you might be thinking, well, but you know it's got a response bias. We do, but this is what you've got to keep in mind. We're always analyzing the original data.

Right? Now the original data contains multiple patterns. We've just removed the response bias from the segmentation, and that just means we're not focusing on the response bias in the segmentation. But continuing to use the original five point scales is just so much easier for the client to understand because it's more obvious, right, than having to explain the standardization.

But the other advantage is you could get some weird effect, but it only appears in the scaled data and it doesn't appear or standardized data and doesn't appear in the original data, it'd be pretty dangerous to form a strategy based on that. So here, we've got very clear differences between the segments. So we've got rid of the response bias, and I said all good before I should've said this problem, but it's all good now. And we could go through that process I showed you and work out if it's a good segmentation or set of clusters if we wanted to.

Case study 3 — text categorization as clustering

The third example we'll look at is text categorization. So we have in this dataset, the first dataset, this data on what people like about their phone company.

And the best way to analyze that today is go in and go text categorization.

Now in the jargon of market research or old person market research, we'd say you can have a single response or multiple response. That is, a person can be allocated to one theme or multiple themes. In the language of cluster analysis, multiple themes is called overlapping clusters, and it's usually the best approach for text data because usually people say multiple things and they can't overlap.

We can create our themes. Now we could manually type them in, but because we're trying to find patterns in the data, we're going to let the algorithm do the work. Now we can edit the prompt, and I'm going to tell it because this is a generic text characterization prompt, I'm going to say, please cluster the data into different groups.

The data shows what they liked about the phone company. Now for those of you who don't work a lot with AI, the more context you give them the prompt, the better the solution, which is why I'm telling it the name. Now note here I'm not telling it how many clusters to create. It can create its own number, but also known that text categorization algorithms are really designed to show all of the variation between people, not all but a a a a good degree of variation between people.

But in most applications of cluster analysis, we're probably only interested in four clusters or something. So I'm going to click create, and then we'll probably end up with too many clusters. I can't guarantee it. It's a it always has its own little bit of magic innovation that it's going to do.

Thinking particularly hard today. Alright. So So it's come back with fourteen clusters right now. I could have changed the prompt until the four, and in fact, I should have if I was doing a real segmentation or clustering type approach. Anyway, I click classify now, and it just works out which responses in which of the clusters.

I hope you find this little circle as entrancing as I do. Otherwise, this has become a quite painful presentation.

Come on. Come on, AI.

So we're very shortly moving from, OpenAI to Google for doing our back end AI. Hopefully, it'll be a little bit faster.

I might wanna add, you know, that thing they say that you should never work with dogs or children when performing, maybe AI. I'm I'm just going to click cancel because I know it's a bit boring watching this. It'll get back to an answer with this shortly, but you get the idea. And we've got lots of webinars on text categorization. I could do another one soon as well.

Choose the best cluster

The last thing that we do, the last step is you create lots of clusterings and choose the best one. So you change which variables are used. If the four cluster solution is not great, look at three, look at five. Play around with the algorithms.

Even hierarchical cluster analysis can do a good job by chance a bit, but it can do a good job. If you're going to choose one, you wouldn't do it, but if you've got the time to explore lots of things, I would even do that. Scaling, I showed you an example of that. Merge categories for categorical variables using tools like PCA or principal or multiple correspondence analysis of input data is another way of preparing the data, which can give you a different solution and different insights.

We have a market segmentation ebook, which focuses very much on the practical side and goes into the interpretation and creation steps in a lot more detail. We will send you a recording of this webinar.

Live audience Q&A

What questions have you got? Please type your questions into the questions field in GoToWebinar, and I've got a couple already.

Mark says, thanks, Jim. Can you suggest ways to make sense of the output from K Means? We're sort of choosing the right number of clusters, but have a tough time using spider visualize, and it'll make sense of mapping things back to attitudinal responses.

Well, I'll look at the k means output itself in a second, but I probably need a little bit more information from you, Mark. And maybe you should, if I don't give you enough information here and you don't provide me more information, maybe just follow-up with an example with me and send me an email to my email address here, because that should be solvable.

Let's have a look at the k means example from before.

So the way the k means outputs work, it's not quite as sophisticated as the segment comparison table, which is why we built the segment comparison table. But there are two sets of outputs here. We're showing the average responses of the data with, again, a color coding. In this case, just the darkness and the blue color tells us which are high or low, and apologies to the colorblind people out there.

So we can see here segment one. It's got highest score in length of experience time in business and the lowest score in cluster one, which is much lower. Now this data was standardized to have a scale, a mean of zero, an average, and an standard deviation of one for each response. So we can see here that the actual variation is not that big between the clusters.

If we go down to lowest price, we've got one segment which has got a really low score for low price. They didn't think low price was very important. It's well outside of one standard deviation, and we've got another two who are kind of moderately high. Now how did I work out quickly the lowest price had the biggest amount of variation?

I did it because we have this column for the k means showing the r squared, and this is telling us how much of the variation in a particular input variable is explained by the clusters.

And so we can see here that these clusters are primarily driven by price. It's very strong and everything the other things here, there are relationships, but they're things that are probably correlated with price. Now if price was very unimportant to me, I could remove price and rerun that analysis. Or getting a little bit more technical, I could scale the price variable by, say, divide if I say took the input variable price and divided it by two, that would reduce its variation, and then it will become less influential in the cluster analysis.

Hope I answered that question. Gosh. A whole lot of questions appeared now. Will a recording be sent out? Yes. It will.

Emily asks, thank you. Is it possible to plot k means cluster analysis? Yeah. You can. It depends what kind of plot you're trying to do. The lots the one type of plot, which I haven't done for a hundred years or so. Slight exaggeration here.

Let's see if hope I can remember how to do it. There's something called a principal component analysis by plot, and we'll go into dimension reduction, principal components analysis by plot. Here we go.

Wrong one. Okay.

What oh, no. That will work. Let me undo.

Okay. So what I'm going to do is I'm going to first create a crosstab where I take the input data, I will do the standardized ones, and then I'll drag across the segments plus the columns.

So now I've got this input table, and let me just and then I visualize it by going into dimension reduction, crossing my fingers. Remember, I haven't done this for a long time. Dimension reduction scatterplot, actually.

Crossing fingers.

No.

Not as clever as I thought.

Alright. Let me click undo there and go back to what I did before. I go principal, components, analysis, biplot.

I'm sorry about this people.

Okay. Now because we've got long labels, we're going to get one really messy chart here.

And so the way we read this is the things in this direction here approach to cluster one.

So cluster three, things in this direction relate to clusters four and two. There's some technical notes on how to read principal component analysis by plots that you can find on the help pages. I don't use them because I find them impossible to explain, so I'm not going to try and sell it to you. So that'd be one kind of analysis thing we could do. Another kind of visualization we could do, which would be much more basic, is do something like a bar crosstab.

And then there are obviously many other types of visualizations that are applicable when you remember that it's just a table. Right? The the output is always just a table.

So any type of visualization could be done with this. I hope I've made sense with that one, Emily, Esther, what techniques can we use to validate the clusters after creating lots of clusterings? Okay. The this is a problem I've spent a lot of time thinking about.

I actually have a PhD on market segmentation. If you can believe something so niche you can have a PhD. This is, I believe, the appropriate set of tools to validate clusters. This thing which the best in all of these things are the most useful clusters.

If you mean, is there a magic number that can be used to choose or validate, such as replicability statistics are one thing people use, things like Kalinski Haribas, things like that.

No.

All of those techniques assume that there are natural clusters and there are never natural clusters and so it's, it would be great if these magic numbers existed but they're just always wrong.

How can clustering results be visualized to communicate results? Asks Esther. I've shown you some visualizations. I mean, to me, that's a a more generic question of how do you visualize any table of data.

I don't think it's specific to k means. I mean, there's a whole lot of processes you go through when you try and communicate them. I don't think the visualization is the place to start. I think the place you have to start when you're doing k means or any form of segmentation or clustering is it's these things here.

Are they easy to name? If they're easy to name, people it's just the name gets people there and you create it. And, you know, you can use a whole lot of things you can do now. Right?

You can feed the data into a chat product and say, this person's segment four. Ask somebody in their segment what the questions are, but questions about who they are. But you can only do that if you've got an easy name. So the naming is the first thing.

Having clean strategies is the second thing. I don't think starting with the visualization is the right place. I I fear the danger you get into when you try and start with visualization is you've actually got a bad solution to begin with. So I would always focus on these two things.

Kobi says, can you also standardize your variables when using land plus? You can for exactly the same reason that I did. Now if we go back to the k means example, when I started the k means, I said because they're all on the same scale, you know, one through five, it's pretty clean data. Late class analysis will tend to perform better when you've got data on different scales automatically, whereas if we use k means, we'd have to first standardize everything to be on the same scale if it wasn't already.

Esther's back again. How do companies usually implement the findings in their marketing effort? Well the way they implement them is if there are clear strategic implications, since they're inspirational, they tend to do what those things imply.

So if we look at cluster, let's have a quick look at this example here, this is very much on the fly. Cluster four, this is a class. So let's say you're a market research company and you've got this solution.

Cluster four, they're looking for people with length of experience in business, breadth of experience in target segments, good reputation in the industry, provides highest data quality, provides data analysis services familiar with the client needs. So if I'm a market research company and I'm trying to go after cluster number four, I clearly need to hire a lot of extremely experienced consultants and demonstrate that we have a very good offering of spread of offering of services. So that would be the example.

It the way you again it all comes back to make sure that your segments have good implications.

Where things go bad is when a researcher clicks the cluster button, doesn't think about the strategy, seeing that as the client's job and inadvertently give them a solution that isn't useful.

Eric says I've done k means clustering a number of times for segmentation in Displayr. I can't think of any instance where I end up with segments who are very large or very small. Is that a coincidence?

That's such an excellent question.

It's a you've actually stumbled upon or through great genius, worked out something kind of cool.

It's nothing to do with us, by the way.

I was talking about the difference between latent class and k means a whole lot early in the presentation. Let's go and find that.

K means technically assumes that all clusters are the same size. Now when I talked about the different shaped models, I talked about the need to specify how likely different shapes are. Now k means starts with an implicit assumption that all clusters are roughly the same size. It's not guaranteed to find that. If the data is radically different from that, it won't being equal, it will focus on finding clusters of exactly the same size. So excellent intuition, Eric. There's a wonderful paper in nineteen sixty seven by somebody called Wolf with two f's on that one.

Gosh. So many questions.

Kirby says anything special I need to do to run Latin class on data I collected through maxdiff or I simply use the utilities?

Another really good question. Look.

The best way to do latent class analysis on MAXDF or quadrant isn't to use the utilities, and the reason for that is utilities are actually estimates. They contain error within them already, and you're better off using special algorithms designed explicitly for MaxDiff. So we in our software have got a special aid class for MaxDiff and a special aid class for choice modeling or conjoint. You need to set up all of the conjoint experiments, the maxdiff experiments in our software, which is a bit of a pain, if you haven't done it before to get the best answer. But and I really need to emphasize this because it's so important.

It's the wrong lens to think of just choosing the best. You need to try lots of different things.

And in general, I tend to find that when I'm doing a a cluster analysis, I I really will you know, if it's an important project, I'll spend a couple of weeks just looking at alternative solutions, and I'll run dozens or hundreds of them. And only a couple of them will be the ones done with the proper late in class solution. And I tend to find that the final solution that I end up is rarely done with one of these best algorithms. It's usually done when I've had to manipulate the data to avoid certain things. So remember with the k means, we found the price was the dominant thing.

You could often have a situation with a conjoint model or a max div model where there's one attribute which is the dominant thing and it shows up in all of the segments, you know, the variation, but it's just irrelevant to the client. There's not much they can do about it and in that case I would tend to try and save it out as important scores and rescale it. It's a big topic there, Coby.

Nav says, could you please link to the segmentation ebook? We will send you out a link when we send you the recording of the webinar, but you can also go to Displayr.com, resources, ebooks and webinars, segmentation, how to do market segmentation, and I will just post that into the chat channel for us all.

Jonathan says, can you briefly explain how you might identify the optimal variables to cluster on especially with high dimensional data? Another great question. The optimal variables are the ones that are interesting to you. Okay?

The they're used to in the world of looking for natural clusters, there are a whole lot of algorithms which try and find out what are the variables with natural clusters and just chuck out variables that don't have natural clusters. But in the world of practical consulting, I think there's rarely a useful approach. The useful approach is to focus on the variables that are interesting to the for the project. So for example, if we're doing a study looking at new product development, we tend to have attributes right to what's important to people.

If we have a study looking at media trends, we might have a look at the type of media usage data. I mean it's all really contextual on the problem. You need to work backwards from the problem that you're trying to solve.

High dimensional data, cluster analysis and late class analysis, all these algorithms work really well with high dimensional data. You can easily run them with hundreds of thousands of variables.

If some of the variables have got a very different scale to others, those variables can tend to dominate the results and a standard protection when you've got very high dimensional data, and by very high dimensional data I mean you know more than about fifty variables, is you might run principal component analysis first or something like that. But that that adds a whole lot of complication because you can make mistakes when you do that which inadvertently make your resulting solution irrelevant. So I would always start by not using them first and then iterating to explore with them later.

Zac says I worked in a segmentation recently and we saw response bias leading to the same type of clusters you showed, one with all high scores, one with low. We didn't think of or to know the standardization method you think that you proposed. Well, I'm sorry, Zach, but now you've got it. Right? And that's the point of these webinars. So you've made my day.

Hopefully, next time, you will do it. Now better software would have alerted you to it, and one day, we'll hopefully have better software. In the k means example, it reads variance explained twenty four percent. Is the variance explained something we should be paying attention to to when evaluating the viability of the results? Is there a threshold we should be aiming for?

No. There isn't.

Let's go and have a look at this in a little more detail.

I'll tell you why it's there. It's probably the better way to approach it. So at the top, it's it's telling us the variance explained, which is twenty four percent. Now should there be a threshold?

In some problems, it's useful to think about thresholds, but the variance explained is really determined by two things, how many variables you have in and how highly correlated they are. So it could be the case that if you've got a thousand say if you've got a thousand variables and they're not correlated very much other, your variance span is going to be less than one percent and that will still be valid. You're still just finding the best summary of that data. So I I wouldn't focus on it too much.

The case where I will sometimes use it is if I'm trying to explore the number of clusters and I might see that if I've got three clusters let's do this.

So four clusters is twenty four percent. When I have four clusters, I must explain more variance or I should explain more variance of three. So it'd explain an extra five percent if I get five clusters.

So the theory is that you could do something called an elbow test, and what that means is you you calculate the variance explained for each number's clusters and you find some point where it jumps massively.

I've never seen it happen or very rarely seen it happen because, again, it's a test that implicitly assumes that we have natural clusters. And so if we had natural clusters, there'll be a point at which the change in variance shows that. But as we've virtually never got natural clusters, that doesn't happen. So note here, we had this graduation nineteen percent to twenty four percent to twenty eight percent. So we're getting at each cluster we get. We're adding explaining a little more of the variance, but it's a diminishing return. And that pretty much always happens, which is why I don't describe how to do that.

Ben says, are there accuracy metrics when we should be mindful of when doing cluster analysis? Accuracy of what? Do you mean variance explained? I don't think I don't believe so. I think you need to focus on those kind of more judgmental things that I mentioned before, sadly. Emily says, if we have multilevel data, a g, multiple countries, can you do a cluster analysis at a higher level, a g, a country? Yes, you certainly can, and it's often a good thing to do.

So if you so let's say you got a sample of ten thousand people and each of those ten thousand people sit in a hundred countries, what you could do is you could aggregate the data by country and then feed that aggregated data into the clustering.

The way you would do that in Displayr is you create the crosstab of the input variables by the country, you'd export that out as a csv file or export it to excel, sorry, and then re import that as raw data, do the clustering then and there.

Thanks for the positive feedback, Jim. Eric says since you brought up PCA, have you found a practical difference between factor analysis and PCA in real life? Does the assumption of looking at variance really make a difference?

Have I found a practical difference between factor analysis and PCA? No.

The real difference is just that people who studied psychology at a university prefer factor analysis because what they were taught. In the real world, the traditional factor analysis algorithms are virtually never used anymore. Everybody uses PCA and even in SPSS, if you choose factor analysis, it runs PCA.

How is LCA for MaxDiff different to regular LCA?' asks Tatiana.

Another great question. The regular latent class analysis performs the analysis on the input variables.

When you do a latent class analysis on maxdiff, you're actually doing two things at the same time. You're estimating your maxdiff, that is you're figuring out how experimental design and the responses lead to differences between or lead to estimates of what people want, and you're estimating the segments at the same time. So you're actually doing everything at the same time. If you do the more common thing that people tend to do is they first estimate the maxdiff utilities and then they put them straight into a k means algorithm.

So they do them as two separate steps. The problem with doing that is you have errors in each step and those errors kind of compound. So the theory is that if you do it all in one step, you'll get a better solution in in a statistical sense. But as I kind of mentioned quite a lot of times, often it's the case that you we won't there's no guarantee it's the most useful solution.

It's the useful solution that is the useful thing we need to focus on. Gosh. So many questions. Wait.

There's another one. Mark. Oh, yes. It's a compliment. Thank you, Mark. Appreciate it.

Sorry this went so long. Thank you for attending the webinar. For all our customers out there, thank you so much for being our customers. If you're not one of our customers, so hopefully you've seen something cool.

And I'll just tell you, go to our website, book a demo, do a trial. We'd love to get you started. Anyway, have a great rest of your day, everybody. Bye now.