Learn all the basics to doing your own tracking research and reporting (fast and error-free).
Watch Part 2: Advanced tracking (longitudinal) research
Learn all the basics to doing your own tracking research and reporting (fast and error-free).
Watch Part 2: Advanced tracking (longitudinal) research
If you're new to tracking analysis and reporting or want to brush up on your basic skills, watch this webinar and master the basics!
The document featured in this webinar can be viewed here.
If you are new to tracking research, also known as analyzing longitudinal data, this webinar is for you.
As always, I am presenting from within Display, as it's designed for interactive presentations. but everything I show, can be done in Q as well. So, if you are a Q user and there are bits you want me to show you in Q, just tell me and I will do it at the end.
Tracking research...
Tracking research involves asking the same set of questions at different points in time. Typically, to a different group of people at each time period.
The idea is to track what changes in he market.
In some markets continuous tackers conduct inerviews every day, week, or month. However, dip sticks, either regular or irregular, are the norm in many markets.
Challenges
Tracking research involves solving challenges that we can put into three buckets:
Data management
Tracking studies have four big data management challenges.
The first is changes in the questionnaire.
A second is clerical errors. Working on tracking studies can be boring. Doing the same thing again and again. Mistakes happen.
The third problem is how you categorize the text data. Or, to use the market research jargon, how you code it.. No two people read text the same way, so if you change who does the text categorization, you can end up with patterns in the data over time that just relate to who performed the cateorization.
And, the fourth big issue is that all the above increase the time and costs of reporting.
So, how do we solve these problems.
It's easier said but done, but the biggest secret is to fight tooth and nail to avoid making unnecessary changes to a tracking questionnaire.
The second point is to work even harder than usual to get a good data file format.
Good file formats likes SPSS and Triple S files contain lots of metadata, such as question wordings, and the order in which options were presented.
If doing a tracking study, changes to the questionnaire are easy to detect and address, if you have these better file formats.
By contrast, when you are using CSV and Excel files, you can make changes to how you collect the data, but have no way of finding these in the raw data, which leads to mistakes.
I'll talk about the next few points. But, a quick notice that more advanced topics, shown in blue, will be covered in the next webinar
Use cumulative ...
OK, let's look at the wrong way to set up your tracker.
You first interview everybody in the first wave.
Then, you copy the project in your data collection software and send the new version off to a second group of people for your second wave.
If you do that, you end up with two data files.
This has two problems.
First, you will need to merge them.
That's usually pretty easy o do.
In Displayr, for example:
In Display: Combine > Data Sets > Merge by Case
But, the bigger issue will be that when you merge them, you will inevitably lose information about how the questionnaire has changed over time.
The smart way forward is to only have the one data collection project that contains all the respondents. So, when you do analysis, you just export all the data in a single file. This is known as a cumulative data file.
Automate as much as you can
I will go into a lot more detail on automation in the advanced webinar. However, in this webinar I will just focus on the ideal end goal.
The end goal of your automation work is that your software, rather than you, do all the automation. This means fewer errors, faster turnaround, and lower costs.
Let's look at an example.
Here's a dashboard. It's for the data we've just been looking at. It's showing us the results for July 2015 to September 2017.
Burger consumption in the past month is at 83%, which rose 4% from the previous month.
Now, I'm going to replace my data file, with a new cumultative file, that contains all my old data and one more quarter.
Update data file
It' will take a moment It's got a lot of thinking to do.
Now, remember, our score was 83%, and this has dropped 78%. For
Everything else has automatically updated. The raw data, the underlying tables, calculations, and all visualizations.
Thatโs where you can get to with automation. But itโs a lot of work. We will return to it in the advanced webinar
Analysis
When we are doing analysis, our key issues are:
There are three sets of tools for solving these problems
Let's dig into theses.
Stat testing over time
We have a set of key questions. For example, here's some data on aided awareness.
We want to look at it over time.
If we have a dip stick, the data may be categorical, and we create a standard crosstab.
The goal of stat testing is to try and help us work out which results aren't just noise.
If you are a true beginner, you might look at this and think. Ah, Arnold's declined in Wave 5 from Wave 4.
But, that's not right.
By default, stat testing is not set up for trackers. Instead, by default, stat tests compare each period with all the other periods, rather than just the one before.
So, how do we rectify this?
One approach is to just delete other columns.
Delete Wave 1 through 3
Now, we can see that the Arnold's difference is no statistically significant. Rather, Bread Basket and Nuovo burger changed.
Or, we can use column comparisons.
Properties Significance > Column comparisons.
So, the previous wave, wave 4, is in column d. So, we look in the Wave 5 column for ay Ds, and, just as we saw before, we can see it only is different for Bread Basket and Nuovo burger.
If we have continuous tracking or regular time periods, we will want to do our analysis with a variable that contains the actual date of the interview.
In Displayr and Q, this will mean we have the data set up as a date variable, which will mean we will see a calendar icon like this.
Displayr's automatically grouped the time into monthly buckets. I've got too much data to see.
So, to simplify the data we will aggregate it. This is good practice. If you have too much data, simplify it.
Let's change that to Quarterly.
Interview date > Date/Time
Aggregation: Quarterly
When we are looking at tracking data, our main focus is always on the very last period, which is Jul-Sep 17.
We're still showing the column comparisons. With this many date periods they'r a pain, and we're better off showing exception tests.
We need to do two things:
First, we turn on the exception tests
Significance > Arrows and Font Colors
As before, these are comparing all the time periods. Note that we are seeing Burger Shack is up. But, this is telling is that Jul-Sep 17 is higher than the rest of the data, rather than just comparing to he previous period. So, it's the same beginner's mistake from before, and we need to tell the software to compare to the previous period.
Significance > Advanced > Compare to
And look, we have a completely different conclusions. The only significant difference is now Pret'a'pane.
Often we want to compare data to where things were at 12 months ago. Here's a couple of reasons for this
Sometimes, it's jus an interesting statistic to look at.
However, in some markets there are strong seasonal effects, and certain brands are stronger in, say, summer. In such markets, we're more interested in comparing to a year ago.
Looking at Burger Chef in the first row, since July September 16, i's gone up from 74% to 71%. Is this statistically significant?
Stat testing: comparing to 12 months ago
But what if we want to compare to a year ago?
I'm going to do this by creating a new variable
+ > Insert Read-Made New Variable(s) > Contrast Periods
Let's drag it across and look at it.
It's showing the last 2 months of the data
So, i'll tell it to focus on quarters
Date aggregation: Quarter
It's now showing the same result we had before for Pret-a-pane. But, we can now control where we compare against.
Versus: Q3 2016
OK, so we can see that quite a few brands are up compared to a year ago.
With such data, it's often nice to show a difference.
In Displayr, we can do this with a Rule.
Rule > Tabe Computations > Difference Between Pair of Columns
Column 1: Q3 2017
Column 2: Q3 2016
And, commonly people may just want to look at the difference.
In this example, we are getting quite different conclusions when comparing to 12 months ago than when just comparing to the previous time period.
So, we need to understand the trend. We will look at the stat testing side of this in the advanced webinar.
But, the key thing is to create a good way of showing all the data, which we will come to shortly.
Weighting is almost always appropriate in tracking studies
In all studies, we get over- and under-representation of different sub-groups. For example, in a tracker, we may have too many men one month, and two few the next.
Weighting can correct for this.
With a tracker, we can expect that every wave we will get such random movement.
Therefore, we pretty much always want to weight a tracker.
Here's some data over time.
Lucky's Pizza shows no significant differences.
What happens when we weight it?
Now we are seeing a difference. This is interesting. The random noise had hidden a pattern, and it's only revealed when we apply the weight.
So, remember, if you can, you should weight your trackers.
We will talk about some more advanced aspects of weighting in our next webinar on tracking.
And, check out our webinar on weighting for more about weighting in general.
Analysis+
Automated stat testing and weighting both help us to sort out the true differences from the random noise.
These should be augmented with attempts to understand causality. That is, trying to figure out what has caused a movement. If you can find other data that that corroborates the differences you see in the tracker, that's a good sign. Examples of other data to look at are:
Marketing activities, such as changes in ad campaigns and distribution.
And, the last thing to do is to decompose changes over time in terms of whether they are driven by changes within sub-groups or changes in sub-group size. There's a nice framework and process for this, which will be the centrepiece of our more advanced webinar on tracking.
Reporting
When we report our tracking, our first key challenge is making it easy for our audience to spot meaningful changes in the data .
Our second challenge is to stop them from seeing patterns in noise.
And, the biggest challenge is stopping them from being bored!
Two great tools for doing this are small multiples and smoothing. We will dig into these shortly.
Another great tool is to have rotational sections of your studies and reporting. E.g., one month maybe you focus on brand health. A second month ad effectveness, and so on.
One particular variant of rotating deep dives is to identify laws of the market, which we will focus on in the advanced webinar.
Spaghetti
OK, so every now and then I see a visualization that's just so horrible I had to take a screen shot.
To prevent embarrassment from all concerned, I've hidden the brand names. It's not a good chart.
In interactive..
This is another line chart showing tracking data. Because it's inteactive it's a bit better.
But, to find a pattern we need to work really hard. Tell me, for example, what's the story with The Crown? What pattern can you see?
Yep, nothing really.
The awesomely named mathematician, Lofti Zadah, had a saying: When the only tool you have is a hammer, everything begins to look like a nail.
And, in the world of PowerPoint and Excel, the line charts like these are the only hammer for time series data.
Fortunately, there's a much better solution, which is small multiples. But, you can't do them in Excel.
"Small multiples"
But, we're not in Microsoft anymore toto, so let's do something better.
Now, have a look. What's the story with the Crown? We can now make out lots of really clear patterns in the data.
Brand awareness spaghetti
Here's our earlier brand awareness data. It's not as much as a mess as the earlier line chart.
Brand awareness small multiples
But the small multiple are still so much better!
Smoothers
Small multiples are the number 1 data visualization tool that you need to know about for tracking.
The number 2 is to use smoothers.
Here's some data on customer churn. It moves around a lot.
We could aggregate to show it quarterly. Or, we could put a trendline on it.
Object inspector > Chart > TREND LINES > Line of best fit : Line
Clearly a straight line tells us something, but it's not doing a great job.
Many make researchers use rolling averages, also known as moving averages to show the underlying trend.
Object inspector > Chart > TREND LINES > Line of best fit : Moving Average
For example, this one is a 6 month moving average.
It's called a smoother, because the resulting trend line is a lot smoother than the original data. The idea is that the smooth data will show make it easier to see the trend.
Unfortunately, this rolling or moving average isn't doing a great job. Notice hat:
We've got no line for the first five periods.
And, notice also that the line doesn't do a great job at showing trends. For example, in the raw data our two big peaks are here and hear
But, the peaks occur in the moving average well after this. What's going on?
Well, this ones a bit sad. Moving averages are one of those things that get taught in undergraduate stats as an introductory topic, to help people understand. But, they're not meant to be used!!!
Much, much better techniques have been invented, such as
Which of these more advanced techniques should you use? Whichever one does a better job at describing the data.
How do they work?
Well, they are better, but they are really complex. So, I'm not gonna even try.
And, what's wrong with the rolling average?
Let's do a worked example.
Why rolling/moving averages are always misleading
We're going to do some very simple math. Do try and follow along, as if you are using rolling averages, I what I will show you is really interesting and surprising.
Let's look at a 6 month rolling average
We calculate the average for October 2019 as the average of the six numbers shown here.
As you hopefully remember, this is the formula for the average.
+
And, the average for November is calculated this way.
++
So, the October number is 6, and November is 6.4. That suggest that once we smooth the data, churn's gone up by 0.4
+++
OK, Now a bit of simple maths.
6.4 - 6
Is the same this line. That is, I've just substituted
that needs to be used. The number 2 is modern smoothers.
Here' I've got some data on customer churn.
Lets plot this data as columns
Insert > Visualization > Column Chart
It jumps around a lot. 99 out of 100 market researchers then go and apply a moving average, also known as a rolling average.
Chart > Trend > Moving average
Displayr's defaulting to the 3 month moving average. This is wiggling around too much.
Let's try 6 months
But, look, the moving average actually does a bad job. Look in June 2019. A huge spike. The moving average shows a small blip up.
According moving average, the data peaked in November 2019. But, look at the actual data. The high points are 2 and 5 months earlier.
What's going on?
The sad truth is that traditional moving averages are just dumb. Statisticians have known for more than 100 years that they are dumb. This is why whenever you see government data or anything done by a good statistician, they don't use moving averages.
There are lots of alternatives.
Three of the popular ones are in Displayr and Q.
Line of best fit: LOESS
One of them is called Locally Weighted Exponential scatterplot smoothing, or loess for short.
It's doing a much better job. It works out that the June 2019 peak was an aberration. And, it picks that since September things have been getting worse.
Another one, with a cooler name, is Friedman's super smoother
Line of best fit: Friedman's super smoother
Or, you can have a cubic spline.
Which is best? Just like with choosing wether to do a 3 or 6 month moving average, it comes down to the story you want to tell.
These techniques are always better than moving averages. Always is a strong word. And I mean it. If your goal is accurately summarize the data, these techniques are provably always better than a moving average.
Nothing comes for free however.
The math is hard. You need to point, click and trust. You are unlikely to ever understand the math. And, unlike a moving average, when you get new data, the historical line will update, so they're not great for setting performance target for bonuses.
Read more