✅ Your data warehouse is more valuable than ever.

Traditional BI assets are not being seen as central to a real-world AI strategy. Nothing can be further from the truth!

I’m betting that your data warehouse is probably more valuable than your organization realizes. The secret is transforming your data warehouse into a crystal ball to predict the future.

✅ Hard financial impacts vs. squishy productivity benefits.

When deploying AI tools like Copilot and ChatGPT across business functions, the expected benefits tend to fall into the category of “productivity benefits.”

Your data warehouse can be central to demonstrable financial benefits to your organization that your CEO will love.

✅ Your KPI knowledge is key.

As the source of clean enterprise data for reporting and dashboards, data warehousing teams have intimate knowledge of the key performance indicators (KPIs) that matter to executives.

This knowledge is central to prioritizing use cases that deliver hard financial benefits to your organization and capture executives’ attention.

✅ You don’t need Data Scientists or AI Engineers.

Successfully transforming your data warehouse is all about two things:

1. Knowledge of your business processes.

2. Deep understanding of your data.

BI and data teams already have both of these. All you need is to augment their existing skills, and you’ll unleash the true value of your data warehouse.

Check out the full webinar recording 👇

Read more

You learned in Part 3 of this tutorial series how the agglomerative hierarchical clustering algorithm calculates the distance between two data points (i.e., rows) in a dataset.

This tutorial will cover the various methods the AgglomerativeClustering class in scikit-learn offers for calculating distances between clusters.

As covered in Part 2 of this tutorial series, the algorithm needs both types of distances to mine the cluster hierarchy (i.e., taxonomy) from a dataset.

These methods for calculating distances between clusters are called linkages. The AgglomerativeClustering class supports four different linkages.

Subscribe now

#1 - Single Link

Understanding each linkage type is quite intuitive when you use a graphical representation. For this tutorial, assume we have the following two clusters:

The first linkage is known as single link.

Single link calculates the distances between all the points in each of the clusters and then chooses the smallest of these distances to represent how similar (i.e., close together) the clusters are.

Graphically, single link would use the following two points for the distance between the clusters because it is the minimum:

Here's how to use single link with the AgglomerativeClustering class:

from sklearn.cluster import AgglomerativeClustering

# Use single link for cluster distances
agg_clustering = AgglomerativeClustering(linkage = 'single')

NOTE - This code is the same whether you’re using Python in Excel, VS Code, or Jupyter Notebook.

#2 - Complete Link

If there’s a linkage that uses the minimum distance between two points in each cluster, it stands to reason that there would be a linkage that uses the maximum distance.

This is known as complete link.

I know.

Why couldn’t they just call them min and max? 🤷‍♂️

Here’s complete link shown graphically:

And in code:

from sklearn.cluster import AgglomerativeClustering

# Use complete link for cluster distances
agg_clustering = AgglomerativeClustering(linkage = 'complete')

#3 - Average Link

The next linkage is the average link. As you might have guessed, this linkage uses the average distance between all data points in two clusters.

To keep the graphic from being overwhelming, I’ve only included the distance lines for two data points from the orange cluster:

Imagine all the distance lines from every data point in the orange cluster to every data point in the green cluster.

The similarity of the clusters is the average of all those distance lines - a smaller average indicates more similar clusters.

Here’s the code to specify using average link:

from sklearn.cluster import AgglomerativeClustering

# Use Ward's method for cluster distances
agg_clustering = AgglomerativeClustering(linkage = 'average')

#4 - Ward Linkage

The last linkage is the AgglomerativeClustering default: the Ward linkage. The Ward linkage (or Ward’s method) is more involved than the previous linkages.

If you’re familiar with the k-means clustering algorithm (e.g., from my Cluster Analysis with Python online course), then Ward linkage will be familiar to you.

The ward linkage uses prototypes. Think of prototypes as being data points that the algorithm “makes up” to be the center of each cluster.

The most common type of prototype is the centroid, which is the average of all data points in the cluster.

Graphically, the orange and green dots are the centroids of the two clusters in our running example:

The ward linkage first evaluates each cluster by finding all the distances (i.e., the dotted lines) between the data points in a cluster and their centroid:

Consider the orange cluster above. There are eight data points in the cluster and eight distances between each data point and the centroid.

The ward linkage takes each one of these distance values and squares them (i.e., the distance multiplied by itself). It then sums all these squared distances.

It then repeats this process for the green cluster.

Lastly, it then adds the sum of the squared distances for the orange cluster to the sum of the squared distances for the green cluster.

Think of this last calculation (i.e., adding the two sums together) as being the baseline.

The ward linkage then combines the two clusters and calculates a new centroid (represented in blue):

The ward linkage then sums up all the squared distances for the new, larger cluster:

Think of the sum of the squared distances for the new, larger cluster as being the new hotness.

Ward linkage calculates the difference between the baseline and the new hotness, treating it as a penalty for combining clusters. Ward linkage looks to make this penalty as small as possible when considering which clusters to combine.

By minimizing the penalty, ward linkage prioritizes combining the most similar clusters.

Even though Ward’s method is the default, here’s the code for reference:

from sklearn.cluster import AgglomerativeClustering

# Use Ward's method for cluster distances
agg_clustering = AgglomerativeClustering(linkage = 'ward')

Which Linkage to Use?

It shouldn’t come as a surprise that using different linkages can produce different clusterings.

So, you may be asking yourself, “Which linkage should I use?”

When it comes to cluster analysis, I’m a big fan of experimenting to see what works best. For example, using different linkages and seeing which produces the most useful clusters.

That being said, here are some rules of thumb:

• Single link is good at handling non-spherical cluster shapes, but is sensitive to outliers.

• Complete link is less susceptible to outliers, but can break large clusters, and it tends to produce spherical clusters.

• Think of average link as being a compromise between the single and complete linkages.

• Think of Ward’s method as being a bit of an improvement over average link.

That’s it for this tutorial.

My next newsletter will be the last in this series. The topic is using ML predictive models to help interpret clusters.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

While it’s tempting to want to feed your data to AI and trust that the output is correct, this is a bad idea for two reasons:

1. AI tools like ChatGPT, Claude, and Copilot often make mistakes in analytics. For example, they often make incorrect assumptions and never tell you about them.

2. You’re accountable for the quality of your analytics. No executive I’ve ever worked with will buy the argument, “Don’t blame me! It’s the AI’s fault.”

That being said, once you’ve developed fundamental skills in data analysis, you can partner with AI tools to accelerate the impact you can make at work.

Introducing Your AI Partner

Here are some screenshots of me partnering with Copilot in Excel to build a sales forecast. Simply paste the prompt into Copilot in Excel to get started:

NOTE - Because of the random nature of AI tools like Copilot in Excel, your experience might not be 100% the same as what you will see in these images.

Also, the prompt is designed to work with LLM-based chat tools like ChatGPT, Claude, Copilot, etc. You’re not limited to Copilot in Excel.

It’s Not Just About Forecasting Sales

While the example images show partnering with AI to forecast sales, the prompt will work for any domain:

• Forecasting sales

• Forecasting product demand

• Forecasting customer service call volume

Best Practices Are Included

All of the best practices you learned about in my 7-part tutorial series are included with the AI prompt. For example, using line charts to assess the characteristics of a time series:

Are you ready to get started?

Let’s start with tips for using the prompt.

Read more

As you saw in the last tutorial​, allowing agglomerative hierarchical clustering to find all possible clusters is usually not a good idea except for the simplest of datasets.

The intuitive way to think about what happened in the last tutorial is to imagine your automobile (or car, as we say in the United States). Specifically, imagine your car’s engine.

For your car’s engine to work optimally, it needs to he tuned. For example, I live in a mountain valley in Montana. My car’s engine likely needs to be tuned differently than the same make/model at sea level (i.e., elevation of 4,800 feet vs. 0 feet).

Similarly, as with a car, you must tune the clustering for the situation at hand. And this tuning will likely differ across datasets.

Don’t worry if this sounds complicated, because it’s really not.

In this tutorial, you will learn a powerful technique for tuning the hierarchical clustering process to find a more optimal number of clusters.

Subscribe now

Loading the Dataset

To follow along (highly recommended), download the CustomerBehavior.xlsx file from the newsletter’s GitHub repository.

NOTE - While I’m using Python in Excel for this tutorial, 99+% of the code is the same whether you use Excel, VS Code, or Jupyter Notebook.

To keep from repeating the same content, I’m going to assume that you’re familiar with Part 4 of this tutorial series.

Here’s how I set up my Python Code worksheet with comments:

Per my usual, I add the Python in Excel code to the right of each comment. The following code loads the CustomerBehavior Excel table into Python:

Here’s the Python code for you to use in your Excel workbook:

# Load the CustomerBehavior Excel table
customer_behavior = xl("CustomerBehavior[#All]", headers = True)

A Problem of Scale

Here’s a small subset of the data used in this tutorial series:

Take a look at the data above, specifically the Income and RecencyInDays columns (i.e., features). Here are working definitions of each of these features:

• Income: Yearly household income in US dollars.

• RecencyInDays: The number of days since the customer’s last purchase.

Now, consider the magnitude of the data values in each of these features. For example, the first row of data has an Income of $58,138 and a RecencyInDays of 58.

Notice how the Income values are far larger than the RecencyInDays values? In technical terms, these two features are on different scales.

Features with different scales are very common in real-world analytics. For example, imagine you’re working in a supply chain organization, managing the shipment of products to customers.

A logistics dataset might contain DistanceShipped and ProductWeight features. If you’re shipping to consumers, it’s likely that DistanceShipped will contain values far larger than the ProductWeight feature (i.e., on different scales).

This is important because the most commonly used clustering techniques (i.e., algorithms) rely on Euclidean distance (see Part 3 for more details). And Euclidean distance works best when all the features are on the same scale.

To achieve the best tuning results (i.e., get the best clustering possible), the data needs to be transformed so all features are on the same scale.

Scaling the Dataset

As it turns out, scaling datasets is so common in real-world DIY data science scenarios that Python’s scikit-learn library provides functionality for you to do this in just a few lines of code.

For example, the StandardScaler class implements one of the most common ways of scaling (i.e., standardizing) data - the Z-score. Many professionals (maybe you) have learned about the Z-score in an introductory statistics class.

For the purposes of this tutorial, the exact mathematics behind the Z-score is not necessary. What is important is the following intuition:

• Clustering techniques work best when all of the features are on the same scale.

• If you notice features that are not on the same scale in your dataset, then you should scale them.

Now, this is super important.

Nothing requires you to scale your data. Technically, it will work just fine. However, to achieve the best results from your cluster analysis, scale the data when appropriate.

The following code demonstrates how to scale your data using the StandardScaler class from the scikit-learn library:

The magic of the code above happens with line 7. This is where the data are transformed (i.e., scaled using Z-scores). Clicking on DataFrame > gives you a preview of the scaled data:

In the preview above, you can see how the Income feature is now on the same scale as all the other features (e.g., an Income value of 0.31465 compared to a RecencyInDays value of 0.31083).

One thing that I should mention if you’re feeling a bit overwhelmed by this.

When in doubt, scale your data before performing your cluster analysis. In general, there’s no downside to scaling your numeric features other than making your computer do some extra work.

With the data scaled, it’s now time to find the optimal number of clusters.

Tuning Hierarchical Clustering

The goal of tuning a clustering algorithm is to determine the optimal number of clusters for a given dataset. Here are the critical things to keep in mind as you perform tuning:

• For any reasonably large (i.e., complex) dataset, you’re very unlikely to find the “correct” number of clusters. Every cluster analysis is wrong to a certain degree. The question is not whether the cluster analysis is “correct”, but whether the insights are useful to the business.

• Think of the optimal number of clusters found to be determined by the exact features and rows of data you use. If these change in any way, then you need to re-tune.

The second bullet deserves a bit of additional context. While it’s intuitive that two different datasets are likely to have a different number of optimal clusters, the idea applies within a single cluster analysis as well.

Consider the dataset used in the tutorial series. Let’s say that you were able to get more rows of data (i.e., more customers). It’s possible that the optimal number of clusters might change as a result of the additional rows, but you wouldn’t know that unless you re-tuned.

Whenever you add/remove features/rows from your datasets, repeat the tuning process and check whether the optimal number of clusters has changed.

By default, the AgglomerativeClustering class is set to find two clusters. In practice, this default setting (i.e., hyperparameter) is usually too low for real-world cluster analysis. However, it represents the starting point for the tuning process.

At a high level, here is the process for tuning agglomerative hierarchical clustering using the n_clusters (i.e., the hyperparameter that controls the number of clusters found):

1. Start with n_clusters = 2 and evaluate the quality of the clusters.

2. Move to n_clusters = 3 and evaluate the quality.

3. Repeat this process with higher values of n_clusters.

4. Pick the n_clusters value with the highest quality score.

The above tuning process is conceptually simple, but there’s a catch.

How do you define “cluster quality”?

In this tutorial series, I will use a cluster quality calculation known as the silhouette coefficient​. This calculation gives each data point (i.e., row in the dataset) a score ranging from -1 to 1.

To implement the process above, a couple of helper objects are a good idea:

# Create a range object to hold cluster values
n_clusters = range(2, 15)

# Empty list to store average silhouette scores
silhouette_scores = []

The n_clusters object contains the values 2, 3, 4, 5, … 14. These are the various numbers of clusters to be tested to find the optimal value.

The silhouette_scores list stores the average silhouette coefficient for each number of clusters. For example, the average silhouette coefficient score will differ between 2 and 5 clusters.

With the helper objects created, here’s the code to implement the tuning process:

from sklearn.cluster import AgglomerativeClustering
from sklearn.metrics import silhouette_score

# Perform a clustering for each value of n_clusters
for num_clusters in n_clusters:
    agg_clustering = AgglomerativeClustering(n_clusters = num_clusters)  

    # Perform the clustering
    clustering = agg_clustering.fit(behavior_scaled_X)

    # Calculate the silhouette score and store it
    silhouette_scores.append(silhouette_score(behavior_scaled_X, 
                                              clustering.labels_))

# Examine the silhouette scores
print(silhouette_scores)

The above code runs rather quickly, but know that the running time can increase dramatically if you’re evaluating many values of n_clusters and/or your dataset is large.

The silhouette score works by looking at each data point in a clustering and then comparing two things:

• The average distance from the data point to all the other data points in the same cluster.

• The average distance from the data point to all the other data points in the next nearest cluster.

This is why the silhouette_score function needs both the original dataset used for the clustering and the resulting cluster assignments (i.e., labels_).

Interpreting silhouette scores is quite simple - scores of 1 are ideal, and scores of -1 are terrible. For example, as shown in the output above, the average silhouette score for 2 clusters is 0.32147, while the average score for 14 clusters is 0.05916.

Rather than looking at the raw average silhouette scores, I prefer to visualize them:

from plotnine import ggplot, aes, theme_bw, geom_line, geom_point

# Build a plot of the silhouette scores
(ggplot(None, aes(x = n_clusters, y = silhouette_scores)) +
 theme_bw() + 
 geom_line() +
 geom_point())

Clicking the PngImageFile > gives you the visualization:

The above visualization shows that the highest average silhouette score is where n_clusters = 2. It is tempting to say that the optimal number of clusters is two and leave it at that.

However, there’s more going on.

Clustering quality metrics should always guide your analysis. For example, the above diagram shows:

• Consistently low values when n_clusters > 5.

• Potentially reasonable values of n_clusters are 2, 3, 4, and 5.

As the DIY data scientist, you should investigate reasonable values. It’s possible that a “suboptimal” value of n_clusters (e.g., 4) might produce the best business interpretation/insights.

Part 7 of this tutorial series will teach you techniques for evaluating the quality of your clusters to generate business insights and determine an optimal number of clusters.

In other words, this ambiguity in the average silhouette scores is job security for humans at the time of this writing. 🤣

👉 Ready to learn more? Check out Part 6 here.

That’s it for this tutorial.

My next newsletter will teach you the various ways agglomerative hierarchical clustering builds clusters (i.e., linkages).

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

After completing the first three tutorials, it’s time to apply what you’ve learned and perform hierarchical clustering on a dataset.

Today’s tutorial will use a dataset of customer behaviors. Specifically, grocery purchases. The dataset also includes customer characteristics (e.g., Income and Age).

To follow along, download the CustomerBehavior.xlsx file from the newsletter’s GitHub repository.

NOTE - While I’m using Python in Excel for this tutorial, 99+% of the code is the same whether you use Excel, VS Code, or Jupyter Notebook.

Subscribe now

Loading the Data

The dataset used in this tutorial series has the tabular form that you need for any analytics work you might perform:

• The rows of the dataset are the logical items to be analyzed. In this dataset, it’s grocery store customers. However, the rows can be anything - claims, users, virtual machines, patients, employees, etc.

• The columns of the dataset are the attributes of the items to be analyzed. Two examples from this dataset are the NumStorePurchases and Age columns.

The following screenshot illustrates a subset of the dataset:

When using Python in Excel, it’s best to organize your Python formulas (i.e., code) to make writing and maintaining your data analysis goodness easier.

I’m a big fan of putting all my Python formulas in a single worksheet, with the code laid out vertically, step by step. First, it’s good to add a dedicated worksheet:

Next, I add comments to my Python worksheets for my own long-term sanity:

I then place my Excel formulas immediately to the right of each comment. In this case, I would click cell C2 to hold the Python formula that loads the CustomerBehavior table from the Excel workbook into Python.

The best way to write Python formulas is to use Excel’s new Python Editor, which you can access from the Ribbon:

Clicking the Python Editor opens a new pane:

Notice how the button at the bottom of the Python Editor references cell C2? Clicking the button inserts a new Python formula in that cell where you write your code:

The code in the image above loads the CustomerBehavior table from Excel into a Python DataFrame (DataFrames are how Python represents entire data tables).

Clicking the disk icon will save (i.e., run) the Python formula:

Clicking on the > in the Python Editor gives you a preview of the DataFrame:

Here’s the Python code for you to use in your Excel workbook:

# Load the CustomerBehavior Excel table
customer_behavior = xl("CustomerBehavior[#All]", headers = True)

Inspecting the Data

Before applying any analytics technique, ensure the dataset meets its requirements.

For example, hierarchical clustering only works with complete datasets. Even if a single cell is missing data, hierarchical clustering will throw an error.

The following code gets summary information about the DataFrame, which is super useful for inspecting a dataset:

Here’s the code for your Excel workbook:

# Get summary info about the DataFrame
customer_behavior.info()

We can use this dataset with hierarchical clustering for the following two reasons:

• There are 2,205 rows of data (i.e., entries in the output above), and no data is missing (i.e., 2205 non-null in the output above). Translate null as meaning missing.

• All of the features are numeric.

In Python, the scikit-learn library is the de facto standard for machine learning, including hierarchical clustering. The scikit-learn library provides the AgglomerativeClustering class for hierarchical clustering of your datasets.

Like most clustering algorithms in scikit-learn (e.g., k-means and DBSCAN), the AgglomerativeClustering class uses Euclidean distance, which only works with numeric columns (i.e., features).

Since both requirements are met, this dataset is ready for hierarchical clustering.

Clustering the Data

With the data inspected and found fit for use with the AgglomerativeClustering class, you can perform the actual clustering with just a few lines of code:

And the code for your workbook:

from sklearn.cluster import AgglomerativeClustering

# Let the algorithm find as many clusters as it can
agg_clustering = AgglomerativeClustering(n_clusters = None, 
                                         distance_threshold = 0)

clustering = agg_clustering.fit(customer_behavior)

A couple of things I need to mention about the above code:

• For this tutorial, I’ll use the default way the AgglomerativeClustering algorithm evaluates the quality of clusters (i.e., I'll use the default linkage).

• I'll also tell the algorithm to find as many clusters as it can.

The above parameters are passed to the AgglomerativeClustering constructor, which tells the algorithm to compute the complete taxonomy (i.e., what the scikit-learn documentation calls the "full tree") from the data.

Visualizing the Clusters

Unfortunately, the scikit-learn library doesn't provide an easy way to create a dendrogram for hierarchical clusterings.

However, the scikit-learn online documentation does give the following code for a custom plotting function:

from scipy.cluster.hierarchy import dendrogram

def plot_dendrogram(model, **kwargs):
    # Create linkage matrix and then plot the dendrogram

    # create the counts of samples under each node
    counts = np.zeros(model.children_.shape[0])
    n_samples = len(model.labels_)
    for i, merge in enumerate(model.children_):
        current_count = 0
        for child_idx in merge:
            if child_idx < n_samples:
                current_count += 1  # leaf node
            else:
                current_count += counts[child_idx - n_samples]
        counts[i] = current_count

    linkage_matrix = np.column_stack(
        [model.children_, model.distances_, counts]
    ).astype(float)

    # Plot the corresponding dendrogram
    dendrogram(linkage_matrix, **kwargs)

Given that we told the AgglomerativeClustering algorithm to find as many clusters as possible (i.e., compute the entire tree), we'll need a large canvas for the dendrogram.

The following code sets this up and calls the custom plotting function:

Clicking PngImageFile > displays the dendrogram:

The above dendrogram illustrates a critical idea in real-world cluster analysis.

To get usable clusters, you must tune the algorithm. This tuning will differ for each dataset you cluster.

Part 6 of this tutorial series will teach you about the various linkage options and how they can impact the number of clusters found.

👉 Ready to learn more? Check out Part 5 here.

That’s it for this tutorial.

My next newsletter will show you how to tune the agglomerative hierarchical clustering algorithm for better results.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

Every clustering algorithm that you commonly use in DIY data science needs some way to calculate the distance (i.e., similarity) between two rows of data.

Euclidean distance is the default method for calculating distance in these algorithms.

Don’t panic! Euclidean distance is a fancy name for something you learned in elementary or middle school - the Pythagorean theorem.

This tutorial will demonstrate how this works using a simple step-by-step graphical approach.

Honestly, you don’t need to be a math genius to understand how this all works.

Subscribe now

Setting the Context

To jog your memory, here’s the hypothetical dataset from the last tutorial:

And the scatter plot of the dataset:

As discussed in the last tutorial, the agglomerative hierarchical clustering algorithm calculates distances between all the pairs of data points in the dataset.

The algorithm finds that the following two data points (i.e., rows) are closest and clusters them:

The tutorial will use these two data points to explain Euclidean distance.

Intuitively, we know the distance between these two data points is represented by the line below:

The Power of the Pythagorean Theorem

Zooming in on these two data points, we can also imagine them as being part of a triangle with each side of the triangle labeled a, b, and c:

As you might recall from school, the Pythagorean theorem gives the mathematical relationship between the lengths of the sides of the above triangle:

From the above images, we now have a way to calculate the distance between the two data points - it’s the length of side c!

Plugging the data points into the Pythagorean theorem will calculate the distance. To do this, we’ll need the data values:

The above image shows which dataset rows correspond with the data points in the scatter plot. The image also displays the data values for each row.

The next step is to think about these data values in terms of the x-axis and y-axis coordinates:

With the coordinates in place, the lengths of the sides of the triangle are just the differences in the x-axis and y-axis coordinates:

The rest is simple plug-and-chug math:

The distance between the Row 2 and Row 5 data points is 0.102.

Easy peasy!

Handling Real-World Datasets

Now, here’s the magic of the Pythagorean theorem.

The Pythagorean theorem is Euclidean distance in two dimensions (e.g., a triangle is a 2D shape). The same ideas behind the Pythagorean theorem scale to more dimensions.

For example, all the visualizations above are 2D. They have an x-axis and a y-axis. Consider a 3D visualization. This adds a z-axis.

As shown above, the calculation uses the differences between the x and y values for the data points. In 3D, this would include a difference for the z values as a third term:

The above pattern scales beyond 3D. In other words, if you have 15 columns in your dataset, there would be 15 different terms under the square root symbol.

The good news is that Python will calculate all this for you automagically.

The important part for you is the intuition of what’s going on with the distance calculation. This is where the Pythagorean theorem helps build your intuition of how clustering determines similarity.

Now you know how agglomerative hierarchical clustering works behind the scenes.

The algorithm iteratively calculates the Euclidean distances between data points and/or clusters to build the taxonomy.

As you might imagine, when your datasets have many columns and rows, your laptop has to do a lot of work performing all the calculations.

The larger your datasets, the longer the algorithm will take to run. However, the time spent increases extremely quickly as you add rows/columns (i.e., there’s a nonlinear relationship between dataset size and running time).

This means that agglomerative hierarchical clustering doesn’t scale well to large datasets. I will cover scaling clustering to very large datasets in a future live crash course.

👉 Ready to learn more? Check out Part 4 here.

That’s it for this tutorial.

My next newsletter will continue this tutorial series by teaching you the Python code to perform your own hierarchical clustering.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

While it’s tempting to want to feed your data to AI and trust that the output is correct, this is a bad idea for two reasons:

1. AI tools like ChatGPT, Claude, and Copilot often make mistakes in analytics. For example, they often make incorrect assumptions and never tell you about them.

2. You’re accountable for the quality of your analytics. No executive I’ve ever worked with will buy the argument, “Don’t blame me! It’s the AI’s fault.”

That being said, once you’ve developed fundamental skills in data analysis, you can partner with AI tools to accelerate the impact you can make at work.

Introducing Your AI Partner

Here are some screenshots of me partnering with ChatGPT to conduct a market basket analysis using a healthcare dataset:

NOTE - Because of the random nature of AI tools like ChatGPT, your experience might not be 100% the same as what you will see in these images.

Also, the prompt is designed to work with LLM-based chat tools like ChatGPT, Claude, Copilot, etc.

It’s Python in Excel-Friendly

I’m a huge fan of Python in Excel because it is the easiest, fastest way for business professionals to have more impact at work using analytics.

So, the prompt is designed to help you perform market basket analyses if you’re using Python in Excel:

NOTE - Because 99+% of Python code is the same whether you use Excel, VS Code, or Jupyter Notebooks, the prompt will help you no matter what tool you use.

Partner with AI Step-by-Step

Because I selected “beginner” mode for the prompt, I received detailed step-by-step instructions on how to perform the market basket analysis:

NOTE - While the screenshots above use a healthcare-specific dataset with ChatGPT, you can use the following prompt with ChatGPT, Claude, Copilot, etc., for any dataset from any industry.

Are you ready to get started?

Let’s start with tips for using the prompt.

Read more

You can think of a cluster as nothing more than a grouping of data points (e.g., rows of data in a table). At a high level, the goal of creating clusters is two-fold:

• Find groups of data points that are very much alike.

• Ensure the groups are as different from each other as possible.

BTW - I will use groups and clusters interchangeably in the tutorial series.

There are two strategies commonly used to perform hierarchical clustering:

• Divisive is a top-down approach.

• Agglomerative is a bottom-up approach.

In real-world DIY data science, agglomerative hierarchical clustering is overwhelmingly used. So, this will be the focus of the tutorial series.

This tutorial will use the same intuitive approach I use with my clients to help you understand how this bottom-up approach works - no math required.

Subscribe now

The Starting Intuition

Imagine you work for a company that sells physical products (e.g., retail) and wants to build a taxonomy from your product data (e.g., a product catalog).

To keep the examples easy to understand, let’s assume your product data consists only of length and width measurements.

Yep. Only two columns (i.e., features).

Don’t worry.

Everything you will learn in this tutorial series applies whether you have two features or 100.

Also, since I will use visualizations in this tutorial, let’s assume the product catalog consists of only six products.

Again, this is OK because what you will learn scales to much larger datasets (e.g., 1,000,000 rows).

Here’s the data:

Learning how clustering techniques (i.e., algorithms) work is much easier using visualizations. Here's the above data visualized as a scatter plot:

Agglomerative hierarchical clustering works by starting with each data point (e.g., row of data) and identifying which other data points are most similar.

As with most clustering algorithms, “similar” is defined as the distance between points:

1. Closer data points are more similar.

2. Distant data points are less similar.

Consider the point in the top right of the visualization. The algorithm calculates the distance (i.e., similarity) between this point and all the other points (I didn’t draw all the lines to reduce clutter):

Compare the above distances to the distances for the point below:

And compare the following:

The algorithm continues in this fashion, calculating distances, to cluster the most similar data points together. The algorithm is greedy, meaning it chooses the best clustering it can find as soon as it can.

This is an important idea.

The clusters found early on may not be the “best” later on. This greediness is necessary for the algorithm to have any chance of running fast enough to be useful.

Greediness is a recurring theme in data science (e.g., decision tree-based machine learning) and works well in practice - so don’t worry! 😁

The First Cluster

Based on all the distance calculations, the algorithm greedily finds the first cluster:

This distance comparison process continues, and the algorithm greedily finds a second cluster:

At this stage, the visualization above shows that four products belong to clusters, while two do not, based on the distances.

Clusters as Data Points

Now, here’s the interesting thing.

Conceptually, the algorithm treats found clusters like individual data points.

For example, in the visualization above, the distance between the top-right data point will be calculated for each cluster, not for individual data points within the clusters.

A future tutorial will teach you the strategies used to make this happen. But for now, it’s not necessary.

Next, the algorithm considers the distances between clusters and any data points that do not belong to clusters.

This is where something interesting happens.

The algorithm finds that the clusters are closer to each other than the two data points that don’t belong to any clusters. In response, the algorithm clusters the clusters:

The visualization above demonstrates how the algorithm constructs the hierarchy of nested clusters. Looking at the data points provides intuition about what’s happening:

• The two unclustered data points have relatively large or small Length and/or Width values.

• The cluster of clusters contains the four data points with more typical values of Length and Width.

• The contained cluster on the left has two data points with relatively low Length values and relatively high Width values.

• The contained cluster on the right has two data points with relatively high Length values and relatively low Width values.

This is how the algorithm mines a taxonomy (i.e., a hierarchy) directly from the data.

However, the algorithm isn’t complete yet because it still needs to cluster all the data. So, the algorithm compares the cluster of clusters to the two remaining data points and finds the following:

At this point, there's only one last cluster that includes all the data points:

In the visualization above, I drew the last cluster using a rectangle simply because it fit better. 🤣

Mapping It Back to the Data

While going through the images above step-by-step helps to build your intuition, the final result is cluttered and hard to read (I added the row numbers from the dataset for context):

This is where a special kind of visualization known as a dendrogram comes in handy:

In the above dendrogram, the numbers along the bottom correspond to the rows of the dataset (i.e., Python starts counting from 0).

When using real (i.e., much larger) datasets, these numbers typically correspond to the number of rows per cluster.

Here’s what the dendrogram shows:

• Rows 2 and 5 are a cluster.

• Rows 1 and 4 are a cluster.

• There’s a cluster that contains the previous two clusters.

• There’s a cluster that contains all the above clusters and row 3.

• There’s one last cluster that contains all the above clusters and row 0.

The heights of the lines in the dendrogram indicate the similarity between clusters. For example, the cluster of rows 2 and 5 is the shortest.

This aligns with what you saw in the previous visualizations: these rows are the closest.

Like most data visualizations, dendrograms break down when the scale becomes large (e.g., many nested clusters). A later tutorial will cover a strategy for optimizing the number of clusters found.

👉 Ready to learn more? Check out Part 3 here.

That’s it for this tutorial.

My next newsletter will continue this tutorial series by teaching you how distances are calculated in agglomerative hierarchical clustering.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

• Data science == machine learning.

• Machine learning == predictive ML models.

• Predictive ML models == production deployments.

Before I get a bunch of 🔥 comments and email replies, let me state something for the record.

When done well, the business value of production ML predictive models can be substantial.

However, these situations are typically the exception rather than the rule. This has been my hands-on experience with my clients and is also reported in industry data collected by TDWI, Forrester, and Gartner.

For example, the percentage of ML projects intended for production but never make it is very high.

Subscribe now

This is unfortunate, because what often gets lost in the discussions about data science is that there are two forms of ML commonly used in business analytics:

1. Supervised Learning: The machine learns from labeled examples.

2. Unsupervised Learning: The machine learns from unlabeled examples.

Supervised Learning is how you craft ML predictive models, such as decision trees and random forests. These models learn from datasets in which each row contains an outcome of interest (i.e., the label).

For example, you work for a governmental agency and want to craft an ML model to predict claims fraud. Every row of your historical dataset needs a label indicating whether a claim was fraudulent.

Supervised Learning gets all the love in social media, but there’s a problem.

Most of the world’s data is unlabeled - including the data in your organization.

So what do you do?

You use Unsupervised Learning.

Introducing Cluster Analysis

More specifically, you use a form of Unsupervised Learning called cluster analysis. Here’s a definition from my favorite machine learning textbook:

Because so much real-world data is unlabeled, cluster analysis is a widely used tool in analytics for discovering structure and generating new insights.

While many forms of cluster analysis have been invented over the years, the three clustering algorithms that are most used in business analytics are:

• K-means clustering

• DBSCAN clustering

• Hierarchical clustering

The third is the subject of this newsletter tutorial series.

Introducing Hierarchical Clustering

Based on the above definition, hierarchical clustering mines groupings from unlabeled datasets. What distinguishes hierarchical clustering is how the mined groupings are defined.

The easiest way to intuit how hierarchical clustering works is to see a typical real-world example:

The image above is a typical representation of a company - an org chart. This is an example of hierarchical clustering. Organizations worldwide cluster employees based on management hierarchies.

BTW - In machine learning terminology, the diagram above is known as a dendrogram and is commonly used to visualize hierarchical clustering results.

Hierarchical clustering can take an unlabeled dataset and mine a hierarchical structure (often referred to as a taxonomy) directly from the data.

You can then analyze the hierarchical clustering to derive new insights based on your business/processes.

For example, consider the highlighted portion of the dendrogram below:

Let’s assume you’re unfamiliar with the above organization and its people. You can use hierarchical clustering to derive insights like:

• “The lower left cluster comprises observations (i.e., employees) with titles indicative of supply chain management functions.”

• “The lower right cluster comprises observations with titles indicative of manufacturing functions.”

• “The upper cluster appears to represent the organization’s manufacturing and supply chain division.”

While a contrived example to be sure, the above illustrates that cluster analysis is a universally applicable skill:

• Marketing: Segmenting customers into groups for more effective campaigns.

• IT Operations: Anomaly detection in network operations and security.

• Text Analytics: Group documents based on similar content.

• Healthcare: Mining patient data for groups to improve outcomes.

The list is endless!

If you’re serious about having more impact at work using data, you want skills with cluster analysis!

👉 Want to learn more? Check out Part 2 here.

That’s it for this tutorial.

My next newsletter will continue this tutorial series by teaching you how the hierarchical clustering technique works.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

​The last tutorial sang the praises of Python in Excel by demonstrating how you can get better exponential triple smoothing (ETS) forecasts than can be achieved using Excel’s native FORECAST.ETS() function.

This week’s tutorial will take it up a notch by demonstrating how Python in Excel enables you to create state-of-the-art forecasts using external drivers.

If you would like to follow along with today’s tutorial (highly recommended), you will need to download the SalesTimeSeries.xlsx file from the newsletter’s GitHub repository.

You will also need, of course, access to Python in Excel.

Subscribe now

Endogenous vs Exogenous Features

When it comes to using data to build better forecasts, the features you use fall into one of two categories. While there are technical definitions for the following, for this tutorial series, I will take an intuitive approach:

• Endogenous means the feature comes from “inside” the time series target values. For example, a moving average calculated from the time series target values is endogenous.

• Exogenous means the features come from “outside” the time series target values. For example, monthly marketing spend is an exogenous feature.

Classic time series models like those produced from Excel’s FORECAST.ETS() function uses only endogenous features.

However, modern business processes are highly complex, with many factors that influence outcomes. These complexities often mean that the best forecasting models will use endogenous and exogenous features.

Because of their power, I start by brainstorming exogenous features with my clients to build state-of-the-art forecasting. This is where Python in Excel is a game-changer.

Preparing the Data

To keep this tutorial from being too long, I will assume you have some familiarity with Python in Excel. If you’re new to Python in Excel, check out Part 6 of this tutorial series for detailed step-by-step instructions.

Using a dedicated worksheet to hold your Python in Excel formulas is a good idea. I typically name my worksheet simply Python Code:

Next, I organize my Python formulas vertically in the worksheet, forming a step-by-step execution order from top to bottom. I also add comments in the worksheet for each step:

The actual Python formulas I place in column C are aligned to the comments. For example, here's the formula for cell C3 to load the SalesAndMarketing table into Python:

And here’s the Python code for your workbook:

# Load the Sales table as a DataFrame
sales = xl("SalesAndMarketing[#All]", headers = True)

The SalesAndMarketing table contains an exogenous feature for the monthly marketing spend:

As covered in Part 6, Python in Excel includes the mighty statsmodels library, which provides a number of powerful forecasting models. These models prefer DataFrames where the index is the timestamp:

And the Python code for your workbook:

# Change the YearMonth column
sales['YearMonth'] = pd.to_datetime(sales['YearMonth'])

# Make YearMonth the index
sales = sales.set_index('YearMonth')
sales = sales.asfreq('MS')
sales = sales.sort_index()

In real-world projects, there is often much more data preparation than you see here (e.g., getting access to the data in the first place).

Forecast KPIs

As covered in Part 3, this tutorial series uses bias and mean absolute error (MAE) to measure forecast accuracy.

The following Python formula creates a utility function to calculate and display these key performance indicators (KPIs):

And the Python code:

# Utility function for our model KPIs
def forecast_kpis(forecasts, targets):
    # Bias calculations
    errors = forecasts - targets 
    
    targets_sum = np.sum(targets)
    bias = np.mean(errors)
    bias_pct = np.sum(errors) / targets_sum 

    # MAE calculations
    mae = np.mean(np.abs(errors))
    mae_pct = np.sum(np.abs(errors)) / targets_sum

    # Display
    print(f'Bias: {bias:0.4f}, {(bias_pct * 100):0.2f}%')
    print(f'MAE: {mae:0.4f}, {(mae_pct * 100):0.2f}%') 

Splitting the Data

As covered in Part 4 of this tutorial series, the best forecasters test their model predictions for accuracy by using a test holdout set with the following characteristics:

• The latest of the time series data.

• Enough data to cover at least one full cycle of any seasonality.

In the case of the time series used in this tutorial, the last full year of data is used as the test holdout set:

Python code:

# Use the first 4 years of endogenous data to train the model
sales_train_y = sales.loc[:'2023-12-01', 'Sales']

# Use the last year of endogenous data to test the model
sales_test_y = sales.loc['2024-01-01':, 'Sales']

As detailed in the code comments above, the Sales column is the target (i.e., endogenous) data for this time series.

The dataset also contains the MarketingSpend feature, which is exogenous data that must also be split using the same date-based logic:

# Get the first 4 years of exogenous data
sales_train_X = sales.loc[:'2023-12-01', 'MarketingSpend']

# Use the last year of exogenous data to test the model
sales_test_X = sales.loc['2024-01-01':, 'MarketingSpend']

With the data split, it's time to discuss the details of using exogenous data for forecasting.

How to Use Exogenous Data

When used properly, exogenous data is a powerful way to increase the accuracy of your forecasts. Think of exogenous data as a way to add additional information that isn’t available from the raw time series targets.

For example, consider the MarketingSpend feature of the tutorial’s dataset. When done correctly, the budget spend on marketing activities should have a positive impact on Sales.

State-of-the-art forecasting models are able to leverage the information contained in exogenous features to create more accurate forecasts than are possible from only using endogenous data.

So, the best forecasters are always on the lookout for:

• Exogenous features that can improve their forecasting models.

• Forecasting techniques that can make maximum use of exogenous features (e.g., machine learning).

However, there’s one big requirement for using exogenous data.

Any exogenous feature you use in your forecasting models must have future data available before you can make a forecast.

This is a bit abstract, so let me cement it with an example. Let’s say you use the MarketingSpend exogenous feature to build a forecasting model using all the data available (i.e., through the end of 2024).

In order to make forecasts for 2025, you would need MarketingSpend values for each month of 2025 that you are trying to forecast. In this example, you could get the 2025 marketing budget numbers from the finance team to use in your forecasts.

Here are some additional examples of exogenous data that are often quite valuable and have future data available:

• Capital investments

• Product launches

• Store openings

• Weather

NOTE - Using weather in your forecasting models typically only works for relatively short forecasting horizons (e.g., up to a week).

Training a SARIMAX Model

Unfortunately, the options for building forecasting models that use exogenous features are limited using native Excel features (e.g., multiple linear regression using the Analysis ToolPak).

Luckily, the mighty statsmodels package provides the Seasonal AutoRegressive Integrated Moving Average with eXogenous regressors (SARIMAX) model that you can use with exogenous data.

Due to space considerations, I can’t cover how SARIMAX works behind the scenes. However, the SARIMAX model is an extension of the popular ARIMA model, and an upcoming tutorial series will cover ARIMA using Python in Excel.

For the purposes of this tutorial, here’s a SARIMAX model that works well with the Sales and MarketingSpend features:

from statsmodels.tsa.statespace.sarimax import SARIMAX

# Specify a SARIMAX model using exogenous features
sales_model = SARIMAX(sales_train_y, exog = sales_train_X,
                      order = (2, 1, 2), 
                      seasonal_order = (0, 1, 0, 12))

# Train the model on the data
sales_model_results = sales_model.fit(method = 'bfgs')

And the Python formula to evaluate the SARIMAX model's predictions against the test holdout:

# Make 12 months of forecasts using exogenous features
forecasts = sales_model_results.forecast(steps = 12, 
                                         exog = sales_test_X)

# What is the model's forecast accuracy?
forecast_kpis(forecasts, sales_test_y)

The first thing to note about the code above is that to make any forecasts, you must pass in the future exogenous data using the exog parameter.

The second thing to note about the code above is that the SARIMAX model’s MAE is better than the ETS model's from Part 6 of this tutorial series.

I know that this tutorial has been a whirlwind, but I wanted to illustrate that if you’re serious about better forecasts, you can’t go wrong building skills with Python in Excel and statsmodels.

👉 Are you ready to partner with AI to build your Excel forecasts?

That’s it for this tutorial series.

My next newsletter will kick off a new tutorial series that teaches real-world machine learning skills with hierarchical clustering.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

If you’re serious about building better forecasts using Microsoft Excel, then you should be excited about the biggest upgrade in Excel’s history - the ability to run formulas written in Python.

Python in Excel is a game-changer for millions of professionals wanting to unleash the power of their data.

Not only does access to Python provide you with powerful ways to build better forecasts, but you also get access to a host of advanced analytics, like:

• Machine learning predictive models

• Mining free-form text

• Cluster analysis

This week’s tutorial shows you just a small sample of what’s possible for your forecasting when you embrace Python in Excel.

If you would like to follow along with today’s tutorial (highly recommended), you will need to download the SalesTimeSeries.xlsx file from the newsletter’s GitHub repository.

You will also need, of course, access to Python in Excel.

NOTE - Even though I’m using Python in Excel for this tutorial, 99+% of the code is exactly the same whether you use Excel, Jupyter Notebook, or VS Code.

Subscribe now

Preparing the Data

When using Python in Excel, it’s best to organize your Python formulas (i.e., code) to make writing and maintaining your data goodness easier.

I’m a big fan of putting all my Python formulas in a single worksheet, with the code laid out step-by-step vertically. First step is to add a worksheet:

Next, I add comments to my Python worksheets for my own long-term sanity:

I then place my Excel formulas immediately to the right of each comment. In this case, I would click cell C2 to hold the Python formula that loads the Sales table.

The best way to write Python formulas is to use the new Python Editor, which you can access from the Ribbon:

Clicking the Python Editor opens a new pane:

Notice how the button at the bottom of the Python Editor references cell C2? Clicking the button inserts a new Python formula in that cell where you write your code:

The code in the image above loads the Sales table from Excel into a Python DataFrame (i.e., how Python represents entire data tables).

Clicking the disk icon will save (i.e., run) the Python formula:

If you enter the code correctly, you should get a new DataFrame:

Clicking on the > in the Python Editor gives you a preview of the DataFrame:

The output above shows 60 rows of data (i.e., 5 years of monthly sales) and two columns: YearMonth and Sales.

Here’s the Python code for you to use in your Excel workbook:

# Load the Sales table as a DataFrame
sales = xl("Sales[#All]", headers = True)

When forecasting with Python in Excel, you usually need to format your DataFrames in a particular way:

• Your timestamps have to be recognized by Python.

• The DataFrame is indexed by the timestamp.

The YearMonth column isn’t recognized by Python as a timestamp because Excel didn’t recognize it as a timestamp. However, this is easily fixed by adding another Python formula to the worksheet:

Conceptually, the Python formula in cell C3 sets up the DataFrame to have an index based on the YearMonth column, where each value corresponds to the first day of each month.

Here’s the Python code to use in your own workbook:

# Change the YearMonth column
sales['YearMonth'] = pd.to_datetime(sales['YearMonth'])

# Make YearMonth the index
sales = sales.set_index('YearMonth')
sales = sales.asfreq('MS')
sales = sales.sort_index()

You can think of a DataFrame index as providing a unique identifier for each row in the table. You can see this in action by hovering over the [PY] in cell C3 and clicking on the card:

Excellent!

With the data now prepared for time series forecasting, here’s the first thing you’ll want to do - check to see if there’s a trend and/or seasonality in the data.

Detecting Trend and Seasonality

As you learned in Part 2 of this tutorial series, detecting trend and seasonality is a critical aspect of building better forecasts. You also learned some simple techniques for utilizing native Excel features to accomplish this.

Python in Excel offers you something so much more powerful: Season-Trend decomposition using Loess (STL).

Python in Excel comes with the mighty statsmodels library. Think of statsmodels as a collection of functions that you can use to access powerful tools to build better forecasts. In most cases, better forecasts are possible than with native Excel features.

This is one of many reasons why many of my clients are embracing Python in Excel.

Accessing the power of STL is simple using Python in Excel:

The Python formula in cell C4 tells STL to analyze the Sales column and that the data is monthly (i.e., period = 12). Also, robust = True tells STL to use a robust version that is resistant to some forms of outliers.

The formula also creates a visualization (i.e., plot) of the results for ease of analysis, specifying a visual that is 10 inches wide and 6 inches high.

Clicking the > in the Python Editor provides the visual:

The above plots clearly show that the time series has a steady upward trend and pronounced seasonality (e.g., a dip in sales every January).

If you’re serious about building better forecasts, you want STL in your tool belt.

If that wasn’t enough, the statsmodels library has a lot more reasons why you want to be using Python in Excel to build your forecasts.

And here’s the code for your own Excel workbook:

from statsmodels.tsa.seasonal import STL

# Perform STL on the sales DataFrame
res = STL(sales['Sales'], period = 12, robust = True).fit()

# Display the results
fig = res.plot()
fig.set_size_inches(10, 6)
plt.show()

Forecast KPIs

You learned about measuring the accuracy of your forecasts in Part 3 of this tutorial series using the bias and MAE calculations.

When using Python in Excel, it’s handy to have a function to calculate and display these KPIs for you:

BTW - I only write the above utility function once and then copy and paste it into my Excel workbooks as needed.

Here’s the code for your own Excel workbook:

# Utility function for our model KPIs
def forecast_kpis(forecasts, targets):
    # Bias calculations
    errors = forecasts - targets 
    
    targets_sum = np.sum(targets)
    bias = np.mean(errors)
    bias_pct = np.sum(errors) / targets_sum 

    # MAE calculations
    mae = np.mean(np.abs(errors))
    mae_pct = np.sum(np.abs(errors)) / targets_sum

    # Display
    print(f'Bias: {bias:0.4f}, {(bias_pct * 100):0.2f}%')
    print(f'MAE: {mae:0.4f}, {(mae_pct * 100):0.2f}%') 

Better Exponential Smoothing

In the last tutorial, you learned about Excel’s FORECAST.ETS() function.

The ETS in the function name stands for exponential triple smoothing and it should come as no surprise that the mighty statsmodels library also supports ETS.

It should also not be surprising that using statsmodels provides you with more opportunities for better ETS forecasts than what is provided by Excel.

By using Python in Excel, you get more fine-grained control over the forecasting model, as you will see in this tutorial. First up, as you learned in Part 4, the data needs to be split:

Here’s the code for your Excel workbook:

# Use the first 4 years of data to train the model
sales_train = sales.loc[:'2023-12-01', 'Sales']

# Use the last year of data to test the model
sales_test = sales.loc['2024-01-01':, 'Sales']

With the data split, building an exponential triple smoothing model is straightforward:

The Python code for your Excel workbook:

from statsmodels.tsa.holtwinters import ExponentialSmoothing

# First exponential triple smoothing model
fit1 = ExponentialSmoothing(sales_train,
                            trend = 'add',
                            seasonal = 'add',
                            seasonal_periods = 12,
                            use_boxcox = True).fit()

# Get the forecasts and KPIs
pred = fit1.forecast(steps = 12)
forecast_kpis(pred, sales_test)

At a high level, there are three things to note about the above code:

• The code uses trend = ‘add’ and seasonal = ‘add’ to align to Excel’s FORECAST.ETS() function.

• The seasonal_periods = 12 indicates that the data is at the monthly grain.

• The use_boxcox = True adds a bit of transformation to the data before the model is built. FORECAST.ETS() is more forgiving than statsmodels, so this is required.

As detailed in Part 5, the above KPIs are an improvement over using Excel’s FORECAST.ETS() function:

• The magnitude of the bias improved from -1.06% to 0.27%.

• The MAE improved from 4.68% to 4.06%.

Excel’s FORECAST.ETS() function uses additive trends and seasonality with exponential smoothing.

However, exponential smoothing also supports multiplicative trends and seasonality. In some scenarios, multiplicative trends and seasonality produce better forecasts:

As shown above, the multiplicative exponential smoothing model is a slight improvement in MAE compared to the additive model.

Here’s the code:

# Multiplicative exponential triple smoothing model
fit2 = ExponentialSmoothing(sales_train,
                            trend = 'mul',
                            seasonal = 'mul',
                            seasonal_periods = 12,
                            use_boxcox = True).fit()

# Get the forecasts and KPIs
pred = fit2.forecast(steps = 12)
forecast_kpis(pred, sales_test)

That’s it for this tutorial.

My next newsletter will show you another reason you want to build better forecasts using Python in Excel - incorporating external drivers.

Stay healthy and happy data sleuthing!

👉 Ready to learn more? Check out Part 7 here.

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

Through this tutorial series, you’ve been learning the fundamentals of time series forecasting using Microsoft Excel.

In this tutorial, you will learn the premier way to build data-driven forecasts using native Excel features - the FORECAST.ETS() function.

If you would like to follow along with today’s tutorial (highly recommended), you will need to download the SalesTimeSeries.xlsx file from the newsletter’s GitHub repository.

Subscribe now

Preparing the Data

The FORECAST.ETS() function requires the use of Excel-formatted dates. The Sales table in the tutorial workbook represents a common situation where Excel doesn't like the format:

As covered in Part 2 of this tutorial series, the answer to this problem is to insert a Date column using the following formula to convert the YearMonth data into an Excel-friendly format:

NOTE - You may need to explicitly format column C as Excel dates if your table doesn't look like the image above.

Establishing the Baseline

As covered in Part 4 of the tutorial series, establishing a baseline is a critical step in crafting the best forecasting models. Unfortunately, not doing this is one of the most common mistakes my clients make:

• All data through December 2023 for training the forecasting baseline.

• All data for 2024 for testing the forecast baseline.

• A simple 6-month moving average as the baseline model.

The first step is to calculate the 6-month moving average for December 2023 by adding an MA(6) Forecast column and entering the following in cell E50:

Next, the forecasts for 2024 are simply the forecasted value for December 2023:

As covered in Part 3, next up is calculating the MA(6) errors in a new column:

And lastly, as detailed in Part 3 and Part 4, calculating the key performance indicators (KPIs) for the MA(6) baseline model for the test dataset:

Here’s what the KPIs tell you:

• The MA(6) baseline typically forecasts values too low for the test dataset.

• On average, the MA(6) baseline is 4.08% too low.

• The typical MA(6) baseline is off by $74,530.

• On average, the MA(6) baseline is off by 6.58%.

With the baseline established, it’s now time to see if the FORECAST.ETS() function can provide uplift (i.e., beat the baseline KPIs).

FORECAST.ETS() Intuition

The FORECAST.ETS() function was added to Microsoft Excel 2016 and empowers millions of professionals to build better data-driven forecasts.

The ETS portion of the function name stands for exponential triple smoothing, a battle-tested forecasting model in use for more than 60 years. For this tutorial series, an intuitive understanding of ETS is enough.

You can think of ETS as learning the following from time series data:

1. The average target of the time series, which is known as the level.

2. The average amount of fluctuation between two consecutive targets in the time series is known as the trend.

3. Patterns that appear at regular intervals (e.g., weekly or monthly) that impact the level known as the seasonality.

Using ETS, each forecast can be thought of as being:

• Forecast = Level + Trend + Seasonality

Now for the intuition around exponential smoothing.

Consider 1 above. Conceptually, this is similar to a moving average forecasting model. However, a moving average model treats each data point equally in the calculation.

ETS introduced the innovation of placing greater weight on recent data points than on older ones. In some cases, ETS will severely penalize older data points to improve forecasting (e.g., exponential smoothing).

If this seems overwhelming, then you’ll probably find this refreshing.

The FORECAST.ETS() function does all this for you automagically.

Using FORECAST.ETS()

With the intuition established, it’s time to apply ETS to the Sales time series dataset.

The FORECAST.ETS() function has some requirements that you need to be aware of. From Microsoft’s website:

“This function requires the timeline to be organized with a constant step between the different points. For example, that could be a monthly timeline with values on the 1st of every month, a yearly timeline, or a timeline of numerical indices.”

This is just a fancy way of saying that your dataset must have a constant grain (e.g., monthly sales) and can’t have any gaps in the dates (e.g., January 2022 is missing).

The easiest way to get started with FORECAST.ETS() is to use only its three required arguments:

• The date for which the forecast is created.

• The historical targets.

• The historical dates.

First, a FORECAST.ETS() column will hold the forecasts. For the tutorial dataset, the first forecast for the test dataset is January 2024 (i.e., cell C51). Here’s the first step of setting up the formula:

Next is adding the historical time series values (i.e., all the Sales values through December 2023):

Last, adding the historical dates:

Hitting <enter> gives the first forecast for the test dataset. The rest of the 2024 forecasts are simply a matter of dragging the formula down:

As you can see in the image above, ETS is far more varied in its forecasts compared to using a simple MA(6) model.

However, are the ETS forecasts better?

FORECAST.ETS() KPIs

As you've learned, calculating forecast errors is the first step in evaluating your forecasts:

The image above shows how to add a new FORECAST.ETS() Error column to the table with the error formula in cell F51. Hitting <enter> and dragging the formula down: RMSE:

And the KPIs for the ETS forecasts:

Yesssss!

Moving to a more complex model has improved the forecast KPIs.

👉 Ready to learn more? Check out Part 6 here.

That’s it for this week.

My next newsletter will show you the biggest thing ever to happen to forecasting with Microsoft Excel - Python formulas.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

Forecasting is about predicting the future. But you don’t have a time machine. So, how can you trust that your forecasts are accurate?

In this week’s tutorial, you will learn how to test your models by simulating the future.

Unfortunately, most professionals (e.g., Finance pros) are not taught how to do this. This is your opportunity to stand out.

If you would like to follow along with today’s tutorial (highly recommended), you will need to download the SalesTimeSeries.xlsx file from the newsletter’s GitHub repository.

Subscribe now

Simulating the Future

To build better forecasts, you must use your data to simulate the future. You do this by splitting your data into two sets:

1. The oldest data becomes what you use to build your forecasts.

2. The newest data becomes what you use to test forecast accuracy.

As I help my clients understand, this strategy is critical to their forecasting success.

First, by using only the oldest data to build your forecasts, you are hoping to discover the underlying patterns that stand the test of time.

Imagine that you’re forecasting sales and have 6 years of monthly data. You use the first 4 years to build and the last 2 years to test.

If your forecasts are accurate, you’ve found these patterns. But that’s not all.

Second, consider a forecasting model that received the first 4 years of sales data.

From the model’s perspective, the future starts with the first month of the test data (i.e., January of year five). This is how you simulate the future.

Is this simulation perfect?

Absolutely not. But it’s the best you can do.

And, as I mentioned above, most professionals haven’t been taught how to do it.

Splitting the Data

The Sales table of the tutorial Excel workbook has 5 years of monthly sales data.

While there is no magical equation that tells you how much data should be split between building your forecasting model (i.e., the training dataset) vs. the test dataset, here are some guidelines:

• Each dataset set should include, at a minimum, one complete cycle of seasonality.

• More data is better than less data.

As covered in Part 2, the Sales data exhibits yearly seasonality. Therefore, here are two reasonable approaches to splitting the data:

1. Use 3 years of data for training and 2 years for testing.

2. Use 4 years of data for training and 1 year for testing.

I will cover option 2 in this tutorial series. Try out option 1 and compare the results.

A Simple Moving Average Model

In Part 3 of this tutorial series, I introduced a simple 6-month moving average as a forecasting model. I will continue to use this MA(6) model as a baseline in this tutorial.

Setting up the MA(6) model in Microsoft Excel is straightforward:

Given my choice of reserving the last year of data for testing (often referred to as the test holdout), I drag the MA(6) formula down the length of the table and stop at the end of 2023:

As covered in Part 3, you can certainly measure the accuracy of the MA(6) model’s forecasts against the historical data.

However, we typically care most about forecast accuracy on the test dataset.

The simplest form of an MA(6) forecasting model will predict the last value for the test dataset. In this case, the value from cell D50:

To put the test dataset forecasts in context, a quick line chart is a very handy tool.

First, select the data for 2024:

And then insert a line chart using the Ribbon:

The straight orange line in the visualization above is the MA(6) forecasted values.

There are a few things worth noting in the line chart:

• The MA(6) forecasts are too high for Jan, Mar, and Apr.

• The MA(6) forecasts are very close for May through Sep.

• The MA(6) forecasts are too low for Oct, Nov, and Dec.

Overall, this doesn’t look too bad based on the visual. However, you don’t want to rely solely on eyeballing the data.

That’s where the KPIs come into the picture.

Simple MA(6) KPIs

I cover the bias and mean absolute error (MAE) KPIs in Part 3 of the tutorial series, so check out the tutorial if you’re not familiar with these calculations.

First, calculating the MA(6) forecasting errors in column E:

And the KPIs for just the test dataset (i.e., only for 2024):

Assuming that the test dataset is a reasonable simulation of the future, here’s what the KPIs tell us about the MA(6) model:

• The forecasts will be, on average, 4.25% too low.

• The forecasts will be, on average, off by $74,530.

• The forecasts will be, on average, off by 6.86%.

A Revised MA(6) Model

What you’ve seen so far isn’t the only way moving averages can be used as forecasting models.

For example, the moving average calculation doesn’t have to stop with the end of the training dataset. You can use the model predictions in later calculations.

Conceptually, the model uses its previous predictions to create new predictions.

This is a bit abstract, so here’s the formula to demonstrate:

Notice in the formula above that the moving average is now calculated from the MA(6) model predictions. Dragging the formula down the length of the table changes the forecasts, errors, and KPIs:

The KPIs tell us that the revised MA(6) model performs worse than the simple MA(6) model.

However, don’t let this deter you from the powerful idea of forecasting models using previous predictions to make future ones.

This technique is used in a large number of forecasting techniques (e.g., Excel’s FORECAST.ETS() function) and is quite powerful - even if it didn’t work out so well in this particular case.

👉 Ready to learn more? Check out Part 5 here.

That’s it for this week.

My next newsletter will build upon everything you’ve learned so far to use Microsoft Excel’s built-in premier forecasting technique - the FORECAST.ETS() function.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

Forecasts are used to make some of the biggest decisions within organizations. Often, these decisions involve many millions of dollars of spending:

• Should we build a new factory?

• Should we open new stores?

• Should we cut budgets?

The list goes on and on.

The last thing you want to do is base these critical decisions on a forecast that isn’t accurate.

In this week’s tutorial, you will learn about two simple (but powerful) measures for assessing the accuracy (i.e., quality) of your forecasting models.

If you would like to follow along with today’s tutorial (highly recommended), you will need to download the SalesTimeSeries.xlsx file from the newsletter’s GitHub repository.

Subscribe now

A Simple Model

As covered in the last tutorial of this series, I’m going to use a 6-month moving average model to provide the baseline forecasts for the rest of this tutorial series.

Here’s how to calculate the forecasts using the Sales table from the tutorial’s Excel workbook:

Dragging the formula down the length of the table populates all the forecasts:

I'm going to use the above forecasts in the accuracy measure calculations in this tutorial.

Your Model’s Bias

When considering the accuracy of a forecasting model, here’s a fundamental question that needs to be answered:

“Are the forecasts typically too high or too low?”

This question concerns the model's bias, and there is a calculation that can tell you how biased your model is, on average.

Not surprisingly, the first thing you need to assess the bias of a forecast model is the forecast errors.

BTW - Other terms for the error are residual and noise.

Despite what term you use, calculating errors is simple. You subtract the target from the forecast:

And dragging the calculation down the length of the table:

As you can see in the image above, the forecasts are both too high (e.g., $36,823 for March 2024) and too low (e.g., -$111,719 in December 2024).

While you can examine/analyze the individual error values, the bias calculation aims at providing a key performance indicator (KPI) for a forecast model’s predictive performance.

Intuitively, the bias KPI is just the average error:

Think of the calculation in cell E64 of the Sales worksheet as combining all the forecasts and telling you, on average, if the forecasts are too low or too high:

The bias calculation shows that the MA(6) forecast model tends to make forecasts that are, on average, too low.

Unfortunately, the value of -$28,539 isn’t super informative by itself.

For example, -$28,539 doesn’t tell you if the bias is very large compared to the targets in your time series.

What can be more informative is calculating the bias percentage (what is often called the scaled bias):

Here’s what’s going on in the bias % calculation - the average error is being divided by the average value of all the targets.

Here’s the bias % for the MA(6) forecasting model:

The bias % provides a better insight into how a model’s bias relates to the magnitude of the targets being forecasted.

In this case, the MA(6) model forecasts are, on average, 2.85% too low.

Your Model’s “Accuracy”

It might have occurred to you that the bias calculation doesn’t really align with the idea of accuracy.

And you’d be right to think that.

By way of analogy, consider what “accuracy” means when throwing darts at a dartboard when the target is the bullseye.

In this scenario, accuracy can be measured as the distance from the bullseye to where a dart landed on the dartboard (e.g., the distance would be zero when you hit the bullseye).

When thought of this way, it doesn’t matter if the dart is too high or too low relative to the bullseye. All that matters is the distance.

Using this analogy, the mean absolute error (MAE) represents how far, on average, the darts are from the bullseye.

This makes MAE a very intuitive way to assess the quality of your forecasts, easily understood by business stakeholders.

Here’s the MAE for the MA(6) forecasts:

If you're unfamiliar, Excel's ABS() function stands for absolute value. Basically, think of this function as turning the negative errors into positive errors and leaving positive errors unchanged:

What the MAE calculation says about the MA(6) model is that the forecasts are off by $60,656 on average.

Unlike the bias calculation, the MAE can be quite informative on its own. For example, MAE can be used as a criterion for determining whether a model will be useful for decision-making.

It can also be super useful to calculate the MAE % (i.e., scaled MAE):

The MAE % is the average absolute error divided by the average target value and tells us that the forecasts are off by 6.07% on average.

However, MAE doesn’t tell us if the forecasts are typically too high or too low.

That’s what the bias tells us.

So, like chocolate and peanut butter, the combination of bias and MAE is better together when evaluating your forecasting models.

What About RMSE?

One last thing I will mention is that I also use another metric to evaluate forecasting models: root mean square error (RMSE).

The RMSE calculation is a little more complicated than the bias and MAE calculations. So, I rarely communicate this KPI to business stakeholders.

The first step is to calculate the squared errors of the forecasts:

Notice how the squared errors can get really large?

RMSE is designed to penalize larger forecast errors more than MAE.

Imagine you have one forecast that is way off (i.e., high error) and all the others are spot-on.

Because the MAE calculation averages errors, it can give a misleading picture of what’s happening in these hypothetical forecasts.

Since RMSE penalizes large errors more than small ones, this will be reflected in the RMSE KPI.

Next up are the final steps of the RMSE calculation:

• Taking the average of the squared errors.

• Taking the square root of the average (i.e., mean).

And scaling RMSE:

Since the interpretation of RMSE is somewhat mathematical, I’m going to discuss using RMSE from a practical perspective.

You can think of RMSE as measuring a different aspect of a model’s forecasts as compared to MAE.

Unlike MAE, there isn’t an intuitive, direct understanding of RMSE. RMSE is most useful when comparing one forecasting model to another.

I will demonstrate using RMSE this way in a later tutorial.

Stay tuned.

👉 Ready to learn more? Check out Part 4 here.

That’s it for this week.

My next newsletter will discuss a critical aspect of building better forecasts - evaluating whether your forecasting model is any good.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

The topic of this week's tutorial is detecting trend and seasonality in a time series using out-of-the-box Excel features.

If you would like to follow along with today's tutorial (highly recommended), you will need to download the SalesTimeSeries.xlsx file from the newsletter's GitHub repository.

Subscribe now

Your First Time Series Model

The best way to learn data-driven forecasting is to start with simple models and gradually add complexity.

Simple forecasting models are also critical for establishing a baseline you use to assess whether a more complex model improves decision-making.

In this tutorial, you will learn about one of the simplest baseline forecasting models - the simple moving average.

Here’s how a moving average works:

• Forecasts are made by taking the average of a fixed number of previous target values.

• As the forecaster, you pick how many previous target values to use in the moving average.

Moving averages, I find, are quite intuitive when you see them in action.

Let’s assume a 6-month simple moving average. Using the data from the Sales worksheet of the tutorial’s Excel workbook:

Hitting the <enter> key will create a new column, which can be renamed Forecast. Dragging the formula down the length of the column produces a collection of forecasts:

The next tutorial in this series will evaluate the predictive performance of a moving-average model. In this tutorial, we’ll use a moving average to determine the trend of a time series.

A common shorthand notation for moving-average models is MA(X), where X denotes the number of previous targets used in the average.

The above 6-month moving-average model is abbreviated as MA(6).

Detecting Trend

In this tutorial, you will learn a simple way of detecting a trend in a time series using an MA(12) model. While this technique is easy to implement using out-of-the-box Excel features, it has some issues.

Luckily, since there’s Python in Excel, you have access to more powerful tools. A later tutorial will demonstrate why my clients are moving to Python in Excel so they can use tools like the mighty Python statsmodels package.

In the Sales worksheet, delete column D and repeat the moving average modeling process above, but using 12 data points instead:

And setting up the column by renaming it to MA(12) and dragging the formula all the way down the length of the table:

To detect a trend, you now need a line chart of all three columns. First, click on any cell in the Sales table. Then insert a line chart using the Ribbon:

And the line chart tells the story of the trend in the time series:

Voila!

The orange MA(12) line clearly has an upward trend.

Detecting Seasonality

As with trends, Python in Excel provides a more sophisticated way of detecting seasonality, which I will cover in a later tutorial.

This week’s tutorial will use out-of-the-box features to help you detect if seasonality is present in a time series.

Here’s the intuition.

The Sales time series data is at the monthly grain, so the goal is to detect whether certain months tend to have higher target values than others and whether there is a pattern among these higher months.

A Box and Whisker plot (i.e., box plot) can be used to visualize this, but the Sales table isn’t set up correctly for Excel box plots. The good news is that setting this up is simple.

First, we need to create a column that Excel recognizes as a date column. Insert a new column named Date between YearMonth and Sales and enter the following formula:

Hitting <enter> populates the column, but you will need to tell Excel to format the column as a date explicitly:

With the Date column created, you can extract the month abbreviations by inserting a Month column and entering the following formula:

Hitting <enter> populates the column:

Now, insert a box plot from the Ribbon:

And here's the resulting visualization:

The box plot shows the range (i.e., distribution) of sales values by each month.

For more information on box plots, check out the Wikipedia article.

The box plot indicates that sales are typically higher in the last three months of each year, with a noticeable decline in January. This is likely due to seasonality in the time series.

👉 Ready to learn more? Check out Part 3 here.

That’s it for this newsletter.

The next newsletter will introduce you to a critical step for better forecasting in Excel: establishing a forecasting baseline.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

So, it probably doesn’t shock you to know that I’ve taken online courses designed for Financial Analysis & Planning (FP&A) professionals. My goal in taking these courses was to understand how business professionals learn about data.

Quite frankly, what I saw in these courses shocked me.

For example, I took an online Excel forecasting course designed for FP&A professionals, and I was appalled by what wasn’t included.

In essence, FP&A professionals did not develop the most important skills for building high-quality forecasts. And this was a course designed to help professionals earn a FP&A certification!

So, I decided to do something about it. Enter this tutorial series.

Subscribe now

Tutorial Materials

This week’s tutorial starts off with fundamental concepts in forecasting. This tutorial is designed for any professional.

The goal is to provide an intuitive introduction to data-driven forecasting using Excel. Therefore, I will limit the math coverage.

However, what you will learn is legit techniques for creating data-driven forecasts using native Excel features.

If you would like to follow along with today’s tutorial (highly recommended), you will need to download the SalesTimeSeries.xlsx file from the newsletter’s GitHub repository.

What is Forecasting?

Here’s a definition of forecasting from Wikipedia:

“Forecasting is the process of making predictions based on past and present data. Later, these can be compared with what actually happens.”

Based on this definition, you would be right to think that any form of predictive analytics is the same as forecasting.

However, the common interpretation of forecasting is creating predictions from time-series data.

You can think of this time series data as being like a crystal ball that allows you to predict what’s going to happen in the future.

For example, predicting monthly sales for the next twelve months using historical data.

Time series forecasting is a universal skill. Here are some examples from various industries and business functions:

• [Supply Chain] - Forecasting product demand.

• [Customer Service] - Forecasting call volume.

• [Healthcare] - Forecasting hospital supplies.

• [IT] - Forecasting server/cloud usage.

• [Finance] - Forecasting revenue.

And the list goes on!

What you will learn in this tutorial series is skills that can be applied to any forecasting problem, regardless of domain.

Therefore, the exact nature of the dataset used in the tutorial series doesn’t matter.

What is a Time Series?

In these tutorials, a time series is defined as a set of measurements collected at regular time intervals. These measurements can include anything, such as sales, demand, advertising spend, claims, etc.

The combination of measurement and a regular time interval is referred to as the grain of the data.

Consider the following snippet of data from the Sales worksheet of the tutorial Excel workbook:

This is a time series dataset at the grain of sales per month.

Most classic (i.e., statistical) time series forecasting techniques work with a single measurement at a time, like the time series pictured above.

However, a time series can also include additional data. For example:

In 2026, my clients want state-of-the-art forecasting (e.g., machine learning) that leverages a wide range of additional data (e.g., promotions, weather, planned expansions, etc.) to improve decision-making.

For example, in the image above, incorporating MarketingSpend data might produce better forecasts than just using Sales data alone.

To keep all of this data straight, I’m going to use the following terminology in this tutorial series:

• Timestamps denote the time period for which the data is applicable. In the image above, YearMonth is the timestamp for the data.

• Targets are the measurement values to be forecasted. In the image above, the Sales column contains the targets.

• Features are additional measurements/data that can be used to improve target forecasts. In the example above, MarketingSpend is a feature.

Time Series Forecasting Models

To craft data-driven forecasts, we use models. Just in case you’re new to the concept of a model, here’s a definition from Wikipedia:

“A model is an informative representation of an object, person, or system.”

This definition is great for this tutorial series, because we’re looking to build an informative representation of the system that produces the targets (e.g., sales).

Notice how the definition uses informative and not definitive?

This is a critical idea embodied by the following famous quote from George Box:

“All models are wrong, but some models are useful.”

Make no mistake. Your forecasting models will always be wrong to some degree.

The question isn’t whether your forecasting model is correct. It’s whether your forecasting model produces “good enough” predictions so that effective decisions can be made.

Ideally, your model is better than the one being used, so better decisions can be made, and you look like a hero at work.

Time Series Characteristics

When thinking about what makes a good forecasting model, we can decompose target values over time into three characteristics:

1. Trend is the tendency for target values to increase/decrease over time.

2. Seasonality is a pattern in the targets that appears in regular intervals (e.g., higher retail sales starting with “Black Friday” each year).

3. The remainder is what is left over from the target values once trend and seasonality are accounted for (i.e., removed from the original target values).

Before I discuss the first two in the context of the tutorial time series, I want to provide some additional context for the remainder.

The remainder is also known by other names:

• Residual

• Noise

• Error

Think of the remainder of a time series as being the result of your model being wrong to some degree (i.e., if your model’s forecasts were perfect, there wouldn’t be any remainder).

OK. The easiest way to build an intuition about trend and seasonality is to use a data visualization.

In the tutorial Excel workbook, select the Sales worksheet and click on any cell within the table:

Now, use the Ribbon to insert a mighty line chart:

First up, the line chart gives us a strong indication that there is a trend in the time series:

Seasonality is a bit more challenging to spot, requiring hovering over the various data points in the line chart and checking the values:

If you inspect the data points, you typically see more sales in the last months of the year than in the other months.

As with so many real-world situations, this is not a cut-and-dried pattern in the data.

Using the tutorial’s time series as a frame of reference, a useful forecasting model will need to accurately predict both the upward trend and the apparent seasonality in the data.

While simultaneously minimizing the remainder as much as possible.

👉 Ready to learn more? Check out Part 2 here.

That’s it for this newsletter.

The next newsletter will cover more robust techniques for characterizing trend and seasonality by introducing simple models.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

• Part 1: Introduction

• Part 2: Communicating Results

• Part 3: Python Code

• Part 4: Targeting Specific Outcomes

As the best way to learn DIY data science skills is through practice, I highly recommend following along with this tutorial in your preferred tool (e.g., Python in Excel).

Subscribe now

Don’t Make This Mistake

When professionals first learn MBA, a common reaction is something like this:

“Wait a minute, Dave. You mean I can only use true/false (i.e., binary) features with MBA? That seems like a huge problem for my analysis/domain/business problem!”

As it turns out, the requirement to use binary features is quite common in do-it-yourself (DIY) data science. Here are two examples:

• Linear/Logistic regression requires transforming categorical data into binary features.

• Many machine learning algorithms offered by Python’s scikit-learn library require transforming categorical data into binary features.

So, don’t underestimate the power of MBA simply because you need to use binary features!

As you build your DIY data science skills, a common theme across the techniques you use will be spending more time working with the data (e.g., cleaning) than you do writing code to perform the actual analysis.

Many years ago, research on data mining projects (think of data mining as what data science was called back then) was conducted. The research showed that between 60-80% of project effort was devoted to tasks like:

• Getting access to the data.

• Understanding the data.

• Cleaning the data.

Based on my experience as an analytics consultant, nothing has changed in 2026.

These days, it’s common to refer to all of these data-related activities as data wrangling.

While all of these activities are critical to success, the most important aspect of data wrangling is feature engineering. Feature engineering is the process of crafting the best data possible for your analysis.

In this regard, the successful use of MBA is no different than logistic regression or random forest predictive models:

The best analytics results come from the best features.

In this tutorial, you are going to learn the following two patterns that I’ve used with MBA to provide insights that have delighted business stakeholders:

• Presence

• Magnitude

Let’s dive in.

Presence Features

This MBA feature pattern is what you’re already familiar with from the tutorials in this series.

BTW - Feature is just another name for column, variable, or attribute.

Using the grocery store example from previous tutorials, consider whether a purchase (i.e., a transaction) includes whole milk.

The following DataFrame illustrates a hypothetical grocery store dataset built around presence features:

While the above dataset uses 1s and 0s to indicate presence, True/False values could be used instead, as Python treats them as equivalent.

One-hot encoding is a common technique for transforming categorical features into a binary representation (e.g., for use with the scikit-learn library).

Imagine you have a dataset tracking the results of the Olympic Games with a Medal feature to track each athlete’s results, like the following:

After one-hot encoding the Medal feature, the resulting DataFrame would look something like this:

The result of one-hot encoding is a presence indicator (e.g., 1/0) and can be used with MBA.

The following code demonstrates how to use one-hot encoding on categorical columns using the Adult.xlsx Excel workbook available in the newsletter’s GitHub repository, which contains data about US residents (e.g., age, occupation, income level, etc.).

Here’s the Python in Excel code:

And here’s the Python code that you can cut and paste:

from sklearn.preprocessing import OneHotEncoder

# Load the Excel table
adult_train = xl("AdultTrain[#All]", headers = True)

# Features to one-hot encode
cat_features = ['Education', 'Occupation']

# Instantiate the encoder
encoder = OneHotEncoder(sparse_output = False)
encoder.set_output(transform = 'pandas')

# Learn encodings and transform data
train_cat = encoder.fit_transform(adult_train[cat_features])

While you can use any Python technology (e.g., Jupyter Notebook) for this tutorial series, many professionals new to analytics find Excel much less intimidating.

In the code above, please note the following:

• The OneHotEncoder object is coded to output a DataFrame - this is usually what you want.

• The OneHotEncoder object is only encoding the two features specified by cat_features.

• The one-hot encoded data is returned and stored as train_cat.

Using Python in Excel, you can quickly look at the resulting DataFrame:

As shown in the card above, the horizontal scroll bar indicates that there are many more one-hot encoded columns to the right, which are not depicted in the image.

Also note the Education_ prefix for each feature name. This naming convention indicates which original feature served as the source of the one-hot encoding (i.e., the Education_10th feature represents the value of 10th coming from the Education feature).

But that’s not all.

Engineering Numeric Presence Features for MBA

There are many instances where you have a numeric feature that can easily be engineered into presence features for MBA. The running grocery store example of this tutorial series is a prime example.

Regardless of the count of a particular product within a transaction (e.g., three bottles of mustard), a simple binary indicator is all that is needed.

There are also times when numeric features do not represent counts, but a simple binary indicator of non-zero status is more than sufficient for MBA.

A great example of this situation is the CapitalGain feature found in the Adult.xlsx Excel workbook. Since the distribution of this feature is highly skewed (i.e., most values are 0), a simple presence feature is highly useful for MBA.

The following Python in Excel code demonstrates how to perform this transformation:

And here’s the Python code that you can cut and paste:

# Create a presence feature for CapitalGain
has_capital_gain = adult_train['CapitalGain'].gt(0)

# Get the counts of True/False
has_capital_gain.value_counts()

As you might imagine, the two examples covered here are not the only ways to create presence features. The possibilities are endless. Here are two more to spark your creativity:

• Detecting the presence of a substring in text data.

• Detecting if a behavior occurred within X amount of time.

Magnitude Features

Your MBA can often be more effective if the magnitudes of numeric features can be included as binary indicators. Making this happen is the result of a two-step process:

• Transforming the values of numeric features into categories.

• One-hot encoding the categories.

The first step is more common than you might imagine in analytics and is often referred to as binning. Here are some examples:

• Creating histograms requires defining bins for numeric feature data.

• Decision tree-based machine learning algorithms bin numeric features as part of their learning process.

• Using deciles to bin numeric feature data for RFM analysis.

I’ll use the last example of RFM analysis to demonstrate a useful technique for creating magnitude features for MBA. First up, RFM stands for:

• (R)eceny

• (F)requency

• (M)onetary

RFM analysis is an old-school marketing technique that classifies customers based on their RFM scores.

The most common (but not only) way to calculate the RFM score is by applying the following logic to each of the three RFM features:

• Classify each value of each feature in terms of deciles (i.e., bin the values into 10 distinct groups).

• The top 10% of values receive a score of 9.

• The next top 10% of values receive a score of 8.

• And so on, until the bottom 10% of values receive a score of 0.

To cement this type of feature engineering, I will demonstrate using the HoursPerWeek feature from the Adult.xlsx Excel workbook.

The HoursPerWeek feature illustrates a common problem when using deciles with business data: there are often many duplicates.

For example, given the business nature of the feature, many of the HoursPerWeek values are 40 (i.e., the standard US work week). When there are many duplicate values, creating deciles becomes impossible.

The following code demonstrates using the rank() method first to transform the data so it can be cleanly binned into deciles.

The code tells rank() to assign a value based on position. For example:

• The first value of 40 found would be assigned a rank of 1.

• The second value of 40 found would be assigned a rank of 2.

• And so on.

The code then calls the pandas qcut() method to divide the rankings into deciles (i.e., q = 10). The following shows the code and the results:

And the Python code for cut and paste:

# Handle all the duplicate values
hours_week_rank = adult_train['HoursPerWeek'].rank(method = 'first')

# Transform the ranks into deciles
hours_week_decile = pd.qcut(hours_week_rank, q = 10, labels = False)
hours_week_decile.value_counts().sort_index()

After transforming the HoursPerWeek feature into categorical magnitudes, the next step is to use a OneHotEncoder to encode the data.

The following code calls the to_frame() method because the OneHotEncoder expects a DataFrame object:

Here’s the Python code for cut and paste:

# Reuse the encoder and make the Series a DataFrame
hours_week_cat = encoder.fit_transform(hours_week_decile.to_frame())

The Python in Excel card gives you a preview of the DataFrame:

Voila! Magnitudes are formatted for your MBA.

As with presence features, there are many possibilities for engineering magnitude features for MBA. Once again, however, I’ve found that using the combination of rank() and qcut() is what I’ve used most often in my analyses.

I hope this tutorial has gotten you excited about the possibilities of using MBA in your own work. As I mentioned in a previous tutorial, I’ve found over the years that insights born of MBA resonate very well with business stakeholders.

👉 Ready to partner with AI to do market basket analysis?

That’s it for this week.

Next week’s newsletter will start a new tutorial series on building better forecasts using Microsoft Excel.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

• Part 1: Introduction

• Part 2: Communicating Results

• Part 3: Python Code

As the best way to learn DIY data science skills is through practice, I highly recommend downloading the data from the newsletter’s GitHub repository and writing the code.

Subscribe now

Market Basket Analysis Mines Your Data

Part 3 of this tutorial series ended with 225 association rules. When you look at these rules, one thing becomes very clear.

MBA is very much a data mining technique that discovers all manner of new and interesting patterns in your data.

This is why grocery store data is a classic example for learning MBA. It’s easy to imagine a grocery store chain looking for new ways to boost revenue, such as creating endcaps in-store or offering coupons.

However, I need to be crystal clear on the following point.

It’s not only retail businesses (e.g., grocery stores) that benefit from mining data for new and interesting patterns using market basket analysis. It’s applicable in healthcare, government, non-profit, and all for-profit industries.

**BUT**

You’re not limited to using MBA just to mine these patterns. You can also use MBA for targeted analysis.

Specifically, you can search any mined association rules for a specific consequent (i.e., the right side of an association rule). While I will again use grocery store data as an example in this tutorial, the general pattern applies to any data you can think of.

BTW - The next tutorial will discuss general feature engineering patterns for MBA that can be applied to any domain.

Targeted Association Rules

While the correct term is consequent, I use a more approachable term when using MBA and explaining the results to my business stakeholders - outcome of interest.

Let’s say our grocery store bought a large quantity of sausage at a low price. To ensure this purchase is profitable, we need to move the product quickly, as our store’s shelf space is limited.

We can use MBA to help us identify opportunities to make this happen by making sausage purchases as the outcome of interest. In this example, the association rule takes the following general form:

• {antecedent} → {sausage}

The technical term for the left side of an association rule is antecedent. At this stage of the analysis, we don’t care about the exact antecedents. The goal is to mine the rules, evaluate them, and then make a decision.

NOTE - For brevity, I’m going to assume you’ve completed Part 3 of this tutorial, so I don’t have to repeat the code here.

The following code demonstrates getting only the association rules for a targeted outcome of interest using Python in Excel:

And here’s the Python code that you can cut and paste:

# Create a mask (i.e., filter) for the outcome of interest
targeted_mask = rules['consequents'] == {'sausage'}

# Filter the rules
targeted_rules = rules[targeted_mask]
targeted_rules

While you can use any Python technology (e.g., Jupyter Notebook) for this tutorial series, many professionals new to analytics find Excel much less intimidating.

Getting a large number of rules for the outcome of interest is common when using MBA. The following code sorts the list in descending order based on the lift metric and converts the rules to strings for ease of reading:

And the code for copy and paste:

# Sort the rules for the outcome of interest
targeted_rules.sort_values(['lift', 'confidence', 'support'], 
                            ascending = False)

# Convert rules to strings for ease of viewing
targeted_rules["antecedents"] = targeted_rules["antecedents"].apply(lambda x: ", ".join(sorted(x)))
targeted_rules["consequents"] = targeted_rules["consequents"].apply(lambda x: ", ".join(sorted(x)))
targeted_rules

When using Python in Excel, you can change the Python Output to write a DataFrame to the worksheet like so:

This makes reading the association rules much easier:

Evaluating Association Rules

When evaluating association rules, here’s the process to follow:

1. Prioritize lift.

2. Look for high levels of confidence.

3. Confirm there’s adequate support.

These metrics were covered in detail in Part 2. Here’s a summary:

• ift tells you how much more likely the outcome of interest is given the antecedent.

• Confidence gives you the reliability of the association rule.

• Support tells you how frequently the itemset occurs.

Breaking this down for the first rule:

1. Customers who purchase whole milk and yogurt together are 2.18 times more likely to buy sausage in the same transaction.

2. 13.17% of Customers who purchase whole milk and yogurt together also purchase sausage.

3. The itemset of {whole milk, yogurt, sausage} represents 0.15% of all transactions.

Here’s the intuition for successfully applying MBA: It’s a balancing act between two opposing forces.

First, higher lift is typically the result of rarer purchases/interactions/behaviors/etc.

Second, higher support is usually associated (no pun intended) with lower lift.

For example, consider the following association rule from the list above:

• {soda} → {sausage}

This association rule has a support metric 4 times higher than the rule with the highest lift. However, this rule shows barely any lift, scoring only 1.014975.

This tradeoff shows how knowledge of business drivers is so essential for the successful application of MBA.

For example, recommending that the store add a refrigerated endcap featuring whole milk, yogurt, and sausage solely based on the highest lift score is unlikely to significantly increase sausage sales.

However, running ads/specials featuring beverages and sausage might produce better results, despite a lower lift, because it’s more practical from a business perspective.

This is just another example of the common theme in DIY data science.

It’s all about combining data, technique (e.g., MBA), and business knowledge. That’s how you stand out at work.

👉 Ready to learn more? Check out Part 5 here.

That’s it for this week.

Next week’s newsletter will continue the tutorial series by discussing engineering features for MBA.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

As the best way to learn DIY data science is through practice, I highly suggest you download the data from the newsletter’s GitHub repository in the MarketBasketPurchases.xlsx file and follow along.

Subscribe now

Loading the Dataset

It’s common for raw transactional data used in market basket analysis to be in the wrong format. The data used in this tutorial demonstrates. Here’s what the data looks like:

If you’re using a technology like Jupyter Notebook, you can get a CSV version of the data from GitHub and load it like this:

import pandas as pd

# Load the raw purchase data
purchases_raw = pd.read_csv('MarketBasketPurchases.csv')

However, if you’re just getting started with Python for analytics, I’m a big fan of using Python in Excel for a number of reasons:

• Python in Excel is included with Microsoft 365 subscriptions.

• Python in Excel runs in the cloud, so you don’t need to mess with a local installation.

• Python in Excel includes battle-tested libraries like pandas, mlxtend, statsmodels, and scikit-learn for powerful analytics.

• 99+% of Python in Excel code is exactly the same as using any other Python technology (e.g., Jupyter Notebooks).

Getting Started with Python in Excel

I want to be crystal clear on this point. You don’t need to use Python in Excel for any of my tutorials, since the Python code is the same.

However, with my 14+ years of analytics experience, I believe that Python in Excel is the easiest way for any professional to start building real-world analytics skills to stand out at work.

The first step in getting started with Python in Excel is to navigate to the Formulas section of the Ribbon and click on the Python Editor:

This will open the Python Editor pane from the right side of Microsoft Excel:

Clicking the Add Python cell… button creates a new kind of Excel formula that will run Python code:

Transforming the Data

As shown in the first image above, the dataset is in long format. That is, each row of data is a single item purchased by a particular customer on a particular day. The combination of MemberID and PurchaseDate identifies all the items purchased in a single transaction (or basket).

Python in Excel comes with the mlxtend library for performing market basket analysis. However, the mlxtend library expects the data to be in a wide format of complete baskets, like the following:

In the image immediately above, the first basket contains 4 items: sausage, whole milk, semi-finished bread, and yogurt.

The following Python in Excel formula loads the raw data from the Purchases table stored in the workbook and transforms the data into a list of lists:

Here’s the code so you can copy and paste it into Python in Excel:

# Load data from Excel table
purchases_raw = xl("Purchases[#All]", headers = True)

# Combine Member_number and Date into a unique transaction ID
purchases_raw = (purchases_raw
                     .assign(TransactionID = lambda df_: df_['MemberID'].astype(str) + "_" + df_['PurchaseDate'].astype(str))
                )

# Group items by transaction as a list of lists
transactions = (purchases_raw.groupby('TransactionID')
                    ['Item']
                    .agg(list)
                    .values
                    .tolist()
               )

NOTE - The first line of code is specific to Python in Excel, so you don’t need it if you’re using a technology like Jupyter Notebook.

A list of lists is a good way to store market basket analysis (MBA) data because each basket contains different counts of items. However, lists of lists don’t work with the most popular MBA algorithm (i.e., apriori) in mlxtend.

Luckily, the mlxtend library can convert the list of lists into a tabular format suitable for MBA with the apriori algorithm:

And here’s the code so you can copy and paste it into Python in Excel:

from mlxtend.preprocessing import TransactionEncoder

# One-hot encode the transactions
transaction_encoder = TransactionEncoder()

transactions_encoded = (transaction_encoder
                            .fit(transactions)
                            .transform(transactions)
                       )

transactions_df = pd.DataFrame(transactions_encoded, 
                               columns = transaction_encoder.columns_)

Clicking the disk icon in the Python Editor saves and runs the Python code. As you will see in the output from cell B3, the data is transformed into a table (i.e., a pandas DataFrame) with 14963 rows (i.e., baskets) and 167 columns (i.e., products).

BTW - In analytics, the term matrix is synonymous with table.

The DataFrame generated by the code in cell B3 is technically a sparse matrix. Here’s why:

• Every item that is purchased in any transaction gets its own column in the DataFrame.

• As most transactions will only contain a small subset of the possible items, most of the DataFrame cells have the value of False (i.e., 0).

• As the Trues (i.e., the 1s) are the information, the DataFrame is sparsely populated with information.

Sparse matrices are very common in real-world DIY data science (e.g., MBA & text analytics) and can be very large (i.e., your laptop needs a lot of memory to create and use them).

So, when using Python in Excel, it’s quite easy to create sparse matrices that are too large to be handled by the secured cloud container.

When this happens, moving to Jupyter Notebooks on your local machine is usually required. The good news is that the code is exactly the same.

However, the dataset in this tutorial can be handled just fine with Python in Excel.

Mining Frequent Itemsets

The next step is to mine the data for frequent itemsets. Intuitively, these are just collections of items that appear frequently across all transactions.

Using mlxtend, we can specify the size of the itemsets we wish to find. For example, we might find the following are frequent itemsets in the dataset:

• {UHT-milk, tropical fruit}

• {whole milk, yogurt, sausage}

• {beef, rolls/buns, citrus fruit, specialty chocolate}

As covered in the previous tutorial, we typically don’t want all of the frequent itemsets for our analyses.

We want only those frequent itemsets that meet the minimum support threshold (e.g., appearing in at least 0.1% of the transactions). Setting a minimum support threshold helps to:

• Analyze commonly occurring itemsets.

• Reduce the number of association rules produced from MBA.

Here’s the code for doing this (for brevity, I’m not going to include Python in Excel code screenshots for the rest of this tutorial):

from mlxtend.frequent_patterns import apriori

# Use a low support threshold due to the small dataset
frequent_itemsets = apriori(transactions_df, min_support = 0.001, 
                            use_colnames = True)

As covered in previous tutorials, MBA is an iterative process. Experimenting with various support thresholds optimizes your results.

Support is a critical measure for the successful use of MBA, because itemsets and association rules with low support may occur purely by chance. However, don’t make the mistake of only considering itemsets and association rules with high support.

For example, you may find that an association rule with a support of 0.1% can lead to very profitable changes to a business process or save your organization a lot of money. Unfortunately, low support thresholds often yield very large numbers of association rules to analyze.

Consequently, when using MBA, a best practice is to start with a relatively low support threshold and then iteratively increase it to reduce the number of association rules you need to analyze.

Mining Association Rules

With the frequent itemsets found, we can now mine them for any association rules. The following code uses a minimum lift threshold of 1.0. As covered in the previous tutorial, lift scores less than 1.0 mean that the right side of the association rule is less likely to happen because of what’s on the left side:

from mlxtend.frequent_patterns import association_rules

# Mine the association rules with a lift threshold
rules = association_rules(frequent_itemsets, metric = 'lift',
                          min_threshold = 1.0)

# Convert rules to strings for ease of viewing
rules["antecedents"] = rules["antecedents"].apply(lambda x: ", ".join(sorted(x)))
rules["consequents"] = rules["consequents"].apply(lambda x: ", ".join(sorted(x)))
rules.head()

Running the above code produces 240 association rules mined from the frequent itemsets using a lift threshold of 1.0. A higher lift threshold would reduce the number of rules, while a lower one would increase it. This is one aspect of MBA that requires experimentation to find an optimal value.

The above code deserves some explanation:

• The antecedent is the left side of an association rule.

• The consequent is the right side of an association rule.

Typically, it's most useful to sort the association rules in descending order based on lift:

# Get the rules in descending order by lift
sorted_rules = rules.sort_values(['lift', 'confidence', 'support'],
                                 ascending = False)
sorted_rules.head()

The sorted rules show the following are the top 4 rules:

• {whole milk, yogurt} → {sausage} with a lift of 2.1829

• {sausage} → {whole milk, yogurt} with a lift of 2.1829

• {whole milk, sausage} → {yogurt} with a lift of 1.912

• {yogurt} → {whole milk, sausage} with a lift of 1.912

Consider the first rule. The lift metric shows that customers who purchase {whole milk, yogurt} together are 2.18 times more likely to purchase {sausage} than when purchasing {whole milk}, {yogurt}, and {sausage} in separate combinations.

This is why, in my experience, business stakeholders love MBA - they can easily understand the insights and form actionable plans based on the insights.

Now, you might have noticed that the 2nd rule is a mirror image of the first:

• {whole milk, yogurt} → {sausage}

• {sausage} → {whole milk, yogurt}

This is an expected outcome of MBA, but not particularly useful. In general, using MBA effectively means having a single item on the right side of the rules.

The following code makes this happen:

# Get rules where only 1 item is on the right side
mask = [c.count(',') == 0 for c in rules['consequents']]
rules_single_consequent = rules[mask]

# Sort the single association rules
rules_single_consequent.sort_values(['lift', 'confidence', 'support'],
                                     ascending = False)

The above code reduces the number of rules from 240 to 225 and produces rules that are much more understandable to business stakeholders.

👉 Ready to learn more? Check out Part 4 here.

That’s it for this newsletter.

Next week’s newsletter will continue the tutorial series by demonstrating how to use MBA for more targeted analyses.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More

If you want to be a successful DIY data scientist, don’t make the same mistake I made early in my data science journey:

Failing to realize that technical skills are a means to an end, not the end itself.

Market basket analysis is a prime example of this.

Like so many who get into data science, my focus in my early days was the “cool stuff”:

• Decision trees

• Random forests

• K-means clustering

• Linear/logistic regression

Don’t get me wrong, all of these are powerful tools useful to any DIY data scientist.

But you know who typically doesn’t care about how you got to the insights?

Your business stakeholders. Here’s what I had to learn the hard way.

The best analytics are those that resonate with business stakeholders and drive changes to business processes - not necessarily the “cool stuff.”

This is where market basket analysis can be so effective.

The results of market basket analysis can be summarized using metrics easily understood by business stakeholders, and the insights are usually perceived as actionable.

This week’s tutorial is all about these metrics and how to use them to communicate your insights with business stakeholders.

Subscribe now

Setting Stakeholder Expectations

We first need to discuss a critical aspect of a successful market basket analysis (MBA): setting stakeholder expectations. I will teach you exactly how I set expectations with my clients in this section.

Like most analytics techniques you will use as a DIY data scientist, MBA mines your data for interesting historical patterns.

In the case of MBA, these patterns take the form of association rules like the following:

• If a patient has condition A, they’re more likely to develop condition B.

• If someone buys a printer, they often buy ink.

• People who buy peanut butter often buy jelly.

It’s critical to understand what these association rules actually mean. While there are mathematical definitions for this, all we need is an intuitive understanding.

Take the second bullet above as an example. Using natural language, we could explain this association rule like so:

“We mined our historical data and found that customers who buy a printer also tend to buy ink as part of the same transaction.

However, it’s important to note that not every customer buys printers and ink in the same transaction.

Lastly, these insights are based on historical data. There’s no guarantee that this behavior will continue in the future.”

The above summarizes that association rules are not causal relationships. Association rules embody likely outcomes based on your historical data.

Continuing the example, the following are possible customer behaviors:

1. The customer buys a printer and ink in the same transaction.

2. The customer buys only a printer.

What the association rule tells you is that, historically, #1 is more likely than #2.

While every business stakeholder dreams of a causal relationship (e.g., for every ad dollar we spend, we make $1.5 dollars in sales no matter what), these are exceedingly rare in real-world analytics.

Instead, data-driven organizations mine their data for associations that can be exploited for economic gain (e.g., more sales or lower costs).

MBA is a powerful tool for mining these associations.

An Example Dataset

Before we dive into the MBA metrics for explaining insights to business stakeholders, we need an example dataset:

• {Bread, Milk}

• {Bread, Diapers, Beer, Eggs}

• {Milk, Diapers, Beer, Cola}

• {Bread, Milk, Diapers, Beer}

• {Bread, Milk, Diapers, Cola}

As mentioned in Part 1 of this tutorial series, MBA works with any kind of data. For example, I’ve successfully used MBA with clients in marketing and product management. However, I will use a grocery store example as it’s a classic way to teach MBA.

As was also mentioned in Part 1, quantities don’t matter. We're looking only for the presence of items in the transaction.

This means we can represent the above dataset using a pandas DataFrame:

How Frequent is the Pattern?

The first MBA metric is support. The support metric tells you how frequently an itemset appears in the dataset.

Using our example data, the following is an itemset that appears in 4 out of 5 transactions:

• {Milk}

The support for this itemset is 4 / 5 = 0.8 or 80%.

Now, consider this itemset:

• {Milk, Diapers, Beer}

The support for this itemset is 2 / 5 = 0.4 or 40%.

Support is a critical measure for the successful use of an MBA, because itemsets and association rules with low support may occur purely by chance.

However, don’t make the mistake of only considering itemsets and association rules with high support (i.e., using a high support threshold).

For example, you may find that an association rule with a support of 10% can lead to very profitable changes to a business process.

Unfortunately, low support thresholds often yield very large numbers of association rules to analyze.

Consequently, when using MBA, a best practice is to start with a relatively low support threshold (e.g., 5%) and then iteratively increase it to reduce the number of association rules you need to analyze.

What’s the Reliability of the Association?

The second MBA metric is confidence. The confidence metric gives you an estimate of the reliability of an association rule given your historical data.

Consider the following association rule:

• {Milk, Diapers} → {Beer}

In natural language, the above rule can be explained to a stakeholder as follows:

“Customers who purchase Milk and Diapers also tend to purchase Beer in the same transaction.”

The confidence metric is calculated by finding the support for all items in the rule:

• The support for {Milk, Diapers, Beer} is 2 / 5 = 0.4 or 40%

Divided by the support for the left side of the rule:

• The support for {Milk, Diapers} is 2 /3 = 0.67 or 67%

Therefore, the confidence for this rule is 0.4 / 0.67 = 0.597 or 59.7%

Here’s how to combine support and confidence using natural language for your business stakeholders:

“About 2/3 of our transactions contain both Milk and Diapers. Of these transactions, 59.7% also contain Beer.”

The first sentence explains the support (or frequency) of the left side of the rule. The second sentence explains the confidence (or conditional frequency) of the right side of the rule.

There’s an urban legend that a grocery store chain uncovered an association rule between Diapers and Beer. In response, the chain created a display in its stores containing Diapers and Beer to increase sales.

While certainly an amusing story, it does underline how MBA can be understood and used by business stakeholders to improve processes.

The Magic of Lift

The third MBA metric is lift. The lift metric addresses a shortcoming of relying only on the support and confidence metrics.

If I’m being completely honest, the lift metric is so magical, it’s usually the primary way I communicate analysis results with my clients.

The problem with the confidence metric is that it doesn’t account for how often the right-hand side of an association rule appears in the dataset.

For example, consider this association rule:

• {Milk, Diapers} → {Cola}

Let’s calculate the confidence for this association rule:

• The support for {Milk, Diapers, Cola} is 2 / 5 = 0.4

• The support for {Milk, Diapers} is 2 / 3 = 0.67

• The confidence is 0.4 / 0.67 = 0.597

So, this association rule looks very similar to the previous one.

**BUT**

Cola purchases are rarer compared to Beer purchases. This is where the lift metric becomes useful.

The lift metric is calculated by dividing the association rule’s confidence by the support for the right side of the rule:

• The support for {Cola} is 2 / 5 = 0.4

• So, the lift for the association rule is 0.597 / 0.4 = 1.49

Now, compare this to the lift for the first association rule:

• Support for {Beer} is 3 / 5 = 0.6

• Lift is 0.597 / 0.6 = 0.995

And here’s where the magic of the lift metric comes into play when explaining your MBA insights to business stakeholders:

“The analysis shows that customers who purchase Milk and Diapers together are 1.5 times more likely to purchase Cola compared to purchasing items independently.

Additionally, the analysis shows that customers who purchase Milk and Diapers together are less likely to purchase Beer than those who purchase items independently.

Based on the analysis, we could experiment with providing coupons for Cola to customers who regularly purchase Milk and Diapers together.”

Now that’s what I call using data to drive impact!

👉 Ready to learn more? Check out Part 3 here.

That’s it for this newsletter.

Next week’s newsletter will continue the tutorial series on market basket analysis and teach you how to perform MBA using Python code.

Stay healthy and happy data sleuthing!

👩‍🏫 Ready to Learn More Analytics Skills?

My paid subscribers have access to exclusive monthly live crash courses that include:

• PDFs of all slides.

• Excel workbooks, code, and data.

• Recordings so you can learn on your schedule.

Here are some examples of my live crash courses:

Learn More