Cluster Analysis in Finance: Group Assets Without Using the Outcome Label
Table of Contents
A portfolio can hold several funds and still make the same bet on the market. State Street, for example, says both SPY and SPYM are designed to track the S&P 500. You can spot that overlap from the funds’ mandates. When the shared exposure is less obvious, cluster analysis can help you find holdings worth examining together. It groups assets by similarities you can measure, such as how their returns moved over the same dates. [1]
The result depends on what you ask the method to treat as similar. The same four fictional assets will form different groups when we change the distance measure, the merging rule, or the return window. That makes clusters useful for flagging possible concentration, provided you check whether the relationships persist before relying on them.
What Does Cluster Analysis Group?
Cluster analysis is an unsupervised learning method. Here, the objects being grouped are assets, and the inputs are measurements you know at the time of grouping. There is no outcome label such as “outperformed next quarter.” The algorithm assigns group labels after it measures similarity.
Start with the portfolio question. To find holdings that tend to rise and fall together, compare their returns across the same dates. To find holdings with similar trading characteristics, use features such as volatility and median daily dollar volume. To find similar economic exposures, compare estimated sensitivities to the market, interest rates or other factors. A volatility feature tells you about the size of moves; by itself it cannot tell you whether two assets move together.
Historical returns up to the decision date can be inputs. Future returns cannot be used to form groups you claim you could have traded earlier. Looking at later outcomes is useful once the rules are frozen. The guide to alternative data in investing shows why a file available at the decision cutoff can lead to a different decision than a version revised later. [2]
How Feature Scaling Changes Which Assets Are Closest
Suppose you want groups with similar volatility and liquidity. In a separate fictional example, three funds have these two features, each measured using information available at the grouping date:
| Fund | Annualized volatility | Median daily dollar volume |
|---|---|---|
| A | 10% | $20 million |
| B | 11% | $50 million |
| C | 15% | $22 million |
Unscaled Euclidean distance takes the square root of the sum of squared differences. Using volatility in percentage points and volume in millions, A to B is √(12 + 302) = 30.02, A to C is √(52 + 22) = 5.39, and B to C is √(42 + 282) = 28.28. A and C look closest. That answer depends on arbitrary units: expressing volume in dollars instead of millions would give it even more influence.
Keep the same distance rule but measure differences in units of 2 percentage points for volatility and $20 million for volume. Now A to B is √(0.52 + 1.52) = 1.58, A to C is √(2.52 + 0.12) = 2.50, and B to C is √(22 + 1.42) = 2.44. A and B are closest. These scale choices are illustrative. A common alternative is to standardize each feature by subtracting its formation-sample mean and dividing by its formation-sample standard deviation. That can stop a large numerical scale from dominating, but does not make all features equally informative. Outliers and several near-duplicate features can still distort the comparison. [2]
Turning Return Correlation Into Distance
If the question is whether holdings move together, use Pearson correlation between return series. Calculate returns over the same dates, at the same frequency and in the currency relevant to the portfolio. In practice, check that the series treat distributions consistently, and deal explicitly with missing observations. A missing return is different from a return of zero. You also need variation in both series to define the correlation.
Distances are zero for perfect positive correlation, about 1.414 for zero correlation, and 2 for perfect negative correlation. A large negative correlation therefore puts two assets far apart under this rule. That makes sense if you want to identify holdings that move in the same direction. It would be a different question to group together all assets with strong relationships regardless of sign. Check your software’s definition: a function called “correlation distance” can also mean 1 − ρ, which is half the squared distance used here. Averaging that quantity can change the order of average-linkage merges. [3]
Hierarchical Clustering vs. K-Means
| Method | How it groups assets | What to watch for |
|---|---|---|
| Agglomerative hierarchical | Start with each asset on its own and repeatedly merge the closest groups. The merge tree, called a dendrogram, can be cut to give a chosen group count or distance threshold. | The linkage rule defines how far apart two groups are. Early merges cannot be undone within that tree. |
| K-means | Choose the number of groups k, assign each asset to the nearest center, and update each center to the mean of its members’ feature vectors. | It seeks to minimize squared Euclidean distances to centers. Different starts can produce different solutions, and elongated groups may be hard to recover. |
Single linkage compares the nearest pair of assets across two groups. Complete linkage compares the farthest pair. Average linkage takes the mean across all pairs connecting the two groups. Ward linkage instead minimizes the increase in within-group dispersion and requires Euclidean geometry. Single, complete and average linkage can work directly with the pair distances in the example below. Ordinary k-means expects coordinates for each asset; feeding it a distance matrix as though it were the original features changes what it clusters. [4]
Worked Example: Four Assets, Two Different Groupings
Start with four fictional assets, A through D. Periods 1–4 are the formation window; the same four return dates are used for every pair. We measure the tendency to move together with signed correlation distance and stop merging when two groups remain. We’ll use the same inputs first with single linkage and then with complete linkage.
| Asset | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| A | 2 | −2 | 0 | 0 |
| B | 6 | −6 | 3 | −3 |
| C | 2 | −2 | 3 | −3 |
| D | 0 | 0 | 2 | −2 |
Each series happens to average zero. For A and B, the sum of matched return products is 2 × 6 + (−2) × (−6) = 24; the last two terms are zero. The sums of squared returns are 8 for A and 90 for B. Their correlation is 24 / √(8 × 90) ≈ 0.8944, so their distance is √[2 × (1 − 0.8944)] ≈ 0.460.
| Pair | Signed correlation distance | Raw Euclidean distance (percentage points) |
|---|---|---|
| A–B | 0.460 | 7.071 |
| A–C | 0.944 | 4.243 |
| A–D | 1.414 | 4.000 |
| B–C | 0.513 | 5.657 |
| B–D | 1.051 | 8.602 |
| C–D | 0.580 | 3.162 |
A and B merge first because 0.460 is the smallest correlation distance. With single linkage, the B–C distance of 0.513 lets C join them before C can join D at 0.580. The two groups are A, B, C and D. With complete linkage, C’s distance to the A–B group is the larger of 0.944 and 0.513, so C and D merge next at 0.580. The two groups are A, B and C, D. A single close connection can pull in C under one rule even when C is relatively far from A.
Now hold complete linkage fixed and change the distance. Compare the raw return vectors by Euclidean distance without rescaling each one. A–B becomes √(42 + 42 + 32 + 32) = 7.071 percentage points. C–D is √(22 + 22 + 12 + 12) = 3.162, the closest pair. A then joins C–D because its farthest distance to a member is 4.243, below both A–B at 7.071 and B’s farthest distance to C–D at 8.602. Now the groups are A, C, D and B. B’s larger swings matter to raw distance; correlation distance compares the pattern after removing magnitude.
How to Choose k and Check Stability
Pick a short list of plausible cluster counts before looking at future returns. Two groups made the example easy to follow; they were not estimated to be optimal. For a larger universe, decide how many groups are useful for the portfolio review and compare neighboring choices. Check group sizes and economic exposures. A silhouette score compares how close each asset is to its own group with how close it is to the nearest alternative group on average. It can help spot a poorly separated grouping; a high average can also reflect a lone outlier. Look at the members and the separation, then see how sensitive the result is to the inputs. [5] [6]
To check stability, take a later non-overlapping block of returns with the same frequency and length. Keep the feature definition, distance, linkage and k fixed, and compare the partitions for the same holdings. You can also track the later correlations of the original close pairs. A re-fit on the later block is a diagnostic; you could not have used its grouping for a decision at the end of the formation window.
| Asset | 5 | 6 | 7 | 8 |
|---|---|---|---|---|
| A | 2 | −2 | 0 | 0 |
| B | 4 | −4 | 6 | −6 |
| C | 2 | −2 | 1 | −1 |
| D | 0 | 0 | 2 | −2 |
With the original signed distance and complete linkage, A–C now merges first: its correlation is 8 / √(8 × 10) ≈ 0.8944, for a distance of 0.460. B–D has a distance of 0.580, below B’s complete-linkage distance to A–C (0.944) and D’s distance to A–C (1.414). The new co-members are A–C and B–D. None of the four pairs that clustered together in either window did so in both: the overlap is 0 / 4. Comparing pairs avoids confusing a membership change with a swap in arbitrary cluster numbers. Four observations per window teach the mechanics; this example does not establish whether a real market relationship is persistent.
How to Use Clusters in a Portfolio Review
Start by totaling how much capital sits in each group. A large share in one group may warrant a closer look at its holdings, benchmarks or factor exposures. If two holdings serve the same role, compare them on dimensions the clustering left out, such as trading costs and liquidity. Different groups can still have substantial exposure to the same risk. Portfolio diversification depends on weights and covariances; correlations alone omit the size of each asset’s swings. Hierarchical risk parity is an example of a separate allocation method that uses a tree and risk estimates to assign weights. Ordinary clustering produces groups, not weights. [7] [8]
If you want to trade a rule based on clusters, test the rule itself. At each historical decision date, build the universe using what was known then, estimate the inputs using data through that date, and lock the weights before the next return period begins. Repeat this walk-forward over different market conditions. Compare out-of-sample volatility, drawdowns, turnover and results after trading costs with a simple benchmark, such as equal weighting under the same universe and trading assumptions. Reserve fresh data for a final test if you tune the rule after seeing earlier validation results. [2]
Common Mistakes
- Clustering price levels to measure co-movement. Compare consistently measured returns when that is the question; two price series can trend upward without matching period-to-period moves.
- Letting future data leak into an unsupervised fit. Do not fit a scale on the full history or select features because they happen to separate next-period winners in the test data. A method can leak information even without a target column.
- Treating a cluster name as a cause. A group that resembles a sector can also reflect a wider market exposure. Check the underlying holdings or factor sensitivities before naming the driver.
What a Return Cluster Can Miss
One pooled correlation describes average linear co-movement over a particular sample. It can hide a relationship that strengthens during losses. Longin and Solnik found evidence of stronger correlation in bear markets in their study of monthly equity-index returns for five countries from 1959 to 1996. That historical finding is a reason to examine downside scenarios separately, not a rule that correlations rise in every downturn. [9]
A short window can make a group sensitive to a few observations. A longer window can blend periods with different market behavior. Try alternative predeclared windows and check whether an important group disappears. If the result changes readily, use it as a prompt to investigate exposures; it is a weak basis for a hard allocation constraint.
Frequently Asked Questions
Sources
- State Street: S&P 500 index exposure (SPY and SPYM benchmark objectives).
- scikit-learn: StandardScaler and Data leakage.
- Banco Central do Brasil Working Paper 351, section 2.2; SciPy correlation distance (different convention).
- scikit-learn: Clustering, sections on k-means and hierarchical clustering.
- Peter J. Rousseeuw, Silhouettes: A Graphical Aid to the Interpretation and Validation of Cluster Analysis (1987).
- Shai Ben-David, Ulrike von Luxburg and Dávid Pál, A Sober Look at Clustering Stability (2006).
- Harry Markowitz, Portfolio Selection (1952), pp. 80–81.
- Marcos López de Prado, Building Diversified Portfolios that Outperform Out-of-Sample (2016).
- François Longin and Bruno Solnik, Extreme Correlation of International Equity Markets (2001).
Disclaimer
This article is for educational purposes. The worked examples use hypothetical returns and do not represent investment recommendations or forecasts.
