Problem
Spotify data scientists group users by usage behavior so that researchers and product teams can cater to specific groups. The input is often new or loosely defined: a first survey, a new data endpoint, a table with thousands of columns. Classical approaches give hundreds of summary tables, do not scale and cannot explain the clusters. PCA followed by k-means gives round, structureless blobs, and k-means splits natural groups arbitrarily.
Example from the article: the author demonstrates the method on the first 10,000 MNIST handwritten digits (784 dimensions), where the true groups are known. The article gives no Spotify user numbers.
Solution in brief
Reduce the data with UMAP, cluster it with HDBSCAN, then "zoom in": repeat UMAP and HDBSCAN on the points of each cluster to find finer sub-clusters. Train an XGBoost one-versus-all model per cluster on the original features and read SHAP plots to see what defines each cluster. The same model assigns new users to clusters without re-running UMAP and HDBSCAN.
Questions this case answers
- How to discover stable, explainable groups of entities in high-dimensional data when classic dimensionality reduction returns a structureless blob
- How to uncover nested structure inside clusters found at the first pass of an unsupervised pipeline
- How to find stable user groups in data with hundreds or thousands of behavioral variables?
- How to find sub-groups inside a cluster that looks like one uniform blob?
Tools and methods
- UMAP — nonlinear dimensionality reduction; replaces PCA.
- HDBSCAN — density-based clustering with noise points; replaces k-means.
- Recursive zoom-in — rerun UMAP and HDBSCAN on one cluster's original points to reveal sub-clusters.
- XGBoost — one-versus-all classifier per cluster, trained on the original features.
- SHAP — Shapley-value summary plots that explain which features define each cluster.
Google or a one-time email code. No password.