Problem
Physicists at the Lebedev Physical Institute (PAMIR experiment) study "families": groups of particles born in cosmic-ray interactions and recorded by detectors. Particles born in one last interaction come from the same height, and the task is to split each family into such groups. The number of groups is unknown, the groups are not spherical, and distance must account for particle energy. Later conclusions about the primary particle depend on these clusters.
Example from the article: in the model family No. 518, agglomerative clustering with a threshold at the 80th percentile of the ER distribution (71) finds three clusters.
Solution in brief
Seven algorithms that do not need the number of clusters were compared on model data, where the true height of each particle is known. Quality is scored by four coefficients built from the "dominant height" of each cluster, and an overall metric combines them. The best algorithm is then checked for stability on real data.
Questions this case answers
- How to choose a clustering algorithm when the number of clusters is unknown, clusters are non-spherical and a custom distance is needed, using a labeled reference set?
- How to verify that an algorithm selected on a labeled model dataset transfers to real unlabeled data?
- How to choose a clustering algorithm when the number of clusters is unknown and clusters have irregular shape?
- How to compare clustering algorithms on labeled data when no single standard metric fits the goal?
Tools and methods
- Agglomerative clustering — merges the closest clusters (average linkage); the stop threshold is a percentile of the ER distribution.
- DBSCAN — density-based clustering; the neighborhood radius is a percentile of the ER distribution.
- PAMIR algorithm — merges the closest pair of points with energy-weighted coordinates until the distance exceeds a boundary of 48.
- Statistical tests — Kolmogorov-Smirnov, Mann-Whitney, Welch's t-test and Levene's test check that model and real data are alike.
Google or a one-time email code. No password.