Long-Tailed Multi-Label Learning: A Benchmark of Evaluation Metrics

Long-tailed multi-label learning(LTMLL) addresses multi-label classification tasks under long-tailed label distributions. Challenges like class imbalance and complex inter-label dependencies make it more difficult to learn effective feature representations. The investigation reveals that most existing studies rely on a single evaluation metric to assess model performance on LTMLL tasks. However, such a metric fails to comprehensively reflect the capability of models in modeling label correlations and achieving accurate instance-level predictions. In this paper, we propose a unified evaluation framework that incorporates multiple multi-label evaluation metrics to better capture the sensitivity to label dependencies and its overall prediction quality at the instance level. Specifically, we analyze and compare these metrics across three key dimensions,i.e., handling class imbalance, expressing label correlations, and instance-level prediction accuracy, by stratifying labels into head, medium, and tail categories. Experiments on VOC-MLT and COCO-MLT using ERM, DB-Focal, DR Loss and LMPT demonstrate that the proposed benchmark enables a more comprehensive evaluation of LTMLL models. © 2025 IEEE.