Neyman-Pearson Classification Methods for Prioritizing Important Categories and Enhancing Robustness to Group-Wise Sample Bias

Addressing error asymmetry and sample bias in classification tasks is crucial, particularly in biomedical applications. Error asymmetry refers to the differing importance of classification errors, e.g., failing to identify a disease-positive case (type I error) can have more severe consequences than misclassifying a negative case (type II error). Group-wise sample bias occurs when certain groups within the dataset are underrepresented or overrepresented, resulting in reduced classification accuracy for underrepresented groups. This project introduces the NPRG (Neyman-Pearson Classification Robust to Groupwise Bias) paradigm to address the challenges of error asymmetry and sample bias in binary classification problems involving multiple groups. This framework aims to control population-level type I errors conditional on the groups and manage maximum type II error disparities. We will derive the oracle NPRG classifier and develop an NPRG umbrella algorithm that adapts to various scoring-type classification methods to achieve these high probability controls in practice. Additionally, the project will extend the NPRG framework to multi-class classification with a hierarchical structure that ranks classes based on their importance. This extension introduces the H-NPRG (Hierarchical NPRG) paradigm, which aims to control both group-conditional under-classification errors (misclassifying cases into less critical classes) and disparities in group-conditional over-classification errors (misclassifying cases into more critical classes). A corresponding H-NPRG umbrella algorithm will be developed to ensure high probability control. In conjunction with this multi-class classification problem, we propose a prediction-based feature ranking criterion, the HNPC (Hierarchical Neyman-Pearson Criterion), which delivers consistent sample-level rankings and remains robust against class-wise sampling bias. The project will develop theories to validate the high probability controls of the proposed umbrella algorithms and the consistency of HNPC. We will also conduct numerical experiments to assess their performance. Additionally, the project will include three case studies (kidney allograft prediction, Alzheimer’s disease stage classification, and COVID- 19 severity classification) to demonstrate the broad applicability and effectiveness of our framework across diverse biomedical contexts.