Skip to main navigation Skip to search Skip to main content

Empirical Insights of Test Selection Metrics under Multiple Testing Objectives and Distribution Shifts

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

3 Downloads (CityUHK Scholars)

Abstract

Deep learning (DL)-based systems can exhibit unexpected behavior when exposed to out-of-distribution (OOD) scenarios, posing serious risks in safety-critical domains such as malware detection and autonomous driving. This underscores the importance of thoroughly testing such systems before deployment. To this end, researchers have proposed a wide range of test selection metrics designed to effectively select inputs. However, prior evaluations of metrics reveal three key limitations: (1) narrow testing objectives, for example, many studies assess metrics only for fault detection, leaving their effectiveness for performance estimation unclear; (2) limited coverage of OOD scenarios, with natural and label shifts are rarely considered; (3) Biased dataset selection, where most work focuses on image data while other modalities remain underexplored. Consequently, a unified benchmark that examines how these metrics perform under multiple testing objectives, diverse OOD scenarios, and different data modalities is still lacking. This leaves practitioners uncertain about which test selection metrics are most suitable for their specific objectives and contexts. To address this gap, we conduct an extensive empirical study of 15 existing metrics, evaluating them under three testing objectives (fault detection, performance estimation, and retraining guidance), five types of OOD scenarios (corrupted, adversarial, temporal, natural, and label shifts), three data modalities (image, text, and Android packages), and 13 DL models. In total, our study encompasses 1,640 experimental scenarios, offering a comprehensive evaluation and statistical analysis. ©2026 Copyright held by the owner/author(s).
Original languageEnglish
Title of host publicationProceedings of the ACM on Software Engineering
Place of PublicationUnited States
PublisherACM
Pages1288-1311
Volume3
EditionFSE
DOIs
Publication statusPublished - Jul 2026
Event34th ACM International Conference on the Foundations of Software Engineering (FSE 2026) - Montreal, Canada
Duration: 5 Jul 2026 → 9 Jul 2026

Conference

Conference34th ACM International Conference on the Foundations of Software Engineering (FSE 2026)
PlaceCanada
CityMontreal
Period5/07/26 → 9/07/26

Bibliographical note

Research Unit(s) information for this publication is provided by the author(s) concerned.

Funding

This work is supported by the National Natural Science Foundation of China under Grant 62502550, the Research Grants Council of Hong Kong (9229029), funds from CityUHK (9229192, 6000871), Shenzhen Science and Technology Program (KJZD20240903095700001), JST CRONOS Grant (No. JPMJCS24K8), and JSPS KAKENHI Grant (No. JP21H04877, No. JP23H03372, and No. JP24K02920), Canada CIFAR AI Chairs Program, the Natural Sciences and Engineering Research Council of Canada.

Research Keywords

  • Deep Learning Testing
  • Test Selection Metrics
  • Empirical Study

Publisher's Copyright Statement

  • This full text is made available under CC-BY 4.0. https://creativecommons.org/licenses/by/4.0/

RGC Funding Information

  • RGC-funded

Fingerprint

Dive into the research topics of 'Empirical Insights of Test Selection Metrics under Multiple Testing Objectives and Distribution Shifts'. Together they form a unique fingerprint.

Cite this