Improving Query and Assessment Quality in Text-Based Interactive Video Retrieval Evaluation

Werner Bailer; Rahel Arnold; Vera Benz; Davide Alessandro Coccomini; Anastasios Gkagkas; Gylfi Þór Guðmundsson; Silvan Heller; Björn Þór Jónsson; Jakub Lokoc; Nicola Messina; Nick Pantelidis; Jiaxin Wu

doi:10.1145/3591106.3592281

Improving Query and Assessment Quality in Text-Based Interactive Video Retrieval Evaluation

Werner Bailer
, Rahel Arnold
, Vera Benz
, Davide Alessandro Coccomini
, Anastasios Gkagkas
, Gylfi Þór Guðmundsson
, Silvan Heller
, Björn Þór Jónsson
, Jakub Lokoc
, Nicola Messina
, Nick Pantelidis
, Jiaxin Wu

Department of Computer Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

4 Citations (Scopus)

Abstract

Different task interpretations are a highly undesired element in interactive video retrieval evaluations. When a participating team focuses partially on a wrong goal, the evaluation results might become partially misleading. In this paper, we propose a process for refining known-item and open-set type queries, and preparing the assessors that judge the correctness of submissions to open-set queries. Our findings from recent years reveal that a proper methodology can lead to objective query quality improvements and subjective participant satisfaction with query clarity. © 2023 ACM.

Original language	English
Title of host publication	ICMR '23 - Proceedings of the 2023 ACM International Conference on Multimedia Retrieval
Editors	Ioannis (Yiannis) Kompatsiaris, Jiebo Luo, Nicu Sebe, Angela Yao, Vasileios Mezaris, Symeon Papadopoulos, Adrian Popescu, Zi (Helen) Huang
Place of Publication	New York, NY
Publisher	Association for Computing Machinery
Pages	597-601
ISBN (Print)	9798400701788
DOIs	https://doi.org/10.1145/3591106.3592281
Publication status	Published - 2023
Event	2023 ACM International Conference on Multimedia Retrieval (ICMR 2023) - Thessaloniki, Greece Duration: 12 Jun 2023 → 15 Jun 2023 https://icmr2023.org/

Publication series

Name	ICMR - Proceedings of the ACM International Conference on Multimedia Retrieval

Conference

Conference	2023 ACM International Conference on Multimedia Retrieval (ICMR 2023)
Abbreviated title	ACM ICMR 2023
Place	Greece
City	Thessaloniki
Period	12/06/23 → 15/06/23
Internet address	https://icmr2023.org/

Research Keywords

benchmarking
evaluation
quality assurance
video retrieval

Access Output

10.1145/3591106.3592281

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Bailer, W., Arnold, R., Benz, V., Coccomini, D. A., Gkagkas, A., Guðmundsson, G. Þ., Heller, S., Jónsson, B. Þ., Lokoc, J., Messina, N., Pantelidis, N., & Wu, J. (2023). Improving Query and Assessment Quality in Text-Based Interactive Video Retrieval Evaluation. In I. Kompatsiaris, J. Luo, N. Sebe, A. Yao, V. Mezaris, S. Papadopoulos, A. Popescu, & Z. Huang (Eds.), ICMR '23 - Proceedings of the 2023 ACM International Conference on Multimedia Retrieval (pp. 597-601). (ICMR - Proceedings of the ACM International Conference on Multimedia Retrieval). Association for Computing Machinery. https://doi.org/10.1145/3591106.3592281

Bailer, Werner ; Arnold, Rahel ; Benz, Vera et al. / Improving Query and Assessment Quality in Text-Based Interactive Video Retrieval Evaluation. ICMR '23 - Proceedings of the 2023 ACM International Conference on Multimedia Retrieval. editor / Ioannis (Yiannis) Kompatsiaris ; Jiebo Luo ; Nicu Sebe ; Angela Yao ; Vasileios Mezaris ; Symeon Papadopoulos ; Adrian Popescu ; Zi (Helen) Huang. New York, NY : Association for Computing Machinery, 2023. pp. 597-601 (ICMR - Proceedings of the ACM International Conference on Multimedia Retrieval).

@inproceedings{327ea35c487641dbb44556211901dc5a,

title = "Improving Query and Assessment Quality in Text-Based Interactive Video Retrieval Evaluation",

abstract = "Different task interpretations are a highly undesired element in interactive video retrieval evaluations. When a participating team focuses partially on a wrong goal, the evaluation results might become partially misleading. In this paper, we propose a process for refining known-item and open-set type queries, and preparing the assessors that judge the correctness of submissions to open-set queries. Our findings from recent years reveal that a proper methodology can lead to objective query quality improvements and subjective participant satisfaction with query clarity. {\textcopyright} 2023 ACM.",

keywords = "benchmarking, evaluation, quality assurance, video retrieval",

author = "Werner Bailer and Rahel Arnold and Vera Benz and Coccomini, \{Davide Alessandro\} and Anastasios Gkagkas and Gu{\dh}mundsson, \{Gylfi {\TH}{\'o}r\} and Silvan Heller and J{\'o}nsson, \{Bj{\"o}rn {\TH}{\'o}r\} and Jakub Lokoc and Nicola Messina and Nick Pantelidis and Jiaxin Wu",

year = "2023",

doi = "10.1145/3591106.3592281",

language = "English",

isbn = "9798400701788",

series = "ICMR - Proceedings of the ACM International Conference on Multimedia Retrieval",

publisher = "Association for Computing Machinery",

pages = "597--601",

editor = "Kompatsiaris, \{Ioannis (Yiannis)\} and Jiebo Luo and Nicu Sebe and Angela Yao and Vasileios Mezaris and Symeon Papadopoulos and Adrian Popescu and Huang, \{Zi (Helen)\}",

booktitle = "ICMR '23 - Proceedings of the 2023 ACM International Conference on Multimedia Retrieval",

address = "United States",

note = "2023 ACM International Conference on Multimedia Retrieval (ICMR 2023), ACM ICMR 2023 ; Conference date: 12-06-2023 Through 15-06-2023",

url = "https://icmr2023.org/",

}

Bailer, W, Arnold, R, Benz, V, Coccomini, DA, Gkagkas, A, Guðmundsson, GÞ, Heller, S, Jónsson, BÞ, Lokoc, J, Messina, N, Pantelidis, N & Wu, J 2023, Improving Query and Assessment Quality in Text-Based Interactive Video Retrieval Evaluation. in I Kompatsiaris, J Luo, N Sebe, A Yao, V Mezaris, S Papadopoulos, A Popescu & Z Huang (eds), ICMR '23 - Proceedings of the 2023 ACM International Conference on Multimedia Retrieval. ICMR - Proceedings of the ACM International Conference on Multimedia Retrieval, Association for Computing Machinery, New York, NY, pp. 597-601, 2023 ACM International Conference on Multimedia Retrieval (ICMR 2023), Thessaloniki, Greece, 12/06/23. https://doi.org/10.1145/3591106.3592281

Improving Query and Assessment Quality in Text-Based Interactive Video Retrieval Evaluation. / Bailer, Werner; Arnold, Rahel; Benz, Vera et al.
ICMR '23 - Proceedings of the 2023 ACM International Conference on Multimedia Retrieval. ed. / Ioannis (Yiannis) Kompatsiaris; Jiebo Luo; Nicu Sebe; Angela Yao; Vasileios Mezaris; Symeon Papadopoulos; Adrian Popescu; Zi (Helen) Huang. New York, NY: Association for Computing Machinery, 2023. p. 597-601 (ICMR - Proceedings of the ACM International Conference on Multimedia Retrieval).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Improving Query and Assessment Quality in Text-Based Interactive Video Retrieval Evaluation

AU - Bailer, Werner

AU - Arnold, Rahel

AU - Benz, Vera

AU - Coccomini, Davide Alessandro

AU - Gkagkas, Anastasios

AU - Guðmundsson, Gylfi Þór

AU - Heller, Silvan

AU - Jónsson, Björn Þór

AU - Lokoc, Jakub

AU - Messina, Nicola

AU - Pantelidis, Nick

AU - Wu, Jiaxin

PY - 2023

Y1 - 2023

N2 - Different task interpretations are a highly undesired element in interactive video retrieval evaluations. When a participating team focuses partially on a wrong goal, the evaluation results might become partially misleading. In this paper, we propose a process for refining known-item and open-set type queries, and preparing the assessors that judge the correctness of submissions to open-set queries. Our findings from recent years reveal that a proper methodology can lead to objective query quality improvements and subjective participant satisfaction with query clarity. © 2023 ACM.

AB - Different task interpretations are a highly undesired element in interactive video retrieval evaluations. When a participating team focuses partially on a wrong goal, the evaluation results might become partially misleading. In this paper, we propose a process for refining known-item and open-set type queries, and preparing the assessors that judge the correctness of submissions to open-set queries. Our findings from recent years reveal that a proper methodology can lead to objective query quality improvements and subjective participant satisfaction with query clarity. © 2023 ACM.

KW - benchmarking

KW - evaluation

KW - quality assurance

KW - video retrieval

UR - https://www.scopus.com/pages/publications/85163689792

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85163689792&origin=recordpage

U2 - 10.1145/3591106.3592281

DO - 10.1145/3591106.3592281

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9798400701788

T3 - ICMR - Proceedings of the ACM International Conference on Multimedia Retrieval

SP - 597

EP - 601

BT - ICMR '23 - Proceedings of the 2023 ACM International Conference on Multimedia Retrieval

A2 - Kompatsiaris, Ioannis (Yiannis)

A2 - Luo, Jiebo

A2 - Sebe, Nicu

A2 - Yao, Angela

A2 - Mezaris, Vasileios

A2 - Papadopoulos, Symeon

A2 - Popescu, Adrian

A2 - Huang, Zi (Helen)

PB - Association for Computing Machinery

CY - New York, NY

T2 - 2023 ACM International Conference on Multimedia Retrieval (ICMR 2023)

Y2 - 12 June 2023 through 15 June 2023

ER -

Bailer W, Arnold R, Benz V, Coccomini DA, Gkagkas A, Guðmundsson GÞ et al. Improving Query and Assessment Quality in Text-Based Interactive Video Retrieval Evaluation. In Kompatsiaris I, Luo J, Sebe N, Yao A, Mezaris V, Papadopoulos S, Popescu A, Huang Z, editors, ICMR '23 - Proceedings of the 2023 ACM International Conference on Multimedia Retrieval. New York, NY: Association for Computing Machinery. 2023. p. 597-601. (ICMR - Proceedings of the ACM International Conference on Multimedia Retrieval). Epub 2023 Jun 12. doi: 10.1145/3591106.3592281

Improving Query and Assessment Quality in Text-Based Interactive Video Retrieval Evaluation

Abstract

Publication series

Conference

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this