Benchmarking for content-based visual information search

The importance of the visual information search problem has given rise to a large number of systems and prototypes being built to perform such search. While different systems clearly have their particular strengths, they tend to use different collections to highlight the advantages of their algorithms. Consequently, a degree of bias may exist, and it also makes it difficult to make comparisons concerning the relative superiority of different algorithms. In order for the field of visual information search to make further progress, a need therefore exists for a standardised benchmark suite to be developed. By having a uniform measure of search performance, research progress can be more easily recognised and charted, and the resultant synergy will be essential to further development of the field. This paper presents concrete proposals concerning the development of such a benchmark, and by adopting an extensible framework, it is able to cater for a wide variety of applications paradigms and to lend itself to incremental refinement.