Caption-Aided Product Detection via Collaborative Pseudo-Label Harmonization

Product detection, which aims to localize products of interest in the advertising images, helps advance many potential Ecommerce applications like product retrieval and recommendation. However, labeling a massive number of fine-grained product categories and accurate product boxes is costly and especially not practical since products are ever-changing on E-commerce websites. In this work, we step forward to train a fine-grained product detector solely supervised by the advertising captions, which are naturally available but often severely flawed and noisy. To reformulate the weakly supervised detection research into a real-world setting, we introduce a large-scale benchmark, named CapProduct, where more than 80,000 product image-caption pairs are collected from E-commerce websites. The fine-grained nature of products and noisy captions in CapProduct make it intractable to excavate valid category labels to train a weakly supervised object detector. To tackle this challenge, we propose a Collaborative Pseudo-Label Harmonization (CoPLH) framework that harmonizes self-mined pseudo labels via modeling the global co-occurrence relationships of products. We construct a collaborative co-occurrence graph based on all training samples to improve the reliability of caption-predicted pseudo-labels as well as benefit the self-training procedure in a weakly supervised setting. Extensive experiments on the CapProduct dataset demonstrate the effectiveness and the superiority of the proposed CoPLH over the state-of-the-art baselines. © 2022 IEEE.