Learning from Web Recipe-image Pairs for Food Recognition : Problem, Baselines and Performance

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

View graph of relations

Related Research Unit(s)


Original languageEnglish
Pages (from-to)1175-1185
Number of pages11
Journal / PublicationIEEE Transactions on Multimedia
Online published29 Oct 2021
Publication statusPublished - 2022


Cross-modal recipe retrieval has recently been explored for food recognition and understanding. Text-rich recipe provides not only visual content information (e.g., ingredients, dish presentation) but also procedure of food preparation (cutting and cooking styles). The paired data is leveraged to train deep models to retrieve recipes for food images. Most recipes on the Web include sample pictures as the references. The paired multimedia data is not noise-free, due to errors such as pairing of images containing partially prepared dishes with recipes. The content of recipes and food images are not always consistent due to free-style writing and preparation of food in different environments. As a consequence, the effectiveness of learning cross-modal deep models from such noisy web data is questionable. This paper conducts an empirical study to provide insights whether the features learnt with noisy pair data are resilient and could capture the modality correspondence between visual and text.

Research Area(s)

  • Context modeling, Data models, Feature extraction, Food recognition, Generative adversarial networks, Image recognition, image-to-image retrieval, image-to-recipe retrieval, Training, Visualization