FoodMask : Real-time food instance counting, segmentation and recognition

Research output: Journal Publications and ReviewsRGC 21 - Publication in refereed journalpeer-review

View graph of relations

Related Research Unit(s)


Original languageEnglish
Article number110017
Journal / PublicationPattern Recognition
Online published4 Oct 2023
Publication statusPublished - Feb 2024


Food computing has long been studied and deployed to several applications. Understanding a food image at the instance level, including recognition, counting and segmentation, is essential to quantifying nutrition and calorie consumption. Nevertheless, existing techniques are limited to either category-specific instance detection, which does not reflect precisely the instance size at the pixel level, or category-agnostic instance segmentation, which is insufficient for dish recognition. This paper presents a compact and fast multi-task network, namely FoodMask, for clustering-based food instance counting, segmentation and recognition. The network learns a semantic space simultaneously encoding food category distribution and instance height at pixel basis. While the former value addresses instance recognition, the latter value provides prior knowledge for instance extraction. Besides, we integrate into the semantic space a pathway for class-specific counting. With these three outputs, we propose a clustering algorithm to segment and recognize food instances at a real-time speed. Empirical studies are made on three large-scale food datasets, including Mixed Dishes, UECFoodPixComp and FoodSeg103, which cover Western, Chinese, Japanese and Indian cuisines. The proposed networks outperform benchmarks in both terms of instance map quality and speed efficiency. © 2023 Elsevier Ltd.

Research Area(s)

  • Food counting, Food instance segmentation, Food recognition