Multi-modal Cooking Workflow Construction for Food Recipes

Liang-Ming Pan; Jingjing Chen; Jianlong Wu; Shaoteng Liu; Chong-Wah Ngo; Min-Yen Kan; Yugang Jiang; Tat-Seng Chua

doi:10.1145/3394171.3413765

Multi-modal Cooking Workflow Construction for Food Recipes

Liang-Ming Pan, Jingjing Chen^*, Jianlong Wu, Shaoteng Liu, Chong-Wah Ngo, Min-Yen Kan, Yugang Jiang, Tat-Seng Chua

^*Corresponding author for this work

Department of Computer Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

26 Citations (Scopus)

Abstract

Understanding food recipe requires anticipating the implicit causal effects of cooking actions, such that the recipe can be converted into a graph describing the temporal workflow of the recipe. This is a non-trivial task that involves common-sense reasoning. However, existing efforts rely on hand-crafted features to extract the workflow graph from recipes due to the lack of large-scale labeled datasets. Moreover, they fail to utilize the cooking images, which constitute an important part of food recipes. In this paper, we build MM-ReS, the first large-scale dataset for cooking workflow construction, consisting of 9,850 recipes with human-labeled workflow graphs. Cooking steps are multi-modal, featuring both text instructions and cooking images. We then propose a neural encoder-decoder model that utilizes both visual and textual information to construct the cooking workflow, which achieved over 20% performance gain over existing hand-crafted baselines.

Original language	English
Title of host publication	MM ‘20
Subtitle of host publication	Proceedings of the 28th ACM International Conference on Multimedia
Publisher	Association for Computing Machinery
Pages	1132-1141
ISBN (Print)	9781450379885
DOIs	https://doi.org/10.1145/3394171.3413765
Publication status	Published - Oct 2020
Event	28th ACM International Conference on Multimedia (MM 2020) - Virtual, Seattle, United States Duration: 12 Oct 2020 → 16 Oct 2020 https://2020.acmmm.org/

Publication series

Name	MM - Proceedings of the ACM International Conference on Multimedia

Conference

Conference	28th ACM International Conference on Multimedia (MM 2020)
Abbreviated title	ACM Multimedia 2020
Country/Territory	United States
City	Seattle
Period	12/10/20 → 16/10/20
Internet address	https://2020.acmmm.org/

Bibliographical note

Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s).

Research Keywords

cause-and-effect reasoning
cooking workflow
deep learning
food recipes
mm-res dataset
multi-modal fusion

Access Output

10.1145/3394171.3413765

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Pan, L.-M., Chen, J., Wu, J., Liu, S., Ngo, C.-W., Kan, M.-Y., Jiang, Y., & Chua, T.-S. (2020). Multi-modal Cooking Workflow Construction for Food Recipes. In MM ‘20: Proceedings of the 28th ACM International Conference on Multimedia (pp. 1132-1141). (MM - Proceedings of the ACM International Conference on Multimedia). Association for Computing Machinery. https://doi.org/10.1145/3394171.3413765

@inproceedings{ed8e5fc876b3468fabfb9a3d59c9effb,

title = "Multi-modal Cooking Workflow Construction for Food Recipes",

abstract = "Understanding food recipe requires anticipating the implicit causal effects of cooking actions, such that the recipe can be converted into a graph describing the temporal workflow of the recipe. This is a non-trivial task that involves common-sense reasoning. However, existing efforts rely on hand-crafted features to extract the workflow graph from recipes due to the lack of large-scale labeled datasets. Moreover, they fail to utilize the cooking images, which constitute an important part of food recipes. In this paper, we build MM-ReS, the first large-scale dataset for cooking workflow construction, consisting of 9,850 recipes with human-labeled workflow graphs. Cooking steps are multi-modal, featuring both text instructions and cooking images. We then propose a neural encoder-decoder model that utilizes both visual and textual information to construct the cooking workflow, which achieved over 20% performance gain over existing hand-crafted baselines.",

keywords = "cause-and-effect reasoning, cooking workflow, deep learning, food recipes, mm-res dataset, multi-modal fusion",

author = "Liang-Ming Pan and Jingjing Chen and Jianlong Wu and Shaoteng Liu and Chong-Wah Ngo and Min-Yen Kan and Yugang Jiang and Tat-Seng Chua",

note = "Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s).; 28th ACM International Conference on Multimedia (MM 2020), ACM Multimedia 2020 ; Conference date: 12-10-2020 Through 16-10-2020",

year = "2020",

month = oct,

doi = "10.1145/3394171.3413765",

language = "English",

isbn = "9781450379885",

series = "MM - Proceedings of the ACM International Conference on Multimedia",

publisher = "Association for Computing Machinery",

pages = "1132--1141",

booktitle = "MM {\textquoteleft}20",

address = "United States",

url = "https://2020.acmmm.org/",

}

Pan, L-M, Chen, J, Wu, J, Liu, S, Ngo, C-W, Kan, M-Y, Jiang, Y & Chua, T-S 2020, Multi-modal Cooking Workflow Construction for Food Recipes. in MM ‘20: Proceedings of the 28th ACM International Conference on Multimedia. MM - Proceedings of the ACM International Conference on Multimedia, Association for Computing Machinery, pp. 1132-1141, 28th ACM International Conference on Multimedia (MM 2020), Seattle, Washington, United States, 12/10/20. https://doi.org/10.1145/3394171.3413765

Multi-modal Cooking Workflow Construction for Food Recipes. / Pan, Liang-Ming; Chen, Jingjing; Wu, Jianlong et al.
MM ‘20: Proceedings of the 28th ACM International Conference on Multimedia. Association for Computing Machinery, 2020. p. 1132-1141 (MM - Proceedings of the ACM International Conference on Multimedia).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Multi-modal Cooking Workflow Construction for Food Recipes

AU - Pan, Liang-Ming

AU - Chen, Jingjing

AU - Wu, Jianlong

AU - Liu, Shaoteng

AU - Ngo, Chong-Wah

AU - Kan, Min-Yen

AU - Jiang, Yugang

AU - Chua, Tat-Seng

N1 - Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s).

PY - 2020/10

Y1 - 2020/10

N2 - Understanding food recipe requires anticipating the implicit causal effects of cooking actions, such that the recipe can be converted into a graph describing the temporal workflow of the recipe. This is a non-trivial task that involves common-sense reasoning. However, existing efforts rely on hand-crafted features to extract the workflow graph from recipes due to the lack of large-scale labeled datasets. Moreover, they fail to utilize the cooking images, which constitute an important part of food recipes. In this paper, we build MM-ReS, the first large-scale dataset for cooking workflow construction, consisting of 9,850 recipes with human-labeled workflow graphs. Cooking steps are multi-modal, featuring both text instructions and cooking images. We then propose a neural encoder-decoder model that utilizes both visual and textual information to construct the cooking workflow, which achieved over 20% performance gain over existing hand-crafted baselines.

AB - Understanding food recipe requires anticipating the implicit causal effects of cooking actions, such that the recipe can be converted into a graph describing the temporal workflow of the recipe. This is a non-trivial task that involves common-sense reasoning. However, existing efforts rely on hand-crafted features to extract the workflow graph from recipes due to the lack of large-scale labeled datasets. Moreover, they fail to utilize the cooking images, which constitute an important part of food recipes. In this paper, we build MM-ReS, the first large-scale dataset for cooking workflow construction, consisting of 9,850 recipes with human-labeled workflow graphs. Cooking steps are multi-modal, featuring both text instructions and cooking images. We then propose a neural encoder-decoder model that utilizes both visual and textual information to construct the cooking workflow, which achieved over 20% performance gain over existing hand-crafted baselines.

KW - cause-and-effect reasoning

KW - cooking workflow

KW - deep learning

KW - food recipes

KW - mm-res dataset

KW - multi-modal fusion

UR - http://www.scopus.com/inward/record.url?scp=85099078424&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85099078424&origin=recordpage

U2 - 10.1145/3394171.3413765

DO - 10.1145/3394171.3413765

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9781450379885

T3 - MM - Proceedings of the ACM International Conference on Multimedia

SP - 1132

EP - 1141

BT - MM ‘20

PB - Association for Computing Machinery

T2 - 28th ACM International Conference on Multimedia (MM 2020)

Y2 - 12 October 2020 through 16 October 2020

ER -

Multi-modal Cooking Workflow Construction for Food Recipes

Abstract

Publication series

Conference

Bibliographical note

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this