Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification

Xin Wu; Yi Cai; Qing Li; Jingyun Xu; Ho-fung Leung

doi:10.1007/978-3-030-02922-7_31

Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification

Xin Wu, Yi Cai^*, Qing Li, Jingyun Xu, Ho-fung Leung

^*Corresponding author for this work

Department of Computer Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

17 Citations (Scopus)

Abstract

Convolutional neural networks (CNN) are widely used in many NLP tasks, which can employ convolutional filters to capture useful semantic features of texts. However, convolutional filters with small window size may lose global context information of texts, simply increasing window size will bring the problems of data sparsity and enormous parameters. To capture global context information, we propose to use the self-attention mechanism to obtain contextual word embeddings. We present two methods to combine word and contextual embeddings, then apply convolutional neural networks to capture semantic features. Experimental results on five commonly used datasets show the effectiveness of our proposed methods.

Original language	English
Title of host publication	Web Information Systems Engineering – WISE 2018
Subtitle of host publication	19th International Conference, 2018, Proceedings, Part I
Editors	Hakim Hacid, Wojciech Cellary, Hua Wang, Hye-Young Paik, Rui Zhou
Publisher	Springer, Cham
Pages	453-467
ISBN (Electronic)	978-3-030-02922-7
ISBN (Print)	978-3-030-02921-0
DOIs	https://doi.org/10.1007/978-3-030-02922-7_31
Publication status	Published - Nov 2018
Event	19th International Conference on Web Information Systems Engineering, WISE 2018 - Dubai, United Arab Emirates Duration: 12 Nov 2018 → 15 Nov 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11233 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	19th International Conference on Web Information Systems Engineering, WISE 2018
Place	United Arab Emirates
City	Dubai
Period	12/11/18 → 15/11/18

Research Keywords

Attention mechanism
Convolutional neural networks
Text classification
Word representation

Access Output

10.1007/978-3-030-02922-7_31

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Wu, X., Cai, Y., Li, Q., Xu, J., & Leung, H. (2018). Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification. In H. Hacid, W. Cellary, H. Wang, H.-Y. Paik, & R. Zhou (Eds.), Web Information Systems Engineering – WISE 2018 : 19th International Conference, 2018, Proceedings, Part I (pp. 453-467). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11233 LNCS). Springer, Cham. https://doi.org/10.1007/978-3-030-02922-7_31

Wu, Xin ; Cai, Yi ; Li, Qing et al. / Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification. Web Information Systems Engineering – WISE 2018 : 19th International Conference, 2018, Proceedings, Part I. editor / Hakim Hacid ; Wojciech Cellary ; Hua Wang ; Hye-Young Paik ; Rui Zhou. Springer, Cham, 2018. pp. 453-467 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{23bc0c9042c54fc68c1ac2282efcdb27,

title = "Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification",

abstract = "Convolutional neural networks (CNN) are widely used in many NLP tasks, which can employ convolutional filters to capture useful semantic features of texts. However, convolutional filters with small window size may lose global context information of texts, simply increasing window size will bring the problems of data sparsity and enormous parameters. To capture global context information, we propose to use the self-attention mechanism to obtain contextual word embeddings. We present two methods to combine word and contextual embeddings, then apply convolutional neural networks to capture semantic features. Experimental results on five commonly used datasets show the effectiveness of our proposed methods.",

keywords = "Attention mechanism, Convolutional neural networks, Text classification, Word representation",

author = "Xin Wu and Yi Cai and Qing Li and Jingyun Xu and Ho-fung Leung",

year = "2018",

month = nov,

doi = "10.1007/978-3-030-02922-7_31",

language = "English",

isbn = "978-3-030-02921-0",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer, Cham",

pages = "453--467",

editor = "Hacid, {Hakim } and Cellary, {Wojciech } and Wang, {Hua } and Paik, {Hye-Young } and Rui Zhou",

booktitle = "Web Information Systems Engineering – WISE 2018",

note = "19th International Conference on Web Information Systems Engineering, WISE 2018 ; Conference date: 12-11-2018 Through 15-11-2018",

}

Wu, X, Cai, Y, Li, Q, Xu, J & Leung, H 2018, Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification. in H Hacid, W Cellary, H Wang, H-Y Paik & R Zhou (eds), Web Information Systems Engineering – WISE 2018 : 19th International Conference, 2018, Proceedings, Part I. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11233 LNCS, Springer, Cham, pp. 453-467, 19th International Conference on Web Information Systems Engineering, WISE 2018, Dubai, United Arab Emirates, 12/11/18. https://doi.org/10.1007/978-3-030-02922-7_31

Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification. / Wu, Xin; Cai, Yi; Li, Qing et al.
Web Information Systems Engineering – WISE 2018 : 19th International Conference, 2018, Proceedings, Part I. ed. / Hakim Hacid; Wojciech Cellary; Hua Wang; Hye-Young Paik; Rui Zhou. Springer, Cham, 2018. p. 453-467 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11233 LNCS).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification

AU - Wu, Xin

AU - Cai, Yi

AU - Li, Qing

AU - Xu, Jingyun

AU - Leung, Ho-fung

PY - 2018/11

Y1 - 2018/11

N2 - Convolutional neural networks (CNN) are widely used in many NLP tasks, which can employ convolutional filters to capture useful semantic features of texts. However, convolutional filters with small window size may lose global context information of texts, simply increasing window size will bring the problems of data sparsity and enormous parameters. To capture global context information, we propose to use the self-attention mechanism to obtain contextual word embeddings. We present two methods to combine word and contextual embeddings, then apply convolutional neural networks to capture semantic features. Experimental results on five commonly used datasets show the effectiveness of our proposed methods.

AB - Convolutional neural networks (CNN) are widely used in many NLP tasks, which can employ convolutional filters to capture useful semantic features of texts. However, convolutional filters with small window size may lose global context information of texts, simply increasing window size will bring the problems of data sparsity and enormous parameters. To capture global context information, we propose to use the self-attention mechanism to obtain contextual word embeddings. We present two methods to combine word and contextual embeddings, then apply convolutional neural networks to capture semantic features. Experimental results on five commonly used datasets show the effectiveness of our proposed methods.

KW - Attention mechanism

KW - Convolutional neural networks

KW - Text classification

KW - Word representation

UR - http://www.scopus.com/inward/record.url?scp=85055802931&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85055802931&origin=recordpage

U2 - 10.1007/978-3-030-02922-7_31

DO - 10.1007/978-3-030-02922-7_31

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 978-3-030-02921-0

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 453

EP - 467

BT - Web Information Systems Engineering – WISE 2018

A2 - Hacid, Hakim

A2 - Cellary, Wojciech

A2 - Wang, Hua

A2 - Paik, Hye-Young

A2 - Zhou, Rui

PB - Springer, Cham

T2 - 19th International Conference on Web Information Systems Engineering, WISE 2018

Y2 - 12 November 2018 through 15 November 2018

ER -

Wu X, Cai Y, Li Q, Xu J, Leung H. Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification. In Hacid H, Cellary W, Wang H, Paik HY, Zhou R, editors, Web Information Systems Engineering – WISE 2018 : 19th International Conference, 2018, Proceedings, Part I. Springer, Cham. 2018. p. 453-467. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-02922-7_31

Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification

Abstract

Publication series

Conference

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this