Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling

Jicheng Li; S. Joe Qin

doi:10.1016/j.compchemeng.2023.108264

Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling

Jicheng Li, S. Joe Qin^*

^*Corresponding author for this work

Department of Data Science

Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review

19 Citations (Scopus)

Abstract

Deep learning models such as the long short-term memory (LSTM) network have been applied for dynamic inferential modeling. However, many studies apply LSTM as a black-box approach without examining the necessity and usefulness of the internal LSTM gates for inferential modeling. In this paper, we use LSTM as a state space realization and compare it with linear state space modeling and statistical learning methods, including N4SID, partial least squares, the Lasso, and support vector regression. Two case studies on an industrial 660 MW boiler and a debutanizer column process indicate that LSTM underperforms all other methods. LSTM is shown to be capable of outperforming linear methods for a simulated reactor process with severely excited nonlinearity in the data. By dissecting the sub-components of a simple LSTM model, the effectiveness of the LSTM gates and nonlinear activation functions is scrutinized. © 2023 Elsevier Ltd

Original language	English
Article number	108264
Journal	Computers & Chemical Engineering
Volume	175
Online published	26 Apr 2023
DOIs	https://doi.org/10.1016/j.compchemeng.2023.108264
Publication status	Published - Jul 2023

Funding

Partial financial support for this work is acknowledged from a Natural Science Foundation of China Project (U20A20189), a General Research Fund by the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. 11303421), an ITF - Guangdong-Hong Kong Technology Cooperation Funding Scheme (Project Ref. No. GHP/145/20), a Math and Application Project (2021YFA1003504) under the National Key R&D Program, a Shenzhen-Hong Kong-Macau Science and Technology Project Category C (9240086), an InnoHK initiative of The Government of the HKSAR for the Laboratory for AI-Powered Financial Technologies, and a Collaborative Research Fund (No. C1143-20G) by RGC of Hong Kong.

Research Keywords

Dynamic inferential modeling
LSTM
Partial least squares
Regularized learning
Subspace identification

RGC Funding Information

RGC-funded

Access Output

10.1016/j.compchemeng.2023.108264

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@article{0f4cdc34c4e8472fb549e63763fc2648,

title = "Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling",

abstract = "Deep learning models such as the long short-term memory (LSTM) network have been applied for dynamic inferential modeling. However, many studies apply LSTM as a black-box approach without examining the necessity and usefulness of the internal LSTM gates for inferential modeling. In this paper, we use LSTM as a state space realization and compare it with linear state space modeling and statistical learning methods, including N4SID, partial least squares, the Lasso, and support vector regression. Two case studies on an industrial 660 MW boiler and a debutanizer column process indicate that LSTM underperforms all other methods. LSTM is shown to be capable of outperforming linear methods for a simulated reactor process with severely excited nonlinearity in the data. By dissecting the sub-components of a simple LSTM model, the effectiveness of the LSTM gates and nonlinear activation functions is scrutinized. {\textcopyright} 2023 Elsevier Ltd",

keywords = "Dynamic inferential modeling, LSTM, Partial least squares, Regularized learning, Subspace identification",

author = "Jicheng Li and Qin, {S. Joe}",

year = "2023",

month = jul,

doi = "10.1016/j.compchemeng.2023.108264",

language = "English",

volume = "175",

journal = "Computers & Chemical Engineering",

issn = "0098-1354",

publisher = "Elsevier B.V.",

}

TY - JOUR

T1 - Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling

AU - Li, Jicheng

AU - Qin, S. Joe

PY - 2023/7

Y1 - 2023/7

N2 - Deep learning models such as the long short-term memory (LSTM) network have been applied for dynamic inferential modeling. However, many studies apply LSTM as a black-box approach without examining the necessity and usefulness of the internal LSTM gates for inferential modeling. In this paper, we use LSTM as a state space realization and compare it with linear state space modeling and statistical learning methods, including N4SID, partial least squares, the Lasso, and support vector regression. Two case studies on an industrial 660 MW boiler and a debutanizer column process indicate that LSTM underperforms all other methods. LSTM is shown to be capable of outperforming linear methods for a simulated reactor process with severely excited nonlinearity in the data. By dissecting the sub-components of a simple LSTM model, the effectiveness of the LSTM gates and nonlinear activation functions is scrutinized. © 2023 Elsevier Ltd

AB - Deep learning models such as the long short-term memory (LSTM) network have been applied for dynamic inferential modeling. However, many studies apply LSTM as a black-box approach without examining the necessity and usefulness of the internal LSTM gates for inferential modeling. In this paper, we use LSTM as a state space realization and compare it with linear state space modeling and statistical learning methods, including N4SID, partial least squares, the Lasso, and support vector regression. Two case studies on an industrial 660 MW boiler and a debutanizer column process indicate that LSTM underperforms all other methods. LSTM is shown to be capable of outperforming linear methods for a simulated reactor process with severely excited nonlinearity in the data. By dissecting the sub-components of a simple LSTM model, the effectiveness of the LSTM gates and nonlinear activation functions is scrutinized. © 2023 Elsevier Ltd

KW - Dynamic inferential modeling

KW - LSTM

KW - Partial least squares

KW - Regularized learning

KW - Subspace identification

UR - http://www.scopus.com/inward/record.url?scp=85154581845&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85154581845&origin=recordpage

U2 - 10.1016/j.compchemeng.2023.108264

DO - 10.1016/j.compchemeng.2023.108264

M3 - RGC 21 - Publication in refereed journal

SN - 0098-1354

VL - 175

JO - Computers & Chemical Engineering

JF - Computers & Chemical Engineering

M1 - 108264

ER -

Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling

Abstract

Funding

Research Keywords

RGC Funding Information

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

ITF: Aerospace System Prognostics and Health Management Model and Telemetry Task Optimization

SZSTIB-C-HK: 深圳地铁客流大数据实时降维建模、拥塞预警与智慧出行方法研究

GRF: Dimension Reduction Modeling Methods for High Dimensional Dynamic Data in Smart Manufacturing and Operations

Cite this

Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling

Abstract

Funding

Research Keywords

RGC Funding Information

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Projects

ITF: Aerospace System Prognostics and Health Management Model and Telemetry Task Optimization

SZSTIB-C-HK: 深圳地铁客流大数据实时降维建模、拥塞预警与智慧出行方法研究

GRF: Dimension Reduction Modeling Methods for High Dimensional Dynamic Data in Smart Manufacturing and Operations

Cite this