Effect of Logistic Activation Function and Multiplicative Input Noise on DNN-kWTA Model

Wenhao Lu; Chi-Sing Leung; John Sum

doi:10.1007/978-981-99-1639-9_17

Effect of Logistic Activation Function and Multiplicative Input Noise on DNN-kWTA Model

^*Corresponding author for this work

Department of Electrical Engineering

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 12 - Chapter in an edited book (Author) › peer-review

Abstract

The dual neural network-based (DNN) k-winner-take-all (kWTA) model is one of the simplest analog neural network models for the kWTA process. This paper analyzes the behaviors of the DNN-kWTA model under these two imperfections. The two imperfections are, (1) the activation function of IO neurons is a logistic function rather than an ideal step function, and (2) there are multiplicative Gaussian noise in the inputs. With the two imperfections, the model may not be able to perform correctly. Hence it is important to estimate the probability of the imperfection model performing correctly. We first derive the equivalent activation function of IO neurons under the two imperfections. Next, we derive the sufficient conditions for the imperfect model to operate correctly. These results enable us to efficiently estimate the probability of the imperfect model generating correct outputs. Additionally, we derive a bound on the probability that the imperfect model generates the desired outcomes for inputs with a uniform distribution. Finally, we discuss how to generalize our findings to handle non-Gaussian multiplicative input noise. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023.

Original language	English
Title of host publication	Neural Information Processing
Subtitle of host publication	29th International Conference, ICONIP 2022, Virtual Event, November 22–26, 2022, Proceedings, Part IV
Editors	Mohammad Tanveer, Sonali Agarwal, Seiichi Ozawa, Asif Ekbal, Adam Jatowt
Place of Publication	Singapore
Publisher	Springer Nature Singapore Pte Ltd.
Pages	202-214
ISBN (Electronic)	978-981-99-1639-9
ISBN (Print)	978-981-99-1638-2
DOIs	https://doi.org/10.1007/978-981-99-1639-9_17
Publication status	Published - 2023
Event	29th International Conference on Neural Information Processing (ICONIP 2022) - Virtual, Indore, India Duration: 22 Nov 2022 → 26 Nov 2022 https://iconip2022.apnns.org/index.php

Publication series

Name	Communications in Computer and Information Science
Volume	1791
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	29th International Conference on Neural Information Processing (ICONIP 2022)
Abbreviated title	29th ICONIP 2022
Place	India
City	Indore
Period	22/11/22 → 26/11/22
Internet address	https://iconip2022.apnns.org/index.php

Bibliographical note

Information for this record is supplemented by the author(s) concerned.

Research Keywords

DNN-kWTA
Logistic activation function
Multiplicative Input Noise
Threshold logic units (tlus)

Access Output

10.1007/978-981-99-1639-9_17

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Lu, W., Leung, C.-S., & Sum, J. (2023). Effect of Logistic Activation Function and Multiplicative Input Noise on DNN-kWTA Model. In M. Tanveer, S. Agarwal, S. Ozawa, A. Ekbal, & A. Jatowt (Eds.), Neural Information Processing: 29th International Conference, ICONIP 2022, Virtual Event, November 22–26, 2022, Proceedings, Part IV (pp. 202-214). (Communications in Computer and Information Science; Vol. 1791). Springer Nature Singapore Pte Ltd.. https://doi.org/10.1007/978-981-99-1639-9_17

Lu, Wenhao ; Leung, Chi-Sing ; Sum, John. / Effect of Logistic Activation Function and Multiplicative Input Noise on DNN-kWTA Model. Neural Information Processing: 29th International Conference, ICONIP 2022, Virtual Event, November 22–26, 2022, Proceedings, Part IV. editor / Mohammad Tanveer ; Sonali Agarwal ; Seiichi Ozawa ; Asif Ekbal ; Adam Jatowt. Singapore : Springer Nature Singapore Pte Ltd., 2023. pp. 202-214 (Communications in Computer and Information Science).

@inbook{a07bba7cc1fe4cbe8991630fa4f6c812,

title = "Effect of Logistic Activation Function and Multiplicative Input Noise on DNN-kWTA Model",

abstract = "The dual neural network-based (DNN) k-winner-take-all (kWTA) model is one of the simplest analog neural network models for the kWTA process. This paper analyzes the behaviors of the DNN-kWTA model under these two imperfections. The two imperfections are, (1) the activation function of IO neurons is a logistic function rather than an ideal step function, and (2) there are multiplicative Gaussian noise in the inputs. With the two imperfections, the model may not be able to perform correctly. Hence it is important to estimate the probability of the imperfection model performing correctly. We first derive the equivalent activation function of IO neurons under the two imperfections. Next, we derive the sufficient conditions for the imperfect model to operate correctly. These results enable us to efficiently estimate the probability of the imperfect model generating correct outputs. Additionally, we derive a bound on the probability that the imperfect model generates the desired outcomes for inputs with a uniform distribution. Finally, we discuss how to generalize our findings to handle non-Gaussian multiplicative input noise. {\textcopyright} The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023.",

keywords = "DNN-kWTA, Logistic activation function, Multiplicative Input Noise, Threshold logic units (tlus)",

author = "Wenhao Lu and Chi-Sing Leung and John Sum",

note = "Information for this record is supplemented by the author(s) concerned.; 29th International Conference on Neural Information Processing (ICONIP 2022), 29th ICONIP 2022 ; Conference date: 22-11-2022 Through 26-11-2022",

year = "2023",

doi = "10.1007/978-981-99-1639-9_17",

language = "English",

isbn = "978-981-99-1638-2",

series = "Communications in Computer and Information Science",

publisher = "Springer Nature Singapore Pte Ltd.",

pages = "202--214",

editor = "Mohammad Tanveer and Sonali Agarwal and Seiichi Ozawa and Asif Ekbal and Adam Jatowt",

booktitle = "Neural Information Processing",

url = "https://iconip2022.apnns.org/index.php",

}

Lu, W , Leung, C-S & Sum, J 2023, Effect of Logistic Activation Function and Multiplicative Input Noise on DNN-kWTA Model. in M Tanveer, S Agarwal, S Ozawa, A Ekbal & A Jatowt (eds), Neural Information Processing: 29th International Conference, ICONIP 2022, Virtual Event, November 22–26, 2022, Proceedings, Part IV. Communications in Computer and Information Science, vol. 1791, Springer Nature Singapore Pte Ltd., Singapore, pp. 202-214, 29th International Conference on Neural Information Processing (ICONIP 2022), Indore, India, 22/11/22. https://doi.org/10.1007/978-981-99-1639-9_17

Effect of Logistic Activation Function and Multiplicative Input Noise on DNN-kWTA Model. / Lu, Wenhao ; Leung, Chi-Sing; Sum, John.
Neural Information Processing: 29th International Conference, ICONIP 2022, Virtual Event, November 22–26, 2022, Proceedings, Part IV. ed. / Mohammad Tanveer; Sonali Agarwal; Seiichi Ozawa; Asif Ekbal; Adam Jatowt. Singapore: Springer Nature Singapore Pte Ltd., 2023. p. 202-214 (Communications in Computer and Information Science; Vol. 1791).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 12 - Chapter in an edited book (Author) › peer-review

TY - CHAP

T1 - Effect of Logistic Activation Function and Multiplicative Input Noise on DNN-kWTA Model

AU - Lu, Wenhao

AU - Leung, Chi-Sing

AU - Sum, John

N1 - Information for this record is supplemented by the author(s) concerned.

PY - 2023

Y1 - 2023

N2 - The dual neural network-based (DNN) k-winner-take-all (kWTA) model is one of the simplest analog neural network models for the kWTA process. This paper analyzes the behaviors of the DNN-kWTA model under these two imperfections. The two imperfections are, (1) the activation function of IO neurons is a logistic function rather than an ideal step function, and (2) there are multiplicative Gaussian noise in the inputs. With the two imperfections, the model may not be able to perform correctly. Hence it is important to estimate the probability of the imperfection model performing correctly. We first derive the equivalent activation function of IO neurons under the two imperfections. Next, we derive the sufficient conditions for the imperfect model to operate correctly. These results enable us to efficiently estimate the probability of the imperfect model generating correct outputs. Additionally, we derive a bound on the probability that the imperfect model generates the desired outcomes for inputs with a uniform distribution. Finally, we discuss how to generalize our findings to handle non-Gaussian multiplicative input noise. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023.

AB - The dual neural network-based (DNN) k-winner-take-all (kWTA) model is one of the simplest analog neural network models for the kWTA process. This paper analyzes the behaviors of the DNN-kWTA model under these two imperfections. The two imperfections are, (1) the activation function of IO neurons is a logistic function rather than an ideal step function, and (2) there are multiplicative Gaussian noise in the inputs. With the two imperfections, the model may not be able to perform correctly. Hence it is important to estimate the probability of the imperfection model performing correctly. We first derive the equivalent activation function of IO neurons under the two imperfections. Next, we derive the sufficient conditions for the imperfect model to operate correctly. These results enable us to efficiently estimate the probability of the imperfect model generating correct outputs. Additionally, we derive a bound on the probability that the imperfect model generates the desired outcomes for inputs with a uniform distribution. Finally, we discuss how to generalize our findings to handle non-Gaussian multiplicative input noise. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023.

KW - DNN-kWTA

KW - Logistic activation function

KW - Multiplicative Input Noise

KW - Threshold logic units (tlus)

UR - http://www.scopus.com/inward/record.url?scp=85161653965&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85161653965&origin=recordpage

U2 - 10.1007/978-981-99-1639-9_17

DO - 10.1007/978-981-99-1639-9_17

M3 - RGC 12 - Chapter in an edited book (Author)

AN - SCOPUS:85161653965

SN - 978-981-99-1638-2

T3 - Communications in Computer and Information Science

SP - 202

EP - 214

BT - Neural Information Processing

A2 - Tanveer, Mohammad

A2 - Agarwal, Sonali

A2 - Ozawa, Seiichi

A2 - Ekbal, Asif

A2 - Jatowt, Adam

PB - Springer Nature Singapore Pte Ltd.

CY - Singapore

T2 - 29th International Conference on Neural Information Processing (ICONIP 2022)

Y2 - 22 November 2022 through 26 November 2022

ER -

Lu W , Leung CS, Sum J. Effect of Logistic Activation Function and Multiplicative Input Noise on DNN-kWTA Model. In Tanveer M, Agarwal S, Ozawa S, Ekbal A, Jatowt A, editors, Neural Information Processing: 29th International Conference, ICONIP 2022, Virtual Event, November 22–26, 2022, Proceedings, Part IV. Singapore: Springer Nature Singapore Pte Ltd. 2023. p. 202-214. (Communications in Computer and Information Science). Epub 2023 Apr 15. doi: 10.1007/978-981-99-1639-9_17