Reinforcement Learning-Driven LLM Agent for Automated Attacks on LLMs

Xiangwen Wang; Jie Peng; Kaidi Xu; Huaxiu Yao; Tianlong Chen

doi:10.18653/v1/2024.privatenlp-1.17

Reinforcement Learning-Driven LLM Agent for Automated Attacks on LLMs

Xiangwen Wang
, Jie Peng
, Kaidi Xu
, Huaxiu Yao
, Tianlong Chen

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

8 Citations (Scopus)

1 Downloads (CityUHK Scholars)

Abstract

Recently, there has been a growing focus on conducting attacks on large language models (LLMs) to assess LLMs’ safety. Yet, existing attack methods face challenges, including the need to access model weights or merely ensuring LLMs output harmful information without controlling the specific content of their output. Exactly control of the LLM output can produce more inconspicuous attacks which could reveal a new page for LLM security. To achieve this, we propose RLTA: the Reinforcement Learning Targeted Attack, a framework that is designed for attacking language models (LLMs) and is adaptable to black box (weight inaccessible) scenarios. It is capable of automatically generating malicious prompts that trigger target LLMs to produce specific outputs. We demonstrate RLTA in two different scenarios: LLM trojan detection and jailbreaking. The comprehensive experimental results show the potential of RLTA in enhancing the security measures surrounding contemporary LLMs. © 2024 Association for Computational Linguistics.

Original language	English
Title of host publication	PrivateNLP 2024 - The Fifth Workshop on Privacy in Natural Language Processing
Subtitle of host publication	Proceedings of the Workshop
Editors	Ivan Habernal, Sepideh Ghanavati, Abhilasha Ravichander,
Publisher	Association for Computational Linguistics
Pages	170-177
ISBN (Print)	9798891761391
DOIs	https://doi.org/10.18653/v1/2024.privatenlp-1.17
Publication status	Published - Aug 2024
Externally published	Yes
Event	5th Workshop on Privacy in Natural Language Processing (PrivateNLP 2024) - Bangkok, Thailand Duration: 15 Aug 2024 → … https://aclanthology.org/2024.privatenlp-1

Publication series

Name	PrivateNLP - Workshop on Privacy in Natural Language Processing, Proceedings of the Workshop

Conference

Conference	5th Workshop on Privacy in Natural Language Processing (PrivateNLP 2024)
Place	Thailand
City	Bangkok
Period	15/08/24 → …
Internet address	https://aclanthology.org/2024.privatenlp-1

Publisher's Copyright Statement

This full text is made available under CC-BY 4.0. https://creativecommons.org/licenses/by/4.0/

Access Output

10.18653/v1/2024.privatenlp-1.17

335447888.pdfFinal Published version, 405 KBLicence: CC BY

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Wang, X., Peng, J., Xu, K., Yao, H., & Chen, T. (2024). Reinforcement Learning-Driven LLM Agent for Automated Attacks on LLMs. In I. Habernal, S. Ghanavati, & A. Ravichander, (Eds.), PrivateNLP 2024 - The Fifth Workshop on Privacy in Natural Language Processing: Proceedings of the Workshop (pp. 170-177). (PrivateNLP - Workshop on Privacy in Natural Language Processing, Proceedings of the Workshop). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.privatenlp-1.17

Wang, Xiangwen ; Peng, Jie ; Xu, Kaidi et al. / Reinforcement Learning-Driven LLM Agent for Automated Attacks on LLMs. PrivateNLP 2024 - The Fifth Workshop on Privacy in Natural Language Processing: Proceedings of the Workshop. editor / Ivan Habernal ; Sepideh Ghanavati ; Abhilasha Ravichander,. Association for Computational Linguistics, 2024. pp. 170-177 (PrivateNLP - Workshop on Privacy in Natural Language Processing, Proceedings of the Workshop).

@inproceedings{e0547bbf39274b58b18db61f1456ab74,

title = "Reinforcement Learning-Driven LLM Agent for Automated Attacks on LLMs",

abstract = "Recently, there has been a growing focus on conducting attacks on large language models (LLMs) to assess LLMs{\textquoteright} safety. Yet, existing attack methods face challenges, including the need to access model weights or merely ensuring LLMs output harmful information without controlling the specific content of their output. Exactly control of the LLM output can produce more inconspicuous attacks which could reveal a new page for LLM security. To achieve this, we propose RLTA: the Reinforcement Learning Targeted Attack, a framework that is designed for attacking language models (LLMs) and is adaptable to black box (weight inaccessible) scenarios. It is capable of automatically generating malicious prompts that trigger target LLMs to produce specific outputs. We demonstrate RLTA in two different scenarios: LLM trojan detection and jailbreaking. The comprehensive experimental results show the potential of RLTA in enhancing the security measures surrounding contemporary LLMs. {\textcopyright} 2024 Association for Computational Linguistics.",

author = "Xiangwen Wang and Jie Peng and Kaidi Xu and Huaxiu Yao and Tianlong Chen",

year = "2024",

month = aug,

doi = "10.18653/v1/2024.privatenlp-1.17",

language = "English",

isbn = "9798891761391",

series = "PrivateNLP - Workshop on Privacy in Natural Language Processing, Proceedings of the Workshop",

publisher = "Association for Computational Linguistics",

pages = "170--177",

editor = "Habernal, \{Ivan \} and Ghanavati, \{Sepideh \} and Ravichander,, \{Abhilasha \}",

booktitle = "PrivateNLP 2024 - The Fifth Workshop on Privacy in Natural Language Processing",

note = "5th Workshop on Privacy in Natural Language Processing (PrivateNLP 2024) ; Conference date: 15-08-2024",

url = "https://aclanthology.org/2024.privatenlp-1",

}

Wang, X, Peng, J, Xu, K, Yao, H & Chen, T 2024, Reinforcement Learning-Driven LLM Agent for Automated Attacks on LLMs. in I Habernal, S Ghanavati & A Ravichander, (eds), PrivateNLP 2024 - The Fifth Workshop on Privacy in Natural Language Processing: Proceedings of the Workshop. PrivateNLP - Workshop on Privacy in Natural Language Processing, Proceedings of the Workshop, Association for Computational Linguistics, pp. 170-177, 5th Workshop on Privacy in Natural Language Processing (PrivateNLP 2024), Bangkok, Thailand, 15/08/24. https://doi.org/10.18653/v1/2024.privatenlp-1.17

Reinforcement Learning-Driven LLM Agent for Automated Attacks on LLMs. / Wang, Xiangwen; Peng, Jie; Xu, Kaidi et al.
PrivateNLP 2024 - The Fifth Workshop on Privacy in Natural Language Processing: Proceedings of the Workshop. ed. / Ivan Habernal; Sepideh Ghanavati; Abhilasha Ravichander,. Association for Computational Linguistics, 2024. p. 170-177 (PrivateNLP - Workshop on Privacy in Natural Language Processing, Proceedings of the Workshop).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Reinforcement Learning-Driven LLM Agent for Automated Attacks on LLMs

AU - Wang, Xiangwen

AU - Peng, Jie

AU - Xu, Kaidi

AU - Yao, Huaxiu

AU - Chen, Tianlong

PY - 2024/8

Y1 - 2024/8

N2 - Recently, there has been a growing focus on conducting attacks on large language models (LLMs) to assess LLMs’ safety. Yet, existing attack methods face challenges, including the need to access model weights or merely ensuring LLMs output harmful information without controlling the specific content of their output. Exactly control of the LLM output can produce more inconspicuous attacks which could reveal a new page for LLM security. To achieve this, we propose RLTA: the Reinforcement Learning Targeted Attack, a framework that is designed for attacking language models (LLMs) and is adaptable to black box (weight inaccessible) scenarios. It is capable of automatically generating malicious prompts that trigger target LLMs to produce specific outputs. We demonstrate RLTA in two different scenarios: LLM trojan detection and jailbreaking. The comprehensive experimental results show the potential of RLTA in enhancing the security measures surrounding contemporary LLMs. © 2024 Association for Computational Linguistics.

AB - Recently, there has been a growing focus on conducting attacks on large language models (LLMs) to assess LLMs’ safety. Yet, existing attack methods face challenges, including the need to access model weights or merely ensuring LLMs output harmful information without controlling the specific content of their output. Exactly control of the LLM output can produce more inconspicuous attacks which could reveal a new page for LLM security. To achieve this, we propose RLTA: the Reinforcement Learning Targeted Attack, a framework that is designed for attacking language models (LLMs) and is adaptable to black box (weight inaccessible) scenarios. It is capable of automatically generating malicious prompts that trigger target LLMs to produce specific outputs. We demonstrate RLTA in two different scenarios: LLM trojan detection and jailbreaking. The comprehensive experimental results show the potential of RLTA in enhancing the security measures surrounding contemporary LLMs. © 2024 Association for Computational Linguistics.

UR - https://www.scopus.com/pages/publications/85204436876

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85204436876&origin=recordpage

U2 - 10.18653/v1/2024.privatenlp-1.17

DO - 10.18653/v1/2024.privatenlp-1.17

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9798891761391

T3 - PrivateNLP - Workshop on Privacy in Natural Language Processing, Proceedings of the Workshop

SP - 170

EP - 177

BT - PrivateNLP 2024 - The Fifth Workshop on Privacy in Natural Language Processing

A2 - Habernal, Ivan

A2 - Ghanavati, Sepideh

A2 - Ravichander,, Abhilasha

PB - Association for Computational Linguistics

T2 - 5th Workshop on Privacy in Natural Language Processing (PrivateNLP 2024)

Y2 - 15 August 2024

ER -

Wang X, Peng J, Xu K, Yao H, Chen T. Reinforcement Learning-Driven LLM Agent for Automated Attacks on LLMs. In Habernal I, Ghanavati S, Ravichander, A, editors, PrivateNLP 2024 - The Fifth Workshop on Privacy in Natural Language Processing: Proceedings of the Workshop. Association for Computational Linguistics. 2024. p. 170-177. (PrivateNLP - Workshop on Privacy in Natural Language Processing, Proceedings of the Workshop). doi: 10.18653/v1/2024.privatenlp-1.17

Reinforcement Learning-Driven LLM Agent for Automated Attacks on LLMs

Abstract

Publication series

Conference

Publisher's Copyright Statement

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this