ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modeling

Kangjie Zheng; Siyu Long; Tianyu Lu; Junwei Yang; Xinyu Dai; Ming Zhang; Zaiqing Nie; Wei-Ying Ma; Hao Zhou

ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modeling

Kangjie Zheng (Co-first Author)
, Siyu Long (Co-first Author)
, Tianyu Lu
, Junwei Yang
, Xinyu Dai
, Ming Zhang^*
, Zaiqing Nie
, Wei-Ying Ma
, Hao Zhou^*

^*Corresponding author for this work

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

4 Citations (Scopus)

Abstract

Protein language models have demonstrated significant potential in the field of protein engineering.However, current protein language models primarily operate at the residue scale, which limits their ability to provide information at the atom level.This limitation prevents us from fully exploiting the capabilities of protein language models for applications involving both proteins and small molecules.In this paper, we propose ESM-AA (ESM All-Atom), a novel approach that enables atom-scale and residue-scale unified molecular modeling.ESM-AA achieves this by pretraining on multi-scale code-switch protein sequences and utilizing a multi-scale position encoding to capture relationships among residues and atoms.Experimental results indicate that ESM-AA surpasses previous methods in protein-molecule tasks, demonstrating the full utilization of protein language models.Further investigations reveal that through unified molecular modeling, ESM-AA not only gains molecular knowledge but also retains its understanding of proteins. Copyright 2024 by the author(s)

Original language	English
Title of host publication	Proceedings of the 41st International Conference on Machine Learning
Editors	Ruslan Salakhutdinov, Zico Kolter, Katherine Heller
Publisher	ML Research Press
Pages	61432-61453
Publication status	Published - 2024
Externally published	Yes
Event	41st International Conference on Machine Learning (ICML 2024) - Messe Wien Exhibition Congress Center, Vienna, Austria Duration: 21 Jul 2024 → 27 Jul 2024 https://proceedings.mlr.press/v235/ https://icml.cc/

Publication series

Name	Proceedings of Machine Learning Research
Volume	235
ISSN (Print)	2640-3498

Conference

Conference	41st International Conference on Machine Learning (ICML 2024)
Place	Austria
City	Vienna
Period	21/07/24 → 27/07/24
Internet address	https://proceedings.mlr.press/v235/ https://icml.cc/

Funding

We would like to thank Qiying Yu and Hanlin Wu from AIR for their insightful discussions on the project.We also thank other members from AIR for their valuable feedback given during the internal seminar.This work is supported by the National Science and Technology Major Project (2022ZD0117502), the National Natural Science Foundation of China (Grant No.62276002), Natural Science Foundation of China (Grant No.62376133) and PharMolix Inc.

Access Output

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Zheng, K., Long, S., Lu, T., Yang, J., Dai, X., Zhang, M., Nie, Z., Ma, W.-Y., & Zhou, H. (2024). ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modeling. In R. Salakhutdinov, Z. Kolter, & K. Heller (Eds.), Proceedings of the 41st International Conference on Machine Learning (pp. 61432-61453). (Proceedings of Machine Learning Research; Vol. 235). ML Research Press. https://dl.acm.org/doi/10.5555/3692070.3694611#core-collateral-info

@inproceedings{7e99f571095b418583294439eede7c8c,

title = "ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modeling",

abstract = "Protein language models have demonstrated significant potential in the field of protein engineering.However, current protein language models primarily operate at the residue scale, which limits their ability to provide information at the atom level.This limitation prevents us from fully exploiting the capabilities of protein language models for applications involving both proteins and small molecules.In this paper, we propose ESM-AA (ESM All-Atom), a novel approach that enables atom-scale and residue-scale unified molecular modeling.ESM-AA achieves this by pretraining on multi-scale code-switch protein sequences and utilizing a multi-scale position encoding to capture relationships among residues and atoms.Experimental results indicate that ESM-AA surpasses previous methods in protein-molecule tasks, demonstrating the full utilization of protein language models.Further investigations reveal that through unified molecular modeling, ESM-AA not only gains molecular knowledge but also retains its understanding of proteins. Copyright 2024 by the author(s)",

author = "Kangjie Zheng and Siyu Long and Tianyu Lu and Junwei Yang and Xinyu Dai and Ming Zhang and Zaiqing Nie and Wei-Ying Ma and Hao Zhou",

year = "2024",

language = "English",

series = "Proceedings of Machine Learning Research",

publisher = "ML Research Press",

pages = "61432--61453",

editor = "Ruslan Salakhutdinov and Zico Kolter and Katherine Heller",

booktitle = "Proceedings of the 41st International Conference on Machine Learning",

note = "41st International Conference on Machine Learning (ICML 2024) ; Conference date: 21-07-2024 Through 27-07-2024",

url = "https://proceedings.mlr.press/v235/, https://icml.cc/",

}

Zheng, K, Long, S, Lu, T, Yang, J, Dai, X, Zhang, M, Nie, Z, Ma, W-Y & Zhou, H 2024, ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modeling. in R Salakhutdinov, Z Kolter & K Heller (eds), Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 235, ML Research Press, pp. 61432-61453, 41st International Conference on Machine Learning (ICML 2024), Vienna, Austria, 21/07/24. <https://dl.acm.org/doi/10.5555/3692070.3694611#core-collateral-info>

ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modeling. / Zheng, Kangjie (Co-first Author); Long, Siyu (Co-first Author); Lu, Tianyu et al.
Proceedings of the 41st International Conference on Machine Learning. ed. / Ruslan Salakhutdinov; Zico Kolter; Katherine Heller. ML Research Press, 2024. p. 61432-61453 (Proceedings of Machine Learning Research; Vol. 235).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - ESM All-Atom

T2 - 41st International Conference on Machine Learning (ICML 2024)

AU - Zheng, Kangjie

AU - Long, Siyu

AU - Lu, Tianyu

AU - Yang, Junwei

AU - Dai, Xinyu

AU - Zhang, Ming

AU - Nie, Zaiqing

AU - Ma, Wei-Ying

AU - Zhou, Hao

PY - 2024

Y1 - 2024

N2 - Protein language models have demonstrated significant potential in the field of protein engineering.However, current protein language models primarily operate at the residue scale, which limits their ability to provide information at the atom level.This limitation prevents us from fully exploiting the capabilities of protein language models for applications involving both proteins and small molecules.In this paper, we propose ESM-AA (ESM All-Atom), a novel approach that enables atom-scale and residue-scale unified molecular modeling.ESM-AA achieves this by pretraining on multi-scale code-switch protein sequences and utilizing a multi-scale position encoding to capture relationships among residues and atoms.Experimental results indicate that ESM-AA surpasses previous methods in protein-molecule tasks, demonstrating the full utilization of protein language models.Further investigations reveal that through unified molecular modeling, ESM-AA not only gains molecular knowledge but also retains its understanding of proteins. Copyright 2024 by the author(s)

AB - Protein language models have demonstrated significant potential in the field of protein engineering.However, current protein language models primarily operate at the residue scale, which limits their ability to provide information at the atom level.This limitation prevents us from fully exploiting the capabilities of protein language models for applications involving both proteins and small molecules.In this paper, we propose ESM-AA (ESM All-Atom), a novel approach that enables atom-scale and residue-scale unified molecular modeling.ESM-AA achieves this by pretraining on multi-scale code-switch protein sequences and utilizing a multi-scale position encoding to capture relationships among residues and atoms.Experimental results indicate that ESM-AA surpasses previous methods in protein-molecule tasks, demonstrating the full utilization of protein language models.Further investigations reveal that through unified molecular modeling, ESM-AA not only gains molecular knowledge but also retains its understanding of proteins. Copyright 2024 by the author(s)

UR - https://www.scopus.com/pages/publications/85203835773

UR - https://www.scopus.com/pages/publications/85203835773#tab=impact

M3 - RGC 32 - Refereed conference paper (with host publication)

T3 - Proceedings of Machine Learning Research

SP - 61432

EP - 61453

BT - Proceedings of the 41st International Conference on Machine Learning

A2 - Salakhutdinov, Ruslan

A2 - Kolter, Zico

A2 - Heller, Katherine

PB - ML Research Press

Y2 - 21 July 2024 through 27 July 2024

ER -

ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modeling

Abstract

Publication series

Conference

Funding

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this