Mission: mmWave Radar Person Identification with RGB Cameras

Ruofeng Liu; Tianshun Yao; Ruili Shi; Luoyu Mei; Shuai Wang; Zhimeng Yin; Wenchao Jiang; Shuai Wang

doi:10.1145/3666025.3699340

Mission: mmWave Radar Person Identification with RGB Cameras

Ruofeng Liu, Tianshun Yao, Ruili Shi, Luoyu Mei, Shuai Wang^*, Zhimeng Yin, Wenchao Jiang, Shuai Wang

^*Corresponding author for this work

Department of Computer Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

1 Citation (Scopus)

Abstract

This paper presents Mission, the first-of-this-kind cross-modal reidentification (ReID) design for mmWave Radar and RGB cameras. Given a person of interest detected by Radar in camera-restricted scenarios, Mission can identify the image of the person from cameras that are ubiquitously deployed in camera-allowed areas. We envision that cross Vison-RF ReID can significantly enrich mmWave human sensing with a wide spectrum of applications in security surveillance, tracking, and personalized services. Technically, we introduce a novel method for cross-modal similarity estimation that exploits inherent synergies between fine-grained 2D images and coarse-grained 3D Radar point clouds to effectively overcome their modal discrepancy. Through extensive experiments, we demonstrated that our proposed system can achieve 85% top-1 accuracy and 90% top-5 accuracy among 58 volunteers. © 2024 Copyright is held by the owner/author(s).

Original language	English
Title of host publication	SenSys '24
Subtitle of host publication	Proceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems
Publisher	Association for Computing Machinery
Pages	309-321
ISBN (Print)	979-8-4007-0697-4
DOIs	https://doi.org/10.1145/3666025.3699340
Publication status	Published - 2024
Event	22nd ACM Conference on Embedded Networked Sensor Systems (SenSys 2024) - The Dragon Hotel Hangzhou, Hangzhou, China Duration: 4 Nov 2024 → 7 Nov 2024 https://sensys.acm.org/2024/

Publication series

Name	SenSys - Proceedings of the ACM Conference on Embedded Networked Sensor Systems

Conference

Conference	22nd ACM Conference on Embedded Networked Sensor Systems (SenSys 2024)
Country/Territory	China
City	Hangzhou
Period	4/11/24 → 7/11/24
Internet address	https://sensys.acm.org/2024/

Bibliographical note

Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s).

Research Keywords

deep learning
millimeter wave
person identification

Access Output

10.1145/3666025.3699340

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Liu, R., Yao, T., Shi, R., Mei, L., Wang, S., Yin, Z., Jiang, W., & Wang, S. (2024). Mission: mmWave Radar Person Identification with RGB Cameras. In SenSys '24: Proceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems (pp. 309-321). (SenSys - Proceedings of the ACM Conference on Embedded Networked Sensor Systems). Association for Computing Machinery. https://doi.org/10.1145/3666025.3699340

@inproceedings{b462051d035043d98b019d99464b8217,

title = "Mission: mmWave Radar Person Identification with RGB Cameras",

abstract = "This paper presents Mission, the first-of-this-kind cross-modal reidentification (ReID) design for mmWave Radar and RGB cameras. Given a person of interest detected by Radar in camera-restricted scenarios, Mission can identify the image of the person from cameras that are ubiquitously deployed in camera-allowed areas. We envision that cross Vison-RF ReID can significantly enrich mmWave human sensing with a wide spectrum of applications in security surveillance, tracking, and personalized services. Technically, we introduce a novel method for cross-modal similarity estimation that exploits inherent synergies between fine-grained 2D images and coarse-grained 3D Radar point clouds to effectively overcome their modal discrepancy. Through extensive experiments, we demonstrated that our proposed system can achieve 85% top-1 accuracy and 90% top-5 accuracy among 58 volunteers. {\textcopyright} 2024 Copyright is held by the owner/author(s).",

keywords = "deep learning, millimeter wave, person identification",

author = "Ruofeng Liu and Tianshun Yao and Ruili Shi and Luoyu Mei and Shuai Wang and Zhimeng Yin and Wenchao Jiang and Shuai Wang",

note = "Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s).; 22nd ACM Conference on Embedded Networked Sensor Systems (SenSys 2024) ; Conference date: 04-11-2024 Through 07-11-2024",

year = "2024",

doi = "10.1145/3666025.3699340",

language = "English",

isbn = "979-8-4007-0697-4",

series = "SenSys - Proceedings of the ACM Conference on Embedded Networked Sensor Systems",

publisher = "Association for Computing Machinery",

pages = "309--321",

booktitle = "SenSys '24",

address = "United States",

url = "https://sensys.acm.org/2024/",

}

Liu, R, Yao, T, Shi, R, Mei, L , Wang, S , Yin, Z, Jiang, W & Wang, S 2024, Mission: mmWave Radar Person Identification with RGB Cameras. in SenSys '24: Proceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems. SenSys - Proceedings of the ACM Conference on Embedded Networked Sensor Systems, Association for Computing Machinery, pp. 309-321, 22nd ACM Conference on Embedded Networked Sensor Systems (SenSys 2024), Hangzhou, China, 4/11/24. https://doi.org/10.1145/3666025.3699340

Mission: mmWave Radar Person Identification with RGB Cameras. / Liu, Ruofeng; Yao, Tianshun; Shi, Ruili et al.
SenSys '24: Proceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems. Association for Computing Machinery, 2024. p. 309-321 (SenSys - Proceedings of the ACM Conference on Embedded Networked Sensor Systems).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Mission

T2 - 22nd ACM Conference on Embedded Networked Sensor Systems (SenSys 2024)

AU - Liu, Ruofeng

AU - Yao, Tianshun

AU - Shi, Ruili

AU - Mei, Luoyu

AU - Wang, Shuai

AU - Yin, Zhimeng

AU - Jiang, Wenchao

AU - Wang, Shuai

N1 - Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s).

PY - 2024

Y1 - 2024

N2 - This paper presents Mission, the first-of-this-kind cross-modal reidentification (ReID) design for mmWave Radar and RGB cameras. Given a person of interest detected by Radar in camera-restricted scenarios, Mission can identify the image of the person from cameras that are ubiquitously deployed in camera-allowed areas. We envision that cross Vison-RF ReID can significantly enrich mmWave human sensing with a wide spectrum of applications in security surveillance, tracking, and personalized services. Technically, we introduce a novel method for cross-modal similarity estimation that exploits inherent synergies between fine-grained 2D images and coarse-grained 3D Radar point clouds to effectively overcome their modal discrepancy. Through extensive experiments, we demonstrated that our proposed system can achieve 85% top-1 accuracy and 90% top-5 accuracy among 58 volunteers. © 2024 Copyright is held by the owner/author(s).

AB - This paper presents Mission, the first-of-this-kind cross-modal reidentification (ReID) design for mmWave Radar and RGB cameras. Given a person of interest detected by Radar in camera-restricted scenarios, Mission can identify the image of the person from cameras that are ubiquitously deployed in camera-allowed areas. We envision that cross Vison-RF ReID can significantly enrich mmWave human sensing with a wide spectrum of applications in security surveillance, tracking, and personalized services. Technically, we introduce a novel method for cross-modal similarity estimation that exploits inherent synergies between fine-grained 2D images and coarse-grained 3D Radar point clouds to effectively overcome their modal discrepancy. Through extensive experiments, we demonstrated that our proposed system can achieve 85% top-1 accuracy and 90% top-5 accuracy among 58 volunteers. © 2024 Copyright is held by the owner/author(s).

KW - deep learning

KW - millimeter wave

KW - person identification

UR - http://www.scopus.com/inward/record.url?scp=85211802665&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85211802665&origin=recordpage

U2 - 10.1145/3666025.3699340

DO - 10.1145/3666025.3699340

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 979-8-4007-0697-4

T3 - SenSys - Proceedings of the ACM Conference on Embedded Networked Sensor Systems

SP - 309

EP - 321

BT - SenSys '24

PB - Association for Computing Machinery

Y2 - 4 November 2024 through 7 November 2024

ER -

Liu R, Yao T, Shi R, Mei L , Wang S , Yin Z et al. Mission: mmWave Radar Person Identification with RGB Cameras. In SenSys '24: Proceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems. Association for Computing Machinery. 2024. p. 309-321. (SenSys - Proceedings of the ACM Conference on Embedded Networked Sensor Systems). Epub 2024 Nov 4. doi: 10.1145/3666025.3699340

Mission: mmWave Radar Person Identification with RGB Cameras

Abstract

Publication series

Conference

Bibliographical note

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this