Keystroke Recognition with the Tapping Sound Recorded by Mobile Phone Microphones

Research output: Journal Publications and ReviewsRGC 21 - Publication in refereed journalpeer-review

2 Scopus Citations
View graph of relations

Detail(s)

Original languageEnglish
Pages (from-to)3407-3424
Journal / PublicationIEEE Transactions on Mobile Computing
Volume22
Issue number6
Online published21 Dec 2021
Publication statusPublished - Jun 2023

Abstract

Mobile phones nowadays are equipped with at least dual microphones. We find when a user is typing on a phone, the sounds generated from the vibration caused by finger's tapping on the screen surface can be captured by both microphones, and these recorded sounds alone are informative enough to localize the user's keystrokes. This ability can be leveraged to enable useful application designs, while it also raises a crucial privacy risk that the private information typed by users on mobile phones has a great potential to be leaked through such a recognition ability. In this paper, we address two key design issues and demonstrate, more importantly alarm people, that this risk is possible, which could be related to many of us when we use our mobile phones. We implement our proposed techniques in a prototype system and conduct extensive experiments. The evaluation results indicate promising successful rates for more than 4000 keystrokes from different users on various types of mobile phones.

Research Area(s)

  • Acoustics, Microphones, Microwave integrated circuits, Mobile handsets, Noise measurement, Privacy, Vibrations, Side-channel attacks, acoustic signal processing, smart devices