Keypad entry inference with sensor fusion from mobile and smart wearables

This paper investigates the threat of password leakage from sensors on mobile phones and smart wearables. We investigate attack methods for recovering a short random number entered on PEDs using a combination of data collected by from microphone, accelerometer and gyroscope in devices on the person entering the number. We take into consideration features based on keypress sounds, from either phone and/or smart watch, in addition to hand movement acceleration and angular velocity captured by the smart watch. We used the fusion features from these three sensor sources to train a Neural Network to recover key entries on a keypad. Our method based on linear acceleration and angular velocity, with acoustic data used to segment individual keystrokes, showed an improved 74% probability to predict a single keystroke in one guess and 52% for a 6-digit PIN in three guesses.