In this post-PC era, wearable devices, such as smart ring, smartwatches, and AR glasses, have been permeated into our daily routine. On one hand, some of the wearable devices are suffering from low interactivity. For example, the input space on low-cost VR head mounted display such as GoogleCardboard are restricted by a button that only provides binary input. On the other hand, the wearable devices were closed to or in contact with the body, providing the potential of capturing body activities to support natural inputs. For example, smart ring could served as a springboard for capturing subtle finger motions to support natural gestural inputs.

Gesture-based interactions offer opportunities for natural and intuitive user experiences with low cognitive and perceptual load required, which could be one potential user-friendly input modality for wearable devices. In this thesis, I identified and explored two deep-learning-based novel sensing approaches for enriching the interactivity of wearable devices: 1) supporting the gesture-based interaction on the surface of a low-cost passive wearable accessory; 2) sensing the gestural inputs on the body surface near the wearable device. I start with exploring designing and sensing on-device gesture to enrich the limited input space on low-cost virtual-reality (VR) head-mounted display (HMD). I presented GestOnHMD, a gesture-based interaction technique and a gesture-classification pipeline that leverages the stereo microphones in a commodity smartphone to detect the tapping and the scratching gestures on the front, the left, and the right surfaces on a mobile VR headset. Taking the Google Cardboard as our focused headset, we first conducted a gesture-elicitation study to generate 150 user-defined gestures with 50 on each surface. We then selected 15, 9, and 9 gestures for the front, the left, and the right surfaces respectively based on user preferences and signal detectability. We constructed a data set containing the acoustic signals of 18 users performing these on-surface gestures, and trained the deep-learning classification pipeline for gesture detection and recognition. Lastly, with the real-time demonstration of GestOnHMD, we conducted a series of online participatory-design sessions to collect a set of user-defined gesture-referent mappings that could potentially benefit from GestOnHMD.

As finger-worn devices, such as smart ring, are vantage springboard for capturing finger motion, I further explore sensing thumb-to-index-finger (T2I) microgestrues with single index-finger-worn smart ring. I present EFRing, an index-finger-worn ring-form device for detecting thumb-to-index-finger microgestures through the approach of electric-field (EF) sensing. Based on the EFRing signal change caused by the T2I motions, we proposed two machine-learning-based data-processing pipelines: one for recognizing/classifying discrete T2I microgestures, and the other one for tracking continuous 1D T2I movements. Our experiments on the EFRing microgesture classification showed an average within-user accuracy of 89.5% and an average cross-user accuracy of 85.2%, for 9 T2I microgestures. For the continuous tracking of 1D T2I movements, our method can achieve the mean-square error of 3.5% for the generic model and 2.3% for the personalized model. Our 1D-fitts'-law target-selection study shows that the proposed tracking method with EFRing is intuitive and accurate for real-time usage. Lastly, we proposed and discussed the potential applications for EFRing.

With the above two projects, I present the design, implementation and evaluation of two novel sensing techniques to support on-surface gestural interactions with two form-factor of wearable devices: mobile head-mounted display and smart ring. My contributions extend the input space on those devices in a intuitive manner, which also unlock a variety of potential applications.