Multimodal Sensors for Hand Gesture Recognition

基於多模態傳感器的手勢識別方法

Student thesis: Doctoral Thesis

View graph of relations

Author(s)

Related Research Unit(s)

Detail(s)

Awarding Institution
Supervisors/Advisors
Award date19 Jun 2020

Abstract

Humans use their fingers to mediate most mechanical interactions between themselves and the world. Decades of research have developed various approaches to utilise hand gestures for human-computer interface applications. Gesture recognition technology enables users to interact with a computer system without physically touching it. Many modalities have been used in gesture recognition as input to touch-less systems including sensor gloves, cameras, and non-invasive muscle activity-based gadgets. This thesis investigates the novel idea of using pressure-based sensors to record bio-acoustics at the human wrist for hand gesture recognition. Two major investigations were conducted: (1) using a sensor array of microphones alone and (2) a multimodal approach by including accelerometers and gyroscopes with a microphone-based sensor array for the classification of hand gestures. In each case, a suitable hardware prototype was designed and prepared in our laboratory, an experimental procedure was devised, and ten human subjects were recruited for data recording.

In the initial investigation, the use of only microphone-based sensor units for recording hand motions from the wrist was studied. After the acoustic data was recorded, features were extracted and classified using Support-vector Machine and Linear Discriminant Analysis methods. The results demonstrate average classification accuracy above 90% using the American Sign Language alphabet and numbers. These results indicate that pressure-based sensing at the human wrist can be utilised for hand gesture recognition.

After the success of using the microphone sensor for hand gesture recognition, further investigations were carried out using a multimodal approach. The objective of this study was to investigate if an armband of multimodal sensors could be worn to detect basic hand gesture input from the wrist. Accelerometer and gyroscope sensors were used along with microphones in a single ring at the wrist. The gesture set consisted of thirteen commonly used daily life gestures. After the multimodal data were recorded using feature extraction and classification, the results demonstrate that microphone-based sensor units increase the classification accuracy if used with other modalities.

Moreover, a clinical study was conducted to detect carpal tunnel syndrome (CTS) using multimodal sensors that include an accelerometer and a gyroscope along with microphones. This study was facilitated by a medical team at a local hospital. The medical team helped to develop the experimental procedure and provided the CTS patients. The preliminary results from this experiment support the idea of using pressure-based sensing for CTS detection.

To summarise, this thesis focuses two important aspects of using microphone-based sensor units as a new modality over the wrist for hand gesture recognition: microphone array and multimodal ring with microphones, an accelerometer, and a gyroscope. In addition, a clinical study for CTS detection using multimodal sensors was also carried out. The results indicate that acoustic signatures from the human wrist can be utilised for hand gesture recognition. The proposed technique demonstrates a promising means to develop a low-cost wearable device using a microphone-based sensing mechanism.