Human Behavior Sensing and Understanding with Sparse Wearable Sensors
Student thesis: Doctoral Thesis
Related Research Unit(s)
Wearable sensors, e.g., accelerometers, gyroscopes, magnetometers, Electromyography (EMG) sensors, can sense human body spots where the sensors are attached to, and generate valuable sensory data. Therefore, existing works design a corpus of applications to sense and understand human behaviors in Internet of Things (IoT) systems, e.g., fitness assessment, elderly care and Human-Computer Interaction (HCI). However, to sense user's fine-grained behaviors, excessive sensors are required to be attached to the user's body usually, leading to a high system deployment cost and maintenance burden. Although existing commercial wearable devices, e.g., smart watch, already integrate many sensors, they can sense one specific spot on the user’s body only. In this dissertation, we study the problem to sense and understand the user's behaviors with sparse wearable sensors and utilize the user's upper body, including arms, fingers and trunk, as a concrete example to investigate the system design. In particular, we first leverage the accelerometer and gyroscope from smart watch to achieve a real-time 3D arm skeleton tracking. The sensors sense the user's wrist merely, while our solution can track the movement of all the key joints on the user's arm by two innovative designs. Next, we further sense and track the movement from the distal end of the arm, i.e., the user's hand. Instead of pasting many sensors to fully cover the user's arm, we track the 3D hand pose of 14 hand skeleton points over time using the EMG sensory data from a commercial armband device only. A key challenge addressed in this work is that the armband EMG sensors inevitably collect mixed EMG signals from multiple forearm muscles because the fixed sensor positions on the device, while prior bio-medical models are built on the isolated EMG signal inputs collected from different forearm spots for different muscles. The fine-grained skeleton tracking achieved above can be used in useful applications directly. In many cases, we should further understand user's activities or behaviors through learning techniques and the tracked skeleton can be also used for the activity inference. However, the sensed data from wearable sensors may be not reliably available, e.g., one wrist-worn wearable device is not worn by the user today and the skeleton from this arm can not be recovered, which means one part of input for the recognition model is missing. To this end, we introduce an effective design to prevent training multiple neural networks for handling all possible missing input combinations. In general, this dissertation carefully studies the sensor sparsity problem when we use wearable devices to sense and understand fine-grained human behaviors by designing new and effective techniques.