Mobile-based Human Computer Interaction Systems with Energy Efficiency

Abstract

With the rapid development in computation and sensing ability, mobile devices, e.g., smart watches, virtual reality (VR) headsets, armbands, and mobile phones are becoming the most prominent platforms for human-computer interaction (HCI). To fully utilize the abundance of information provided by mobile sensing platforms, existing works explore multi-modal interfaces, including gaze tracking, hand tracking, and electroencephalography (EEG) signals from non-invasive brain interfaces, for more natural and immersive interactions. However, the mobility hardware design of these devices imposes many challenges on these interaction modalities, namely, limited computation ability compared with their desktop counterparts, noisy and context dependent sensor data, and restricted battery life. These constraints may lead to low tracking accuracy and discontinuity of service, reducing the effectiveness and experience of interactions.

In this dissertation, we first identify learning efficiency and energy efficiency as two primary objectives in mobile interactions. And we propose solutions to improve the robustness and reliability of newly emerged mobile interfaces, including iris tracking and EEG-based brain interaction. Particularly, we achieve fine-grained iris tracking using commercial phone RGB cameras. We address key issues when processing the eye model with RGB cameras, including a semi-supervised framework to accurately extract eye iris boundary from noisy pixels. And we built a gaze tracking prototype to demonstrate its capability to enable accurate gaze interaction on mobile phones. Furthermore, to achieve a longer battery life, we design an intelligent workload-aware power management framework based on dynamic voltage and frequency scaling (DVFS) to improve the power efficiency of dynamic workloads on mobile devices. The key observation is that the workload characteristics could be described and learned in a hyperspace composed of multiple dimensions derived from hardware metadata, forming the contextual indicators that benefit the exploring of processors' optimal voltage-frequency settings for different workloads.
In summary, this dissertation provides a careful study towards energy-efficient multi-modal human-computer interactions for mobile devices, investigating the solutions and enabling insightful application designs.

Date of Award	25 Apr 2023
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Zhenjiang LI (Supervisor)