XSolar: A Generative Framework for Solar-based Human Gesture Sensing via Wearable Signals

Solar cell-based gesture recognition is emerging as an innovative approach to facilitate seamless human-machine interactions. However, the primary challenge is the scarcity of datasets for solar-based gesture recognition. To address this issue, we introduce XSolar, an innovative cross-modal gesture recognition framework that utilizes Inertial Measurement Unit (IMU) data to generate the equivalent solar photocurrent signals for the corresponding gestures. The core concept is to harness the readily available IMU signals from modern wearable devices to create solar photocurrent response signals. Nonetheless, this process presents several technical challenges, including the disparity between solar photocurrent and IMU signal characteristics, the inherent noise in solar gesture sensing, and the complex nature of human gestures. To navigate these challenges, our first step is to establish a methodology that processes both IMU and photocurrent data to capture essential features of gestures reliably. Subsequently, we introduce a generative model that converts IMU data into synthetic photocurrent signals. Finally, we implement a Convolutional Neural Network (CNN) model designed to refine gesture recognition accuracy. Our experimental findings confirm that XSolar consistently delivers an impressive 92.65% accuracy, showcasing its robustness. © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.