A Ubiquitous 3D Input System based on μIMU-Vision Fusion Technology

Ubiquitous Computing is a post-desktop model of human-computer interaction in which information processing has been extensively integrated into everyday objects and activities. On the other hand, Intelligence Devices (e.g., smart phone, tablet, 3D TV etc.) are now becoming substitutes for the desktop computers. Hence, the traditional input devices of desktop computers such as the keyboard may someday completely be replaced by small and ubiquitous input devices. Therefore, human-to-computer natural interaction input devices play an increasingly crucial role to 3D user interface for mobile devices and digital home systems, and other fringe applications such as scientific visualization, virtual reality, and multimedia systems.For this project, we propose to develop a novel Ubiquitous 3D Input System based on MEMS motion sensing and computer vision technologies. This novel human-to-computer interactive device aims to capture human hand-writing or drawing motions in real-time and also communicates with a computing platform (i.e., an iPad, iPhone, for labtop PC, etc.) to process the human motion trajectories for character recognition. Products which aspire to capture the new market of ubiquitous input devices for mobile applications have begun to surface commercially but most of them are still limited to only 2D interactive space, and also are hampered by at least three problems: 1) requirement of specialized and bulky accessories (e.g., special whiteboards or papers are need) that hinder the technologies for becoming highly popular; 2) cannot provide long-term stability with inertial measurement unit (IMU) technology due to inertial sensors' integration and random noise errors; 3) relative high cost of the products which makes them inaccessible to many professional or home users.The proposed system will allow end-users to have the total freedom of interacting with computing devices in 3D space ubiquitously, i.e., they can use the system anywhere and on any surface, without the need for specialized papers or whiteboards, or being constrained to only creating their input to computing device in 2D space. Within 12 months, our team will demonstrate a 3D Digital Input System that has a geometry similar in shape and size as a conventional whiteboard marker and with the associated software tools for sensor fusion and for hand-gesture motion tracking and recognition.