MirrorTablet: Exploring a low-cost mobile system for capturing unmediated hand gestures in remote collaboration

Direct and natural images of hand gestures have been shown to benefit remote collaboration on physical tasks in several settings, including ad-hoc ones. However, to capture such unmediated hand gestures within collaborative tasks, existing approaches require stationary hardware systems or heavily instrumented mobile devices, making them unfeasible for use at ad-hoc workplaces. We present MirrorTablet, a low-cost mirror-based system taking advantage of the built-in front-facing camera of a tablet to capture the user’s unmediated hand interactions on and above the screen. This system requires minimal instrumentation of the tablet and can be easily (un-)mounted, making it suitable for mobile usage. A user study with ten pairs of participants on a helper-worker setup working on construction tasks showed that MirrorTablet improved task completion time and had positive effects on participants’ perceived workload when working with unfamiliar tasks compared to using a common sketch-only interface. In addition, qualitative feedback yielded design considerations on hand visualization for mobile device remote collaboration on physical tasks.