Analyses of the Reflection Cues for Glass Surface Detection
- Rynson W H LAU (Principal Investigator / Project Coordinator)Department of Computer Science
- Benjamin Wan Sang WAH (Co-Investigator)
DescriptionTransparent glass surfaces are ubiquitous. We see them in outdoor scenes (e.g., shop display windows and building glass doors) and indoor scenes (e.g., windows and shower doors). They typically do not have their own visual appearances but reveal those behind them. Due to this lack of a consistent visual appearance, existing object detection methods often fail to detect them, but the objects behind them instead. If such detectors are used in a robot (or a drone) to assist navigation, the robot can easily crash onto glass surfaces, endangering itself and people around it. While there are some methods proposed for detecting small glass objects, which have well-defined shapes or boundary features for detection, glass surfaces do not possess such properties. Although a few methods use an ultrasonic sensor for detecting the presence of glass surfaces, they are only for indoor applications and cannot be used to produce depth maps of the scene for real-time applications, due to low sampling rates and interferences. We observe that humans often rely on observing some reflection artifacts on a glass surface in order to be aware of its presence. Hence, in this project, we propose to apply the intrinsic property of glass surfaces (that they reflect as well as transmit light) for glass surface detection, and to conduct in-depth studies of glass reflections for tackling the problem automatically. Our goal is to be able to detect these glass surfaces with high accuracy for real-time applications, in both indoor and outdoor environments. As a preliminary work, we have demonstrated in our CVPR 2021 paper that through detecting reflections, a deep-learning model may be guided to locate where the glass surfaces are. However, this preliminary study only considers glass reflections in general, which may not always be reliable due to the diverse degrees of reflection in indoor/outdoor scenes. In this proposal, we will explore different types of glass reflection and formulate them as cues for glass surface detection. In particular, we will focus on the following three types of reflection, and develop models to extract them for reliable detection. 1.Depth Reflection Analysis: As a glass surface both reflects and transmits light, the depths of a glass surface returned by a depth sensor may consist of both reflected and transmitted depths. Hence, the depth field collected over a glass surface is expected to be noisier than a non-glass surface. In this part of the research, we will investigate this property of the depth fields on glass/non-glass surfaces. We study how to model it, and how it correlates with the RGB image reflection property so as to combine them for joint learning. 2.Double Reflection Analysis: A piece of glass has two contact surfaces (the two sides of it). When a light ray hits the first contact surface, it is divided into two rays. One represents the transmission ray and the other reflection ray. When the transmission ray hits the second contact surface, it is once again divided into a transmission ray and a reflection ray. As a result, there are a total of two reflected rays and one transmitted ray. As these two reflected rays are shifted by a small amount, the reflected content from a piece of glass surface should be blurrier than the transmitted content, and the degree of blur should depend on the thickness of the glass. In this part of the research, we will investigate how to detect this double reflection blur and how we may model it as a cue for glass surface detection. 3.Multi-view Reflection Analysis: We observe that humans sometimes move their heads left and right to sense how the reflection may change in order to gather more confidence in determining the presence of a glass surface. Multi-view analysis can typically produce a stronger cue for detection, but requires handling more input images. In this part of the research, we will investigate how we may fuse this multi-view reflection information for a more reliable detection of glass surfaces. We will also study if there is an optimal number of views that can produce the best overall performance. At the end of the project, we will deliver models that incorporate the three types of reflection cues for glass surface detection. We expect these cues to complement each other, leading to better performances over our preliminary model, with both indoor and outdoor scenes.
|Effective start/end date||1/10/23 → …|