Automatic Detection of Safety Violations using Vision and Knowledge

Description

The construction industry is notorious for its poor safety performance, which accounts for approximately 20% of all workplace deaths worldwide. As accidents often result from safety violations, regularly checking and monitoring construction workers' behaviors and surrounding environments to identify the safety violations are regarded as an effective way of improving construction job site safety performance. However, traditional safety checking and monitoring usually rely on manual observation and analysis based on the safety rules, which is time-consuming and labor-intensive. Moreover, manual safety checking and monitoring are subjective and prone to human error. Image-based surveillance technology has recently attracted attention in construction safety research as a non-intrusive and cost-effective monitoring technique. However, the application of image-based safety monitoring in construction practice is limited because 1) the existing applications usually focus on a single safety violation (e.g., non-hardhat-use, uncovered openings) with straightforward reasoning rules hardcoded. The links between the low-level objects (e.g., hardhat) and high-level safety knowledge (e.g., a specific inspection rule in the standard) are missing to support more complex safety inspection tasks; 2) the existing image-based safety violation detection tools in construction research often lack a comprehensive understanding of rich context information in the images, limiting the automatic safety violation detection performance. Therefore, an improved vision-based construction safety violation detection approach similar to the human reasoning process is needed.The objective of this project is to develop an approach that can support multi-safety violation detection from job site images. As falling-from-height is a leading cause of accidents on construction job sites, work-at-height will be used as the scope to demonstrate the approach development in this project. To tackle the limitations of existing research in construction safety, the following research questions will be answered: 1) How to present the complex safety inspection rules in a knowledge graph that can be understood by the computer, supporting the image-based reasoning? 2) What model could support the computer to understand the scenes in the job site images required for safety violation detection? 3) How to induce the safety violation based on the knowledge graph and the extracted scene information? To answer those questions, an improved knowledge graph structure will be designed, and the safety handbooks' knowledge statements will be converted into a knowledge graph accordingly. Then, a scene description generation model will be designed and trained to allow the computer to understand the job site image, and provide the description in texts. Next, an automatic annotation model will be designed to convert the texts into tuples, which are fed into the knowledge graph-based safety violation reasoning mechanism designed by the project team. Finally, an integrated system prototype will be developed and demonstrated using the images collected from actual job sites.If successful, the proposed image-based construction safety violation detection approach could help integrate human violation detection intelligence into autonomous job site safety surveillance for improving safety performance. Meanwhile, as the data analysis process in safety violation detection gets more automatic, it can significantly reduce the safety workload, addressing the workforce shortage problems in the construction industry. The proposed approach can also be extended to other aspects (e.g., job site productivity analysis, security management, etc.) in the architecture and civil engineering industry.