Image captioning in chinese for construction activity scene understanding using a pre-trained cross-modal language model

With the popularity of surveillance cameras, many vision-based artificial intelligence (AI) agents have been applied to construction projects, significantly improving management efficiency and workers' productivity. However, only a few works study scene understanding because it is one of the most challenging topics of intelligent monitoring. Besides, as a big construction country, China lacks corresponding AI research based on Chinese in the construction field, which seriously hinders the further development of China's construction industry. Therefore, this paper proposes a Vision-based BERT (V-BERT) model for construction activity scene understanding. A Chinese caption dataset named Images of Jobsite Daily Activity and Chinese Captions (IJDACC) is created to verify V-BERT's performance. Some data augmentation operations are then used to enlarge the training set further. Two evaluation systems are established to evaluate V-BERT's comprehensive performance. The experimental results show the V-BERT achieves state-of-the-art performance in the construction area with an average performance improvement of 171.20%.