Towards Smart Visual Sensor Data Representation with Intelligent Sensing in the Internet of Video Things

Description

Recent years have witnessed a tremendous amount of visual sensors deployed in ambient environments, which have greatly powered the internet-of-things with the advent of powerful 5G wireless networks. The astronomical amount of internet of video things (IoVT) pose great challenges to the storage, transmission, and utilization of visual sensor data around the internet. Despite demonstrated success in video coding and processing, in this project, we demonstrate that the representation capability of IoVT sensor data can be largely extended by exploiting the front-end intelligent sensing with enhanced embedded video analytics capabilities, and more importantly, the subsequent video data utilization pipeline can greatly benefit from such representation paradigm. Generally speaking, there are three distinguished properties of IoVT sensor data 1) the data are acquired in an uncontrolled physical environment without sophisticated equipment and elaborate acquisition preparations; 2) the data are featured with high volume and low-value density, instead of being entirely valuable with high utility; 3) the data possess underlying regularities, as they are generated without human intervention instead of being carefully processed and edited. As such, we propose a new scheme based on intelligent sensing at the front-end, in an effort to provide highly conceptual and extremely compact representation with the consideration of these characteristics. The pipeline is composed of four modules, including visual data tracing with representation-in- loop, utility-oriented resource allocation and data-driven based inference at the encoder side, as well as unsupervised deep reconstruction in an effort to better reform the visual data at the decoder side.  The application scope of the proposed research is much broader than data compression. The distribution of analytics and inference modules at the front-end can greatly shrink the data volume such that only the valuable data are concentrated on, which could optimize the whole sensing and networking infrastructures in terms of data volume, transmission latency, energy efficiency, etc. Moreover, the proposed scheme can also benefit video storage and transmission by exploiting external redundancies across different sensors. With the design of algorithms based on multidisciplinary approaches and promising preliminary verifications, we are confident that the proposed scheme can achieve a very compact representation of the all-weather, all-time, all-coverage mega-scale visual sensor data. More importantly, beyond data representation, the proposed scheme can also contribute to the technology innovations of front-end smart visual sensing, which will play more and more critical roles in numerous personal-scale and industrial-scale applications.