Towards Automatic Design of Deep Neural Networks

Artificial Intelligence has been enjoying a remarkable resurgence in recent years, which is mainly reinvigorated by a family of machine learning techniques called deep neural networks (DNNs). DNNs feature sophisticated architectures which endow them with exceptionally distinguished learning capabilities, although their performances often heavily depend on good settings of architecture and parameters. Although learning millions of parameters in a DNN is no longer an issue by leveraging on modern high performance computing (HPC) facilities, how to determine the most suitable DNN architecture (prior to parameter learning) to tackle a given task remains an open problem. It has been a common practice to manually design the architecture of an DNN via trial and error. However, this design paradigm is not effective for DNN users with insufficient design experience, and even experienced DNN researchers may not be omnipotent to handle the ever-growing number, variety, scale, and complexity of applications. Design automation thus provides a promising way to address these issues, which should be capable of systematically taking into account the design criteria, methodologies, approaches, and efficiency.With the rapid advancement of computing facilities, design automation for DNNs has become computationally feasible recently. This project aims to make a big step forward in this research area by devising an automatic DNN architecture design framework, which can automatically search for desirable architectures subjected to multiple design criteria that reflect design requirements in different aspects, while exploring and exploiting reusable design knowledge in various forms to a great extent. To make the best use of design knowledge to assist in the current design task, multiple related subtasks will be derived from the current design task and solved simultaneously as a multi-task optimisation problem so that the concurrent solving processes of these subtasks can facilitate one another via design knowledge transfer to ultimately augment the efficiency and effectiveness of solving the current design task. Under this framework, customized approaches will be devised for designing different types of DNNs and implemented in a computationally efficient and scalable way on GPU clusters (a modern HPC facility). The efficacy of the proposed techniques will be demonstrated on both benchmark problems and real-world applications related to sensor and voice signal processing. Ultimately, the project aims at developing automatic design of DNNs into a common choice of techniques in practice such that the pervasive utilities of DNNs can be boosted to generate greater and broader socio-economic benefits.