Achieving Lightweightness and Data Efficiency in Deep Reinforcement Learning for Compound Agent Learning

Abstract

Currently, Deep Reinforcement Learning (DRL) has emerged as a prominent technology in the field of artificial intelligence. It learns an optimal strategy for a given task through online interaction with the environment, enabling efficient decision-making and control in complex environments. DRL has proven to be the state-of-the-art (SOTA) approach for various application tasks such as complex robot control, autonomous driving, and solving combinatorial optimization problems.

However, practical applications of DRL still face three significant technical limitations. Firstly, a limitation in DRL is how to learn a better DRL policy from a given reward function, aiming for higher learning efficiency. As we know, DRL training relies on a reward function, and extracting more knowledge from a given reward function to facilitate the learning of a better policy has been a long-standing challenge in the DRL community. Secondly, DRL encounters difficulty in effectively generalizing across different tasks. In practical applications like autonomous driving, each task has distinct characteristics, such as weather conditions and road conditions, resulting in substantial task differences. Training a separate DRL model for each task becomes impractical. Thus, achieving effective generalization of a DRL model across different tasks is an open challenge in the DRL community. Lastly, the resource consumption of DRL makes its deployment on devices without advanced hardware impractical. In real-world tasks like StarCraft, learning a task often requires a large DRL model, which consumes significant storage and computational resources. Therefore, reducing the number of model parameters while minimizing performance trade-off is also an ongoing challenge in the DRL community.

Addressing the aforementioned limitations, namely improving learning efficiency, achieving efficient generalization, and reducing model consumption, is crucial for advancing the field of DRL. This thesis aims to individually investigate these limitations and propose novel solutions.

Firstly, the thesis explores the Multi-Reward Architecture (MRA) in DRL, a promising framework that enhances the learning efficiency of DRL. The thesis proposes a generic framework to improve existing MRA methods. Specifically, the proposed approach includes adding a policy branch corresponding to the original reward function, allowing MRA to learn from each sub-reward branch without disregarding the experience provided by the original reward. The Asynchronous Advantage Actor Critic (A3C) algorithm is employed to learn time-varying weights for all policy branches. These weights are transformed into a one-hot vector, facilitating the selection of the appropriate policy branch for action generation. Extensive experiments demonstrate the effectiveness of the method in enhancing three SOTA MRA methods across four tasks, considering factors such as episode reward, success rate, score difference, and episode duration.

Secondly, the thesis explores lifelong DRL, the current best paradigm for achieving cross-task generalization. The thesis presents a generic framework that aims to facilitate existing lifelong DRL methods in achieving few-shot generalization. Specifically, the proposed approach incorporates selective experience reuse, leveraging encountered state experiences to improve adaptation training for new tasks. Additionally, a softmax function is applied to the target Q-values to enhance the accuracy of evaluated Q-values, resulting in more optimal policies. Furthermore, the discrepancy in data distribution between policy and off-policy samples is measured and reduced, leading to improved adaptation efficiency. Through extensive experiments on three typical benchmarks, the method enhances the performance of six lifelong DRL methods by at least 25\% in terms of training speed, episode return, and average return across all episodes.

Finally, the thesis investigates Dynamic Sparse Training (DST), a prominent approach in the DRL community that enables lightweight DRL models during both training and inference. The thesis proposes a new topology adaptation strategy to improve existing DST methods in DRL, enhancing policy performance while preserving sparsity. The proposed approach involves developing a novel approach to calculate the importance of connections within the model. The sparse network topology is dynamically adjusted by dropping existing connections and introducing new connections based on their respective importance values. Through validation on eight widely-used simulation tasks, the method improves two SOTA DST approaches by up to 70\% in terms of episode return and average return across all episodes, across various levels of sparsity.

In summary, the solutions proposed in this thesis have been extensively validated through numerous experiments and effectively improve the performance of existing methods in terms of return, success rate, score difference, and episode duration, among other metrics. These proposed methods hold broad application prospects and can be applied to various current applications of DRL, including robotics, autonomous driving, gaming, and combinatorial optimization problems.

Date of Award	15 Jul 2024
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Jianping WANG (Supervisor) & Qingfu ZHANG (Supervisor)