Modeling Paradigms for Adaptive Decentralized Traffic Control via a Rollout Reinforcement Learning Approach

This paper proposes a dynamic modeling framework that integrates rollout reinforcement learning (RL) with tractable traffic models for adaptive decentralized traffic control. The traffic control problem is represented as a Markov decision process, aiming to minimize network delay under stochastic traffic demand through acyclic signal plans. To address the instability issue of model-free RL methods, we incorporate a model-based rollout component into the RL optimization framework, where the point queue and spatial queue models are adopted to explicitly represent network flow propagation. This approach enables more reliable estimations of traffic states and system costs through model-based rollout simulations derived from real data within the planning horizon. Fed with the rollout traffic states, a state-dependent surrogate function is proposed to parameterize the unknown optimal value function, which significantly mitigates the curse of dimensionality when recursively calculating the Bellman equation. A multi-step temporal difference learning method is employed to train the neural network-based surrogate function with realized transitions collected from the external environment, enhancing the accuracy of the value function approximation while correcting model-plant mismatch. Additionally, the framework adopts a decentralized architecture, enabling asynchronous operation across intersections that reduces computational burden while maintaining network-level coordination. Validation on three urban networks demonstrates that the proposed controllers can effectively mitigate delays and queue spillback even in oversaturated scenarios. Notably, the point queue model achieves performance comparable to the spatial queue model with high precision, highlighting that the proposed framework is generic across different traffic models even without sophisticated calibration and complex propagation laws. © 2026 IEEE.