Q-learning-based Schedule Generation for Excavating Hard Rock Tunnels under Resource Constraints

Description

The development of underground structures (e.g., transportation tunnels and sewage treatment caverns) enhances land availability and reduces environmental impacts. In recent years, the demand for underground structures has rapidly increased in Hong Kong; however, due to significant uncertainties in ground conditions, planners for these projects frequently encounter cost overruns, schedule delays, and associated risks. To overcome these challenges, planners would obtain significant benefits (e.g., reduction of delays and their associated risks) from adapting stochastic programming and feedback control approaches. As a fundamental step, this proposed research focuses on implementing such approaches for hard rock tunnels under resource constraints. Formal implementation of such approaches requires planners to make two specific decisions: (1) schedules and (2) updating cycles of ground conditions. Although the repetitive and comprehensive generation and evaluation of schedules are necessary for the decision-making, existing methods have the following limitations: (1) existing construction process simulation methods cannot consider multiple activity allocation rules for different states (i.e., the excavation progress for each phase) and modify the rules as excavation progresses and (2) existing genetic algorithms are limited to reusing the outcomes for each state from previous learning. To address these gaps, this research proposes a framework that supports the formal adaptation of such approaches for excavation scheduling of hard rock tunnels under resource constraints. To consider uncertain ground conditions under the given equipment fleet, schedules are defined as optimal activity allocation rules for each state. This framework will consist of two methods: (1) a Q-learning-based schedule generation method that enhances the reusability of outcomes from previous learning and (2) an estimation method for durations resulting from schedules modified on the basis of the ground condition updating cycle, which allows for the application of multiple rules for different states. The schedule generation method will allow planners to rapidly identify optimal rules for specific states via the Q tables. In addition, the method for estimating durations will enable planners to rapidly and consistently estimate durations resulting from schedule modification based on the ground condition updating cycle. By conducting three case studies, the research team will identify how the adaptation of stochastic programming and feedback control impacts the reduction of excavation durations, which are also related to time-dependent costs, and their associated risks. In the long term, this research will be extended to complex underground structures and thus will support the development of a variety of underground structures in urban and adjacent fringe areas.