Modeling human driving behaviors plays a key role in many domains, including "teaching" autonomous vehicles to drive, simulating testing environments for autonomous driving, and analyzing some human traffic phenomena. From these three perspectives, (1) imitation learning approaches have been widely studied to imitate human driving behaviors and serve as the motion planning module of the autonomous driving system, and (2) a huge amount of human-like car-following models and lane-change models have been developed to simulate and analyze human-like traffic flow. Modeling human driving behaviors in the former task serves as a driving instructor to "tell" autonomous vehicles what behaviors should perform and focus on the imitation of a general human driver. While in the latter task, they pay more attention on modeling different types of drivers and vehicles to simulate realistic and diverse traffic environments.

This dissertation also focuses on the above two tasks. From the perspective of controlling the autonomous vehicles to drive like a human driver, we study both the decision-making task and the motion planning task. Specifically, the driving style is integrated to improve the human-like decision-making model. The mathematical-based motion planning algorithm is combined with the imitation learning based approach to benefit from both of them. From the perspective of simulating diverse and realistic traffic environments, we study the human-like car-following model, which can be assigned with different explicit driving styles.

First, we develop a motion planning scheme that integrates the algorithmic approach with the learning-based approach, where the former guarantees safety, and the latter one helps to generate human-like trajectories. The sampling-based algorithm is selected as the base of our approach. A generative model is applied to generate sample points near the human trajectory for the sampling-based algorithmic approach. We also empirically study the relationship between the number of sample points and the complexity of the environment.

Second, we design a human-like discretionary lane-change (DLC) decision-making model with driving style awareness. The driving style of the surrounding vehicles and the ego vehicle are all considered using Driving Operational Pictures (DOP). Besides the driving style, the contextual traffic factors, which are used in most of the rule-based decision-making models, are also integrated into our approach. By doing so, our model can imitate human drivers' decision-making maneuvers by learning the driving style of the ego vehicle and considering the driving style of surrounding vehicles.

Third, we develop a novel generative hybrid car-following (CF) model, which achieves high accuracy in characterizing dynamic human CF behaviors and is able to generate realistic human CF behaviors for any given observed or even unobserved driving style. The mathematical-based CF model Intelligent Driver Model (IDM) is calibrated with time-varying parameters to characterize human CF behaviors of different driving styles accurately. Neural processes (NP) are applied to generate human CF behaviors with stochasticity. The relationship between the intermediate variable of NP and time-varying parameters of IDM is explored and modeled to infer CF behaviors of unobserved driving styles.

The proposed approaches in this dissertation will contribute to both autonomous driving and transportation systems. The motion planning approach will guarantee safety while imitating human driving behaviors for autonomous vehicles. The lane-change decision-making model will improve the interactions between the ego vehicle and surrounding human driving vehicles. The hybrid CF model will contribute to simulating customized realistic and diverse traffic environments for analyzing traffic phenomena and testing autonomous vehicles.