During the past few years, deep learning models have shown great success in natural language processing tasks. By embedding one hot token vector into low-dimensional space, deep learning models can grasp the semantics of text during end to end learning process. However, in task oriented dialogue system (TOD), the annotated text data is scarce and usually not enough for these data-hungry deep learning models. Therefore, pre-trained model approaches are proposed to overcome the limitation. In this thesis, we delve into pre-trained model based approaches for task oriented dialogue system, according to how external knowledge is incorporated into the system, i.e., API database, FAQ knowledge base, and hybrid system. In particular, we investigate two modules in TOD based on API database, including intent detection and dialogue state tracking, and two modules in TOD based on FAQ knowledge base, including knowledge selection and knowledge grounded response generation.

Task oriented dialogue system based on API database consists of several modules, including intent detection, slot filling, dialogue state tracking, dialogue policy management, and natural language generation. In this thesis, we mainly investigate the module of intent detection and dialogue state tracking. Intent detection aims to detect the intent of user’s utterance. The limitation of this module is that the annotated dataset of this module is not large enough. Thus, we propose to fully utilize annotated data in a different but related domain, i.e., natural language inference (NLI). We pre-train the model on NLI dataset, then replace the final linear layer with an SVM layer, and finally train the model on intent detection dataset. Experiment result shows that our method can detect user’s intent with high accuracy, stability and generalization ability. Dialogue state tracking aims to track the state of the TOD in the form of slot value pairs. The performance is evaluated by slot accuracy and joint goal accuracy. We find that the slot accuracy is still based on exact matching with variants list and many semantically correct slot values are judged as incorrect. Therefore, we propose to make use of the power of contextualized embedding of pre-trained language model to overcome the limitation. We also propose constructive sampling to provide supervision clues for the model, and propose reformulation to make use of real data in a related domain. Experiment result shows that our two methods can evaluate dialogue state tracking accurately and correlate well with human judgements.

Task oriented dialogue system based on FAQ knowledge base consists of several modules, including knowledge seeking turn detection, knowledge selection, and knowledge grounded response generation. In this thesis, we mainly investigate the module of knowledge selection and knowledge grounded response generation. Knowledge selection aims to select proper FAQ knowledge given the dialogue context. The limitation of this module is that there is a domain gap between the dialogue knowledge joint data and the general text data that pre-trained language model is pre-trained on. Thus, we propose to use domain adapted pre-training to bridge the domain gap. We also propose to use dynamic masking probability to further facilitate the optimization process. Experiment result shows that our method can improve the performance of downstream knowledge selection task by a large margin. Knowledge grounded response generation aims to generate system response given dialogue context and selected FAQ knowledge. The limitation of this module is that the decoding process of the generator model is probabilistic and may lead to hallucination effect. Therefore, we propose to train a ranker model with negative sampling, and then ranks the generated responses of the generator model to improve performance. Experiment result shows that our method can improve the quality of generated responses in terms of all evaluation metrics.

Hybrid task oriented dialogue system is a combination of TOD based on API database and TOD based on FAQ knowledge base. It can incorporate knowledge from both sources and can be built on the basis of these two kinds of systems.