To protect computer networks, intrusion detection systems (IDSs) are widely deployed to identify various threats. With the evolution of adversarial tools and techniques, network intrusions are currently becoming more and more sophisticated. This makes an isolated IDS vulnerable to many complicated or multi-step attacks. To boost the detection performance, collaborative intrusion detection network (CIDN) is developed by allowing an IDS node to exchange data with each other. Due to the distributed architecture, insider attacks are a big threat for CIDNs, where an attacker has the authorized access within the network. Trust management schemes are necessary and vital to help evaluate the reputation of a node. However, these schemes may still be vulnerable to some advanced insider attacks.

In this thesis, we focus on challenge-based trust mechanism, which evaluates the trustworthiness of IDS nodes by measuring the satisfaction between the received feedback and the expected answers. We firstly design two specific advanced attacks to compromise such mechanism.

- PMFA. Passive message fingerprint attack enables a set of malicious nodes to communicate with each other and distinguish a normal request from challenges. Then, malicious nodes can maintain their reputation by sending untruthful feedback to only a normal request.

- SOOA. Special On-Off attack allows malicious nodes to keep responding normally to one node while acting abnormally to another node. This can affect the process of trust evaluation in another IDS node.

Our experimental results demonstrate that both types of attacks can compromise challenge-based CIDNs in practice, i.e., malicious nodes remain undetected by keeping their trust values above the threshold. To defend against such threat, we accordingly propose some approaches to enhance the robustness of trust management, especially challenge-based trust mechanism for CIDNs.

- Intrusion sensitivity. We develop a notion of intrusion sensitivity to measure different capability of detecting specific attacks for IDS nodes, and then design a sensitivity-based trust management scheme for enhancing CIDNs, through leveraging supervised learning and expert knowledge to help automatically assign the value of sensitivity. We identify that it can help improve the detection of insider attacks like PMFA and SOOA, whereas it is still a challenge for these nodes to automatically assign the values.

- Honey challenge. Focused on PMFA, we design an approach called Honey Challenge to enhance the security of challenge-based trust mechanism through delivering challenges in a similar way of sending normal requests, in such a way malicious nodes cannot accurately identify the normal requests.

- Message verification. We propose a compact but efficient approach to help identify PMFA and SOOA through inserting a verification alarm into each normal request. As the verification alarm is randomly added, it increases the cracking difficulty for insider attackers.

In the evaluation, we compare the performance of several supervised classifiers in assigning sensitivity values and investigate our sensitivity-based trust model under different attack scenarios. The results indicate that applying intrusion sensitivity can help identify both PMFA and SOOA by highlighting the impact of expert nodes. We also explore the performance of honey challenge and message verification in both simulated and practical environments. Our evaluation demonstrates that both approaches can help enhance the robustness of challenge-based CIDNs against PMFA and SOOA. In the end, we further discuss some challenges on the improvement of collaborative intrusion detection.