Maintaining a diagnostic knowledge-based system : A control theoretic approach

Diagnostic knowledge-based systems are used in a variety of application domains to classify incoming events as malicious or benign. Over time, the effectiveness of the system may decrease as the application environment changes and (in some cases) malicious attempts are made to misuse it. Hence frequent adjustments to the system with knowledge by a human become necessary. We study the problem of assessing the optimal amount of effort a human expert should expend to continuously recalibrate the system over a finite planning horizon. Using the ROC curve to derive a measure for system quality, we minimize total costs by balancing misclassification costs on the one hand against the cost of human intervention on the other. An additional novelty in our problem is that we explicitly model system deterioration due to intentional misuse. The problem is cast as an optimal control model in which the goal is to choose the rate at which human effort must be expended to minimize total cost. We find that the optimal solution usually follows a bang-bang plus singular structure. The singular structure of the solution is especially interesting because it suggests a steady-state level of human effort for problem with an infinite time horizon.