Utility theory, minimum effort, and predictive coding

Research in text classification (a.k.a. predictive coding) usually focuses on the design of algorithms for training a text classifier from manually coded data, and for automatically classifying, via the trained classifier, large amounts of uncoded data. Very little attention, if any, has been given to what comes next, i.e., to supporting human annotators in inspecting (and correcting if appropriate) the automatically classified documents with the goal of reducing the amount of classification error present in the data. In this talk I will present recent research aimed at minimizing the amount of human inspection effort needed to reduce the classification error down to a desired level. The fact that for many applications false positives and false negatives weigh differently calls for an approach to this task based on utility theory.