Preferential text classification: learning algorithms and evaluation measures

In many applicative contexts in which textual documents are labelled with thematic categories, a distinction is made between the primary and the secondary categories that are attached to a given document. The primary categories represent the topics that are central to the document, while the secondary categories represent topics that the document somehow touches upon, albeit peripherally. This distinction has always been neglected in text categorization (TC) research. We contend that the distinction is important, and deserves to be explicitly tackled. The contribution of this paper is three-fold. First, we propose an evaluation measure for this preferential text categorization task, whereby different kinds of misclassifications involving either primary or secondary categories have a different impact on effectiveness. Second, we establish baseline results for this task on a well-known benchmark for patent classification in which the distinction between primary and secondary categories is present; these results are obtained by using state-of-the-art learning technology such as multiclass SVMs (for detecting the unique primary category) and binary SVMs (for detecting the secondary categories). Third, we improve on these results by using a recently proposed class of algorithms explicitly devised for learning from training data expressed in preferential form, i.e. in the form 'for document d_i, category c' is preferred to category c' '; this allows us to distinguish between primary and secondary categories not only in the testing phase but also in the learning phase, thus differentiating their impact on the classifiers to be generated.