This paper explores how both annotation procedures and automatic
detection (i.e. classifiers) can be used to assess the consistency of textual literary
concepts. We developed an annotation tagset for the ‘literary comment’ – a
frequently used but rarely defined concept – and its subtypes (interpretative
comment, attitude comment and metanarrative/metafictional comment) and
trained a multi-output and a binary classifier. The multi-output classifier shows
F-scores of 28% for attitude comment, 36% for interpretative comment and
48% for meta comment, whereas the binary classifier achieves F-scores up to
59%. Crucially, both our annotation and the automatic classification struggle
with the same subtypes of comment, although annotation and classification
follow completely different procedures. Our findings suggest an inconsistency
in the overall literary concept ‘comment’ and most prominently the subtypes
‘attitude comment’ and ‘interpretative comment’. As a best-practice-example,
our approach illustrates that the contribution of Digital Humanities to Literary
Studies may go beyond the automatic recognition of literary phenomena.
