Measuring Literary Quality. Proxies and Perspectives

Computational studies of literature have adopted approaches from statistics and social sciences to perform large scale studies of fiction, and recent work has sought to approximate the success of literary texts using some proxy for literary quality, such as collections of human judgments, sales-numbers or lists indicating canonicity. However, most quantitative studies of literary quality use one such measure as a golden standard of literary judgement without fully reflecting on what it represents. Conclusions drawn from these studies are nonetheless bound to mirror a particular conception of literary quality associated with the chosen metric. To address this issue, we provide a discussion of the interrelation of various “proxies of literary quality” within a corpus of novels published in the US in the late 19th and 20th century, performing correlations and comparisons across 14 different proxies. We start with a heuristic distinction between expert-based literary judgments, such as those represented by college syllabi and literary anthologies, and crowd-based judgments, such as GoodReads’ ratings, and explore the differences between these and other proxies that fall in-between, such as library holding numbers, prestigious literary prizes, and classics book series. Our findings suggest that works favored in expert-based judgments tend to score lower on GoodReads, while those longlisted for awards tend to score higher and enjoy greater circulation in libraries. Generally, two main kinds of “quality perception” emerge as we map the literary judgment landscape: one associated with canonical literature, and one with more popular literature, which may indicate that judgements of canonicity or literariness are not equal to popularity among readers. Additionally, our study suggests that prestige in genre-literature, as represented by main genre-fiction awards such as the Hugo or World Fantasy Award, constitute distinct proxies on their own, though more closely aligned to popular than canonical proxies.