What about research quality? A discussion on qualitative judgement and quantitative criteria in research assessment

The reform of research assessment since the San Francisco Declaration on Research Assessment (2012) and the Agreement on Reforming Research Assessment (ARRA) (2022) has focused on the contrast between quantity and quality. The Coalition for Advancing Research Assessment (CoARA) was set up to promote the principles and commitments contained therein. The actions of ARRA signatories are set out in the four core commitments that constitute the heart of the reform and its guiding framework. In particular, the second commitment states to "base research assessment primarily on qualitative evaluation for which peer review is central, supported by the responsible use of quantitative indicators" when meaningful and relevant. The ARRA identifies peer review as "the most robust known method for assessing research quality” and has the advantage of being in the hands of the research community. CoARA and the ARRA thus place qualitative judgement - as opposed to quantitative indicators - at the centre, implying the existence of an opposition between 'quality' and 'quantity'. This calls for a cultural change based on the principle that it is necessary to publish less (abandoning the 'publish or perish' logic) and publish better (making results, data and processes transparent, accessible and reproducible, and paying more attention to the integrity of research). The aim of this change is to produce research that is more robust, more rigorous, more responsible. In short, of higher quality. This approach has been criticised. One criticism, in the many debates following presentations on the reform that I attended, is that quantitative indicators are objective and qualitative judgement is subjective and arbitrary. This paper aims to refute both objections starting from the second. To do this, it draws on the definition of quality - and qualitative - by R.M. Pirsig, who devoted his entire life to defining a Metaphysics of Quality.