doi: 10.1633/JISTAP.2025.13.2.4 Citation BibTeX RIS Mendeley

Scientometric Evaluation of Research Institutions: Identifying the Appropriate Dimensions and Attributes for Assessment

• Christian Schlögl (Department of Operations and Information Systems, University of Graz, Graz, Austria)
• Wolfgang G. Stock (Department of Information Science, Heinrich Heine University Düsseldorf, Düsseldorf, Germany)
• Gerhard Reichmann (Department of Operations and Information Systems, University of Graz, Graz, Austria)

Many descriptions and evaluations of research institutions apply publication and citation indicators, which can also be found in most of the popular rankings of universities all over the world. When comparing different institutions, the question arises whether scientometric indicators, and institutional rankings derived from them, are really valid. In our paper we discuss various dimensions of scientometric analyses which have a more or less strong impact on the results of research evaluations on the scientometric meso-level (e.g., department or university level). Concerning research output (based upon publications), we found nine different dimensions, namely time period, size of the institution, representatives of the institution, data source, language, document types and their weighting, co-authorship, document length, and access option. Concerning research impact (based upon citations), there are 14 different dimensions. Using the simple example of two university departments, we present different rankings while varying a few attributes of the suggested dimensions. As the results show, there are large differences between the performed rankings. Each ranking stresses a different aspect; hence there is not the one and only valid ranking. All the popular global university rankings work with arbitrary selections of indicator combinations, making the results more or less arbitrary. The greatest issues are the rankings’ incomplete empirical bases, the non-consideration of fractional counting of authors, and – most importantly – the disregarding of the institutions’ sizes.