LibGuides: Impact Metrics: Controversies

Limitations and Controversies

The world has entered an approach of systematic statistical evaluations, and so has academia. This is called the Publish or Perish (POP) mindset.

Keep in mind that the abuse of metrics can produce effects such as:

Researchers break up their production in order to increase their chances of being cited
Researchers focus on trendy areas to find more funding
Researchers can manipulate indicators by using self-citation or by partnering with others
A researcher who excels in his field could be excluded from a scientific department even if his research would advance knowledge because he was judged non-productive in terms of quantity
The use of metrics can be a way to evaluate the visibility, but no metrics can replace a human opinion to evaluate the work of a researcher

"In this short video, Dr Kevin Lalor, School of Social Sciences and Law, Dublin Institute of Technology, highlights some of the limitations of use of journal impact data in the social sciences and humanities and all the types of publication that are missed"

The publication pressure put on academics conducted them to ask for the improvement of evaluation criteria and appropriate indicators. Academics defend taking into account the quality of the work in relation to the quantity produced.

The San Francisco Declaration on the Evaluation of Research (DORA) "was initiated at the 2012 Annual Meeting of the American Society for Cell Biology by a group of editors and publishers of scholarly journals, recognizes the need to improve the ways in which the outputs of scientific research are evaluated". It covers all scientific disciplines and denounces the increasing use of metric tools (see poster presenting the objectives of DORA).

In 2015, the Leiden Manifesto sets out ten principles to guide research evaluation. (Hicks, D., Wouters, P., Waltman, L., Rijcke, S. D., & Rafols, I. (2015). Bibliometrics: the Leiden Manifesto for research metrics.)

The Metric Tide report (UK) was published in 2015. This report identifies 20 recommendations for UK research institutions to use "responsible metrics".

The Coalition for Advancing Research Assessment (COARA) drew up an agreement where 350 organizations from over 40 countries were involved. It was finalized in July 2022 and the signatories committed to work together to develop new assessment criteria, methods, and tools. It has been signed by several Swiss universities.

Other initiatives:

Snowball metrics - "Standardized research metrics – by the sector for the sector" is a collaboration between Elsevier and several universities (UK, US and Australia / New Zealand).

HuMetricsHSS - Humane Metrics in Humanities and Social Sciences (HuMetricsHSS) initiative is an exploratory project (at the moment) to rethink humane indicators of excellence in the humanities and social sciences.

Links

The Declaration on Research Assessment (DORA) "recognises the need to improve the ways in which the outputs of scholarly research are evaluated". List of organizations or individuals signatories of the declaration.

The LSE Impact Blog is "a hub for researchers, administrative staff, librarians, students, think tanks, government, and anyone else interested in maximising the impact of academic work in the social sciences and other disciplines".

The Metrics Toolkit provides information on research metrics in all disciplines. Among other things, you will find information about the sources of each metric, how they are calculated and how to use (or not use) them. You'll also find information on how each metric is calculated, where you can find it and how it should (or shouldn't) be applied.

The ORCID Blog to follow updates of ORCID activities.

Readings