4.6 Review

Reviewers' Ratings and Bibliometric Indicators: Hand in Hand When Assessing Over Research Proposals?

Journal

PLOS ONE
Volume 8, Issue 6, Pages -

Publisher

PUBLIC LIBRARY SCIENCE
DOI: 10.1371/journal.pone.0068258

Keywords

-

Funding

  1. National RD Plan [TIN2008-03180-E]
  2. FPU Grant of the Ministerio de Economia y Competitividad of the Spanish Government

Ask authors/readers for more resources

Background: The peer review system has been traditionally challenged due to its many limitations especially for allocating funding. Bibliometric indicators may well present themselves as a complement. Objective: We analyze the relationship between peers' ratings and bibliometric indicators for Spanish researchers in the 2007 National R&D Plan for 23 research fields. Methods and Materials: We analyze peers' ratings for 2333 applications. We also gathered principal investigators' research output and impact and studied the differences between accepted and rejected applications. We used the Web of Science database and focused on the 2002-2006 period. First, we analyzed the distribution of granted and rejected proposals considering a given set of bibliometric indicators to test if there are significant differences. Then, we applied a multiple logistic regression analysis to determine if bibliometric indicators can explain by themselves the concession of grant proposals. Results: 63.4% of the applications were funded. Bibliometric indicators for accepted proposals showed a better previous performance than for those rejected; however the correlation between peer review and bibliometric indicators is very heterogeneous among most areas. The logistic regression analysis showed that the main bibliometric indicators that explain the granting of research proposals in most cases are the output (number of published articles) and the number of papers published in journals that belong to the first quartile ranking of the Journal Citations Report. Discussion: Bibliometric indicators predict the concession of grant proposals at least as well as peer ratings. Social Sciences and Education are the only areas where no relation was found, although this may be due to the limitations of the Web of Science's coverage. These findings encourage the use of bibliometric indicators as a complement to peer review in most of the analyzed areas.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.6
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

Review Computer Science, Interdisciplinary Applications

An empirical review of the different variants of the probabilistic affinity index as applied to scientific collaboration

Zaida Chinchilla-Rodriguez, Yi Bu, Nicolas Robinson-Garcia, Cassidy R. Sugimoto

Summary: Responsible indicators are crucial for research assessment, requiring transparency and accuracy to ensure fairness and reproducibility. This paper reviews different variants of the Probabilistic Affinity Index (PAI), demonstrating its utility in scientific collaboration and providing comparisons of implementation results. The selection of a particular variant depends on the research question, with recommendations for appropriate use in specific contexts.

SCIENTOMETRICS (2021)

Article Sport Sciences

Body mass index and aerobic capacity: The key variables for good performance in soldiers

Eva Orantes-Gonzalez, Jose Heredia-Jimenez, Manuel Escabias

Summary: This study analyzed the influence of aerobic capacity, body composition, and leg strength on performance during an obstacle test with and without load carriage in soldiers. The results indicated that VO2max and BMI are the best predictors of obstacle course performance with and without load carriage, while jump height was not a significant factor.

EUROPEAN JOURNAL OF SPORT SCIENCE (2022)

Article Environmental Studies

Researchers' institutional mobility: bibliometric evidence on academic inbreeding and internationalization

Vit Machacek, Martin Srholec, Marcia R. Ferreira, Nicolas Robinson-Garcia, Rodrigo Costas

Summary: This study proposes institutional mobility indicators based on researchers' mobility flows. The findings reveal that institutions in Central, Eastern, and Southern Europe have the highest share of insiders, while institutions in Northern America and Western and Northern Europe have a higher share of foreign outsiders. Foreign outsiders are most common in small and wealthy countries. No disciplinary differences are observed.

SCIENCE AND PUBLIC POLICY (2022)

Article Communication

The year we became unsustainable: Analysis of Spanish production in Sustainability (2020)

Rafael Repiso, Adoracion Merino-Arribas, Alvaro Cabezas-Clavijo

Summary: This study examined the publication patterns of Spanish authors in the journal Sustainability, finding that a majority of papers were published in special issues edited by authors of the same nationality, with some articles having minimal relevance to the sustainability theme. Additionally, it was noted that a significant portion of the output came from the same editor or colleagues from the same institution, raising debates about publishing papers on other topics.

PROFESIONAL DE LA INFORMACION (2021)

Article Computer Science, Interdisciplinary Applications

Introducing a bibliometric index based on factor analysis

Pilar Valderrama, Evaristo Jimenez-Contreras, Manuel Escabias, Mariano J. Valderrama

Summary: This study developed a Weighted Factor Index through factor analysis with VARIMAX rotation to provide a new classification for journals, different from the traditional Journal Impact Factor. By incorporating 16 metrics from three different databases, the index summarizes information from all indicators and was applied to rank journals in the Information Science & Library Science category.

SCIENTOMETRICS (2022)

Article Multidisciplinary Sciences

A value creation model from science-society interconnections: Archetypal analysis combining publications, survey and altmetric data

Irene Ramos-Vielba, Nicolas Robinson-Garcia, Richard Woolley

Summary: This paper examines the interplay between science and society and proposes a conceptual model for value creation. The study finds that there are different ways of connecting science and society in different scientific fields.

PLOS ONE (2022)

Article Computer Science, Interdisciplinary Applications

COVID-19 and the scientific publishing system: growth, open access and scientific fields

Gabriela F. Nane, Nicolas Robinson-Garcia, Francois van Schalkwyk, Daniel Torres-Salinas

Summary: This study models the growth of scientific literature related to COVID-19 and provides predictions based on ARIMA and exponential smoothing models. These predictions are crucial for scientists, politicians, and citizens to make scientifically informed decisions when facing a large corpus of scientific literature.

SCIENTOMETRICS (2023)

Article Information Science & Library Science

Valuation regimes in academia: Researchers' attitudes towards their diversity of activities and academic performance

Nicolas Robinson-Garcia, Rodrigo Costas, Gabriela F. Nane, Thed N. van Leeuwen

Summary: Evaluation systems are heavily criticized for misusing bibliometric indicators, leading to a culture where academics strive to meet the expected standards. This study investigates the alignment between researchers' values and expectations with the evaluation system. Through a multiple case study, combining curriculum analysis and semi-structured interviews, the research examines how departments in Dutch universities balance their own valuation regimes with the evaluation schemes. The study proposes a model to understand the impact of discipline and career stage on the diversity of academic activities, concluding that the misalignment is not solely due to metric abuse, but also a lack of contextualized and adaptable evaluation tools.

RESEARCH EVALUATION (2023)

Article Mathematics

Scalar Variance and Scalar Correlation for Functional Data

Cristhian Leonardo Urbano-Leon, Manuel Escabias, Diana Paola Ovalle-Munoz, Javier Olaya-Ochoa

Summary: In this paper, we study the problem of comparing functional data in Functional Data Analysis (FDA) where the existing summary statistics are not sufficient. We present a coherent redefinition of some common summary statistics to solve comparability problems and discuss the functional forms coherence of some statistics already present in the FDA. Simulation study and application examples are provided to demonstrate the effectiveness and consistency of our proposed variance.

MATHEMATICS (2023)

Review Communication

The bibliometric journey towards technological and social change: A review of current challenges and issues

Daniel Torres-Salinas, Nicolas Robinson-Garcia, Evaristo Jimenez-Contreras

Summary: This article reviews the current trends and challenges in the field of bibliometrics, covering six aspects: the explosion of databases, the inflation of metrics, its relationship to Data Science, searching for meaning, evaluative bibliometrics, and diversity and profession. The evaluation encompasses three dimensions of the bibliometrics field: technological, theoretical, and social. Finally, the article advocates for the principles of evaluative bibliometrics, balancing the power of metrics with expert judgment and science policy.

PROFESIONAL DE LA INFORMACION (2023)

Article Computer Science, Interdisciplinary Applications

ChatGPT and the potential growing of ghost bibliographic references

Enrique Orduna-Malea, Alvaro Cabezas-Clavijo

Summary: This letter warns about the use of ChatGPT to generate fake bibliographic references, which may lead to the presence of non-existent papers being cited in research publications available online. While the extent of the problem is still uncertain, both journals and publishers should remain vigilant in order to prevent the proliferation of false references.

SCIENTOMETRICS (2023)

Letter Computer Science, Interdisciplinary Applications

Bibliometric denialism

Daniel Torres-Salinas, Wenceslao Arroyo-Machado, Nicolas Robinson-Garcia

SCIENTOMETRICS (2023)

Article Mathematics, Interdisciplinary Applications

Comparison of Positivity in Two Epidemic Waves of COVID-19 in Colombia with FDA

Cristhian Leonardo Urbano-Leon, Manuel Escabias

Summary: The study uses functional data methodology to compare differences between two waves of COVID-19 contagion in Colombia, proposing an alternative statistical test that performs well in small samples and provides an overall assessment of the significance of positivity curves.

STATS (2022)

Article Computer Science, Information Systems

A Comprehensive Dataset of the Spanish Research Output and Its Associated Social Media and Altmetric Mentions (2016-2020)

Wenceslao Arroyo-Machado, Nicolas Robinson-Garcia, Daniel Torres-Salinas

Summary: This paper presents data on research publications and social media impact of scientists affiliated with Spanish institutions between 2016 and 2020. It also includes information on researchers whose work gained significant attention on social media and non-academic outlets. The datasets provide bibliographic metadata, subject areas, altmetric indicators, and information on researchers and their publications.
Article Communication

Coverage and distribution of altmetric mentions in Spain: a cross-country comparison in 22 research fields

Daniel Torres-Salinas, Nicolas Robinson-Garcia, Wenceslao Arroyo-Machado

Summary: This study presents a cross-country comparison of altmetric coverage between Spain and a selection of 16 countries in 22 research fields. The results show that altmetric coverage in Spain is similar to that in other countries, but it is lower in areas such as Arts & Humanities and Social Sciences.

PROFESIONAL DE LA INFORMACION (2022)

No Data Available