4.5 Review

Latent Markov modeling applied to grant peer review

Journal

JOURNAL OF INFORMETRICS
Volume 2, Issue 3, Pages 217-228

Publisher

ELSEVIER SCIENCE BV
DOI: 10.1016/j.joi.2008.05.003

Keywords

latent Markov model; latent class analysis; peer review; multi-stage evaluation process; reliability

Ask authors/readers for more resources

In the grant peer review process we can distinguish various evaluation stages in which assessors judge applications on a rating scale. Research on the grant peer review process that considers its multi-stage character scarcely exists. In this study we analyze 1954 applications for doctoral and post-doctoral fellowships from the Boehringer Ingelheim Fonds (B.I.F.), which are evaluated in three stages (first: evaluation by an external reviewer; second: internal evaluation by a staff member; third:final decision by the B. I. F. Board of Trustees). The results of a latent Markov model ( in combination with latent class analysis) show that a fellowship application has a chance of approval only if it is recommended for support already in the first evaluation stage, that is, if the external reviewer's evaluation is positive. Based on these results, a form of triage or pre-screening of applications seems desirable. (C) 2008 Elsevier Ltd. All rights reserved.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.5
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

Article Computer Science, Interdisciplinary Applications

Diversity and interdisciplinarity: Should variety, balance and disparity be combined as a product or better as a sum? An information-theoretical and statistical estimation approach

Ruediger Mutz

Summary: Diversity is a central concept in ecology, social sciences, and bibliometrics. This study proposes a probability-based diversity indicator that reconceptualizes the components of diversity as entropy masses, addressing an inconsistency issue in existing indicators. The overall diversity of research projects in terms of entropy is estimated, with journal articles being the most balanced output type across research areas.

SCIENTOMETRICS (2022)

Article Multidisciplinary Sciences

How relevant is climate change research for climate change policy? An empirical analysis based on Overton data

Lutz Bornmann, Robin Haunschild, Kevin Boyack, Werner Marx, Jan C. Minx

Summary: This study examines the connection between climate change research and policy-making. It found that climate change papers cited in policy documents receive more citations on average. The study also observed an impact of international climate policy cycles on policy document publication and identified similar research fields between the scientific and policy communities.

PLOS ONE (2022)

Article Multidisciplinary Sciences

Same-gender citations do not indicate a substantial gender homophily bias

Alexander Tekles, Katrin Auspurg, Lutz Bornmann

Summary: Previous studies have shown that papers written by male scientists receive more citations than those written by female scientists, and this could be explained by a gender homophily bias, where scientists tend to cite others of the same gender. However, this bias may have been overestimated as previous studies have overlooked structural aspects, such as the gender composition of research topics. When controlling for research topics at a detailed level, there is little evidence to support a gender homophily bias in citation decisions. This study highlights the importance of controlling for gendered specialization in research topics when investigating gender bias in science.

PLOS ONE (2022)

Review Computer Science, Interdisciplinary Applications

Use of bibliometrics for research evaluation in emerging markets economies: a review and discussion of bibliometric indicators

Hamdi A. Al-Jamimi, Galal M. BinMakhashen, Lutz Bornmann

Summary: The governments of emerging market countries have invested significantly in scientific research, resulting in high research volume, impact, and international collaboration. It is important to evaluate their research performance using indicators such as productivity, impact, and collaboration. These indicators are crucial for developing research directions, policies, and meeting management needs.

SCIENTOMETRICS (2022)

Article Communication

The Social Systems Citation Theory (SSCT): A proposal to use the social systems theory for conceptualizing publications and their citation links

Iman Tahamtan, Lutz Bornmann

Summary: The normative theory of citing views citations as rewarding tools, while the social constructivist theory sees them as persuasion tools. However, existing theories fail to fully explain all citation motives, leading to the proposal of a new theory called social systems citation theory (SSCT). SSCT integrates previous theories and models, distinguishing authors' motives as belonging to psychological systems and publications with citation links as belonging to social science systems. The theory explains the autonomous operation of these systems and their interaction, offering a framework for empirical bibliometric studies.

PROFESIONAL DE LA INFORMACION (2022)

Article Computer Science, Information Systems

Revolutions in science: The proposal of an approach for the identification of most important researchers, institutions and countries based on co-citation reference publication year spectroscopy exemplified at research on physical modelling of Earth's climate

Lutz Bornmann, Robin Haunschild, Werner Marx

Summary: Reference Publication Year Spectroscopy (RPYS) is a bibliometric method used to reveal the historical roots of research topics or fields. It identifies the most frequently referenced publications within a specific reference publication year, instead of the most highly cited papers. This study demonstrates how RPYS can be applied to identify important researchers, institutions, and countries in breakthrough research, using the example of climate modeling and global warming prediction.

JOURNAL OF INFORMATION SCIENCE (2023)

Article Multidisciplinary Sciences

Anchoring effects in the assessment of papers: An empirical survey of citing authors

Lutz Bornmann, Christian Ganser, Alexander Tekles

Summary: In this study, we empirically investigated the influence of numerical information as anchors, such as citation impact, on the assessment of cited papers. We conducted a survey among corresponding authors and assigned them to different treatment groups receiving various additional numerical information. Our results suggest that the assessment of paper quality is influenced by the citation impact information, but not by other numerical information like an access code or journal impact.

PLOS ONE (2023)

Article Computer Science, Interdisciplinary Applications

John Mearsheimer's academic roots: a reference publication year spectroscopy of a political scientist's oeuvre

Anton Gruber, Alexander Tekles, Lutz Bornmann

Summary: This paper aims to analyze the academic roots of political scientist John J. Mearsheimer using the method of Reference Publication Year Spectroscopy (RPYS). By compiling a list of his most cited works and examining temporal peaks, the study reveals both the foundation of his theory of International Relations and his focus on current events and conflicts.

SCIENTOMETRICS (2023)

Article Computer Science, Interdisciplinary Applications

Which papers cited which tweets? An exploratory analysis based on Scopus data

Robin Haunschild, Lutz Bornmann

Summary: Many altmetric studies have examined the frequency of paper mentions on Twitter. This study investigates the citations of tweets in papers to assess their potential relevance. However, the findings suggest a low number of citations and indicate that tweets are more often used as study objects rather than influential content. The subject areas with the most citations of tweets are Social Sciences, Arts and Humanities, and Medicine, with COVID-19/corona pandemic being the dominant topic.

JOURNAL OF INFORMETRICS (2023)

Article Computer Science, Interdisciplinary Applications

Efficiency of universities and research-focused institutions worldwide: The introduction of a new input indicator reflecting institutional staff numbers

Lutz Bornmann, Sabine Gralka, Felix de Moya Anegon, Klaus Wohlrabe

Summary: One of the core indicators in scientometrics is the number of papers published by a unit within a given period. To properly assess such indicators, it is necessary to consider the unit's available resources. This study introduces a new input indicator, the number of unique authors mentioned on the institutions' papers, and calculates efficiency scores for over 3100 institutions from the higher education sector using this new indicator and output indicators. The results show a strong correlation between the new input indicator and institutional staff numbers.

JOURNAL OF INFORMETRICS (2023)

Article Multidisciplinary Sciences

Are quality assessments in science affected by anchoring effects? The proposal of a follow-up study

Lutz Bornmann, Christian Ganser

Summary: This study aims to empirically examine the assessment of scientific papers using the anchoring-and-adjustment heuristic. By surveying corresponding authors and analyzing their adjustments to the anchor value, the research investigates whether the evaluation can be influenced by both quality-related information and unrelated numerical factors. The study seeks to shed light on whether bibliometrics or other numerical measures create the social order they intend to measure.

PLOS ONE (2023)

Article Computer Science, Interdisciplinary Applications

Identification of potential young talented individuals in the natural and life sciences: A bibliometric approach

Robin Haunschild, Lutz Bornmann

Summary: This study proposes a citation-free method for identifying potential young talented individuals based on their early publication performance. Three different indicators and their combinations were used to define potential talent. The best performing indicator combination was applied to identify young potentially talented individuals who published their first paper between 2007-2011, resulting in a set of 46,200 individuals that can be downloaded for free.

JOURNAL OF INFORMETRICS (2023)

Article Computer Science, Interdisciplinary Applications

What do we know about the disruption index in scientometrics? An overview of the literature

Christian Leibel, Lutz Bornmann

Summary: The paper provides a comprehensive review of the original disruption index (DI1) and its variants in scientometrics. It explains the technical and theoretical properties of these indices and examines their validity and limitations. The review highlights the higher convergent validity of modified index variants compared to DI1 and emphasizes the need for further research on the validity of disruption scores.

SCIENTOMETRICS (2023)

Article Information Science & Library Science

Applied usage and performance of statistical matching in bibliometrics: The comparison of milestone and regular papers with multiple measurements of disruptiveness as an empirical example

Felix Bittmann, Alexander Tekles, Lutz Bornmann

Summary: The paper discusses the use of statistical matching as an alternative to linear regression models in bibliometrics to estimate effects and remove bias. Through analyzing milestone papers and nonmilestone papers, the study examines disruption indicators and evaluates the performance of different matching algorithms in terms of covariate balancing and disruptiveness assessment. Results show that coarsened exact matching (CEM) and entropy balancing (EB) perform well in balancing covariates, while DI5 and DEP are effective in evaluating disruptiveness of published papers.

QUANTITATIVE SCIENCE STUDIES (2022)

Article Information Science & Library Science

Investigating dissemination of scientific information on Twitter: A study of topic networks in opioid publications

Robin Haunschild, Lutz Bornmann, Devendra Potnis, Iman Tahamtan

Summary: This study investigates public attention to opioid scholarly publications on Twitter using topic networks. The results show that Twitter users mainly discuss opioid publications using generic terms, and there is a significant presence of bot accounts in the discussion. The study also finds that topic networks generated by bot and nonbot accounts overlap, indicating that excluding bot accounts may not be necessary for generating topic networks.

QUANTITATIVE SCIENCE STUDIES (2022)

Article Computer Science, Interdisciplinary Applications

Enhancing semantic text similarity with functional semantic knowledge (FOP) in patents

Hao Teng, Nan Wang, Hongyu Zhao, Yingtong Hu, Haitao Jin

Summary: In this paper, a new method based on functional semantic knowledge (FOP) is proposed for patent similarity calculation. Furthermore, patent STS datasets are processed and released as benchmarks. Preliminary results show that FOP-based methods are more suitable for STS tasks when combined with IPC codes, weights' assignments, and patent pre-trained vectors.

JOURNAL OF INFORMETRICS (2024)

Article Computer Science, Interdisciplinary Applications

An automatic and association-based procedure for hierarchical publication subject categorization

Cristina Urdiales, Eduardo Guzman

Summary: Subject categorization of scientific publications is important for evaluating paper quality. Traditional mechanisms for categorization have been questioned, and a new method based on association rules is proposed. The method automatically defines publication categories based on the repetition or absence of relevant descriptors. The empirical study in the field of Physical Sciences and Engineering shows that the proposed method produces consistent and suitable categorization results.

JOURNAL OF INFORMETRICS (2024)