Review
Social Sciences, Mathematical Methods
Elena A. Erosheva, Patricia Martinkova, Carole J. Lee
Summary: This study focuses on the impact of reviewer agreement on the quality of the peer review process and highlights two important issues when using IRR to evaluate peer review quality. It is found that zero local IRR estimates are more likely when considering subsets of top-quality proposals, but this does not mean arbitrariness in peer review. When complete ranges of proposals are considered, IRR estimates are above 0.6, indicating good reviewer agreement.
JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES A-STATISTICS IN SOCIETY
(2021)
Article
Psychiatry
Jeffrey D. Shahidullah, James Custer, Oscar Widales-Benitez, Nazan Aksan, Carly Hatchell, Jeffrey Newport, Karen Dineen Wagner, Eric A. Storch, Cynthia Claassen, Amy Garrett, Irma T. Ugalde, Wade Weber, Charles B. Nemeroff, Paul J. Rathouz
Summary: This study aims to describe the rater training protocol for administering semi-structured psychiatric interviews in large, multi-site studies and evaluate its effectiveness. The training involved synchronous and asynchronous learning modules, and certification included critique of mock scale administration. The protocol resulted in strong interrater reliability across all measures.
PSYCHIATRY RESEARCH
(2023)
Article
Engineering, Industrial
Christian Lins, Sebastian Fudickar, Andreas Hein
Summary: The OWAS method shows high agreement in assessing arm postures at over 98%, but lower agreement for leg postures (66-97%) and upper body postures (80-96%). There was no significant difference between raters with and without intense prior training in physical therapy, indicating general reliability especially for raters with non-specialized backgrounds but weaknesses in detecting specific postures.
APPLIED ERGONOMICS
(2021)
Article
Environmental Sciences
Daniel W. Smith, Samantha Wind, Igor A. Markov, Stephen Atwii Ongom, Jennifer Davis
Summary: Ensuring the functionality of drinking water supplies in low-and-middle-income countries has always been a challenge. A study on handpump functionality tests in Uganda found that the 10-minute leakage rate had the highest inter-rater reliability, followed by pump capacity. Flowrate, commonly measured manually, had poor inter-rater reliability. Based on the inter-rater reliability results, a fully quantitative procedure and an ordinal scale of physical handpump functionality were proposed.
SCIENCE OF THE TOTAL ENVIRONMENT
(2023)
News Item
Multidisciplinary Sciences
Max Kozlov
Summary: The US biomedical agency has decided to stop scoring researchers' expertise and institutions during grant evaluations.
Review
Management
Thomas Feliciani, Michael Morreau, Junwen Luo, Pablo Lucas, Kalpana Shankar
Summary: This study aims to explore how factors relating to grades and grading affect the correctness of choices made by review panels among submitted proposals, and identify interventions in panel design that can increase the correctness of choices. The results of the experiment indicate that increasing the number of grades used by panel members and giving panels a greater capacity for discriminating among proposals can improve the correctness of choices, while differences in grading standards among panel members do not significantly decrease correctness.
Article
Health Care Sciences & Services
Heeae Seo, Jeongseon Kim, Changseon Yu, Hyoungwon Lim
Summary: Assessing muscle tone is crucial for the diagnosis, prognosis, and treatment planning of developmental disabilities in children. The myotonometer, an assessment tool used in this study, showed good reliability for measuring muscle tone in children with DD. The intra-rater and inter-rater reliability were excellent, indicating that the myotonometer can objectively assess muscle tone and be used in clinical practice.
Article
Multidisciplinary Sciences
Chonnanid Limsakul, Kiattisak Sengchuai, Rakkrit Duangsoithong, Nattha Jindapetch, Jermphiphut Jaruenpunyasak
Summary: This study assessed the reliability of electromyography and range of motion measurements obtained using a knee exercise monitoring system. The results showed moderate to excellent inter- and intra-rater agreement for electromyography, but a small agreement bias for range of motion measurements.
Review
Medicine, Research & Experimental
Phillip Staibano, Jennifer Ham, Jennifer Chen, Han Zhang, Michael K. Gupta
Summary: This study conducted a systematic review and meta-analysis to determine the inter-rater reliability of thyroid ultrasound criteria. The findings suggest moderate inter-rater reliability, but the results are influenced by poor study quality and a lack of standardization.
Article
Education, Scientific Disciplines
Amir H. Sam, Michael D. Reid, Viral Thakerar, Mark Gurnell, Rachel Westacott, Malcolm W. R. Reed, Celia A. Brown
Summary: The study found that there is high inter-rater agreement among assessors from different regions in the UK when evaluating simulated student performances and expected standards for 'clear fail,' 'borderline,' 'clear pass,' and 'good' levels.
Article
Surgery
Chin-Hung Wei, Shih-Chiang Shen, Yih-Cherng Duh, Kuei-Yen Tsai, Hsin-An Chen, Shih-Wei Huang
Summary: The Moorthy checklist (MC) and laparoscopic skill competency assessment tool (LS-CAT) are reliable tools for evaluating laparoscopic suturing in different backgrounds. LS-CAT is particularly reliable for medical students and trainees. The current assessment model of single measurement by multiple raters provides excellent reliability.
SURGICAL ENDOSCOPY AND OTHER INTERVENTIONAL TECHNIQUES
(2022)
Article
Agriculture, Dairy & Animal Science
Laura Van Driessche, Gilles Fecteau, Julie Arsenault, Lea Miana, Younes Chorfi, Marianne Villettaz-Robichaud, Pierre Helie, Sebastien Buczinski
Summary: The study aimed to determine the inter-rater reliability of scoring systems used to detect abomasal lesions in veal calves, comparing macroscopic and histological lesions. Various scoring systems were used to assess 76 abomasa retrieved from veal calves, with lesions classified into erosions, ulcers, and scars. The results showed a variable agreement among raters for the presence or absence of a lesion, with better agreement observed when lesions were combined within certain areas. Poor to moderate agreement was found for the number of lesions. The study highlights the importance of developing a reliable scoring system to effectively assess abomasal lesions in veal calves.
Article
Otorhinolaryngology
Ingi Min, Hyeonseong Woo, Jae Yoon Kim, Tae-Lim Kim, Yookyung Lee, Won Kee Chang, Se Hee Jung, Woo Hyung Lee, Byung-Mo Oh, Tai Ryoon Han, Han Gil Seo
Summary: This study aimed to develop a standardized protocol for the assessment of videofluoroscopic dysphagia scale (VDS) and to demonstrate its inter-rater and intra-rater reliability. The protocol was developed by dysphagia experts and 60 patients from three medical centers were recruited to evaluate its reliability. The results showed excellent inter-rater and intra-rater reliabilities for the VDS, regardless of the evaluators' experience, VFSS equipment, and dysphagia etiologies. The VDS can be a useful assessment scale in the quantitative analysis of dysphagia based on VFSS findings.
Article
Medicine, General & Internal
Steffen Mickenautsch, Stefan Rupf, Ivana Miletic, Ulf Tilman Straehle, Richard Sturm, Faheema Kimmie-Dhansay, Kata Vidosusic, Veerasamy Yengopal
Summary: This study aimed to establish the inter-rater reliability of the Composite Quality Score (CQS-2) and compare it with the first version (CQS-1). Four independent raters evaluated 45 clinical trial reports using both versions. The results showed that both CQS-1 and CQS-2 had very high inter-rater reliability, and there was no significant difference between the two versions. This suggests that the CQS-2 may be a suitable appraisal tool for prospective controlled clinical therapy trials.
FRONTIERS IN MEDICINE
(2023)
Article
Integrative & Complementary Medicine
Ayako Maeda-Minami, Tetsuhiro Yoshino, Yuko Horiba, Tomonori Nakamura, Kenji Watanabe
Summary: This study aimed to assess the inter-rater reliability of Kampo medicine pattern diagnosis for chronic diseases and found that modularization may improve the reliability of diagnosis.
JOURNAL OF ALTERNATIVE AND COMPLEMENTARY MEDICINE
(2021)
Article
Dentistry, Oral Surgery & Medicine
Vlaho Brailo, Patrick McKnight, Alexander R. Kerr, Giovanni Lodi, Peter B. Lockhart
Review
Integrative & Complementary Medicine
Eric Jacobson, Lisa Conboy, Dolma Tsering, Monica Shields, Patrick McKnight, Peter M. Wayne, Rosa Schnyer
JOURNAL OF ALTERNATIVE AND COMPLEMENTARY MEDICINE
(2019)
Article
Integrative & Complementary Medicine
Rosa N. Schnyer, Patrick McKnight, Lisa A. Conboy, Eric Jacobson, Anna T. Ledegza, Mary T. Quilty, Roger B. Davis, Peter M. Wayne
JOURNAL OF ALTERNATIVE AND COMPLEMENTARY MEDICINE
(2019)
Article
Dentistry, Oral Surgery & Medicine
Jean-Luc C. Mougeot, James M. Davis, Jing Zhao, Kathleen A. Sullivan, Martin H. Thornhill, Patrick E. McKnight, Casey Stephens, Peter B. Lockhart
ORAL SURGERY ORAL MEDICINE ORAL PATHOLOGY ORAL RADIOLOGY
(2020)
Article
Dentistry, Oral Surgery & Medicine
Peter B. Lockhart, Martin H. Thornhill, Jing Zhao, Larry M. Baddour, James Davis, Patrick E. McKnight, Gregg H. Gilbert, Rahma Mungia, Jean-Luc Mougeot
JOURNAL OF THE AMERICAN DENTAL ASSOCIATION
(2020)
Article
Psychology, Clinical
Todd B. Kashdan, David J. Disabato, Fallon R. Goodman, James D. Doorley, Patrick E. McKnight
PSYCHOLOGICAL ASSESSMENT
(2020)
Article
Psychology, Social
Todd B. Kashdan, David J. Disabato, Fallon R. Goodman, Patrick E. McKnight
PERSONALITY AND INDIVIDUAL DIFFERENCES
(2020)
Article
Psychology, Social
Todd B. Kashdan, Fallon R. Goodman, David J. Disabato, Patrick E. McKnight, Kerry Kelso, Carl Naughton
PERSONALITY AND INDIVIDUAL DIFFERENCES
(2020)
Article
Education & Educational Research
Timothy Curby, Patrick McKnight, Lisa Alexander, Simone Erchov
ASSESSMENT & EVALUATION IN HIGHER EDUCATION
(2020)
Article
Psychology, Experimental
James D. Doorley, Todd B. Kashdan, Lisa A. Alexander, Dan Blalock, Patrick E. McKnight
MOTIVATION AND EMOTION
(2019)
Article
Psychology, Multidisciplinary
Giacomo Bono, Jeffrey J. Froh, David Disabato, Dan Blalock, Patrick McKnight, Samantha Bausert
JOURNAL OF POSITIVE PSYCHOLOGY
(2019)
Article
Psychology, Multidisciplinary
Allison M. Bock, Kelly B. Cartwright, Patrick E. McKnight, Allyson B. Patterson, Amber G. Shriver, Britney M. Leaf, Mandana K. Mohtasham, Katherine C. Vennergrund, Robert Pasnak
FRONTIERS IN PSYCHOLOGY
(2018)
Article
Ergonomics
Samuel S. Monfort, John J. Graybeal, Amanda E. Harwood, Patrick E. McKnight, Tyler H. Shaw
THEORETICAL ISSUES IN ERGONOMICS SCIENCE
(2018)
Article
Psychology, Experimental
Todd B. Kashdan, Fallon R. Goodman, Melissa Stiksma, Cayla R. Milius, Patrick E. McKnight
Article
Psychology, Social
Todd B. Kashdan, Melissa C. Stiksma, David J. Disabato, Patrick E. McKnight, John Bekier, Joel Kaji, Rachel Lazarus
JOURNAL OF RESEARCH IN PERSONALITY
(2018)