id	author	title	date	pages	extension	mime	words	sentences	flesch	summary	cache	txt
work_7stioriya5fr5f32pb2hrenvqm	Carole J. Lee	A Kuhnian Critique of Psychometric Research on Peer Review	2012	13	.pdf	application/pdf	5563	579	50	Until these empirical-cum-philosophical analyses are done, it will remain unclear the extent to which low interrater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers. Of the empirical research available on peer review, one of the "most basic, broadly supported, and damning" aspects is the failure for independent studies on single-rater reliability rates for grant review across disciplines. Some psychometrically oriented researchers suggest that levels of interrater reliability for expert reviewers should be at about 0.8 (or even 0.9; Marsh et al. appropriate for editors and grant panels to rely on differences in reviewer expertise in the evaluation of a submission. were true, we would expect low interrater reliabilities along evaluative dimensions, as researchers have discovered (Scott 1974; Marsh and Ball 1989; unclear the extent to which low interrater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers. "The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation." Behavioral and Brain Sciences 14:119–86.	./cache/work_7stioriya5fr5f32pb2hrenvqm.pdf	./txt/work_7stioriya5fr5f32pb2hrenvqm.txt