BIAS IN PEER-REVIEW:
A review of the article Sexism and Nepotism in Peer-review, by Christine Wennerås and Agnes Wold (Nature 387:341-343)
Keywords: gender bias, Sweden, Medical Research Council, bibliographic analysis, impact, ISI.
Department of Biochemistry, University of Victoria, B.C., Canada,
Received December 2, 1997, published December 4, 1997
Within the scientific community, peer review is the chief method for evaluating both research grant applications and manuscripts for journal publication. The existence of bias in peer review is generally acknowledged (see Henneberg 1997), but conclusive evidence of unfairness is hard to come by. Moreover, there is no widely accepted alternative to peer review for assessing the merit of proposed or completed scientific work. Thus, a recent article in Nature (Wennerås and Wold 1997) is of interest because it not only provides evidence of massive gender bias in the the peer review of research grant applications to Sweden’s Medical Research Council (MRC), but suggests what may prove to be a more trustworthy and democratic method than the traditional form of peer review for assessing the scientific competence of research grant applicants.
The article, which appears in Nature, rather oddly under the heading “Commentary”--as though it were an opinion piece, not the original research report that it actually is, deals primarily with the question of why, in applications to the MRC for post-doctoral research grants, women have a success rate only half that of men. The authors investigated this question within the context of the broader issue of why women in Sweden account for 44% of biomedical PhD's but only 7% of professional positions in the biomedical field. To answer the question, they took advantage of Sweden’s Freedom of the Press Law to obtain access to the evaluation sheets completed by members of 11 review panels, which, among them, assessed a total of 114 MRC post-doctoral research grant applications (62 from men, 52 from women).
Each application consisted in a curriculum vitae, a bibliography and a research proposal. These provided the basis for a numerical rating according to three criteria: the applicant’s scientific competence, the relevance of the proposed research and the quality of the proposed research methodology. The product of the three scores, assigned independently by each reviewer, yielded an overall score. The mean of the scores awarded by review panel members provided a final score, according to which the applicants were ranked.
Overall, applications from women were scored lower than those from men on all three criteria, the greatest deficiency being in scientific competence. As an individual’s scientific competence is generally considered related to the number and quality of their scientific publications, the investigators examined the effect of the applicants’ gender on the relationship between a bibliographic measure of scientific achievement and the reviewers’ assessment of scientific competence.
The principle bibliographic measure of scientific achievement was Total Impact, which is the number of journal articles published by an applicant, adjusted for the Impact of the journals in which the articles appeared. Thus, one article in a journal with an Impact factor of 1.0 would yield a Total Impact of 1.0. The Impact of a journal is based on the mean frequency with which articles in that journal are cited in over three thousand journals reviewed annually by the Institute for Scientific Information, Inc. (Journal Citation Reports). Impact thus provides one measure of the significance that the research community as a whole attributes to articles in the various journals. For example, Nature and Science have Impact factors of more than 20, whereas the better specialist journals in most fields have an Impact factor of 2 to 3.
Using multiple regression analysis, the authors found that applicants’ Total Impact scores were positively and significantly related to their scientific competence as assessed by the MRC reviewers. The y intercept of the regression curve, which corresponds with the reviewers’ assessment of scientific competence for applicants with a Total Impact of zero, was 2.09 on a four-point scale, for women versus 2.30 for men. As the assessed competence of women increased by 0.0033 points for every Total Impact point, a woman had to out-produce a man by 64 Total Impact points to be considered as competent. This sum represents three articles in a high Impact journal, such as Nature or Science--journals that most researchers are proud to publish in once or twice in their careers, or twenty or more articles in specialist journals. Considering that the mean Total Impact score of the applicants was about 40, the regression model indicates that a woman must be 2.5 times more productive ((40+64)/40) than a man to be considered equally competent.
Regression analysis was also used, to examine the influence on the reviewer’s assessment of scientific competence of the applicant’s nationality, type of education, scientific field, university affiliation, affiliation with a review committee member, and experience. Of these factors, only the applicant’s affiliation with a committee member significantly affected the assessment of scientific competence. Remarkably, an applicant affiliated with a review panel member was advantaged over other applicants in perceived scientific competence to the same degree that male applicants were over female applicants, notwithstanding that the review panel member concerned was excluded from participation in the evaluation of their affiliate’s application.
The paper concludes with an appeal for the development of a peer review system with built-in resistance to the weaknesses of human nature. Surprisingly, though, it does not discuss what is perhaps the most striking implication of the study; namely, that in the evaluation of scientific competence, bibliometric measurement alone may provide the least biased form of peer review, and certainly the most democratic. Further, in comparison with other forms of peer evaluation, appropriate bibliographic analysis may prove both more realistic, because it depends solely on past achievement rather than the assessment of future promise, and less costly to implement, because it requires no specially convened review panel.
Henneberg, M. 1997. Peer review: the holy office of modern science. naturalSCIENCE, Volume 1, Article 2. http://naturalscience.com/ns/articles/01-02/ns_mh.html
Wennerås, C., and A. Wold. Nepotism and sexism in peer-review. 1997. Nature 387: 341-343.
Journal Citation Reports, Institute For Scientific Information, Inc., Philadelphia.
Your comment on this article is invited and should be addressed to: firstname.lastname@example.org. For further information on submitting a contribution to naturalSCIENCE, please see the Author Guide