Reflections on Grant Peer Review
The Squeaky Wheel and Other Concerns

by Harry Brodie

(Issue 5; posted April 4, 1997; archived April 18, 1997)

Recently I came across another article from a disgruntled applicant complaining about the National Institutes of Health (NIH) peer review system (see references). The NIH process seems to attract criticism even though it is arguably the most open and objectively managed one extant. A similar program run by the National Science Foundation (NSF) does not assign priority scores to applications and the recently established Army Breast Cancer Research Program (ABCRP) does not construct a pay line for funding.

I shall argue in this essay that scientists should spend less time trying to fix a peer review process that "ain't broke" and concentrate more on the second stage of the two-level review process. In the first stage, expert panels judge applications based on intrinsic merit alone; in the second, special councils also consider programmatic reasons to fund applications with poorer priority scores. At NSF, with no priority scores to constrain them, program officers exchange their review hats for program hats and choose applications for funding from the pool of well-regarded proposals. (See sidebar for a fuller description.) In the army program, funding decisions are made mainly by an integration panel that, unencumbered by a pay line, is less influenced by the assigned priority score than are most NIH institutes. At the NIH, depending on the program, up to 20 percent or more of applications with relatively poorer priority scores have received special funding consideration for programmatic reasons. Yet the criteria for making these decisions remain encased in the proverbial "black box," shrouded in mystery.

Peer review itself was in a similar black box until the mid-1970s, when the indiscretions of President Richard Nixon precipitated the enactment of the Privacy Act of 1974. One benefit of the act was to force the release of the scientific peer review group summary statement, and its priority score, to the applicant. Until then, it was difficult to get reliable feedback on study section recommendations. After summary statements began being released, there was increased demand for more information about the comments used to justify the priority score assigned. The document grew in length from less than one page to three or more pages. While preparing such reports became time-consuming for NIH staff, and led to the proliferation of rebuttal letters from disgruntled applicants, it is generally agreed that the applicant, reviewers, and indeed the entire grant review system benefits from the more open dialog that resulted. Even if the Privacy Act were repealed today, it is unlikely that NIH would wish to return to the previous closed system.

In contrast to the analyses and attacks that peer review undergoes, the second, programmatic review stage gets little attention from most scientists. Nevertheless, some scientists who understand the system will promote their applications directly to staff, and through it to the second-stage council, before and after peer review has taken place. NIH administrators often claim that special consideration for high program relevance is given relatively rarely. However, in the late 1980s and early 1990s, when obtaining funding for R01-type research applications was more difficult than it is today, institutes began to fund more applications that fell beyond the normal pay line. The National Institute on Aging (NIA), perhaps the most aggressive in this regard, was funding such applications in the 20 percent range. The rationale at the time was that research on aging needed a more targeted approach than did, say, the basic research funded by the National Institute of General Medical Sciences (GMS). With higher success rates for applications today, the rationale to fund out of line is more tenuous, but even now the National Institute of Child Health and Human Development (CHD), which awards a smaller percentage of R01 applications than many of the other institutes, is prepared to fund up to 25 percent of its applications out of line. It must be remembered that the total funding pie does not change, and when a programmatic decision is made to fund an application beyond the pay line, another unidentified application with a better priority score is passed over.

The ABCRP appears to have a programmatic intervention policy akin to that of the NIA in the early nineties. The effort is a targeted one in which program relevance has a very important role. The program does not deal with pay lines, and the integration panel has roamed rather freely up the priority score ladder in search of applications with strong relevance to breast cancer. It seeks applications with high risk/high gain and prospects for significant success in the shorter term.

There is nothing intrinsically wrong with applying high program relevance to applications after peer review. The problem is that applicants have no feedback as to whether or how criteria are being applied. The situation reminds me of the problem the courts have had in developing a definition of pornography. Supreme Court justice Stewart Potter said that he couldn't define it but he knew it when he saw it. Public accountability would be strengthened if priority scores of funded applications were released along with application titles and names of the principal investigators. Then unsuccessful applicants and, through them, the community at large could gain a deeper understanding of what subjects and approaches institutes think are especially relevant to their programs. Unfortunately, when release of the priority score was sought under the Freedom of Information Act in the 1970s, a judge ruled that the score was merely an internal working number with little relevance to funding! To force release, it would take determination and/or financial resources to appeal this decision to a higher court.

Funding authorities are undoubtedly reluctant to release priority scores because doing so would open up the process and probably promote analysis and criticism. But we know that the more open systems become, the more objective and responsive they are. The NIH coped with release to the applicant of the summary statement and priority score when forced to do by law, and would adapt to the release of the priority score of funded proposals to the general public as well. It is ironic that the public has access to the actual funded application but not to the score that was instrumental in getting it funded. With the availability of this missing link, there are several important benefits. First of all, full disclosure would improve confidence in the entire process. The additional knowledge would educate the applicant as to how funding agencies set their priorities, and this would result in more relevant applications. Conversely, funding agencies will have to ensure that their decisions are free of undue bias, knowing that their actions could be monitored more readily. It could be argued that if applicants thought only about program concerns, administrators would be over-directing science. This should not be a concern since scientific merit, i.e., the priority score, will always be the prime consideration for funding. However, applicants will be more aware of programmatic concerns and will highlight these more forcefully in their proposals. The enhanced awareness of program concerns and the ensuing dialog can only be beneficial to the applicant, the agency, and the scientific enterprise in general.

Harry Brodie has been a principal investigator on NIH grants and contracts, and he supervised study sections at NIH in the Division of Research Grants for over fourteen years. He currently is a writer and also works on peer review projects in the Washington, D.C., area.

Science in Air and Space: NASA's Science Policy Guide details the policies governing that agency's research, including the role of advisory groups and peer review.

NIH's Committee on Improving Peer Review has released an extensive report on grant-application rating.

The NIH National Institute on Aging has a brief statement on grant funding and review.

A recent letter to The Scientist suggests criteria to be considered in reviews.

See the Beagle Poll on Discretionary NIH funding.