Workshop Reveals Flaws in Peer Review Bulletin

The National Academy of Sciences (NAS) held an all day workshop Nov. 18 that brought together regulators, academics, industry and public interest groups to discuss the Office of Management and Budget’s (OMB) draft bulletin on peer review. By the end of the day, presenters and participants had expressed various concerns about the impact of the bulletin as currently written and uncovered fundamental flaws with the policy. OMB’s Aug. 29 draft bulletin on peer review proposes a more uniform standard of peer review for all federal agencies. The bulletin would establish a variety of strict requirements for agencies conducting peer review including that all "significant regulatory information" be peer reviewed, that a public comment period be added to all peer reviews and that agencies obtain input and approval of their peer review policies from OMB’s Office of Information and Regulatory Affairs (OIRA) and the White House Office of Science and Technology Policy (OSTP). Numerous participants noted that OMB had not fully established a need for the overarching peer review policy. OMB makes several claims in the bulletin’s introductory statement that federal agencies either do not have satisfactory peer review policies or are not effectively applying them. However, workshop participants noted that OMB does not present any proof or examples of these claims. It was argued that if OMB does not establish and clearly define a problem, then the appropriate course of action cannot be determined and the bulletin should be dropped. Several ex-regulators noted that the bulletin, as currently written, would create more problems than it solves and would allow more mischief and delay within an already difficult regulatory process. One speaker noted that the bulletin would provide already powerful industry groups with greater access and influence over the science upon which regulations are based. Another ex-regulator expressed great concern that OMB seems to be placing itself as the arbiter of science and information quality when there are several other agencies or organizations with greater experience in peer review and quality assurance of scientific data such as National Institutes of Health, the Food and Drug Administration, the Environmental Protection Agency or even the NAS, which hosted the workshop. OMB’s claim that the costs of implementing the peer review policy would be negligible was raised several times during the workshop. John Graham, administrator of OIRA, stood by this assessment during the workshop. However, several participants disagreed with this claim and asserted that the cost would be considerable. Though peer review is traditionally a professional courtesy performed by experts for free, it was proposed that only financial compensation would enable agencies to acquire the vast number of reviewers needed under the new policy. The irony of OMB, a fervent advocate of cost-benefit analysis for regulations, basically dismisses the issue of cost for its draft bulletin failed to amuse attendees who seemed outraged at the hypocrisy. Participants also criticized the process by which OMB developed the draft bulletin. The primary complaint seemed to be that OMB did not consult either agencies or academic peer reviewers prior to drafting the bulletin. At least one panelist believed that OMB should withdraw the draft bulletin and start from scratch while engaging agencies and academics. The recommended process would first catalog the various forms of peer review and then allow agencies to review each other’s peer review policies to identify strengths and weaknesses as well as recommend more focused policies to improve the process. Some other concerns that were raised and discussed during the workshop included:
  • Adequate disclosure of conflicts of interest;
  • OMB assuming oversight of emergency health decisions that circumvent peer review;
  • The usefulness of a public comment process during peer review;
  • Whether OMB has the legal authority to propose these peer review policies.
Essentially, both academics and regulators at the NAS workshop expressed serious concerns about the draft bulletin and considered it deeply flawed. Implementing the bulletin would be problematic and more costly than anticipated with very few, if any, benefits. Indeed the bulletin might cause more problems, in the form of additional delays in the regulatory process and lost objectivity in the peer review process. Finally, many attendees seemed to prefer that OMB scrap the draft bulletin and start a stakeholder inclusive process to identify peer review problems of federal agencies and develop policies to fix those problems. The public comment period for the draft bulletin on peer review remains open until Dec. 15.
back to Blog