In the regulatory process, decision makers rely on a wide variety of scientific, technical and analytical documents such as risk assessments, cost-benefit analyses and reports. The reliability and impartiality of these documents is paramount in developing regulations that best protect our economy, civil liberties, environment and public health.

Peer review is a tool used by regulatory decision makers to ensure the accuracy and appropriateness of these documents. Peer review adds a degree of authority and comfort to often complex scientific and technical issues. Peer review also strengthens the final assessments produced by agencies. These are important objectives in pursuing thorough and thoughtful rulemakings.

Who performs peer review?
Peer reviewers must be experts in the field of the assessment they are reviewing. Peer reviewers must possess enough experience and knowledge of a subject to assess the strengths and faults of an assessment. Peer reviewers should also be knowledgeable enough to identify factors which may be absent from an assessment. Impartiality and a lack of conflicts of interest are also important qualities for peer reviewers to possess.

Peer reviewers are usually chosen by the federal agency that produced the document being reviewed. In some cases, the agency will choose a scientific body (such as the National Academies of Science), which will then choose individual peer reviewers and/or conduct the peer review.

How is a peer review conducted?
The make-up and conduct of a peer review panel varies widely depending on the nature of the assessment being reviewed, the importance of the need for review and the potential use of the assessment.

A White House bulletin on peer review describes just how wide-ranging peer review processes can be:

Peer review may take a variety of forms, depending upon the nature and importance of the product. For example, the reviewers may represent one scientific discipline or a variety of disciplines; the number of reviewers may range from a few to more than a dozen; the names of each reviewer may be disclosed publicly or may remain anonymous (e.g., to encourage candor); the reviewers may be blinded to the authors of the report or the names of the authors may be disclosed to the reviewers; the reviewers may prepare individual reports or a panel of reviewers may be constituted to produce a collaborative report; panels may do their work electronically or they may meet together in person to discuss and prepare their evaluations; and reviewers may be compensated for their work or they may donate their time as a contribution to science or public service.

For large, complex reports, different reviewers may be assigned to different chapters or topics. Such reports may be reviewed in stages, sometimes with confidential reviews that precede a public process of panel review. As part of government-sponsored peer review, there may be opportunity for written and/or oral public comments on the draft product.

What does a peer reviewer evaluate?
In evaluating a document, peer reviewers evaluate a variety of factors. One of the most commonly reviewed types of documents is scientific assessments. In this case, peer reviewers judge the hypothesis developed and scientific methods used in the assessment. In this and other cases, reviewers may judge research design and the quality of data used in the document.

Regardless of the scope of the document, one of the most important factors peer reviewers evaluate is the appropriateness of assumptions made by the document's authors. Any scientific or technical assessment includes certain assumptions or inferences the authors must make to proceed with their assessment or to draw meaningful conclusions. Peer reviewers pay careful attention to these assumptions because they are often a point of subjectivity in the assessment.

What does a peer reviewer produce?
Peer reviewers produce a critique of the document. The critique includes the peer reviewers' judgment of any evaluated factor. The critique often includes recommendations for improvements the authors can make.

The peer reviewers' critique is important for a variety of reasons. The document's authors use the critique to assess the quality of their work and improve the end product. Decision-makers use the critique to gauge the extent to which they may rely on a given document to make decisions about pursuing a rulemaking. Outside experts can use the critique to justify journal publication or to determine the appropriateness of a document's acceptance in the scientific or technical community. The public can use the critique to gain a better understanding of a rulemaking and to determine whether that rulemaking is responsive to public needs.

When is peer review required?
A legal requirement for the peer review of any agency assessment does not exist. Agencies generally choose to subject an assessment to peer review for the purposes of providing transparency to the scientific process and strengthening its end product.

However, the White House mandates peer review in some cases. In December 2004, the White House Office of Management and Budget (OMB) issued its "Final Information Quality Bulletin for Peer Review." OMB issued the Bulletin with the stated goal of improving the "quality and credibility of the government's scientific information." The Bulletin requires agencies to peer review "influential scientific assessments," which it defines as having the potential to create "substantial impact on important public policies or private sector decisions."

While the peer review process for influential scientific assessments is left at the discretion of the agencies, in select cases, the Bulletin requires the use of an OMB-defined peer review process. For these "highly influential scientific assessments," OMB requires agencies follow the procedures set out in the Bulletin. The OMB-defined process primarily concerns the selection of reviewers but also requires agencies to include an opportunity for public comment on this subset of assessments. According to the Bulletin, "A scientific assessment is considered 'highly influential' if the agency or the OIRA Administrator determines that the dissemination could have a potential impact of more than $500 million in any one year."

OMB's decision has met with opposition. Critics have charged OMB with micromanaging agencies' peer review procedures in the absence of statutory authority to do so. Critics also argue a public comment period is inappropriate for a scientific review of highly technical information.

Other regulatory tools