
What Others Are Saying About PART
by Guest Blogger, 2/22/2006
OMB Watch is not alone in criticizing the White House's Program Assessment Rating Tool. See what others have had to say recently about this flawed measure.
PART Punishes Programs for Following the Law
Clay Johnson, OMB deputy director, when asked in a congressional hearing, "[I]s it possible for a program to get a poor rating simply because it does what's required by statute and not necessarily what OMB might like for that program to do?":
Yes.
--Accountability and Results in Federal Budgeting: Hearing Before the Subcomm. on Federal Financial Management, Government Information & Int'l Security of the Senate Comm. on Homeland Security & Gov't Affairs, 109th Cong. (2005), 2005 WL 1409975 (F.D.C.H.) (colloquy between Sen. Carper and Clay Johnson III).
PART Is Divorced from Reality
From ThinkProgress:
1) "Federal Emergency Management Agency: Disaster Recovery":
The Department of Homeland Security's Recovery program ensures that individuals and communities affected by disastes [SIC] of all sizes, including catastrophic and terrorist events, are able to return to normal function with minimal suffering and disruption of services. PERFORMING: Adequate (one star)
Reality--Reuters:
With no clear recovery plan in sight five months after Hurricane Katrina, many victims are simply hanging on, waiting anxiously for signs that their neighborhoods are either reviving or turning into permanent ghost towns.
2) "Preparedness--Grants and Training Office National Exercise Program":
Prepare Federal, state, and local responders to prevent, respond to, and recover from acts of terrorism by providing the tools to plan, conduct, and evaluate exercises. PERFORMING: Effective (three stars)
Reality--GAO:
Although the [National Response Plan] framework envisions a proactive national response in the event of a catastrophe, the nation does not yet have the types of detailed plans needed to better delineate capabilities that might be required and how such assistance will be provided and coordinated.
3) "Federal Emergency Management Agency: Disaster Response":
The Department of Homeland Security's Response program is designed to quickly, efficiently and effectively provide support to State, Tribal, and local governments, and Federal response teams in the event of a natural or manmade disaster, emergency or terrorist event. PERFORMING: Adequate (one star)
Reality--Washington Post:
Four years after the Sept. 11, 2001, attacks, administration officials did not establish a clear chain of command for the domestic emergency; disregarded early warnings of a Category 5 hurricane inundating New Orleans and southeast Louisiana; and did not ensure that cities and states had adequate plans and training before the Aug. 29 storm, according to the Government Accountability Office.
PART Continues the Bush War on Science
Statement of Dr. Genevieve Matanoski, EPA Science Advisory Board, to Subcommittee on Environment, Technology, and Standards, House Committee on Science, March 11, 2004, available on Westlaw at 2004 WL 506081 (by subscription only):
[A]fter evaluating PART
summaries for several research programs, our conclusion is that
PART may, at this time, have a limited capacity to inform budget
decisions on research programs. The Board is concerned with the
manner in which the weighting formula in PART seems to influence
the full analysis and thus favor programs with short-run results
over those having long term results. There is also concern that
an evaluator's subjective considerations might be able to bias
those weights and the rating itself.
Specifically, it appears that the weighting formula in the PART
favors programs with near-term benefits at the expense of
programs with long-term benefits. Since research inevitably
involves more long-term benefits and fewer short-term benefits,
PART ratings serve to bias the decision-making process against
programs such as STAR ecosystem research, global climate change
research, and other important subjects. The PART seems to be
intended as a formula for predictions about likely program
success. However, the weights that the PART assigns to different
program characteristics do not seem to have been validated
systematically against the contribution of each program
characteristic to any independent objective measure of program
success. If the weights in the tool are arbitrarily assigned, the
PART may have characteristics that could lead to biases in
evaluation that are related to the subjective judgments of its
designers. We believe that the tool should be reviewed to
determine its adequacy for its use in supporting budget
decisions.
As the Board observed significant decreases in science and
research funding, it also noted a substantial resource increase
in the State and Tribal Assistance Grant account (STAG) for an
initiative for retrofitting school busses. The Board does not
challenge the worthiness of this program, rather it notes that it
has no information on the science supporting this initiative. The
Board trusts that the benefits of this program have been
rigorously reviewed.
The real issue here is how research programs (and others) are to
be evaluated and whether a different metric is necessary for
basic vs. applied research programs. Also, of interest is whether
research results should be evaluated separately from the outcomes
of programs they are intended to support? Although the Board did
not directly evaluate the PART itself, it is of obvious
difficulty to conceive of a simple quantitative metric that could
be applied across the broad areas of ecosystem quality, human
health effects, endocrine effects, and technology development.
The question is even more complex when you consider that some
research is intended to develop limited data in the short-run to
fill a specific knowledge gap and other research is intended to
provide an understanding of whole systems in the long-term.
Research program measurement is even more difficult because the
knowledge and methods developed by EPA, especially ORD's
researchers, are not usually directly applied by ORD, rather they
are often used by others to support decisions on a broad suite of
diverse statutory mandates. Thus, we believe that evaluations of
the performance of research programs will need to consider the
specific factors of each program that the research is intended to
support. Further, it is unlikely that simple formulas will be
able to handle this task well. It is more likely that realistic
research program performance assessment will need to be a
combination of quantitative metrics and other information and
analyses which is then evaluated by groups of experts with
relevant knowledge.
I note that the NAS, in its review of STAR, also had concerns
with quantitative routines used in performance assessments and
noted that "The Committee judges that expert review by a group
of people with appropriate expertise is the best method of
evaluating broad research programs, such as the STAR program."
