Environmental Factor, January 2011, National Institute of Environmental Health Sciences
NIEHS completes draft of metrics manual
By Matt Goad
Drew heads a branch in the NIEHS Division of Extramural Research and Training that analyzes the effectiveness of grants ranging from basic research to environmental justice. (Photo courtesy of Steve McCaw)
NIEHS-funded research helps advance public health in many ways, and the Partnerships for Environmental Public Health (PEPH) program is helping grantees measure just how many ways. Now available for public review, the new PEPH Evaluation Metrics Manual is designed to help grantees identify and measure the key activities, outputs, and impacts of their partnership work.
"We are helping grantees understand and map out what their influence is and demonstrate the effects of their research," said Christie Drew, Ph.D., chief of the NIEHS Program Analysis Branch (PAB)(https://www.niehs.nih.gov/research/supported/dert/pab/index.cfm).
PEPH is letting NIEHS employees and grantees get a look at the first draft of the manual on Jan. 10 from noon to 1:30 p.m. in the Rodbell Auditorium at NIEHS. The session is open to all NIEHS and U.S. Environmental Protection Agency staff, as well as the general public. During the presentation, Drew will provide an overview of the manual and facilitate discussions to obtain feedback on the content and metrics.
Over the next few months, PAB will present the manual to several other NIH centers, as well as the Centers for Disease Control and Prevention's evaluation workgroup. The PAB staff will also conduct webinars for groups such as the Association of State and Territorial Health Officials and National Environmental Health Association.
"This is the first draft of the manual," said Kristi Pettibone, an evaluator with the PAB. "What we want to do now is to take it out to the folks who are doing this work, and get their thoughts and ideas about how to make it even better."
During the first PEPH grantee meetings, researchers talked about the challenges associated with measuring and documenting their successes in environmental public health activities. PAB began drafting the manual in June 2009 in response to these challenges. "Commonly, scientists measure the success of their research by counting their publications, but this is not the only way," Drew said.
As the accountability culture of Washington expands, the need for evaluation also expands, and people want to be able to show what grants have accomplished.
Other metrics that PEPH grantees could use to demonstrate achievements include documenting testimony before a legislative committee; interactions with the target audience; activities; outcomes and impacts related to partnership building; product development and dissemination; and education and training activities.
Not all about numbers
"The science of measurement is not all about numbers," Drew explained. "We have worked hard in the manual to focus on narrative and descriptive metrics, in addition to the numbers."
Grantees sometimes think they have to demonstrate major changes, such as reductions in asthma cases or school absence rates. But it can be difficult for grantees to measure such long-term goals, according to Drew. The Evaluation Metrics Manual shows grantees how to identify and measure key activities and short-term outputs associated with their projects. This enables grantees to show progress along the way to reaching their long-term goals.
"Although we have provided lots of ideas and metrics related to environmental public health activities," Drew said, "we want grantees to understand that these measures are just examples of some of the many ways to implement and measure the environmental impact of their work."
(Matt Goad is a contract writer with the NIEHS Office of Communications and Public Liaison.)