In a paper published Aug. 2 in the journal Environmental Health Perspectives, a team of National Toxicology Program (NTP) scientists presented a comprehensive approach for evaluating the effects of research over time, using its work on hexavalent chromium as a case study.
NTP research on hexavalent chromium led to California’s adoption, in 2014, of the nation’s first-ever water quality standard for the compound. The California congressional delegation, Environmental Protection Agency, and Department of Health Services nominated the substance for NTP study in 2000 and 2001.
Tracing ripple effects
The authors of the case study took a detailed look at the ripple effects of the nomination of hexavalent chromium and the resulting technical report, which found that hexavalent chromium in drinking water caused cancer in rodents. The authors said that hexavalent chromium made a good case study, because it balanced records availability with enough lag time to see the full impact of a high profile project.
“We think NTP studies are having an impact on public health, but we wanted to be able to prove it,” said Mary Wolfe, Ph.D., NTP deputy division director for policy. “So we developed an approach to measure the effectiveness of NTP’s programs, using hex chromium as a case study.”
Developing a conceptual model
The new paper offers a conceptual model for objectively measuring immediate, or proximal, impacts; intermediate impacts; and long-range, or distal, impacts of NTP research. For each of these categories, the authors listed specific products or outcomes that can be measured.
In addition to the peer-reviewed publication, NTP received input on the model and case study findings from stakeholders, including the NTP Board of Scientific Counselors, in December 2014. The authors also showcased the study as a poster at the 2015 Society of Toxicology annual conference.
“We developed an approach with methodical steps, to evaluate research products and outcomes,” the team wrote in its conclusion. “[This] approach can be applied to future projects and is adaptable to different NTP research programs.”
From nomination to regulation
Using a variety of tools, the team examined scientific studies, NIH grants, and documents from other agencies and groups, to find references to NTP publications on hexavalent chromium. The authors identified limitations in existing tools, and recommended development of text and web mining tools to improve efficiency of future assessments.
The study began by analyzing requests for information and web page views of NTP reports, as a way to gauge immediate awareness from stakeholder groups in industry, academia, international organizations, and state, and federal agencies. Although NTP only began tracking web page views in 2011, the team was able to track NTP responses to information requests as far back as 2005.
Intermediate outcomes included citations in journal articles and book chapters, NIH grants, and public documents. The team used the commercial indexing services Web of Science and Scopus, as well as NIH databases, and Google. For NTP technical and toxicity reports, which are not fully indexed by the commercial services, the team examined full-text archives in PubMed Central.
Searches with Google and Lexis Advanced turned up evidence of long-range outcomes of the NTP findings on hexavalent chromium in water, both in text and citation lists of proposed laws, committee reports, and regulations. The authors concluded that, in addition to actions by the state of California, the NTP studies on hexavalent chromium influenced regulatory activity in other states, and helped shape decisions in federal agencies.
Citation: Xie Y, Holmgren S, Andrews DM, Wolfe MS. 2016. Evaluating the impact of the U.S. National Toxicology Program: a case study on hexavalent chromium. Environ Health Perspect; doi:10.1289/EHP21 [Online 2 August 2016].
(Eddy Ball, Ph.D., is a contract writer in the NIEHS Office of Communications and Public Liaison.)