Preventable health risks, such as smoking, obesity, and physical inactivity, cause up to half of all deaths in the United States. To reduce these risks and the associated burden of disease, the NIH spends approximately 20% of its annual budget on prevention research. To increase the scope, quality, dissemination, and impact of this research, the Office of Disease Prevention (ODP) recently developed a Strategic Plan. The plan outlines our key priorities, including identifying unmet research needs (Strategic Priority II), supporting methods development and collaborations (Strategic Priorities III and IV), and disseminating and promoting actionable results (Strategic Priorities V and VI). Achieving these priorities is dependent on our ability to accurately characterize and rigorously monitor the NIH prevention research portfolio (Strategic Priority I). As we enter the strategic plan's second year, I am pleased to report on our progress with Strategic Priority I.

No automated tools currently exist that provide an in-depth categorization of NIH-funded prevention research grants simultaneously across a number of dimensions such as study rationale, independent and dependent variables, entities studied, study setting, population focus, research design, and type of prevention research. Developing new methods for classifying prevention research will enable a more meaningful portfolio analysis and will inform program planning. The ODP—in collaboration with the NIH Office of Portfolio Analysis (OPA), the NIH Center for Information Technology (CIT), and the ODP's support contractor—is developing a new portfolio analysis tool to rapidly and accurately characterize NIH-funded prevention research. The tool, which is estimated to take at least three years to complete, is being developed in four phases:

1. Develop a taxonomy, or framework, to classify funded prevention research grant abstracts.
2. Manually code abstracts to identify a set of "gold standard" examples.
3. Use these examples to develop a machine-learning algorithm within the Portfolio Learning Tool (PLT).
4. Validate the machine-learning algorithm developed in step 3 by having it classify new grant abstracts not previously coded manually; then compare the results to those obtained by manual coding.

ODP's Strategic Priority I Team has completed the first phase. A new taxonomy and accompanying 26-page protocol (which provides instructions, definitions, and examples) was created to facilitate the accurate, standardized classification of grant abstracts. This taxonomy is a set of nearly 150 non-mutually exclusive topics grouped into eight categories (study rationale, independent and dependent variables, entities studied, study setting, population focus, research design, and type of prevention research). Following the instructions specified in the protocol, coders read each grant abstract and select all topics that apply within each category. The ODP has developed a comprehensive team-based approach to coding abstracts, which includes reaching a consensus within a team of three coders.

ODP's Strategic Priority I Team is now working with ODP's support contractor to manually apply the taxonomy to thousands of research grant abstracts. The ODP, in collaboration with its support contractor, developed a custom software called the Prevention Abstract Classification Tool (PACT). This web-based platform can be accessed via tablet computers and captures individual and team coding, as well as calculating inter-rater reliability. The goal of this second phase is to identify both positive and negative examples of prevention studies to serve as training data for the machine-learning algorithm used by the PLT—a portfolio analysis tool developed by CIT and now administered by the NIH Office of Portfolio Analysis. To date, 450 gold-standard grant abstracts have been identified, and an additional 500 await validation. With the exemplars identified to date, the OPA and the ODP have piloted three iterations of the PLT algorithm.

The Strategic Priority I Team has given several presentations to NIH audiences about their progress, which have generated a lot of attention. Many Institutes, Centers, and Offices have expressed interest in adopting or adapting the taxonomy protocol, PACT, the team-coding process, and/or the PLT to assess and monitor their own portfolios.

Rigorous portfolio analysis will enable the identification of funding patterns and trends, as well as research areas that may benefit from targeted investments. Those investments can address important modifiable risk factors and reduce disease burden. NIH-funded prevention research has led to many notable public health successes. We hope to accelerate these advances by more effectively assessing, monitoring, and leveraging the NIH prevention research portfolio.

David M. Murray, Ph.D.
Associate Director for Prevention
Director of the Office of Disease Prevention