INTRODUCTION

Funding is a prerequisite for biomedical research. In the USA, the National Institutes of Health (NIH) is the most significant public funding agency that supports basic science, clinical, and translational research. The Research Project Grant (R01) is the agency’s oldest and most widely used funding mechanism. NIH makes its merit-based funding decisions using a scoring system that explicitly emphasizes scientific impact, as judged by independent panels of experts.1 For scientists applying for grants, receiving an R01 award can be a career-transforming event that indicates the acquisition of a national reputation in the investigator’s field of research. In academic institutions, securing grant funding often puts researchers on a path for successful tenure and promotion.2 Similarly, an academic institution’s reputation and ranking also hinge on the number of research grants and the total amount of the awards it receives.3 While there is a keen appreciation of R01 awards’ prestige confer on scientists and their home institutions, little is known about the longer-term impact of the work that the grant funding generates.

For the vast majority of R01 grants, peer-reviewed publications are the most essential research products. Without publications, research findings cannot be disseminated, and knowledge gained or translated into practice. Thus, the numbers of publications and persistence of publications over time of a funded grant not only gauge the investigative team’s scientific output but also reflect the grant’s impact. The number of publications from a grant by itself, however, does not adequately convey the quality and impact of the work it generates. Not all publications are equal—articles published in high-impact journals tend to be cited more frequently, thus having more chance to affect future research. For a similar reason, a journal’s impact factor (JIF) is not an appropriate measure of a grant’s impact either, because JIF is calculated as the average number of annual citations of all the articles in the journal received during the past 2 years, while most of the articles published in the journal are unrelated to the grant. For a grant, the number of citations that its publications generate indicates how well its findings or methods have been adopted into the mainstream thinking of science. For this reason, we used the actual number of citations of the papers produced by a grant as one of the measures for its impact. Similarly, the concept of the Hirsch Index or h-index4 can be naturally extended to funded grants, to provide the maximum value of h, the number of the most frequently cited papers produced by a grant that is cited at least h times. In a sense, the h-index combines quantity with quality into a single metric. It has been shown that a researcher’s h-index predicts future achievement better than the number of papers or the number of citations.5 We contend that these metrics (publications, citations, and h-indices), when used together, provide a more complete picture of the grant’s productivity and its overall impact on science (Fig. 1).

Figure 1
figure 1

Publications, citations, and impact provide a more holistic picture of the performance of a grant than funding alone.

Full size image

Good science takes time. It often takes years for a funded grant to ascertain the experimental data and to publish its findings in peer-reviewed journals. For example, randomized clinical trials can only be published after the completion of follow-up, which often takes multiple years of effort. Basic science discoveries that challenge the existing paradigm tend to face more scrutiny and thus taking longer to publish. Dissemination and adoption are even slower; they may take decades to influence science. Nonetheless, good studies do change how we think and practice. To adequately assess the impact of a funded grant, one must allow a more extended time window to examine the changes in relevant metrics after the award.

In the current report, we present the results of an inception cohort of first NIH-funded R01 grants awarded in the year 2000 and follow the post-award trajectories of publications, citations, and the h-indices over the subsequent 20 years. By first R01 we did not mean the first award to the investigator, but an initial award conferred in 2000, with the exception of grants renewed in that year. In following this cohort of grants, we hope to offer a more objective picture, or normative data, on the productivity, adoption, and the impact of publicly funded biomedical research grants in the USA.

METHODS

Data Acquisition and Metric Calculation

An inception cohort was created by identifying all R01 grants that were initially funded by the NIH in the year of 2000. The cohort of awards had a date of origin in 2000. Data from subsequent renewals are included because they shared the same R01 grant number. Such an inclusion is reasonable because papers and citations generated by the renewals are attributable to the original award.

We searched the NIH RePORTER database to ascertain the information on R01 grants funded by all 24 NIH institutes in 2000. The publications associated with these awards were retrieved from the NIH-RePORTER for each of the grants. The number of citations for publications reported before July 4, 2020, was obtained from the ISI Web of Knowledge (Clarivate Analytics, Thomson Reuters). We then counted the total number of citations associated with papers produced by each grant. The number of years from publication until 2020 was used to calculate the citation years. We calculated the citations for papers that had at least 5 years since publication. We then repeated the same calculation for papers that were published for at least 10 and 15 years. We also calculated the citations regardless of the time since publication. Finally, we calculated the h-index of each award by year. The h-index was proposed by Hirsch based on an author’s most cited papers and the number of citations they received in other publications.4 As proposed by Hirsch, the h-index attempts to measure both the productivity and the impact of the scientist’s work. The h-index was defined as the maximum value of h such that a given award had published h papers that each have been cited at least h times. Here, we report the h-index of the award—not the scientist—to simultaneously measure both the productivity and the impact of the award on science.

Data Analysis

We presented the mean numbers of publications and citations over time, and h-indices by a 5-year increment. We also presented the distribution information of these metrics. The time course of each metric was presented graphically. To assess the correlations among the different domains of publications, citations, and impact, we calculated the Pearson’s correlation coefficients and graphed the relations between publication and citation; publication and h-index; and finally, citation and h-index.

RESULTS

In the year 2000, NIH awarded 4451 R01 grants to 4266 investigators in 490 institutions. Of the 24 National Institutes of Health, more than 60% of the awards were granted by six Institutes: National Heart Lung and Blood Institute (566), National Cancer Institute (553), National Institute of General Medical Sciences (430), National Institute of Allergy and Infectious Diseases (408), National Institute of Neurological Disorders and Stroke (339), and National Institute of Diabetes and Digestive and Kidney Diseases (339). The total amount awarded was $1,338,492,284. The median award amount was $ 270,725 (interquartile range $221,235 to $337,500).

Publications

A total of 55,053 publications were reported to NIH that were associated with the eligible grants. Of these, 406 were missing sufficient information to be linked to identifiable publications on PubMed. However, PubMed Identification Numbers (PMIDs) were available for 56,987 (99.3%) of 57,393 publications. Of these, 52,470 (96%) were credited to one award, 2036 (3.7%) to two awards, 129 (0.24%) to 3 awards, and 8 (0.01%) to 4 awards, 2 to 5 awards, 1 to 6 awards, and 1 to 10 awards. The 54,647 uniquely identifiable publications were therefore credited 56,987 times to various awards.

The cumulative number of publications in the first 5 years was 25,174 (44% of total) (Fig. 2). At 10 years, the number rose to 40,717 (71% of total), at 15 years 50,948 (89% of total). The median number of publications produced by an R01 award was 5 at 5 years, 7 at 10 years, and 8 at 15 years and onwards. The 25th percentiles were 2, 3, 4, and 4 at five, ten, fifteen, and twenty years, respectively. Although the 25th percentiles did not change much, there were much greater changes in the 75th, 90th, 95th, and 99th percentiles (Fig. 2). Figure 2 shows that compared to awards below the median in cumulative publication output, there was a much greater change over time in the awards above the median.

Figure 2
figure 2

Dot plot of the distribution of publications at various time points. The leftmost graph is the cumulative number of publications at 5 years and the rightmost the cumulative number of publications for all years. Each hollow circle represents the cumulative output of an R01 grant. The three lower lines are 25th, 50th, and 75th percentiles. The three upper dotted lines are the 90th, 95th, and 99th percentiles. The values are shown next to each line.

Full size image

Figure S1 shows the cumulative number of publications over time by quartiles. The first quartile (Q1) represents those awards that published 4 or fewer papers, the second quartile (Q2) between 5 and 8 papers, the third quartile (Q3) between 9 and 17 papers, and the fourth quartile >17 papers. In the first three quartiles, there were some awards that produced early and some late. The top quartile had a handful of awards that started early and kept producing over the 20 years. One-quarter of the awards produced 18 publications or more. One percent of the awards produced 78 publications or more. Cumulatively, the number of publications (n=5254) from the top 1% of the awards (n=42) was equal to the number of publications from the bottom 40% (n=1653) of the awards, pointing to the large variability in productivity among awards.

Citations

Since initial funding in 2000, until 2020, the cumulative number of citations was 3,705,553 (Fig. 3) over 736,811 citation-years, with a median citation of 441 per award (25th, 75th percentiles 156, 1061). Awards with at least 5 years of data since publication numbered 50,725, and these produced 1,337,922 citations, at 10 years 1.74-fold more (2,321,705 Citations for 41,052 awards) and at 15 years 1.70-fold more (2,388,806 Citations for 29,852 awards). The median number of citations produced by an R01 award at 5 years was 154, which was nearly doubled at 10 years (median 296), and nearly tripled over the entire period of observation (median 441). The 25th percentiles of the citation number were 55, 105, and 121 at 5, 10, and 15 years. The incremental changes in the 75th, 90th, 95th, and 99th percentiles at the corresponding time points were much larger (Fig. 3).

Figure 3
figure 3

Dot plot of the distribution of citations at various time points.

Full size image

Figure S2 shows the cumulative numbers of citations over time by quartiles of citation scores. Each line represents a cumulative citation number for an award. The citation scores’ trajectories are highly variable regardless of whether the award was in the top or bottom quartiles. The top 1% of the awards (n=42) generated as many citations (n=400,768) as the bottom 51% (n=2138) of the awards, again pointing to a large variation in the adoption of science.

Impact

The impact of the grant measured by the h-index in 2020 is shown as a dot plot in Figure 4. The median value of the h-index was 7 (25th, 75th percentiles 4, 12). The 90th, 95th, and 99th percentiles were 20, 25, and 37, respectively. The mean h-index was 9.1 (SD 7.72); the maximum h-index was 64. The time course of the h-indices by quartiles is shown in Figure S3. Half the awards had an h-index of ≤7, implying that the seventh most cited publication by the award was cited at least 7 times. The top quartile awards had at least the fourteenth most cited publication cited at least 14 times. The trajectories of h-index growth were also highly variable.

Figure 4
figure 4

Dot plot of the distribution of h-index.

Full size image

Associations Between Publications, Citations, and Impact

Two-way scatter plots between publication, citations, and h-index are shown in Figure S4. The correlation coefficients are shown. The large variability (heteroscedasticity) among these metrics—especially among the higher end of productivity and citations—suggests that they measure different domains of the scientific output. The heat map displays in two dimensions, the three-way relations among publications, citations, and the h-index, and affirms the multidimensional nature of the measurements.

DISCUSSION

Herein, we reviewed the relations between the funding of a Research Project Grant, the number of publications it generated, the numbers of citations they received, and the impact they had on science. In the year 2000, NIH awarded 4451 R01 grants, with a combined cost of over $1.3 billion, produced over 55,000 publications and over 3.7 million citations. About 44% of the publications emerged within the first 5 years, 71% within 10 years, and 89% within 15 years of funding. The cumulative number of citations for a typical award was 441 times. The median h-index over 20 years for an award was 7; this means that the 7th most cited publications were cited at least 7 times. There was large variability in the cumulative number of publications, citations, and impact, as well as in the time course of these metrics. The number of publications was only loosely in sync with the number of citations they had received, and even less with the impact on science. Thus, the numbers of publications, citations, and h-index measured different aspects of an award. The current paper seeks to draw attention to these domains so that a more comprehensive assessment can be rendered on a grant’s eventual impact on science. The awards’ varying performance is by itself noteworthy, and why some awards so remarkably outperformed others in all three domains remains to be elucidated.

The current study has several implications. How grant award is used in faculty hiring, promotion, and tenure needs a more careful examination; the number and total dollar amount do not tell the full story. We take the view that an award enables an investigator to perform the proposed research; the funding itself does not necessarily assure research productivity, peer adoption, and eventual scientific impact. A grant’s performance in these latter domains should carry more weight than just securing the funding. We contend that a combination of the three metrics examined in this paper provides reasonably accurate and objective quantification of a grant’s research productivity and scientific impact. Discovery of what makes some grants more successful than others is not a question we intended to answer in this study; however, answers to this question would have considerable implications on how scientific research should be supported.6, 7

There are some limitations to the study. Publications that were not reported in association with an NIH award were missed; therefore, the publications’ impact could not be accounted for. Conversely, multiple awards may have supported the same papers. When a publication acknowledged two or more concurrent awards, we assigned full credit to each of the multiple awards; this would favor those publications that are part of a consortium funding by multiple R01 awards. Furthermore, if publications emerged from awards outside the initial R01 funding period, productivity may be inflated. Extended funding periods from renewed grants, however, also represent a form of productivity that should be credited to the initial awards, as we have done in the current study. Non-NIH funding sources may be used to support publications, which we attributed only to the NIH grant. Publications are not the only product of the grant’s output. Patents, data sets, software, and the researcher’s influence on policy and practice are other domains that were not assessed in the current report.8 Additionally, among its other limitations,9 the h-index tends to differ by fields of research.4 This said, the homogeneity of our sample, i.e., all awards were for biomedical research and funded in the same year, perhaps alleviates some of the concerns. Finally, the reasons for large variations in productivity, citation, and impact metrics cannot be answered by this descriptive study but should be examined in future reports.

Notwithstanding these limitations, we put forward a framework for evaluating publicly funded biomedical research grant’s productivity and impact over a 20-year time window. We present a conceptual framework built upon the three pillars of productivity, citation, and impact. By providing normative data associated with research output and impact over an extended time interval, we believe that funding agencies might be able to use the data to more effectively assess research output of funded projects. This same framework can be useful if faculty-level normative data are generated at the time of promotion to assist academic institutions more objectively evaluate research achievements in promotion and hiring decisions.