0
Original Investigation |

Cost-effectiveness of Combined Outreach for the Pneumococcal and Influenza Vaccines FREE

Marcia Weaver, PhD, MA; James Krieger, MD, MPH; Joseph Castorina, MD, MPH; Mary Walls, MPH; Sandy Ciske, MN
[+] Author Affiliations

From Public Health: Seattle & King County, Seattle, Wash (Drs Weaver and Krieger and Ms Ciske); Departments of Health Services (Drs Weaver and Krieger), Medicine (Dr Krieger), and Psychosocial and Community Health Nursing (Ms Ciske) and Nursing School (Ms Walls), University of Washington, Seattle; and Northeast Health Care Quality Foundation, Dover, NH (Dr Castorina).


Arch Intern Med. 2001;161(1):111-120. doi:10.1001/archinte.161.1.111.
Text Size: A A A
Published online

Background  We conducted a cost-effectiveness analysis as part of a randomized, controlled trial of a community-based outreach initiative to promote the pneumococcal and influenza vaccines for people aged 65 years or older.

Methods  The analysis was based on primary data from the trial on the increase in vaccination rates and cost of the intervention, and published estimates of the effectiveness of the vaccines and cost of treatment. We performed partial stochastic analyses based on the confidence intervals (CIs) of the effectiveness of the intervention and of the vaccines.

Results  The cost-effectiveness ratio of the combined-outreach initiative as implemented was $35 486 per quality-adjusted life-year (QALY), whereas it was $53 547 per QALY for the pneumococcal vaccine and $130 908 per QALY for the influenza vaccine. In partial stochastic analyses, the quasi-CI of the combined-outreach initiative ranged from $15 145 to $152 311 per QALY. The cost-effectiveness ratio of the intervention targeted to people who had never received the pneumococcal vaccine or who had not received the influenza vaccine in the previous year was $11 771 per QALY, with a quasi-CI of $3330 to $46 095 per QALY. With the use of the projected cost of replicating the intervention, the cost-effectiveness ratio was $26 512 per QALY for the initiative as implemented and $7843 per QALY for a targeted initiative.

Conclusions  The community-based outreach initiative to promote the pneumococcal and influenza vaccines was reasonably cost-effective. Further improvements in cost-effectiveness could be made by targeting the initiative or through lessons learned during the first year that would reduce the cost of the initiative in subsequent years.

Figures in this Article

PNEUMOCOCCAL and influenza infections continue to cause substantial morbidity and mortality,15 and immunization against these infections remains an important public health activity. Several major advisory groups recommend the pneumococcal vaccine (Pneumovax)57 and the influenza vaccine69 for people aged 65 years and older, and research has shown that the vaccines are cost-effective. The pneumococcal vaccine was cost saving in preventing pneumococcal bacteremia among people aged 65 years and older.10 The influenza vaccine was cost saving or the net cost per person vaccinated was about $7# for people aged 65 years and older.1114 (A number sign [#] denotes that the estimates were adjusted to 1996 dollars15 to the extent information in the original article was available to do so.)

Less is known about the cost-effectiveness of interventions to promote the vaccines, and there has never been a cost-effectiveness analysis (CEA) of a community-based outreach initiative. Previous researchers analyzed interventions by providers or insurers to promote the influenza vaccine, including letter or postcard reminders from physicians' offices,1618 telephone reminders from nurses,17 personal reminders from physicians,17 and letter reminders from community pharmacists.19 Estimates of the cost per additional person vaccinated ranged from $5# to $20# for the intervention to promote the vaccine and from $14# to $32# for the intervention plus the vaccine.

We conducted a CEA as part of a randomized controlled trial of a community-based outreach initiative to promote the pneumococcal vaccine and influenza vaccine for people aged 65 years and older.20 The research presented below extends the literature on the cost-effectiveness of interventions to promote vaccines with 3 innovations. (1) It was the first CEA of a community-based outreach initiative in which effectiveness was measured in quality-adjusted life-years (QALYs). (See Krieger et al20 for estimates of the cost per person vaccinated.) (2) It was conducted from a societal perspective and conformed to the reference case recommended by the US Panel on Cost-effectiveness in Health and Medicine.21 It was based on primary data from a randomized controlled trial on the increase in the vaccination rate and the cost of the intervention and published estimates of the effectiveness of the vaccines in preventing illness and mortality and costs of treatment. (3) It demonstrated how the methods currently recommended for CEA can be used to add costs and effectiveness across more than 1 vaccine in a combined outreach initiative, or more generally across more than 1 CEA for any combined intervention.

RANDOMIZED CONTROLLED STUDY OF COMBINED OUTREACH FOR THE PNEUMOCOCCAL AND INFLUENZA VACCINES

The population, interventions, and outcome measures of the trial are described briefly below. Pls see Krieger et al20 for a report of the randomized controlled trial that conforms to the recommendations of Begg and colleagues.22

Study Site and Population

The intervention was conducted at an urban senior center in Seattle, Wash, from October 14 to November 22, 1996. Participants were recruited from the senior center membership and a marketing database with seniors who resided in the 5 contiguous ZIP codes. Potential participants were mailed an invitation letter and a baseline questionnaire. Those who returned the survey and met the study inclusion criteria (aged 65 years or older and residence in the targeted ZIP code areas) were enrolled. No exclusion criteria were applied. As shown in Figure 1, 1246 people were enrolled and randomized at baseline and 1083 people completed the follow-up survey. Randomization was by systematic allocation of alternate respondents to either control or intervention groups.

Place holder to copy figure label and caption
Figure 1.

Study participants for the randomized controlled study of combined outreach for the pneumococcal and influenza vaccines. Reprinted by permission of Elsevier Science from Krieger et al.20 Copyright 2000 by American Journal of Preventive Medicine.

Graphic Jump Location
Intervention

For the intervention group, a project coordinator sent each participant a specially designed educational brochure and a postage-paid reply card for tracking immunization status. If the participant replied that he or she had not been vaccinated or if the card was not returned, a senior volunteer called the participant. Senior volunteers received training about the pneumococcal and influenza vaccines and were responsible for contacting 20 to 25 participants. The volunteers received technical support from the project coordinator.

Other vaccine promotion activities at the senior center were available to both the intervention and control groups, including a volunteer nurse who was on site for 2 hours each weekday to give vaccines free of charge, and announcements about vaccines appeared in the newsletter and were announced at events. Vaccines were also given at other community sites, such as pharmacies and grocery stores, and the regional Medicare provider review organization sent letters to all African Americans in King County (in which Seattle is located) who were enrolled in Medicare in October 1995 and October 1996 to encourage them to get the influenza vaccine.

Outcome Measures

The primary outcomes were the proportion of individuals who reported receiving the pneumococcal vaccine and the proportion who reported receiving the influenza vaccine. Use of self-report data on immunization status is a common practice.23 Data were collected through the baseline survey in September 1996 and the follow-up survey in March 1997.

MODEL FOR THE CEA

The CEA of the combined-outreach initiative was constructed from 2 independent analyses: (1) the pneumococcal vaccine against pneumococcal bacteremia and (2) the influenza vaccine against influenza and pneumonia. Figure 2 is a model that shows these 2 relationships, as well as a possible relationship between the pneumococcal vaccine and pneumococcal pneumonia. The analysis does not include the pneumococcal vaccine against pneumococcal pneumonia, because the vaccine's effectiveness among people aged 65 years and older is controversial. The results of a recent clinical trial showed that the vaccine was not effective among the elderly who had previously been treated as inpatients for community-acquired pneumonia.24 In addition, a recent meta-analysis of randomized controlled trials showed that the vaccine was not effective in reducing the incidence of definitive pneumococcal pneumonia or mortality among high-risk patients, which included the elderly.25 (See the "Comment" section.)

Place holder to copy figure label and caption
Figure 2.

Model showing the relationship between the vaccines and the illnesses and outcomes they were intended to prevent.

Graphic Jump Location

There is a decision tree for each independent CEA. A sample decision tree for the influenza vaccine is displayed in Figure 3. The decision tree begins on the left side and shows the initial randomization of participants into either the intervention or control group. Moving to the right, participants made a decision about getting a vaccine, but the probability of getting a vaccine was higher for the intervention group than the control group. Next, some participants became ill and others did not, but the probability of becoming ill was lower for people who were vaccinated than for those who were not. Finally, some participants died and other did not, but the probability of dying was lower for people who were vaccinated than for those who were not, because the vaccine eliminated influenza and pneumonia as a cause of death when it was effective. The last branch of the tree embodies the assumption that the vaccine affected the incidence of the illnesses, but not the severity when they occurred. The last branch also reflects the competing risk analysis in which the vaccine affected mortality from influenza and pneumonia, but not from other causes.26

Place holder to copy figure label and caption
Figure 3.

Decision tree for an intervention to promote only the influenza vaccine.

Graphic Jump Location

The decision tree for bacteremia (not shown) would be similar to Figure 3, but it would have 6 cycles rather than 1 cycle, because the duration of immunity for the pneumococcal vaccine was 6 years and the duration for the influenza vaccine was 1 year.

For each independent CEA, we performed 5 different calculations. First, we performed 2 calculations for each group: (1) the total cost of the intervention (when relevant), vaccine (including treatment of side effects), and treatment of illnesses, and (2) the total QALYs lost because of vaccine side effects, morbidity, and mortality. Then, between the intervention and control groups, we calculated the marginal cost and marginal effectiveness, which were the difference between the costs and the QALYs, respectively, of the groups. Finally, we calculated the cost-effectiveness ratio, which was the ratio of marginal cost to marginal effectiveness.

For the CEA of the combined outreach initiative, we performed the same 5 calculations with the use of the sum of the costs and effectiveness of each independent analysis. The cost of the intervention, however, was counted only once. In addition, when people got the influenza and pneumococcal vaccines at the same time, the participants' expenditures for transportation and the social cost of the participants' and caregivers' time to obtain the vaccines were counted only once. The CEA of the combined outreach initiative did not allow for an interaction between the effectiveness of the pneumococcal vaccine and the influenza vaccine. Previous research suggested that there was no incremental effect of the pneumococcal vaccine in preventing pneumonia or pneumococcal pneumonia among elderly people who received the influenza vaccine,27 but neither was there a reduction in effectiveness.28

UNCERTAINTY

The major sources of uncertainty that were incorporated into the model were the effectiveness of the intervention and the effectiveness of the vaccines. To address this uncertainty, we performed partial stochastic CEAs.29 For these analyses, we calculated quasi–confidence intervals (CIs) for the cost-effectiveness ratio based on the CIs of the estimates of the effectiveness of the intervention and the vaccines. The measure of effectiveness was the difference in rates (vaccination rates in the case of the intervention and incidence rates of the illnesses in the cases of the vaccines), and the upper and lower bounds in effectiveness were based on the CI for the difference in rates. Values of effectiveness of the intervention and the vaccines used in the base case, lower bound, and upper bound estimates are listed in Table 1. The choice of these values is justified in the following subsections.

We performed 3 sets of estimates based on 3 sets of CIs: (1) the effectiveness of the intervention, (2) the effectiveness of the vaccine, and (3) the effectiveness of the intervention and the vaccine. For example, the lower bound of the quasi-CI of the first set of estimates was calculated by means of the upper bound of effectiveness of the intervention.

Table Graphic Jump LocationTable 1. Point Estimates and Confidence Intervals for the Effectiveness of the Intervention to Promote the Pneumococcal and Influenza Vaccines and the Effectiveness of the Vaccines

Values of other variables and their sources are listed in Table 2. For many variables, a 1-way sensitivity analysis in which the values were changed within reasonable bounds did not change the cost-effectiveness ratio by more than $1000. Examples include the cost of the vaccines, the frequency of influenza epidemic years, and probability of a bed-disability day from influenza and pneumonia. These variables are identified with an asterisk in Table 2. For other variables, changing the values would substantially change the cost-effectiveness ratio. These variables include the discount rate; the cost of the intervention, for which the 1-way sensitivity analyses are reported; and the incidence of and mortality rate from bacteremia. (See the "Comment" section.)

Table Graphic Jump LocationTable 2. Values of Other Variables Used in the Cost-effectiveness Model
Effectiveness of the Community-Based Outreach Initiative to Promote the Pneumococcal and Influenza Vaccines

The point estimate and the CIs for the effectiveness of the intervention were obtained from the randomized controlled trial. The measure of effectiveness was the difference in vaccination rates between the intervention and control groups; for the vaccination rates, please see Krieger et al.20 The trial showed that the effectiveness varied by vaccine and by prior immunization status, so we performed 2 sets of estimates for the cost-effectiveness: (1) one set for the intervention as implemented and (2) one set for an intervention that would be targeted to people who had never received the pneumococcal vaccine or who did not receive the influenza vaccine in the previous year. For the reference case of the first set of estimates, the intervention was 15% effective in promoting the pneumococcal vaccine and 6% effective in promoting the influenza vaccine. For the second set, the intervention was 21% effective in promoting the pneumococcal vaccine and 27% effective in promoting the influenza vaccine. The second set of estimates embodies the assumption that the effectiveness of the intervention for those subsamples could be extended to a sample as large as the full sample in a targeted outreach initiative.

Effectiveness of the Pneumococcal Vacine Against Pneumococcal Bacteremia

Estimates of effectiveness and CIs were based on a case-control study37 in which a significant negative relationship was found between effectiveness of the vaccine and number of years since vaccination and between effectiveness and patient's age. Using those data, Sisk et al10 estimated a declining, linear relationship between effectiveness and immunity during a 6-year period for 3 age strata: 65 to 74 years, 75 to 84 years, and 85 years or older. For the reference case, we assumed that effectiveness was the weighted average of the 65- to 74-year and 75- to 84-year age strata reported by Sisk et al,10 where the weights were the numbers of people in each 10-year age interval in the stationary population.30 We used the average for these 2 strata because the average age of the participants of the trial was 75 years.20 Only a few participants were aged 85 years and older, and the available evidence suggested that the vaccine was not effective for people aged 85 years or older.37 Note that the outcome measure reported by Shapiro et al37 and Sisk et al10 was incidence of pneumococcal bacteremia, and we assumed that their estimates could be applied to hospitalizations and mortality.

Effectiveness of the Influenza Vaccine

The point estimate and CI were based on the randomized controlled trial of the efficacy of the influenza vaccine among people aged 60 years and older in whom the vaccine reduced the incidence of influenza by 50%.31 For the reference case estimate, we assumed that the vaccine reduced disability days, outpatient visits, hospitalizations, and mortality by 50%. Although the outcome measure of the trial was incidence of influenza, this estimate was within the range of estimates from case-control studies for the effectiveness of the vaccine in reducing hospitalizations12,38,39 and mortality.12,38,40

DISCOUNT RATE

We followed the recommendations of the Panel on Cost-effectiveness in Health and Medicine and discounted all future costs and benefits in the reference case by 3%.21 The discount rate reflects the fact that people prefer to receive benefits and save costs in the present relative to the future. Also, in accord with their recommendations, we conducted a 1-way sensitivity analysis of the CEA with the use of a discount rate of 5%.

COST OF THE COMBINED, COMMUNITY-BASED OUTREACH INITIATIVE TO PROMOTE THE PNEUMOCOCCAL AND INFLUENZA VACCINES

The main elements of the intervention were the letter reminders, computer system for tracking vaccine status, volunteer training, and telephone calls to seniors who were not immunized. Consequently, the estimate of the cost of the intervention focused on those elements and excluded the cost of the on-site vaccinations and announcements at senior center activities that were available to both the intervention and control groups. The analysis also excluded research costs associated with conducting the randomized, controlled trial. There were 5 categories of costs of the intervention: (1) staff, (2) volunteers, (3) advisory committee, (4) computer tracking system, and (5) materials. Sources of data and assumptions for the first 3 categories are described below. The computer tracking system and materials were valued at their purchase prices, and the cost of the computer tracking system was amortized over 5 years.

Staff at the senior center and Public Health kept records of the time spent on the project, either on time sheets designed for the project or on their personal calendars. Their time was valued at their salary plus benefits. As recommended,41 the estimates also included a 7% markup for general overhead, which is the overhead rate for Public Health. We estimated that 37.5% of all time at meetings was devoted to the intervention, based on interviews with the staff.

Volunteers also kept records of the amount of time they spent calling seniors who were not vaccinated, either on time sheets designed for the project or on senior center records. The senior center kept good records of volunteer contributions because much of the budget depended on matching funds. To conform to the reference case, their time was valued at the mean weekly earnings of people aged 65 years and older36 divided by 40, which was $9.80 per hour. (Earning data by sex were not available for this age group.) The estimated wage was based on the assumption of a 40-hour work week, which may not have been valid for people aged 65 years and older because it is an age that is associated with retirement. To check the validity of this estimate, we also collected primary data from participants about work outside the home and hourly wages as part of surveys before and after the intervention. The estimated wage was identical to the average hourly wage of the participants.

The advisory committee costs were somewhat specific to community-based interventions because of the time and effort required to establish trust among community members, health agencies, and researchers. We used sign-in sheets to track time spent at meetings. Their time was valued at $9.80 per hour for members of the senior center and at salary plus benefits for representatives of health agencies and researchers. We estimated that 37.5% of the time at meetings was devoted to the intervention.

The estimated total cost of the intervention was $25 076. The cost of each of the 5 categories are reported in Table 3. As shown, $22 523 or 90% of the total cost, was for personnel. Seventy-seven percent of total cost was for staff at the senior center and Public Health, 10% for volunteers, and 3% for the advisory committee.

Table Graphic Jump LocationTable 3. Cost of the Community-Based Outreach Initiative to Promote the Pneumococcal and Influenza Vaccines

We also projected that the intervention could be replicated for about $19 315 on the basis of lessons learned during the first year of implementation. Those lessons included (1) hiring a project coordinator to work full-time for 6 weeks during the outreach initiative rather than part-time for 5 months and (2) eliminating the computer tracking system, because it was just as efficient to record the information manually.

INFORMED CONSENT

All protocols for the randomized controlled trial of the vaccine to promote the vaccines were approved by the University of Washington Human Subjects Protection Division, Seattle, and informed consent was obtained from each participant.

Beginning with the reference case results, Table 4 reports the total cost and total effectiveness for the intervention group and the control group, as well as marginal cost, marginal effectiveness, and cost-effectiveness ratio for 6 estimates. The first 3 estimates in the top half of the table were for the combined outreach initiative, pneumococcal vaccine, and influenza vaccine as implemented. The ratio of the intervention as implemented was $35 486 per QALY for the combined outreach initiative, whereas it was $53 547 per QALY for the pneumococcal vaccine and $130 908 per QALY for the influenza vaccine. As shown, the marginal effectiveness of the pneumococcal vaccine was greater than that of the influenza vaccine, reflecting the relatively greater effect of the intervention on pneumococcal vaccine coverage than influenza vaccine coverage. The marginal effectiveness of the combined outreach initiative was the sum of the independent analyses (aside from rounding error), because combined outreach did not alter the effectiveness of the intervention or the vaccines. The marginal cost was roughly the same for all 3 estimates, because the cost of the intervention was the same for all of them.

Table Graphic Jump LocationTable 4. Cost and Effectiveness Results for the Reference Case

The 3 estimates in the bottom half of the table were for the combined outreach initiative, pneumococcal vaccine, and influenza vaccine if the intervention could be targeted to seniors who had never received a pneumococcal vaccine or who had not received the influenza vaccine in the previous year. For each of these estimates, the marginal costs were lower and the marginal effectiveness was higher than in the estimates for the intervention as implemented. The cost-effectiveness ratios were $11 771 per QALY for the combined outreach initiative, $38 030 per QALY for the pneumococcal vaccine, and $22 431 per QALY for the influenza vaccine.

The quasi-CIs estimated from the partial stochastic analyses are reported in Table 5. For the intervention as implemented, the quasi-CI of the pneumococcal vaccine that was calculated from the CI of effectiveness of the intervention ranged from $37 305 to $82 364 per QALY. The interval that was calculated from the CI of effectiveness of the vaccine ranged from $40 498 to $144 380 per QALY. The range was greater for the latter, because the CI of effectiveness of the vaccine was not symmetric and the lower bound of effectiveness of the vaccine was less than 50% for 5 of 6 years. Finally, the interval that was calculated jointly from CIs of effectiveness of the intervention and the vaccine ranged from $27 944 to $218 750 per QALY. The range of the quasi-CI that was calculated jointly was greater than the sum of the ranges calculated independently, because the effectiveness of the vaccine had a multiplicative rather than an additive effect.

Table Graphic Jump LocationTable 5. Quasi–Confidence Intervals Estimated in Partial Stochastic CEA*

For the intervention as implemented, the quasi-CI of the influenza vaccine that was calculated from the CI of effectiveness of the intervention ranged from $67 666 to $409 553 per QALY. The interval that was calculated from the CI of effectiveness of the vaccine ranged from $98 179 to $171 428 per QALY. The range was greater for the former, because the lower bound of the CI of effectiveness of the intervention was only 2% and the denominator of that ratio was relatively small. Finally, the interval that was calculated jointly from the CIs of effectiveness of the intervention and the vaccine ranged from $49 606 to $528 852 per QALY.

For the intervention as implemented, the quasi-CI of the combined outreach initiative that was calculated jointly from the CIs of effectiveness of the intervention and the vaccines ranged from $15 145 to $152 311 per QALY. For the targeted intervention, the quasi-CI of the combined iniatiative that was calculated jointly ranged from $3330 to $46 095 per QALY.

In a 1-way sensitivity analysis using a 5% rather than a 3% discount rate, the cost-effectiveness ratio of the combined outreach initiative was $40 830 per QALY for the intervention as implemented, and $13 428 per QALY for the targeted intervention. The higher discount rate increases the ratios, because it decreases the value of the future benefits in the denominator and it decreases the value of future cost savings in the numerator.

In a 1-way sensitivity analysis using $19 315 as the projected cost of replicating the intervention, the cost-effectiveness ratio for the combined outreach initiative was $26 512 per QALY for the intervention as implemented and $7843 per QALY for the targeted intervention.

The CEA demonstrated that the combined community-based outreach initiative to promote the pneumococcal and influenza vaccines had a lower cost per QALY and a narrower quasi-CI than an intervention to promote either vaccine alone. As implemented, the cost-effectiveness ratio of the combined outreach initiative was $35 486 per QALY, whereas it was $53 547 per QALY for the pneumococcal vaccine and $130 908 per QALY for the influenza vaccine. The merits of the combined outreach initiative were even more evident when we assessed the uncertainty of the estimates.

There is no single criterion of a cost-effectiveness ratio below which an intervention should be adopted. The US Panel on Cost-effectiveness in Health and Medicine recommended comparisons relative to other interventions,42 as presented in the preceding paragraph. As an alternative comparison, note that the median cost-effectiveness ratio of preventive interventions surveyed at the Harvard Center for Risk Analysis was $47 590# per QALY,43 and the cost-effectiveness ratio of the combined outreach initiative was lower than that median.

Canadian experts classified interventions into 3 categories by their cost-effectiveness ratios: (1) less than $28 000# per QALY, (2) $28 000# to $140 000# per QALY, and (3) more than $140 000# per QALY.44,45 A ratio in the first category was strong evidence for adoption of an intervention. A ratio in the second category was moderate evidence for adoption of an intervention; some interventions in the second category were routinely provided and others were not. A ratio in the third category was weak evidence for adoption. According to their categories, the combined outreach initiative, as well as independent interventions for the pneumococcal and influenza vaccines, would be in the second category of cost-effectiveness ratios. Considering the uncertainty of the estimates, however, the quasi-CI of the combined outreach initiative ($15 145 to $152 311 per QALY) was more or less within the second category, whereas the quasi-CIs of an intervention for only the pneumococcal vaccine ($27 944 to $218 750 per QALY) was between the second and third categories and an intervention for only the influenza vaccine ($49 606 to $528 852 per QALY) was mostly in the third category.

The analysis also demonstrated that the cost-effectiveness ratio could be reduced by targeting the intervention or lowering the cost of the intervention. The cost-effectiveness ratio of a combined outreach initiative targeted to people who had never received the pneumococcal vaccine or who had not received the influenza vaccine in the previous year was $11 711 per QALY, with a quasi-CI of $3330 to $46 095 per QALY, which was more or less within the first category of interventions. For the intervention as implemented, participants were selected from a list of people aged 65 years and older who lived in specific ZIP code areas. For the targeted intervention, Medicare could provide a list of people for whom claims had been submitted for the pneumococcal and the influenza vaccines each year, and participants could be selected from the list of people aged 65 years and older who were not on the Medicare list in 2 previous years for the pneumococcal vaccine or the previous year for the influenza vaccine. Alternatively, managed care plans and physicians could report lists of patients who received the vaccines each year, and participants could be selected from the list of people aged 65 years and older who were not on the managed care plan and physician list. Note that the cost of the intervention reported above did not include the cost of identifying people for the targeted intervention.

Finally, the CEA of the combined, community-based outreach initiative highlighted the importance of evaluating interventions to promote vaccines and other preventive care. The CEA of the pneumococcal vaccine showed that it was cost saving,10 and CEAs of the influenza vaccine suggested that it was cost saving or that the net cost per vaccine was about $7#.1114 Promoting the vaccines however, was costly, especially among urban, underserved communities. When the cost of the intervention to promote the vaccines was included, even the combined outreach initiative was in the second category of cost-effectiveness ratios44 rather than being cost saving.

There were some limitations to the CEA, which we note below. The CEA did not include a 1-way sensitivity analysis with the unrelated medical costs of survivors. The US Panel on Cost-effectiveness in Health and Medicine concluded that the reference case may exclude these costs, but a researcher should conduct a 1-way sensitivity analysis whenever he or she believes that including them may make a significant difference to the analysis.21 Meltzer46 demonstrated that the CEA should include all future medical and nonmedical costs and provided a method of estimating the bias from omitting them. Using Meltzer's estimate of the medical expenditures for a 75-year-old, we estimated that omitting unrelated medical costs of survivors reduced the cost-effectiveness ratios by $11 000. The bias primarily reflected the lower value of average health measured in QALYs among seniors (0.71 on a scale from 0 to 1) and was essentially the same for the combined outreach initiative, pneumococcal and influenza vaccines. Similarly, Sisk et al10 reported that including unrelated medical costs increased the cost-effectiveness ratio of the pneumococcal vaccine by $9600# and $12 900# in the base-case analysis of people 65 to 74 years old and 75 to 84 years old, respectively. Using Meltzer's estimate of the present value of future net resource use per year of life saved for a 75-year-old, we estimated that omitting all future costs reduced the cost-effectiveness ratios in our study by $24 000.

As noted earlier, the cost-effectiveness ratios for the pneumococcal vaccine and the combined outreach initiative were sensitive to changes in the values of the incidence of and mortality from bacteremia. Unfortunately, the only available evidence was from community-specific studies, because there was no nationwide surveillance for pneumococcal bacteremia.47 We used the base-case values from Sisk et al10 so that our estimates could be compared with estimates that focused only on the costs of the vaccine. Note, however, that the value of the incidence of bacteremia we used was on the upper end of the range of available estimates, whereas the value of mortality from bacteremia was in the middle of the range of available estimates.

The CEA did not include the effect of the pneumococcal vaccine against pneumococcal pneumonia, because the vaccine's effectiveness among people aged 65 years and older is controversial. A meta-analysis of randomized controlled trials showed that the pneumococcal vaccine was not effective against pneumococcal pneumonia for high-risk patients, but it raised the possibility that immunocompetent elderly people should not be characterized as high risk.25 Future research that focuses on the immunocompetence of the elderly rather than on their risk of infection may show that the pneumococcal vaccine is effective against pneumococcal pneumonia among the immunocompetent elderly. Such evidence would reduce the cost-effectiveness ratio for the pneumococcal vaccine and the combined outreach initiative.

Finally, the value of the effectiveness of the influenza vaccine on mortality in our estimates was limited to mortality from pneumonia and influenza. Again, we chose this approach so that our estimates could be compared with previous research that focused only on the costs of the vaccine.14 This approach assumed that most of the illness attributed to influenza during an epidemic would be reported with diagnostic codes for pneumonia and influenza. There was evidence from case-control studies, albeit mixed, that the influenza vaccine affected mortality from other causes in addition to pneumonia and influenza. The vaccine significantly reduced mortality from all causes12 and hospital deaths from all causes38 among elderly people living in the community. Mullooly et al,11 however, reported that the influenza did not significantly reduce hospital deaths among high-risk elderly living in the community. Limiting the effectiveness to mortality from pneumonia and influenza could represent an upward bias in the cost-effectiveness ratio.

The community-based outreach initiative to promote the pneumococcal and influenza vaccines was reasonably cost-effective at $35 486 per QALY. Further improvements in cost-effectiveness could be made by targeting the intervention to people who had never received the pneumococcal vaccine or who had not received the influenza vaccine in the previous year. Lessons learned during the first year of implementation would also reduce the cost of the outreach initiative in subsequent years.

Accepted for publication July 28, 2000.

This research was funded by the US Centers for Disease Control and Prevention, Atlanta, Ga, cooperative agreement U50/CCU011820-02 (Urban Research Centers), and United Way of King County, Seattle, Wash.

We thank Carol Allen and Bob Burnside of the Central Area Senior Center, the senior volunteers, and the Project Advisory Group (Russ Alexander, MD, MPH, Ross Burks [deceased], Eric Coleman, MD, MPH, Mary Davis, Seawellow Holland, Peter Houck, MD, MPH, Tom Koepsell, MD, MPH, Rachel Pitts, RN, BSN, and Pamela Owens-Williams, MSW) for their generous contributions to this project. We also thank the anonymous reviewers of this article.

Corresponding author: Marcia Weaver, PhD, MA, 937 Broadway Ave E, Seattle, WA 98102 (e-mail: mweaver@u.washington.edu)

Fedson  DS Adult immunization: summary of the National Vaccine Advisory report. JAMA. 1994;2721133- 1137
Barker  WH Excess pneumonia and influenza associated hospitalizations during influenza epidemics in the United States, 1970-78. Am J Public Health. 1986;76761- 765
Sullivan  KMMonto  ASLongini Jr  IM Estimates of the US health impact of influenza. Am J Public Health. 1993;831712- 1716
Fedson  DS Pneumococcal vaccination in the prevention of community-acquired pneumonia: an optimistic view of cost-effectiveness. Semin Respir Infect. 1993;8285- 293
Centers for Disease Control and Prevention, Prevention of pneumococcal disease: recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Morb Mortal Wkly Rep. 1997;46 ((RR-8)) 1- 24
US Preventive Services Task Force, Guide to Clinical Preventive Service. 2nd ed. Baltimore, Md Williams & Wilkins1996;
American College of Physicians Task Force on Adult Immunization and Infectious Diseases Society of America, Guide for Adult Immunization.  Philadelphia, Pa American College of Physicians1994;
Centers for Disease Control and Prevention, Prevention and control of influenza: recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Morb Mortal Wkly Rep. 1996;45 ((RR-5)) 1- 24
Canadian Task Force on the Periodic Health Examination, Canadian Guide to Clinical Preventive Health Care.  Ottawa, Ontario Canada Communication Group1994;
Sisk  JEMoskowitz  AJWhang  W  et al.  Cost-effectiveness of vaccination against pneumococcal bacteremia among elderly people. JAMA. 1997;2781333- 1339
Mullooly  JPBennett  MDHornbrook  MC  et al.  Influenza vaccination programs for elderly persons: cost-effectiveness in a health maintenance organization. Ann Intern Med. 1994;121947- 952
Nichol  KLMargolis  KLWuorenma  JVon Sternberg  T The efficacy and cost-effectiveness of vaccination against influenza among elderly persons living in the community. N Engl J Med. 1994;33778- 784
Centers for Disease Control and Prevention, Final results: Medicare influenza vaccine demonstration—selected states, 1988-1992. MMWR Morb Mortal Wkly Rep. 1993;42 ((R-31)) 601- 604
US Congress, Office of Technology Assessment, Cost-effectiveness Analysis of Influenza Vaccination.  Washington, DC Office of Technology Assessment1981;
US Bureau of Labor Statistics, Consumer price index—all urban consumers. US city average, medical care, not seasonally adjusted. Consumer Price Index [serial online]. Available at: ftp://146.142.4.23/pub/news.release/history/cpi.011497.news. Accessed November 2, 2000.
Buchner  DMLarson  EBWhite  RF Influenza vaccination in community elderly: a controlled trial of postcard reminders. J Am Geriatr Soc. 1987;35755- 760
McDowell  INewell  CRosser  W Comparison of three methods of recalling patients for influenza vaccination. CMAJ. 1986;135991- 997
Frank  JWMcMurry  LHenderson  M Influenza vaccination in the elderly, 2: the economics of sending reminder letters. Can Med Assoc J. 1985;132516- 518521
Grabenstein  JDHartzema  AGGuess  HAJohnston  WPRittenhouse  BE Community pharmacists as immunization advocates: cost-effectiveness of a cue to influenza vaccination. Med Care. 1992;30503- 513
Krieger  JCastorina  JCiske  SWalls  MWeaver  M Increasing influenza and pneumococcal immunization rates: a randomized controlled study of a senior center-based intervention. Am J Prev Med. 2000;18123- 131
Weinstein  MCSiegel  JEGold  MRKamlet  MSRussel  LB Recommendations of the Panel on Cost-effectiveness in Health and Medicine. JAMA. 1996;2761253- 1258
Begg  CCho  MEastwood  S  et al.  Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA. 1996;276637- 639
Centers for Disease Control and Prevention, Influenza and pneumococcal vaccination levels among adults aged ≥65 years—United States, 1997. MMWR Morb Mortal Wkly Rep. 1998;47 ((R-38)) 797- 802
Ortqvist  AHedlund  JBurman  L-A  et al.  Randomised trial of 23-valent pneumococcal capsular polysaccharide vaccine in prevention of pneumonia in middle-aged and elderly people. Lancet. 1998;351399- 403
Fine  MJSmith  MACarson  CA  et al.  Efficacy of pneumococcal vaccination in adults: a meta-analysis of randomized controlled trials. Arch Intern Med. 1994;1542666- 2677
Chiang  CL Competing risks in mortality analysis. Annu Rev Public Health. 1991;12281- 307
Honkanen  POKeistinen  TMiettinen  L  et al.  Incremental effectiveness of pneumococcal vaccine on simultaneously administered influenza vaccine in preventing pneumonia and pneumococcal pneumonia among persons aged 65 years or older. Vaccine. 1999;172493- 2500
Grilli  GFuiano  LBiasio  LR  et al.  Simultaneous influenza and pneumococcal vaccination in elderly individuals. Eur J Epidemiol. 1997;13287- 291
O'Brien  BJDrummond  MFLabelle  RJWillan  A In search of power and significance: issues in the design and analysis of stochastic cost-effectiveness studies in health care. Med Care. 1994;32150- 163
Anderson  RN United States abridged life tables, 1996. Natl Vital Stat Rep [serial online]. December24 1998;47 ((13)) Available at: http://www.cdc.gov/nchs/data/nvs47_13.pdf. Accessed May 29, 1999.
Govaert  TMEThijs  CTMCNMasurel  NSprenger  MJWDinant  GJKnottnerus  JA The efficacy of the influenza vaccination in elderly individuals: a randomized double-blind placebo-controlled trial. JAMA. 1994;2721661- 1665
Abt Associates Inc, Cost Effectiveness Study of Medicare Coverage of the Influenza Vaccine: Final Report.  Washington, DC Health Care Financing Administration1993;Contract 500-89-0049.
Nichol  KLLind  AMargolis  KL  et al.  The effectiveness of vaccination against influenza in healthy, working adults. N Engl J Med. 1995;333889- 893
US National Center for Health Statistics, Final mortality statistics, 1996. Natl Vital Stat Rep [serial online]. November10 1998;47 ((9)) 18- 25Available at: http://www.cdc.gov/nchs/data/nvsr47_9.pdf. Accessed December 8, 1999.
Erickson  PWilson  RShanon  I Years of healthy life. Statistical Notes. April1995;No. 71- 14
US Bureau of Labor Statistics, Labor Force Statistics from the Current Population Survey, Table 2: median usual weekly earning of full-time wage and salary workers by age, race, Hispanic origin, and sex, third quarter 1996 averages, not seasonally adjusted. Curr Pop Survey [serial online]. Available at: http://stats.bls.gov/news.release/wkyeng.t02.htm. Accessed February 7, 1997.
Shapiro  EDBerg  ATAustrian  R  et al.  The protective efficacy of polyvalent pneumococcal polysaccharide vaccine. N Engl J Med. 1991;3251453- 1460
Fedson  DSWajda  ANicol  JPHammond  GWKaiser  DLRoos  LL Clinical effectiveness of influenza vaccination in Manitoba. JAMA. 1993;2701956- 1961
Foster  DATalsma  AFurumoto-Dawson  A  et al.  Influenza vaccine effectiveness in preventing hospitalization for pneumonia in the elderly. Am J Epidemiol. 1992;136296- 307
Fleming  DMWatson  JMNicholas  SSmith  GESwan  AV Study of the effectiveness of the influenza vaccination in the elderly in the epidemic of 1989-90 using a general practice data base. Epidemiol Infect. 1995;115581- 589
Balas  EAKretschmer  RACGnann  W  et al.  Interpreting cost analyses of clinical interventions. JAMA. 1998;27954- 57
Siegel  JEWeinstein  MCRussell  LB  et al.  Recommendations for reporting cost-effectiveness analysis. JAMA. 1996;2761339- 1341
Tengs  TOAdams  MEPliskin  JS  et al.  Five-hundred life-saving interventions and their cost-effectiveness. Risk Anal. 1995;15369- 390
Laupacis  AFeeny  DDetsky  ASTugwell  P How attractive does a new technology have to be to warrant adoption and utilization? tentative guidelines for using clinical and economic evaluations. CMAJ. 1992;146473- 481
Laupacis  AFeeny  DDetsky  ASTugwell  P Tentative guidelines for using clinical and economic evaluations revisited. CMAJ. 1993;148927- 929
Meltzer  D Accounting for future costs in medical cost-effectiveness analysis. J Health Econ. 1997;1633- 64
Breiman  RFSpika  JFNavarro  VJDarden  PMDarby  CP Pneumococcal bacteremia in Charleston County, South Carolina: a decade later. Arch Intern Med. 1990;1501401- 1405

Figures

Place holder to copy figure label and caption
Figure 1.

Study participants for the randomized controlled study of combined outreach for the pneumococcal and influenza vaccines. Reprinted by permission of Elsevier Science from Krieger et al.20 Copyright 2000 by American Journal of Preventive Medicine.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 2.

Model showing the relationship between the vaccines and the illnesses and outcomes they were intended to prevent.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 3.

Decision tree for an intervention to promote only the influenza vaccine.

Graphic Jump Location

Tables

Table Graphic Jump LocationTable 1. Point Estimates and Confidence Intervals for the Effectiveness of the Intervention to Promote the Pneumococcal and Influenza Vaccines and the Effectiveness of the Vaccines
Table Graphic Jump LocationTable 2. Values of Other Variables Used in the Cost-effectiveness Model
Table Graphic Jump LocationTable 3. Cost of the Community-Based Outreach Initiative to Promote the Pneumococcal and Influenza Vaccines
Table Graphic Jump LocationTable 4. Cost and Effectiveness Results for the Reference Case
Table Graphic Jump LocationTable 5. Quasi–Confidence Intervals Estimated in Partial Stochastic CEA*

References

Fedson  DS Adult immunization: summary of the National Vaccine Advisory report. JAMA. 1994;2721133- 1137
Barker  WH Excess pneumonia and influenza associated hospitalizations during influenza epidemics in the United States, 1970-78. Am J Public Health. 1986;76761- 765
Sullivan  KMMonto  ASLongini Jr  IM Estimates of the US health impact of influenza. Am J Public Health. 1993;831712- 1716
Fedson  DS Pneumococcal vaccination in the prevention of community-acquired pneumonia: an optimistic view of cost-effectiveness. Semin Respir Infect. 1993;8285- 293
Centers for Disease Control and Prevention, Prevention of pneumococcal disease: recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Morb Mortal Wkly Rep. 1997;46 ((RR-8)) 1- 24
US Preventive Services Task Force, Guide to Clinical Preventive Service. 2nd ed. Baltimore, Md Williams & Wilkins1996;
American College of Physicians Task Force on Adult Immunization and Infectious Diseases Society of America, Guide for Adult Immunization.  Philadelphia, Pa American College of Physicians1994;
Centers for Disease Control and Prevention, Prevention and control of influenza: recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Morb Mortal Wkly Rep. 1996;45 ((RR-5)) 1- 24
Canadian Task Force on the Periodic Health Examination, Canadian Guide to Clinical Preventive Health Care.  Ottawa, Ontario Canada Communication Group1994;
Sisk  JEMoskowitz  AJWhang  W  et al.  Cost-effectiveness of vaccination against pneumococcal bacteremia among elderly people. JAMA. 1997;2781333- 1339
Mullooly  JPBennett  MDHornbrook  MC  et al.  Influenza vaccination programs for elderly persons: cost-effectiveness in a health maintenance organization. Ann Intern Med. 1994;121947- 952
Nichol  KLMargolis  KLWuorenma  JVon Sternberg  T The efficacy and cost-effectiveness of vaccination against influenza among elderly persons living in the community. N Engl J Med. 1994;33778- 784
Centers for Disease Control and Prevention, Final results: Medicare influenza vaccine demonstration—selected states, 1988-1992. MMWR Morb Mortal Wkly Rep. 1993;42 ((R-31)) 601- 604
US Congress, Office of Technology Assessment, Cost-effectiveness Analysis of Influenza Vaccination.  Washington, DC Office of Technology Assessment1981;
US Bureau of Labor Statistics, Consumer price index—all urban consumers. US city average, medical care, not seasonally adjusted. Consumer Price Index [serial online]. Available at: ftp://146.142.4.23/pub/news.release/history/cpi.011497.news. Accessed November 2, 2000.
Buchner  DMLarson  EBWhite  RF Influenza vaccination in community elderly: a controlled trial of postcard reminders. J Am Geriatr Soc. 1987;35755- 760
McDowell  INewell  CRosser  W Comparison of three methods of recalling patients for influenza vaccination. CMAJ. 1986;135991- 997
Frank  JWMcMurry  LHenderson  M Influenza vaccination in the elderly, 2: the economics of sending reminder letters. Can Med Assoc J. 1985;132516- 518521
Grabenstein  JDHartzema  AGGuess  HAJohnston  WPRittenhouse  BE Community pharmacists as immunization advocates: cost-effectiveness of a cue to influenza vaccination. Med Care. 1992;30503- 513
Krieger  JCastorina  JCiske  SWalls  MWeaver  M Increasing influenza and pneumococcal immunization rates: a randomized controlled study of a senior center-based intervention. Am J Prev Med. 2000;18123- 131
Weinstein  MCSiegel  JEGold  MRKamlet  MSRussel  LB Recommendations of the Panel on Cost-effectiveness in Health and Medicine. JAMA. 1996;2761253- 1258
Begg  CCho  MEastwood  S  et al.  Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA. 1996;276637- 639
Centers for Disease Control and Prevention, Influenza and pneumococcal vaccination levels among adults aged ≥65 years—United States, 1997. MMWR Morb Mortal Wkly Rep. 1998;47 ((R-38)) 797- 802
Ortqvist  AHedlund  JBurman  L-A  et al.  Randomised trial of 23-valent pneumococcal capsular polysaccharide vaccine in prevention of pneumonia in middle-aged and elderly people. Lancet. 1998;351399- 403
Fine  MJSmith  MACarson  CA  et al.  Efficacy of pneumococcal vaccination in adults: a meta-analysis of randomized controlled trials. Arch Intern Med. 1994;1542666- 2677
Chiang  CL Competing risks in mortality analysis. Annu Rev Public Health. 1991;12281- 307
Honkanen  POKeistinen  TMiettinen  L  et al.  Incremental effectiveness of pneumococcal vaccine on simultaneously administered influenza vaccine in preventing pneumonia and pneumococcal pneumonia among persons aged 65 years or older. Vaccine. 1999;172493- 2500
Grilli  GFuiano  LBiasio  LR  et al.  Simultaneous influenza and pneumococcal vaccination in elderly individuals. Eur J Epidemiol. 1997;13287- 291
O'Brien  BJDrummond  MFLabelle  RJWillan  A In search of power and significance: issues in the design and analysis of stochastic cost-effectiveness studies in health care. Med Care. 1994;32150- 163
Anderson  RN United States abridged life tables, 1996. Natl Vital Stat Rep [serial online]. December24 1998;47 ((13)) Available at: http://www.cdc.gov/nchs/data/nvs47_13.pdf. Accessed May 29, 1999.
Govaert  TMEThijs  CTMCNMasurel  NSprenger  MJWDinant  GJKnottnerus  JA The efficacy of the influenza vaccination in elderly individuals: a randomized double-blind placebo-controlled trial. JAMA. 1994;2721661- 1665
Abt Associates Inc, Cost Effectiveness Study of Medicare Coverage of the Influenza Vaccine: Final Report.  Washington, DC Health Care Financing Administration1993;Contract 500-89-0049.
Nichol  KLLind  AMargolis  KL  et al.  The effectiveness of vaccination against influenza in healthy, working adults. N Engl J Med. 1995;333889- 893
US National Center for Health Statistics, Final mortality statistics, 1996. Natl Vital Stat Rep [serial online]. November10 1998;47 ((9)) 18- 25Available at: http://www.cdc.gov/nchs/data/nvsr47_9.pdf. Accessed December 8, 1999.
Erickson  PWilson  RShanon  I Years of healthy life. Statistical Notes. April1995;No. 71- 14
US Bureau of Labor Statistics, Labor Force Statistics from the Current Population Survey, Table 2: median usual weekly earning of full-time wage and salary workers by age, race, Hispanic origin, and sex, third quarter 1996 averages, not seasonally adjusted. Curr Pop Survey [serial online]. Available at: http://stats.bls.gov/news.release/wkyeng.t02.htm. Accessed February 7, 1997.
Shapiro  EDBerg  ATAustrian  R  et al.  The protective efficacy of polyvalent pneumococcal polysaccharide vaccine. N Engl J Med. 1991;3251453- 1460
Fedson  DSWajda  ANicol  JPHammond  GWKaiser  DLRoos  LL Clinical effectiveness of influenza vaccination in Manitoba. JAMA. 1993;2701956- 1961
Foster  DATalsma  AFurumoto-Dawson  A  et al.  Influenza vaccine effectiveness in preventing hospitalization for pneumonia in the elderly. Am J Epidemiol. 1992;136296- 307
Fleming  DMWatson  JMNicholas  SSmith  GESwan  AV Study of the effectiveness of the influenza vaccination in the elderly in the epidemic of 1989-90 using a general practice data base. Epidemiol Infect. 1995;115581- 589
Balas  EAKretschmer  RACGnann  W  et al.  Interpreting cost analyses of clinical interventions. JAMA. 1998;27954- 57
Siegel  JEWeinstein  MCRussell  LB  et al.  Recommendations for reporting cost-effectiveness analysis. JAMA. 1996;2761339- 1341
Tengs  TOAdams  MEPliskin  JS  et al.  Five-hundred life-saving interventions and their cost-effectiveness. Risk Anal. 1995;15369- 390
Laupacis  AFeeny  DDetsky  ASTugwell  P How attractive does a new technology have to be to warrant adoption and utilization? tentative guidelines for using clinical and economic evaluations. CMAJ. 1992;146473- 481
Laupacis  AFeeny  DDetsky  ASTugwell  P Tentative guidelines for using clinical and economic evaluations revisited. CMAJ. 1993;148927- 929
Meltzer  D Accounting for future costs in medical cost-effectiveness analysis. J Health Econ. 1997;1633- 64
Breiman  RFSpika  JFNavarro  VJDarden  PMDarby  CP Pneumococcal bacteremia in Charleston County, South Carolina: a decade later. Arch Intern Med. 1990;1501401- 1405

Correspondence

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).
Submit a Comment

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 21

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Topics
PubMed Articles