Estimating Influenza Vaccine Effectiveness in Community-Dwelling Elderly Patients Using the Instrumental Variable Analysis Method FREE

Influenza causes substantial mortality in people 65 years or older.^1- 4 Annual vaccination is recommended to reduce the burden of influenza in this age group.^5- 6 However, the evidence in support of vaccinating older adults against influenza stems primarily from observational studies, which are susceptible to bias.⁷

Past observational studies suggest that influenza vaccines reduce all-cause mortality in the elderly by approximately 50%.⁸ Recent studies observing similar mortality reduction among vaccinated individuals during non–influenza seasons suggest potential bias in such studies.^9- 13 Because influenza has been estimated to account for less than 10% of all deaths during winter periods, it seems implausible that vaccination could reduce all-cause mortality during influenza seasons by approximately 50%.¹⁴ A recent Cochrane review concluded that the available evidence is of poor quality¹⁵; thus, the true effectiveness of influenza vaccination in the elderly population is uncertain.¹⁶

Individuals who engage in health-promoting behaviors may be more likely to get vaccinated, whereas very sick individuals may be less likely to receive vaccine.^9- 10,17 This difference in underlying health status between vaccinated and unvaccinated individuals may artificially overestimate vaccine effectiveness.^9,17- 18 Traditional analytical strategies (eg, stratification, restriction, matching, and regression modeling) cannot adjust for unobserved confounders or control for nonrandom treatment allocation.¹⁹ No randomized controlled trials to examine the effectiveness of influenza vaccines against mortality have been conducted. The objective of this study was to use instrumental variable (IV) analysis, a method designed to control for unmeasured confounding, to obtain unbiased estimates of influenza vaccine effectiveness against all-cause mortality and hospitalizations for pneumonia and influenza (P&I) in the elderly population.

We conducted a population-based cohort study for 9 influenza seasons (2000-2001 to 2008-2009) using health administrative databases in Ontario. Setting the index date as the start of each influenza season, we included all persons older than 65 years. We excluded institutionalized individuals, those with invalid health card numbers, and those with no contact with the health care system within 3 years before an index date. All cohort members (approximately 1.4 million per influenza season) had free access to influenza vaccines, hospital care, physician services, and prescription medications. Ethics approval for the study was obtained from the Sunnybrook Health Sciences Centre Research Ethics Board, Toronto.

Patient records were linked using unique, anonymous, encrypted identifiers across multiple Ontario health administrative databases containing information on all publicly insured, medically necessary hospital and physician services. These included the Discharge Abstract Database,²⁰ which contains detailed information on diagnoses and procedures for hospitalizations; the Ontario Health Insurance Plan database,²¹ which contains billings claims from approximately 98% of Ontario physicians that include patient diagnosis codes and services for outpatient visits; the Ontario Drug Benefit database for outpatient drug prescriptions²²; and the Registered Persons Database for patient demographic information and deaths.²³

Influenza seasons were defined using influenza surveillance data provided by a network of sentinel laboratories. In accordance with previous studies, the start and end dates of influenza seasons were defined as the first and last occurrences of 2 consecutive weeks with positive findings for at least 5% of weekly influenza isolates tested.^10,24 Post–influenza seasons were defined as July 1 to September 30 after each influenza season. Individuals were followed up from the beginning of the influenza season (index date) to the end of the post–influenza season separately for each influenza season.

Influenza vaccination status was ascertained from physician billing claims. We also considered general vaccination codes when they exceeded baseline rates (representing annual influenza vaccination campaigns) because there was gradual uptake of the influenza-specific vaccination codes after their introduction in 1998. Although these claims include other vaccines, during annual influenza vaccination campaigns most claims (96%) likely represent influenza vaccination.¹⁰ We used self-reported influenza vaccination as the criterion standard to determine that the combination of influenza-specific and general vaccination codes among individuals older than 65 years has performance values of 0.75 for sensitivity, 0.90 for specificity, 0.96 for positive predicted value, and 0.54 for negative predicted value.²⁵ Influenza vaccines are generally administered from October through March, meaning that individuals may have been vaccinated after the start of an influenza season (the index date).

The primary outcome was all-cause mortality during influenza season. As a secondary outcome, we examined P&I hospitalizations (codes 480-487 from the International Classification of Diseases, Ninth Revision, and codes J10-J18 from the International Statistical Classification of Diseases, 10th Revision) coded in any diagnostic field. Hospitalizations for P&I were analyzed as a composite end point with mortality because mortality is a more serious outcome than hospitalization, making it inappropriate to censor the hospitalization analyses for mortality. In addition, the factors causing mortality were likely an exacerbation of those causing hospitalization, so that these events were not independent. We considered only the first admission during each influenza season or post–influenza season.

We collected data on more than 100 baseline (index date) characteristics, including demographics, comorbidities, health care utilization, current prescription medication use, and receipt of specific medical procedures.

Comorbidities were defined according to an adaptation of the adjusted clinical group as any mention of the diagnosis in the outpatient or hospitalization data sets within 3 years of the index date.²⁶ The comorbidities of interest were chronic health conditions (ie, cancers, cardiovascular diseases, respiratory diseases, anemias, renal diseases, diabetes mellitus, and immune disorders) that put individuals at elevated risk for complications owing to influenza, as identified by Canada's National Advisory Committee on Immunizations.⁵ Diagnoses of other chronic conditions (ie, cerebrovascular disease, chronic liver disease, rheumatologic diseases, and history of aspiration) were also considered. All models also controlled for the number of hospitalizations in the 3 years before the index date, and post–influenza season models controlled for any hospitalization occurring from the index date through the start of the post–influenza season.

All models used individuals as the unit of analysis. We developed a baseline patient risk index using logistic regression to predict outcomes during influenza season, incorporating all baseline patient characteristics (C = 0.87 for mortality; C = 0.85 for P&I hospitalization or mortality). We used standardized differences, expressed as percentages, to compare characteristics between vaccinated and unvaccinated patients.²⁷ Statistical models adjusted for all covariates described in the preceding section. We performed a sensitivity analysis excluding 4.5% of vaccinated individuals who were immunized after the index date.

We used logistic regression models to estimate relative outcome rates during influenza season among study participants who did or did not receive influenza vaccination. Each season was analyzed separately. We repeated this analysis to compare outcomes during post–influenza seasons among those who survived until the start of the post–influenza season. Using a pooled random-effects model, we computed a summary measure of effect combining all 9 influenza seasons.²⁸

Instrumental variable analysis is an econometric method used to remove the effects of hidden bias in observational studies.^19,29 An IV has the following 2 key characteristics: it is highly correlated with treatment and does not independently affect the outcome, so it is not associated with measured or unmeasured patient health status.

We used census subdivision (CSD)–specific influenza vaccine coverage as the IV. There were 377 CSDs (ie, municipalities or the equivalent) in Ontario before 2006 and 385 CSDs subsequently. Influenza vaccine coverage was defined as the percentage of individuals older than 65 years living in a CSD who received influenza vaccination for that influenza season. We excluded individuals whose CSD of residence could not be identified and those living in a CSD with fewer than 100 elderly individuals. We constructed this IV separately for each influenza season because of year-to-year variation in vaccine coverage across regions. Study participants were assigned to the vaccine coverage of their CSD of residence as the IV. For reporting purposes, we grouped the CSDs into quintiles of vaccination coverage to facilitate comparisons of patient characteristics across regions of high and low levels of vaccination.

The IV behaves like natural randomization of patients to regional vaccination groups that differ in their likelihood of receiving influenza vaccination. Unlike randomization, the difference in the likelihood of treatment is not 100%, and one can explore but not prove that the groups are similar in unmeasured patient characteristics. Rather than compare patients with respect to whether they received influenza vaccination, IV analysis compares groups of patients that differ in the likelihood of receiving influenza vaccination.^19,29 Excellent nontechnical expositions of the use of geographical IVs exist.^19,29- 30

Because our outcomes are binary variables, we performed IV modeling using 2-stage multivariable probit regression models. For each influenza season, we estimated the IV-adjusted odds of each outcome, with CSD-level influenza vaccine coverage serving as the IV. We transformed the results of the probit regression models into odds ratios (ORs) by multiplying the coefficients by a scaling factor of 1.6, which approximates the log OR coefficients of a logistic regression model.³¹ We repeated the IV analysis on outcomes during the post–influenza seasons and pooled the results across all influenza seasons to compute a summary measure of effect. Owing to a greater proportion of individuals living in urban areas in the highest influenza vaccination quintile, we conducted a sensitivity analysis restricted to urban CSDs.

We performed statistical analyses using commercially available software (SAS, version 9.2,³² and the procedure IVPROBIT in STATA, version 9.2³³). All tests were 2-tailed. We used P < .05 as the level of statistical significance.

During the course of 9 influenza seasons, we followed up 1.3 million to 1.5 million elderly individuals each year, totaling 12.6 million person–influenza seasons of observation (Table 1). On average, 58.2% of the cohort was vaccinated annually. We observed 130 532 deaths and 62 913 P&I hospitalizations during influenza season. Vaccinated individuals had more outpatient visits in the preceding year, a higher prevalence of recorded comorbidities, and greater use of prescription medication and medical procedures (Table 2).

Influenza vaccine coverage ranged from 0% to 79% across CSDs. The CSDs with vaccination coverage ranging from 0% to 5% represented 2.8% of CSDs but only 0.2% of residents in the study. Coverage increased consistently from the lowest to the highest quintiles (Table 3). The partial F statistics from the first-stage IV regression models were greater than 950. Although there was a notable difference in rural residence and small differences in specific risk factors, mean predicted influenza season outcome rates—our summary measures of patient risk—were remarkably consistent across quintiles for all influenza seasons. The balance in the distribution of the measured risk factors across CSDs provides reasonable evidence to infer that unmeasurable risk factors are also likely balanced across CSDs. This inference lends support to regional influenza vaccine coverage being a valid and strong IV.

Using logistic regression and pooling across influenza seasons, we determined that influenza vaccination was associated with a 33% reduction in mortality during influenza season (adjusted OR, 0.67 [95% CI, 0.62-0.72]) (Table 4). Influenza vaccination was also significantly associated with mortality reduction after influenza season (adjusted OR, 0.85; [95% CI, 0.83-0.86]).

Instrumental variable analysis showed that the pooled adjusted association between influenza vaccination and mortality during influenza season was not significant (adjusted OR, 0.94 [95% CI, 0.84-1.03]) (Table 4). Restricting the analysis to urban areas led to similar results (adjusted OR, 0.93 [95% CI, 0.82-1.04]). According to the IV method, influenza vaccination was not associated with decreased mortality during post–influenza seasons (adjusted OR, 1.13 [95% CI, 1.07-1.19]).

Excluding individuals vaccinated after the index dates did not alter the results (adjusted OR using logistic regression, 0.68 [95% CI, 0.63-0.73]; adjusted OR using IV analysis, 0.94 [95% CI, 0.84-1.04]).

Results for the composite outcome of P&I hospitalization and death demonstrated reductions when using logistic regression (Table 5). According to IV analysis, the composite outcome was reduced during influenza seasons (adjusted OR, 0.86 [95% CI, 0.79-0.92]) but not during post–influenza seasons (1.02 [ 0.97-1.06]).

We found influenza vaccination of community-dwelling elderly individuals to be associated with reductions in all-cause mortality and the composite of P&I hospitalization and mortality during and after influenza seasons when using standard regression modeling. The latter suggests the presence of bias because influenza vaccines should have no benefit in the absence of circulating influenza. In contrast, IV analysis indicated that influenza vaccination was not associated with decreased mortality during or after the influenza season. However, it was associated with a reduction in combined P&I hospitalizations and mortality during influenza seasons but not post–influenza seasons. The estimates of vaccine effectiveness obtained using IV analysis appear to be less biased than those from standard regression modeling. We conclude that influenza vaccines may not reduce all-cause mortality among elderly individuals during influenza season but may reduce the combined rates of P&I hospitalization and death.

The estimates of influenza vaccine effectiveness obtained in this study by using standard regression modeling are comparable to estimates from other observational studies: 43% to 55% for all-cause mortality and 31% to 48% for the composite of P&I hospitalization and death.^8,34- 36 However, a 40% to 50% reduction in all-cause mortality (not just deaths attributable to influenza) is implausible if influenza accounts for no more than 10% of deaths during winter months.¹⁴ The results of our IV analysis suggest that influenza vaccination may potentially be associated with a 6% reduction (with as much as a 16% reduction or as little as no reduction) in all-cause mortality during influenza seasons and is associated with a 14% reduction in P&I hospitalization or death, which is a more specific outcome. These estimates are more plausible and are similar to a 4.6% reduction in all-cause mortality and a 9% reduction in P&I hospitalizations during influenza season observed in recent studies using case-centered logistic regression, which has been proposed as a bias-minimizing technique.^11,37

The association between influenza vaccination and outcomes during non–influenza seasons is evidence that previous observational studies are biased.^9- 11 Because there is minimal influenza circulation outside of influenza seasons, vaccination should confer no benefits during these periods (all OR estimates should be close to the null value of 1.00). However, several studies have demonstrated that vaccinated individuals are approximately 20% to 35% less likely to die during post–influenza seasons,^9- 11 which is comparable to our estimate of 15% when using standard regression models. Although our study showed that vaccinated individuals had a higher prevalence of recorded comorbidities and greater use of prescription medications and medical procedures, the protective effect of vaccination during noninfluenza seasons demonstrates the persistence of bias despite adjustment for the factors we described. With IV analysis, influenza vaccination was found not to be associated with reductions in post–influenza season P&I hospitalization or death. This finding increases the confidence that our estimates for the association between vaccination and outcomes during influenza seasons using the IV analysis are unbiased. The estimate for all-cause mortality during post–influenza seasons being significantly greater than 1.00 was unexpected and raises the possibility that we may have underestimated the true effectiveness of the vaccine for this outcome.

Our season-specific estimates also warrant discussion. Based on the framework proposed by Simonsen et al,⁷ influenza vaccine effectiveness is expected to be lower when the vaccine does not match the predominant circulating strains and higher during more severe influenza seasons. The 2003-2004 season saw the emergence of the influenza A/Fujian(H3N2) drifted strain that was responsible for 92% of the characterized isolates in Canada but was not included in that season's vaccine.³⁸ However, that season was also the most severe among the 9 influenza seasons included in this study, with 0.70 all-cause deaths and 0.40 P&I hospitalizations per 1000 population per week of influenza season compared with means of 0.66 and 0.32, respectively, for the other 8 seasons. It is uncertain which of these factors contributed more toward the estimates of vaccine effectiveness for that season being among the largest or whether the finding is due to chance.

In terms of interpretation, IV analysis estimates the treatment effect on the marginal population, defined as patients who would receive influenza vaccination in regions with higher but not lower influenza vaccine coverage.³⁹ In our study, such individuals would be more likely to be vaccinated if they had better access to influenza vaccination clinics and/or physicians who are more inclined to encourage influenza vaccination.

There have been 2 previous attempts to use IV analysis to estimate influenza vaccine effectiveness among elderly individuals. One study tried various IVs but found that none satisfied the assumptions of a valid IV.⁴⁰ Another study used the diagnosis of arthritis or gout as the IV and found that it satisfied both key assumptions; in that study, influenza vaccination was associated with a nonsignificant 10% mortality reduction (confidence intervals were not provided).⁴¹

Our study had several notable strengths other than the application of the IV analysis method. By including all community-dwelling elderly individuals in Ontario for 9 influenza seasons, we analyzed more than 12 million person–influenza seasons of observation, making our study one of the largest ever conducted to assess influenza vaccine effectiveness. The large sample allowed for more than 100 confounding variables to be adjusted for without compromising statistical power.

Nonetheless, several limitations in our study merit emphasis. First, because approximately 25% of influenza vaccines administered to the elderly are not recorded in the Ontario Health Insurance Plan physician claims database (they were provided outside physician offices, eg, at public health clinics and pharmacies), individual-level vaccination status and regional vaccine coverage may have been misclassified to a certain extent.²⁵ Such misclassification may have biased our estimates toward the null. Second, cause-specific mortality data were unavailable, so we could not use P&I or respiratory mortality as more specific outcomes. With respect to the IV method, one can strongly infer but not prove that the IV is not related to unmeasured illness variables. However, the fact that there was little relationship with measured variables implies that, for an unmeasured variable to bias these results, it needs to be associated with the outcome and uncorrelated with any of the measured variables.

Application of IV analysis methods to linked health administrative data sets is a valid method for estimating an unbiased estimate of influenza vaccine effectiveness in elderly individuals without the challenges associated with conducting a randomized controlled trial. The results of our study suggest that influenza vaccination may be associated with reductions in the composite of P&I hospitalization and mortality, but not with all-cause mortality alone, among elderly individuals. Nevertheless, current guidelines recommending annual vaccination against influenza for elderly individuals should remain in place until more definitive evidence has been amassed, as even small reductions in outcomes during influenza season as a result of influenza vaccination would be worthwhile because it is a generally safe and relatively low-cost intervention. Improved estimates of influenza vaccine effectiveness will permit better decision making at the individual and population levels and may spur the development of more effective strategies for reducing the burden of influenza.