Critical Appraisal of Clinical Practice Guidelines Targeting Chronic Obstructive Pulmonary Disease FREE

CHRONIC obstructive pulmonary disease (COPD) is widely prevalent in developed and developing countries.¹ From the National Health Interview Survey conducted in the United States in 1993,² the prevalence of COPD approximated 5%. A Canadian health survey reflected that, in 1994-1995, 6% of the population aged 55 years and older acknowledged the diagnosis of COPD having been made by a health professional.³ Similarly, European studies⁴ have indicated that 4% to 6% of the adult population has clinically relevant COPD. As COPD is so prevalent, treatment approaches recommended by professional organizations are likely to affect the attitudes and behaviors of health care professionals and the use of health care resources associated with its management.

Representative examples that have clinical and financial implications include the use of ipratropium bromide vs β₂-agonists, the indications for inhaled corticosteroids and oral theophylline, the benefit of oxygen therapy for transient nocturnal or exercise desaturation, and the use of noninvasive ventilation in end-stage disease.

Several organizations have developed practice guidelines to assist clinicians in making decisions about the management of COPD. Practice guidelines are defined as "systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances."⁵(p1) They are intended to improve the process of health care and health outcomes, to decrease practice variation, and to optimize resources use.⁶ Despite the previously described intentions, it is sometimes unclear whether such guidelines do actually influence patient outcomes.⁷ Poor scientific quality ("validity") of the practice guidelines or the lack of a cogent implementation strategy may contribute to their failure to influence outcomes.⁷ Primary criteria defining the validity of a clinical practice guideline (CPG) include whether an explicit, sensible process was used to identify, select, and combine the evidence supporting its recommendations. It is also necessary for the important management options and outcomes to be clearly specified.⁸

We recently appraised CPGs for the management of COPD published by professional organizations to compare them and to explore potential sources of discrepancy among their recommendations.

We searched MEDLINE (from January 1990 to May 1999) for CPGs related to the overall management of COPD using the core strategies of: (1) lung diseases, obstructive; and (2) guideline—publication type or guideline*—text word. We also contacted content experts and professional organizations to retrieve documents not listed in MEDLINE. We selected only the guidelines targeting the comprehensive management of COPD and excluded guidelines that addressed specific components of the disease, such as respiratory rehabilitation or home oxygen use. We excluded reports that were secondary publications of practice guidelines, individual overviews, original investigations, editorials, and letters to the editor. Practice guidelines published in other languages (French, Norwegian, German, Spanish, and Polish) were translated into English.

We selected, from the 12 instruments that measure the scientific quality of practice guidelines,⁹ the one instrument that provided data supporting its validity and reliability (the "Appraisal Instrument for Clinical Guidelines" developed by Cluzeau et al¹⁰). This instrument had satisfactory internal reliability and was able to differentiate the components of guideline development that contributed to the overall guideline quality. It measures 3 methodological dimensions: (1) the rigor of development, (2) the context and content, and (3) the extent to which dissemination and implementation have been addressed during development. Four reviewers (Y.L., I.F., D.B., and R.S.G.) used this instrument to independently appraise the COPD practice guidelines.

Three reviewers (Y.L., I.F., and D.B.) also examined the guidelines for specific components relating to the management of COPD, including the following: (1) the initial assessment of the patients, (2) smoking cessation, (3) vaccination, (4) pharmacological management, (5) oxygen therapy, (6) rehabilitation, (7) surgical therapy, (8) management of acute exacerbations, and (9) α₁-antitrypsin replacement therapy. These components were classified as "recommended," "not recommended," "mentioned without any firm recommendation," or "not mentioned at all." For inhaled bronchodilators, we noted the priority ranking that was attached to their use in the initial bronchodilator prescription. Where the guidelines agreed on a management approach, we examined their recommendations regarding its application. Agreement among the reviewers for the methodological quality score and the strength of recommendations was measured using κ statistics. Once all the reviewers had appraised the content and quality of the guidelines, they shared the results of their assessment. Whenever disagreement was identified among the reviewers, it was resolved following a discussion involving all of them.

We identified 15 CPGs published between August 1992 and May 1999.^11- 25 Seven^{12,14,16,18,20- 21,24} of the 15 guidelines were published in languages other than English. When we applied the appraisal instrument to these guidelines, we noted many limitations in their scientific validity.

None of the guidelines met the primary criteria of validity, ie, none were truly evidence-based. In only 1 guideline did the authors mention that they conducted MEDLINE searches to retrieve relevant literature; however, neither the search strategy nor the study selection criteria were detailed. One guideline included an explicit statement about how the background evidence was synthesized and categorized. Sources of external funding for guideline development were clearly identified by 7 of the 15 professional organizations. In only 1 of them did the authors clearly mention that the "sponsorship did not influence the activities of the group." Mention of an independent review (other than a possible review related to their being published in peer-reviewed journals) was included in 6 guidelines. None of the guidelines was pilot tested, and only 1 included a date for reviewing or updating. A trend in the improvement of guidelines development methods over time was not clearly apparent. With few exceptions, there was no clear indication that local and cultural influences had modulated any of the organizations' recommendations.

The objectives of the guidelines were stated in 10 documents. All provided a clinical definition of COPD that would clearly identify the population to which the guidelines were meant to apply. Only 1 of the guidelines included an estimate of the expenditures likely to be associated with the recommended management.

Two guidelines suggested possible methods for implementation.

Table 1 summarizes our observation relating to the content of the 15 practice guidelines. Clarity of the recommendations was often lacking. This was reflected by the moderate level of agreement among the 3 reviewers on the recommendations attached to each management component (κ median, 0.41; interquartile range, 0.21-0.85). Disagreement often stemmed from trying to interpret phrases such as "may be used" or "can be considered." All the guidelines recommended smoking cessation, and almost all recommended influenza vaccination. Recommendations for vaccination against pneumococcus varied. Other areas of controversy included the preferential use of β₂-agonists vs anticholinergic agents as first-line bronchodilators, the indication for mucolytics, the role of inhaled corticosteroids, and the prescription of oxygen therapy for patients with transient desaturation during sleep or exercise. Lung volume reduction surgery for emphysema was an especially good example of a management strategy that has been widely accepted by some despite considerable debate regarding its indications and effectiveness (in the absence of a single randomized controlled trial). Guidelines issued after the publication of the report²⁶ that relaunched interest in this intervention also varied in their recommendations, with 4 supporting it, 3 not mentioning it at all, and 5 being equivocal.

When unanimity existed regarding a particular recommendation, we often identified discrepancies in its application. For example, regarding respiratory rehabilitation, the recommendations ranged from no application rule at all to specific recommendations based on the measurement of PCO₂ or the response to a trial of oral glucocorticoids (Table 2).

The methodological quality of CPGs that address the comprehensive management of COPD is limited. This conclusion is in support of the observation by Shaneyfelt et al²⁷ that during the past decade, guidelines published in peer-reviewed medical literature have not always adhered to methodological standards. The selection, evaluation, and synthesis of the scientific evidence are the items being most in need of improvement. It is likely that the discrepancies among the practice guidelines have arisen from different interpretations of the medical literature. Given the frequency with which guidelines were developed by "experts" who relied heavily on their knowledge or opinions of published work rather than a systematic review of the literature,²⁸ the guidelines likely reflected individual enthusiasm and biases. These may not necessarily be synonymous with current knowledge based on available evidence. Inevitably, discrepancies will arise based on different interpretations of the medical literature, as was identified by Antman et al²⁹ in a comparison of the results of meta-analyses of randomized controlled trials for the treatment of myocardial infarction and by the recommendations of clinical experts.

Criteria defining the quality of a CPG are yet to be fully validated. Guidelines' developers themselves may not agree on the methods perceived by others to be optimal. Also, the "optimal methods" may be too onerous to be implemented even by professional organizations.³⁰ We selected the Appraisal Instrument for Clinical Guidelines developed by Cluzeau et al¹⁰ to measure the scientific quality of the guidelines because it is the only instrument for which data supporting its validity were available. In the absence of a gold standard of guideline quality, the validity of this instrument was determined by the authors' finding of (1) significant correlations between the scores obtained from the instrument and their global assessment of a selection of 60 guidelines; and (2) higher scores for national guidelines than for local guidelines, a result that met their a priori prediction. The items included in the instrument we used in this study and the criteria selected by Shaneyfelt et al²⁷ are similar and encompass those that would be important to most guideline users.³⁰ Among these criteria, the generation of evidence-based recommendations is perceived as an important initial step in the guideline development process³¹ and should become a primary criterion of guideline quality.

The interest in evidence-based practice guidelines is not restricted to methodologists. Grol et al³² recently found that family practitioners were most likely to comply with clear, evidence-based recommendations, whereas vague, controversial recommendations, especially those requiring a change in existing practice, were less likely to be followed. When reviewing information on disease management, clinicians preferred brief summaries of the major recommendations with a synopsis of the underlying evidence for the expected benefits and risks.³³

The development of evidence-based guidelines presents several major challenges. A systematic review of the literature is time-consuming and expensive, a task expanded by the inclusion of several components of management in a single document. Existing systematic reviews and meta-analyses will often obviate the need or reduce it to updating.³¹ If more than 1 systematic review has been published, conflicting interpretations may emerge. There may be insufficient well-designed trials to guide clinical practice, in which case professional judgment and group consensus can fill gaps of knowledge provided that major assumptions or areas of uncertainty are acknowledged.³⁴ This involves a summary of the evidence available, including its susceptibility to bias. An example of such a classification scheme is found in Table 3. An organization may support lung volume reduction surgery in its guidelines despite the absence of a single randomized controlled trial. The decision may be based on the results of encouraging case series and the availability of financial and human resources. However, such a recommendation should also detail the level of evidence attached to it (in this case, level 3 at best). Another organization might consider the evidence insufficient to include lung volume reduction surgery in its guidelines. In both cases, the decision would be based on the evidence available at the time of the guideline publication.

For the 26 components of the management of COPD summarized in Table 1, we observed several differences in recommendations for which there is most often neither a "right" nor a "wrong" answer. Some of the differences may stem from the evolution, from 1992 to 1999, of the scientific knowledge regarding the effectiveness of several interventions. We submit that the only misleading recommendations are those for which strong data establishing the lack of efficacy are available.

Finally, even well-conducted systematic reviews are insufficient for guidelines until they are interpreted in the context of local factors, such as patient preferences and the health care setting in which the recommendations are being made.³⁵ Thus, although respiratory rehabilitation improves important domains of the quality of life of patients with COPD, practice guidelines must consider that less than 2% of the population with COPD per annum has access to such programs.³⁶ The British Thoracic Society provided an interesting example of such a situation in stating that "although some patients undoubtedly benefit from rehabilitation, facilities are limited and, until more UK data are available, firm recommendations as to who should be treated cannot be made."²²(pS13)

It is our view that the compilation (and regular update) of systematic reviews that address the management of COPD is highly desirable. Improved access to medical databases, either through the Cochrane Collaboration (an international initiative designed to prepare, maintain, and disseminate systematic reviews on health care³⁷) or by peer-reviewed medical information on the Internet, may be instrumental in improving the quality of CPGs. This effort is to avoid the situation in which firm recommendations are issued years after the evidence is available.³⁸ Such a resource document would provide professional organizations in the developed and developing world with the evidence to create their own guidelines in the context of available health care resources, attitudes, and beliefs by the physicians and patients. It would also be a valuable teaching instrument for physicians and nonphysician health professionals.

For patient care to be influenced, guidelines must be disseminated and implemented by those in practice. Their impact can then be evaluated. The lack of uniformity of the recommendations for many components of the management of COPD complicates the dissemination and implementation of comprehensive guidelines. At present, local implementation of guidelines for specific components of the management of COPD, such as the indications for home oxygen use or respiratory rehabilitation, would appear more promising.

The possibility that CPGs might be used to establish the "prevailing standard of care" and, by implication, to determine what is appropriate practice is of concern. Until now, in jurisdictions such as Canada and the United Kingdom, minimum acceptable standards of care have been determined from responsible customary practice, not from guidelines.^39- 40 In these countries, discrepancies between guidelines on COPD likely reflect the widespread variations in practice. It is, therefore, unlikely that such guidelines will influence the legal standard of care.³⁹ We should nevertheless be mindful of the potential of guidelines to affect the clinical and financial resources allocated to the disease. In the United States, in a study⁴¹ of 259 malpractice claims between 1990 and 1992, CPGs were more often used for inculpatory purposes than for exculpatory purposes. In France, mandatory medical practice guidelines were introduced in 1994 as a way of containing costs and standardizing patient care. Physicians who do not comply with several CPGs (including guidelines targeting asthma and long-term oxygen therapy for chronic respiratory tract insufficiency) can be fined.⁴² Whether practice guidelines are used within a medicolegal context or as a teaching and resource instrument for those who provide or fund health care, they should meet the criterion of scientific quality.²⁸

The purpose of this communication is not to endorse any particular management strategy for COPD, but rather to highlight that existing guidelines are not evidence-based. They vary in their recommendations for specific interventions and likely reflect the biases of selective experience rather than scientific knowledge. Practitioners, policy makers, and patients would benefit from an evidence-based resource document that summarized the literature and identified the gaps in our knowledge and the discrepancies between evidence and clinical practice. In addition to integrating evidence of effectiveness with local availability, guidelines should also address approaches to their implementation and an evaluation of the impact of their recommendations on COPD.

Accepted for publication July 28, 2000.

This study was supported by Glaxo Wellcome (Canada) Inc, Mississauga, Ontario, which was not otherwise involved in the preparation of the manuscript.

Corresponding author: Yves Lacasse, MD, MSc, Centre de Pneumologie L'Hôpital Laval, 2725 Chemin Ste-Foy, Ste-Foy, Quebec, Canada G1V 4G5 (e-mail: Yves.Lacasse@med.ulaval.ca)