0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Research Letters |

Comparative Analysis of Different Approaches to Report Diagnostic Accuracy FREE

Pieter Vermeersch, MD, PhD; Xavier Bossuyt, MD, PhD
[+] Author Affiliations

Author Affiliations: Laboratory Medicine, University Hospitals Leuven, Leuven, Belgium.


Arch Intern Med. 2010;170(8):734-735. doi:10.1001/archinternmed.2010.84.
Text Size: A A A
Published online

The accuracy of a diagnostic test is traditionally reported in terms of sensitivity and specificity and predictive values in scientific articles, systematic reviews, and test inserts. Physicians tend, however, to confuse sensitivity and specificity, which are independent of the prevalence, with positive and negative predictive values, which are dependent on the prevalence.1

An alternative way of expressing diagnostic accuracy is the likelihood ratio (LR).2 Likelihood ratios offer the advantage that they are not dependent on the prevalence of the disease, in contrast to positive and negative predictive values, and can be used for nondichotomous test results.35 Steurer and coworkers1 found that reporting diagnostic accuracy as LRs expressed in nontechnical language is more intelligible to clinicians, although a subsequent randomized trial by the same group found no difference in posttest probability estimates between LRs in nontechnical language and sensitivity and specificity.6

Most practicing physicians do not use the Bayesian approach to calculate posttest probability because they consider it impractical and difficult to calculate.7 The use of a graphic representation of the posttest probability as a function of the pretest probability could overcome these problems. This approach has, however, not been studied. While such approach is ill suited for printed laboratory reports, the current advances in information technology allow laboratories to provide this graphic representation “one mouse click away” to clinicians when results are checked electronically.

We developed a 3-item questionnaire. Each question asked the respondent to estimate the posttest probability and included a pretest probability in the form of a prevalence and results of diagnostic accuracy. Depending on the question, the diagnostic accuracy was presented as sensitivity and specificity, LR in nontechnical terms, or graphically. The questionnaire, in English translation, is shown in Figure 1.

Place holder to copy figure label and caption
Figure 1.

The questionnaire and the answers by clinicians (given as number and percentage of responses) to questions 1, 2, and 3.

Graphic Jump Location

The questionnaire was distributed to 117 general practitioners and 55 specialists in internal medicine, including trainees, at 2 separate conferences on continuing medical education. The program did not include any specific information regarding the performance of diagnostic tests. The chairman asked participants to complete the questionnaire during the small breaks between the speakers. The questionnaires were collected immediately after the last speaker of the session.

Of the 172 eligible clinicians, 123 filled in the questionnaire (response rate, 72%). The clinicians answering the questions had on average more than 25 years of experience as a physician and had on average more than 40 patient contact hours per week. The correct answer to question 1 was “approximately 20%” (19.6% exactly). The second question described the same test as the first question, but the diagnostic accuracy information was described as an LR in nontechnical language (Figure 1). Since the LR equals sensitivity/(1 − specificity), the LR for the first test was 9.5. The correct answer for question 2 was also “approximately 20%” (20.4% exactly).

The third question provided a graphic representation of the posttest probability as a function of the pretest probability. The test depicted was relatively similar to the tests in the 2 previous questions, with an LR of 12.4 for a positive result. Given the higher prevalence of 20%, the correct answer was “approximately 80%” (75.5% exactly). The test depicted in question 3 is an IgA antitissue transglutaminase assay used in our laboratory (diagnostic accuracy in routine clinical practice).

The proportion of physicians who gave a correct answer was calculated for each of the 3 questions. Physicians who indicated that they did not know the answer to a particular question were not included for this calculation. A Fisher exact test was used to evaluate significant differences in proportions. Calculations were performed using GraphPad QuickCalcs (http://www.graphpad.com) from GraphPad Software (GraphPad Software Inc, La Jolla, California).

Most clinicians grossly overestimated the probability of disease when the diagnostic accuracy information was given as sensitivity and specificity (question 1). Most clinicians (81%) indicated that the probability of disease was approximately 90%. The response by younger (≤40 years old) and older physicians (≥41 years old) was comparable (7% vs 8% correct answers, respectively).

When the diagnostic accuracy information of the same test was provided as an LR in nontechnical terms (question 2), the number of correct answers increased significantly from 8% to 34% (P < .001; Figure 1). Of the 93 clinicians who answered questions 1 and 2, 43 (46%) gave the same estimate twice, 41 (44%) gave a lower estimate for question 2 than for question 1, and only 9 (10%) gave a higher estimate for question 2 than for question 1.

With a graphic approach (question 3), 73% of the responding clinicians gave the correct answer (P < .001 vs questions 1 and 2). The number of nonresponders for question 3 (38%) was higher than for question 1 (15%) (P = .002) and question 2 (22%) (P = .03).

Our study demonstrates that a graphic approach is the most efficient way to convey diagnostic accuracy information to clinicians. The lack of familiarity with a graphic approach, indicated by the higher proportion of persons who could not answer question 3 compared with question 1, is hardly surprising, since previous studies found that most clinicians also lack familiarity with LRs.7 Because the graphic approach does not require calculations to estimate the posttest probability, in contrast to the use of sensitivity and specificity or LRs, we believe that the graphic approach is much easier to explain to clinicians (Figure 2). Since most physicians in industrialized countries consult their laboratory results electronically, a graphic representation could be provided “one mouse click away” to clinicians.

Place holder to copy figure label and caption
Figure 2.

A graphic depiction of pretest vs posttest probability of the disease. Using this type of graph, one can calculate the posttest probability of the disease for any given pretest probability for a specific laboratory test. For example, if the pretest probability is 10% (the patient has a 10% chance of having the target disorder) and the depicted test result is positive, then the posttest probability of the disease is 60% (the patient has a 60% chance of having the target disorder if the test result is positive).

Graphic Jump Location

Correspondence: Dr Vermeersch, Laboratory Medicine, University Hospitals Leuven, Herestraat 49, B-3000 Leuven, Belgium (pieter.vermeersch@uz.kuleuven.be).

Author Contributions:Study concept and design: Vermeersch and Bossuyt. Acquisition of data: Vermeersch. Analysis and interpretation of data: Vermeersch. Drafting of the manuscript: Vermeersch. Critical revision of the manuscript for important intellectual content: Vermeersch and Bossuyt. Statistical analysis: Vermeersch. Study supervision: Bossuyt.

Financial Disclosure: None reported.

Steurer  JFischer  JEBachmann  LMKoller  Mter Riet  G Communicating accuracy of tests to general practitioners: a controlled study. BMJ 2002;324 (7341) 824- 826
PubMed Link to Article
Bossuyt  X Clinical performance characteristics of a laboratory test: a practical approach in the autoimmune laboratory. Autoimmun Rev 2009;8 (7) 543- 548
PubMed Link to Article
Boyd  JC Statistical analysis and presentation of data. Price  CP  RHEvidence-Based Laboratory Medicine: Principles, Practice, and Outcomes. 2nd ed. Washington, DC AACC Press2007;113- 140
Vermeersch  PBlockmans  DBossuyt  X Use of likelihood ratios can improve the clinical usefulness of enzyme immunoassays for the diagnosis of small vessel vasculitides. Clin Chem 2009;55 (10) 1886- 1888
PubMed Link to Article
Vermeersch  PVercammen  MHolvoet  AVande Broeck  IDelforge  MBossuyt  X Use of interval-specific likelihood ratios improves clinical interpretation of serum FLC results for the diagnosis of malignant plasma cell disorders. Clin Chim Acta 2009;410 (1-2) 54- 58
PubMed Link to Article
Puhan  MASteurer  JBachmann  LMter Riet  G A randomized trial of ways to describe test accuracy: the effect on physicians' post-test probability estimates. Ann Intern Med 2005;143 (3) 184- 189
PubMed Link to Article
Reid  MCLane  DAFeinstein  AR Academic calculations versus clinical judgments: practicing physicians' use of quantitative measures of test accuracy. Am J Med 1998;104 (4) 374- 380
PubMed Link to Article

Figures

Place holder to copy figure label and caption
Figure 1.

The questionnaire and the answers by clinicians (given as number and percentage of responses) to questions 1, 2, and 3.

Graphic Jump Location
Place holder to copy figure label and caption
Figure 2.

A graphic depiction of pretest vs posttest probability of the disease. Using this type of graph, one can calculate the posttest probability of the disease for any given pretest probability for a specific laboratory test. For example, if the pretest probability is 10% (the patient has a 10% chance of having the target disorder) and the depicted test result is positive, then the posttest probability of the disease is 60% (the patient has a 60% chance of having the target disorder if the test result is positive).

Graphic Jump Location

Tables

References

Steurer  JFischer  JEBachmann  LMKoller  Mter Riet  G Communicating accuracy of tests to general practitioners: a controlled study. BMJ 2002;324 (7341) 824- 826
PubMed Link to Article
Bossuyt  X Clinical performance characteristics of a laboratory test: a practical approach in the autoimmune laboratory. Autoimmun Rev 2009;8 (7) 543- 548
PubMed Link to Article
Boyd  JC Statistical analysis and presentation of data. Price  CP  RHEvidence-Based Laboratory Medicine: Principles, Practice, and Outcomes. 2nd ed. Washington, DC AACC Press2007;113- 140
Vermeersch  PBlockmans  DBossuyt  X Use of likelihood ratios can improve the clinical usefulness of enzyme immunoassays for the diagnosis of small vessel vasculitides. Clin Chem 2009;55 (10) 1886- 1888
PubMed Link to Article
Vermeersch  PVercammen  MHolvoet  AVande Broeck  IDelforge  MBossuyt  X Use of interval-specific likelihood ratios improves clinical interpretation of serum FLC results for the diagnosis of malignant plasma cell disorders. Clin Chim Acta 2009;410 (1-2) 54- 58
PubMed Link to Article
Puhan  MASteurer  JBachmann  LMter Riet  G A randomized trial of ways to describe test accuracy: the effect on physicians' post-test probability estimates. Ann Intern Med 2005;143 (3) 184- 189
PubMed Link to Article
Reid  MCLane  DAFeinstein  AR Academic calculations versus clinical judgments: practicing physicians' use of quantitative measures of test accuracy. Am J Med 1998;104 (4) 374- 380
PubMed Link to Article

Correspondence

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
Submit a Comment

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Collections