2. Part II: Is the
Therapy Clinically Useful?
The preceding section presented a list of criteria which
readers can use to differentiate studies that are likely
to be valid from those that may not be. Studies which do
not satisfy most of the methodological filters are usually
best ignored. This section considers how therapists should
interpret those trials which satisfy most of the methodological
filters. The message is that it is not sufficient to look
simply for evidence of a statistically significant effect
of the therapy. You need to be satisfied that the trial measures
outcomes that are meaningful, and that the positive effects
of the therapy are big enough to make the therapy worthwhile.
The harmful effects of the therapy must be infrequent or
small so that the therapy does more good than harm. Lastly,
the therapy must be cost-effective.
Of course, for a trial to be useful it must investigate meaningful
effects of treatment. This means that the outcomes must be
measured in a valid way. In general, because we usually judge
the primary worth of a treatment by whether it satisfies
patients’ needs, measurement outcomes should be meaningful
to patients. Thus a trial which shows that low-energy laser
lowers serotonin levels is much less useful than one which
shows that it reduces pain, and a trial which shows that
motor training reduces spasticity is much less useful than
one which shows it enhances functional independence.
The size of the therapy's
effect is obviously important, but often overlooked. Perhaps
this is because many readers of
clinical trials do not appreciate the distinction between "statistical
significance" and "clinical significance".
Or perhaps it reflects the preoccupation of many authors
of clinical trials with whether "p < 0.05" or
not. Statistical significance ("p < 0.05") refers
to whether the effect of the therapy is bigger than can reasonably
be attributed to chance alone. That is important (we need
to know that the observed effects of therapy were not just
chance findings) but on its own tells us nothing about how
big the effect actually was. The best estimate of the size
of the effect of a therapy is the average difference between
groups. Thus, if a hypothetical trial on the effects of mobilisation
reports that shoulder pain, as measured on a 10 cm visual
analogue scale, was reduced by a mean of 4 cm in the treatment
group and 1 cm in the control group, our best estimate of
the mean effect of treatment is a 3 cm reduction in VAS (as
4 cm minus 1 cm is 3 cm). Another hypothetical trial on muscle
stretching before sport might report that 2% of patients
in the stretch group were subsequently injured, compared
to 4% in the control group. In that case our best evidence
is that stretching reduced the risk of injury by 2% (as 4%
minus 2% is 2%). Readers of clinical trials need to look
at the size of the reported effect to decide if the effect
is big enough to be clinically worthwhile. Remember patients
often come to therapy looking for cures (of course this generalisation
may not hold in all areas of clinical practice) - most are
not interested in therapies which have only small effects.
There is an important subtlety in looking at the size of
a therapy's effects. It applies to studies whose outcomes
are measured with dichotomous outcomes (dichotomous outcomes
can have one of two values, such as dead or alive, injured
or not injured, admitted to nursing home or not admitted;
this contrasts with variables such as VAS measures of pain,
which can have any value between and including 0 and 10).
Many studies that measure dichotomous outcomes will report
the effect of therapy in terms of ratios, rather than in
terms of differences. (The ratio is sometimes called a "relative
risk" or "odds ratio" or "hazard ratio",
but it comes by other names as well). Expressed in this way,
the findings of our hypothetical stretching study would be
reported as a 50% reduction in injury risk (as 2% is half
of 4%). Usually the effect of expressing treatment effects
as ratios is to make the effect of the therapy appear large.
The better measure is the difference between the two groups.
(In fact, the most useful measure may well be the inverse
of the difference. This is sometimes called the "number
needed to treat" because it tell us, on average, how
many patients we need to treat to prevent one adverse event
- in the stretching example the NNT is 1/0.02 = 50, so one
injury is prevented for every 50 subjects who stretch).
Many studies do not report the harmful effects of therapies
(ie, the "side effects" or "complications" of
therapy). That is unfortunate, because the absence of reports
of harmful effects is often interpreted as indicating that
the therapy does no harm, but clearly that need not be so.
Glaziou and Irwig (BMJ
311: 1356-1359, 1995) have argued that the effects of therapy are usually
most pronounced when given to patients with the most severe
conditions (for example, bronchial suction might be expected
to produce a greater reduction in risk of respiratory arrest
in a head-injured patient with copious sputum retention than
in a head-injured patient with little sputum retention).
In contrast, the risks of therapy (in this case, from raised
intracranial pressure) tend to be relatively constant, regardless
of the severity of the condition. Thus a therapy is more
likely to do more good than harm when it is applied to patients
with severe conditions, and therapists should be relatively
reluctant to give a therapy which has potentially serious
side effects when the patient has a less serious condition.
In practice, it is often difficult for clinical trials to
detect harmful effects, because harmful effects tend to occur
infrequently, and most studies will have insufficient sample
sizes to detect harmful effects when they occur. Thus, even
after good randomised controlled trials of a therapy have
been performed there is an important role for large scale "monitoring" studies
which follow large cohorts of treated patients to ascertain
that harmful events do not occur excessively. Until such
studies have been performed, therapists should be wary about
applying potentially harmful therapies, particularly to patients
who stand to gain relatively little from the therapy.
An extra level of sophistication in critical appraisal involves
consideration of the degree of imprecision of estimates of
effect size offered by clinical trials. Trials are performed
on samples of subjects that are expected to be representative
of certain populations. This means that the best a trial
can provide is an (imperfectly precise) estimate of the size
of the treatment effect. Clinical trials on large numbers
of subjects provide better (more precise) estimates of the
size of treatment effects than trials on small number of
subjects. Ideally readers should consider the degree of imprecision
of the estimate when deciding what a clinical trials means,
because this will often affect the degree of certainty that
can be attached to the conclusions drawn from a particular
trial. The best way to do this is to calculate confidence
intervals about the estimate of the treatment effect size,
if these are not explicitly supplied in the trial report.
[A tutorial on how to calculate and interpret confidence
intervals about common measures of effect size is currently
being prepared. In the meantime the interested reader could
consult Sim, J and Reid, N. (1999). Statistical inference
by confidence intervals: issues of interpretation and utilization.
Physical
Therapy, 79, 186-195. Readers who are confident
(sorry) with confident intervals may find it useful to download
PEDro's confidence interval calculator by clicking here.
The calculator is in the form of an Excel spreadsheet].
The last part of
deciding the usefulness of a therapy involves deciding if
the therapy is cost-effective. This is particularly
important when health care is paid for, or subsidised, by
the public purse. There will never be enough resources to
fund all innovations in health care (probably not even all
good innovations). Thus the cost of any therapy is that money
spent on it cannot be spent on other forms of health care.
Sensible allocation of finite funds involves spending money
where the effect per dollar is greatest. Of course a therapy
cannot be cost-effective if it is not effective. But effective
therapies can be cost-ineffective. The methods used determine
cost-effectiveness are outside this author's expertise, and
it is probably better if I defer to more authoritative sources.
If you are interested, you might like to read:
Drummond
MF, Richardson WS, O'Brien BJ, Levine M, Heyland D (1997).
User's guide to the medical literature:
XIII. How to use an article on economic analysis of clinical
practice: A. Are the results of the study valid? JAMA
277: 1552-1557.
O'Brien, BJ, Heyland D, Richardson WS, Levine M, Drummond
MF (1997). User's guide to the medical literature: XIII.
How to use an article on economic analysis of
clinical practice: B. What are the results and will they help me in caring for
my patients? JAMA 277: 1802-1806.
To summarise this section:
Statistical significance does not equate to clinical usefulness.
To be clinically
useful, a therapy must:
· affect outcomes that patients
are interested in
· have big enough effects to be worthwhile
· do more good than harm
· be cost-effective
If you want to read further on assessing effect size, you
could consult:
Guyatt GH, Sackett DL, Cook DJ (1993). User's guide to
the medical literature: II. How to use an article about
therapy or prevention: B. What were the results
and will they help me in caring for my patients? JAMA
271: 59-63.
Two other (rather
less authoritative) papers, by the current author, are:
Herbert RD (2000).
How to estimate treatment effects from
reports of clinical trials. I: Continuous outcomes. Australian
Journal of Physiotherapy 46: 229-235.
Herbert RD (2000). How to estimate treatment
effects from reports of clinical trials. II: Dichotomous
outcomes. Australian
Journal of Physiotherapy 46: 309-313. |
