Heterogeneity: we can't live with it, and we can't live without it

As hard as it is to believe now, it was permissible not long ago to sell a prescription drug (in the USA) without providing evidence that it worked; all that was needed was evidence it was safe. Then, KefauvereHarris changed everything. Acting largely in response to the thalidomide tragedy, Estes Kefauver, a senator from Tennessee, and representative Oren Harris from Arkansas introduced the amendment that now bears their names. The amendment required convincing evidence of efficacy, as well as safety, before a drug could be brought to market. It was signed into law by John F Kennedy on 10 October 1962. The effect of KefauvereHarris on clinical practice is unequivocal; its effect on the shaping of clinical evidence has been less obvious. Although it would be hard to prove a strong causeeeffect relation between enactment of the amendment and emergence of controlled trials as the evidentiary ‘gold standard,’ the timing is right. Reports of controlled trials that meet reasonable methodological standards began to enter the medical literature in 1948, but their numbers were small until 1966djust a few years after KefauvereHarrisdat which point their numbers soared, increasing about fivefold over the next 14 years. (The Cochrane Collaboration, which has been a powerful force in establishing the dominance of controlled trial evidence, clearly did not play a role in this surge, since it came into existence only in 1993.) The rapid rise in healthcare spending in the 1960s and 1970s almost certainly reinforced the need for hard evidence of efficacy to avoid wasting medical resources. However, it is hard to avoid the conclusion that after 1962, clinical research went increasingly where the money was, and the money was flowing increasingly from industry as it set about meeting the new legislative requirement for strong experimental evidence that its products actually work. Since biological and social heterogeneity among trial participants interferes with the detection of true causeeeffect relationships, controlled trials are carefully designed to minimise or eliminate heterogeneity’s effects, primarily by aggregating the outcomes from many participants rather than drawing inferences from outcomes in individual persons. This intentional ‘heterogeneity blindness’ in controlled trials has brought with it a progressive shift from qualitative documentation of the concreted meticulous narrative case reports and case series describing illness in individual patients, which had dominated the medical literature for decadesdto quantitative assessment of the abstractdsophisticated statistical inferences about summary effect size in average patients. Determining effect sizes by merging outcome data from groups of study participants obscures the reality that clinical interventions rarely work for everyone and under all circumstances. In large part, clinicians are unable to face that reality because they lack information about the biological, experiential and environmental sources of that heterogeneity, and the impact of that heterogeneity on clinical responses. When an intervention known to be effective in study populations fails to work in a particular patient, clinicians therefore have little choice but to move to alternative interventions on the basis of pragmatic rules of thumb (heuristics). 5 The shift in medical evidence from finegrained narrative to sweeping statistical inference calls to mind analogous shifts in perspective that have followed certain other landmark changes in information systems. For example, when satellite cameras became available, they aggregated billions of bits of information from the earth’s surface into images from space, which have created a new, emergent reality about land use, oceans, Correspondence to Dr Frank Davidoff, Institute for Healthcare Improvement, 143 Garden Street, Wethersfield, CT 06109, USA; [email protected]