Interpreting hazards: the increasing importance of "antidote to anecdote" in managed care.

A September 2009 report commissioned by a British biscuit manufacturer identified the dangers of a seemingly harmless activity in which citizens of the United Kingdom (UK) engage on a daily basis—English tea time. The report was based on a “national survey” of more than 1,000 UK residents selected and queried about their tea time habits using an unreported methodology, coupled with laboratory testing of the physical properties of various biscuits. Costs from a societal perspective were based on standard pricing for items and services that are necessary, according to the project’s “team of experts,” to treat biscuit-related injuries—for example, anesthetic spray to treat burns sustained while dunking the biscuit in hot tea; heat packs to ease back strain caused by bending to pick up a dropped biscuit fragment; and visits to a National Health Service physician to treat an errant crumb lodged in an eye, ear, or windpipe. The report even included a statistical model, the Biscuit Incident Threat Evaluation (BITE), which predicted the risk associated with eating particular biscuits, accompanied by a disclaimer noting that the model “has been calculated based upon only the most obvious types of injuries and based upon common biscuit-eating behavior types. It is strongly recommended that the individual consult the biscuit manufacturer and perform a self assessment of their particular risk prior to eating biscuits.”1 It appears unlikely that the BITE model will ever become a Medline-indexed publication. Nonetheless, its dissemination bears the markings of challenges that sometimes threaten the efforts of managed care organizations (MCOs) to identify and promote cost-effective therapies. The BITE findings were widely reported in the popular press, with headlines that included “Crumbs: Half of Britons Injured by Their Biscuits” and “Brits in Killer Biscuit Warning.”2,3 Consumers visiting the sponsoring company’s website were told that “there are hundreds of biscuit related injuries treated by UK doctors every year” and were encouraged to use the BITE tool lest they “risk it for a biscuit.”4 And, in a pattern that is familiar to most MCO decision makers, the most dangerous biscuit identified by the “study” was a particular brand of custard cream cookie, a competitor to a purportedly safer treat whose manufacturer funded and published the work.1 MCO decision makers are routinely flooded with requests to modify coverage and benefit design policies in response to purportedly reliable evidence about clinical and economic risks. Typically, the MCO decision maker is told that the increased acquisition cost for a given treatment or benefit design change is outweighed by clinical or economic benefits attributable to reduction in the risk of high-cost medical events, indicated by odds ratios (ORs) or hazard ratios (HRs). The challenge for a decision maker is to assess a vast amount of information in the usually limited amount of time available. The MCO’s intent, as expressed by Jackson and Barbuto in a June 2008 JMCP commentary on the lack of evidence for treating migraine with botulinum toxin, is to apply “antidote to anecdote.”5 That is, the ideal decisionmaking process relies on quantitative evidence reported in the peer-reviewed research literature in lieu of press reports and the ever-increasing volume of subjective and sometimes unreliable information provided on blogs. The key question for a decision maker is this: How accurate is the peer-reviewed evidence that is presented to support the proposed policy? This editorial examines some of the most common problems in claims of risk reduction and provides a checklist for hazard avoidance by MCO decision makers (Table 1).6