Understanding the Role of Big Data in the CV Risk Calculator Controversy

Medical scholars on both sides of the very public dispute over new disease prevention guidelines from the American College of Cardiology (ACC) and the American Heart Association (AHA) expected controversy.

Major revisions to treatment protocols typically generate at least some points of genuine dispute, leaving individual professionals to weigh the evidence for themselves until new research and analysis answers questions definitively.

In this particular case, the points of actual contention are few, though serious. The vast majority of the guidelines, along with the vast majority of the procedures for creating them, have won widespread support among those who study atherosclerotic cardiovascular disease. Indeed, even the most famous critics of the new recommendations have praised them for their emphasis

on preventing stroke as well as heart disease, their efforts to differentiate risk by race and gender, and their efforts to translate research into suggestions¹:

• Studies correlating low-density lipoprotein (LDL) cholesterol and disease find no specific thresholds separating safety from danger, so the guidelines omit any mention of specific LDL targets along the lines of <70 mg/dL or <100 mg/dL.

• Drug trials have found that many nonstatin medications can lower LDL levels, but no such drug has demonstrated the ability to lower the risk of stroke or heart disease like statins, which is why the new guidelines rarely recommend considering them, not even for statin-intolerant patients with dangerous LDL levels.

• Trials of statins, on the other hand, have distinctly demonstrated that they prevent strokes and myocardial infarction associated with hypertension, diabetes, and other risk factors beyond LDL levels; thus, the new guidelines suggest statins for many patients with healthy cholesterol profiles.^2,3

• Analyses that found no evidence of benefit to statins given to older patients have also been incorporated into the new guidelines, which do not recommend them for patients over age 75 years. Such recommendations have raised objections among conservative clinicians who think the changes too radical and from people who wanted more emphasis on lifestyle and less on medication.

Among thought leaders, however, all the suggestions above have received a warm reception. The only real controversial points are formulas that combine nearly a dozen factors to estimate each patient’s 10-year risk of atherosclerotic cardiovascular disease and a recommendation that any patient whose calculated 10-year risk exceeds 7.5% consider statin therapy.

A viewpoint article published by The Lancet on November 19, 2013, just a few days after the ACC and AHA unveiled the new guidelines, began the debate on both those points, starting with the decision to base discussions about statin use on a combined calculation. “No trial of statin therapy has ever used a global risk prediction score as an enrollment criterion, so basing prescriptions on such a metric might be difficult to defend in an evidence-based climate,” wrote Paul Ridker, MD, and Nancy Cook, PhD, both professors at Harvard University’s Brigham & Women’s Hospital.

“In addition, trial data contradict statements that high absolute risk always predicts statin efficacy… The CORONA⁴ and AURORA⁵ trials enrolled individuals with very high vascular risk in the settings of heart failure or renal failure and found no evidence of event reduction with statin therapy despite large reductions in LDL cholesterol.”

Other critics have expressed particular frustration that high risk scores generated almost entirely by a patient’s age can easily cross the 7.5% threshold and thus trigger serious consideration of statin therapy.⁶ After all, they argue, no trial has ever found that statins prevent atherosclerotic cardiovascular disease associated with aging.

Donald Lloyd-Jones, MD, ScM, who co-chaired the risk assessment work group, thinks many of the critics simply misunderstand, or haven’t read, the guidelines. “The guidelines don’t order anyone to use statins. They merely indicate when it makes sense for doctors and patients to consider statins because there is evidence of benefit for patients in those groups,” said Lloyd-Jones, Eileen M. Foell Professor and chairman of the Department of Preventive Medicine at Northwestern University’s Fienberg School of Medicine.

“If your total 10-year risk is lower than 7.5%, the costs associated with statins may outweigh the benefits for many patients, so there may be no reason to use them. Once your risk exceeds 7.5%, though, it may make sense to use them, depending on why your risk exceeds 7.5%.

“If hypertension or diabetes or high LDL contributes to your risk, then statins probably make sense. If age is your only risk factor, then statins may not make sense and these guidelines aren’t pushing statins on you. I just don’t understand how anyone who had bothered to read the documents in full rather than news reports could think otherwise.”

The other major criticism of the guidelines concerns the accuracy of the risk calculator and, again, traces its origin to the Lancet piece. In it, Ridker and Cook report on their efforts to validate the risk calculator on 3 large medical data sets: the Women’s Health Study (WHS), the Physicians’ Health Study (PHS), and the Women’s Health Initiative Observation Study (WHI-OS).

The risk calculator received the initial information from each cohort participant and made predictions that Ridker and Cook then compared against actual patient results. The calculator overestimated risk by 75% to 150% in all 3 cohorts, Ridker and Cook wrote, predicting roughly double the actual amount of observed disease across the 3 groups. “It is possible that as many as 40—50% of the 33 million middle-aged Americans targeted by the new ACC/AHA guidelines for statin therapy do not actually have risk thresholds that exceed the 7.5% threshold suggested for treatment. Miscalibration to this extent should be reconciled and addressed in additional external validation cohorts before these new prediction models are widely implemented.”

Those charges have made much news, but the risk calculator’s creators and backers stand firmly behind their work.

To build the calculator, the ACC/AHA committee started with a list of factors proved to increase the risk of atherosclerotic cardiovascular disease and found several major medical cohorts that collected all the relevant data on large numbers of typical American patients for at least a decade: the Framingham Heart Study, Atherosclerosis Risk in Communities, Coronary Artery Risk Development in Young Adults, and the Cardiovascular Health Study.

They then crunched untold billions of numbers until they had the tool that best translated starting-point risk factors into accurate projections of actual disease occurrence. After finalizing the calculator, the committee attempted to validate it just as Ridker and Cook later did, by seeing how its predictions matched reality among patients in other cohorts, namely the Multiethnic Study of Atherosclerosis and Reasons for Geographic and Racial Differences in Stroke studies. These validation efforts also found that the calculator systematically overestimated risk. The overestimates tended to be small for people at low risk, but often exceeded 50% for patients at higher risk.

Committee members began to investigate and quickly found a plausible explanation for the error: many of the patients with high initial risk in both cohorts had started taking statins, which had prevented many of them suffering the strokes and heart problems that the calculator predicted. They suspect the same explanation would account for some of the overestimation that Ridker and Cook found and that missing data, along with the unusual patient populations of the WHS, PHS and WHI-OS cohorts, would explain most of the rest.

“We considered using all 3 of those cohorts, either to build or test the risk calculators, but we decided against it on grounds that they lacked the data needed to be helpful and could well end up being misleading,” said David Goff, MD, PhD, FACP, FAHA, dean of the Colorado School of Public Health and co-chair of the Risk Assessment Committee.

The data shortfalls ranged from the duration of patients’ records (less than a decade) to the information recorded (incomplete accounts of some risk factors and medications) to the way the information was collected (often selfreported). The patient populations of all 3 cohorts posed even greater concerns. Neither WHS nor PHS nor WHI-OS tracks anything like a normal cross-section of Americans. All 3 abound with the sort of affluent and educated patient who consistently enjoys far better health than poorer people with ostensibly similar medical starting points.

Barring both a decision by Ridker and Cook to hand over all their data and calculations and the discovery that Ridker and Cook devised a very clever way to compensate for the unusually healthy patient populations, Goff and his colleagues see little reason to scrap a risk calculator that, they say, works on normal populations better than any other available tool.

That said, the ACC-AHA committee did take a step to guard against the possibility of systematic risk overestimation. It decided to raise the risk threshold for the consideration of statin therapy from the 5% that many felt the evidence supported to the 7.5% that made the final cut. “It’s basically impossible to believe the calculator massively overestimates the number of patients who should consider statins when you look at the big picture,” said Goff. “The risk calculator would suggest that about a third of Americans aged 40 to 75 are at enough risk to consider statins. In reality, about a third of all Americans will die from heart disease or stroke and 60% of us will have a major vascular event.”

The controversies over the risk calculator and other aspects of the new guidelines have themselves sparked a sort of meta-controversy, not about the actual guidelines but about the process of their creation. A number of critics have suggested the very existence of serious criticisms, even if the critics turn out to be wrong, indicates that the guideline committees failed to create the sort of bulletproof process that guarantees accurate recommendations and their rapid acceptance.

“Every time a controversy plays out like this, it hurts patients. Many people will leap from the assertion that this calculator recommends statins too frequently to the conclusion that they should always refuse to use statins,” said Steven Nissen, MD, chairman of the Department of Cardiovascular Medicine at the Cleveland Clinic. “The committees should have taken extra steps, like independent peer review for previously unpublished material, to avoid controversies that could set us back years.”

But avoiding controversy has proved very difficult for many bodies that have written guidelines in the past. Debate still rages about the US Preventive Services Task Force (USPSTF) recommendation that women under 50 years not get routine yearly mammograms,⁷ and it is beginning to heat up over new suggestions on when to treat high blood pressure.⁸

There is, unfortunately, no widely accepted blueprint for writing comprehensive and accurate guidelines beyond a brief primer from the Institutes of Medicine (IOM).⁹ Still, the ACC/AHA committees think they created a strong process for tracking down relevant research, evaluating its merit, resolving apparent conflicts, acknowledging hazy areas, and translating it all into easily understood recommendations. Indeed, the process has won the groups praise from the IOM, which is trying to update and expand its guidelines for writing guidelines.

“They were talking about guideline writing committees that had designed good processes,” Lloyd-Jones said. “And they used us as their example.” References

1. Ridker PM, Cook NR. Statins: new American guidelines for prevention of cardiovascular disease [published online November 20, 2013]. Lancet. 2013;382(9907):1762-1765. doi:10.1016/S0140.

2. Colhoun HM, Betteridge DJ, Durrington PN, et al. Primary prevention of cardiovascular disease with atorvastatin in type 2 diabetes in the Collaborative Atorvastatin Diabetes Study (CARDS): multicenter randomized placebo-controlled trial. Lancet. 2004;364(9435):685-696.

3. Sever PS, Dahlöf B, Poulter NR, et al. Prevention of coronary and stroke events with atorvastatin in hypertensive patients who have average or lower-than-average cholesterol concentrations, in the Anglo-Scandinaviran Cardiac Outcomes Trial-Lipid Lowering Arm (ASCOTLLA): a multicenter randomised controlled trial. Lancet. 2003;361(9364):1149-1158.

4. Kjekshus J, Dunselman P, Blideskog M, et al. A statin treatment of heart failure? controlled rosuvastatin multinational study in heart failure (CORONA): Study design and baseline haracteristics. European Journal of Heart Failure. 2005;7(6):1059-1069.

5. Fellstrom BC, Jardine AG, Schmieder RE, et al. Rosuvastatin and cardiovascular events in patients underdoing hemodialysis. N Eng J Med. 2009;360(14):1395-1407.

6. Husten L. Controversy erupts over accuracy of cardiovascular risk calculator for guidelines. Forbes website. http://www.forbes.com/sites/larryhusten/2013/11/18/controversy-eruptsover-accuracy-of-cardiovascular-risk-calculatorfor-guidelines/. Published November 18, 2013. Accessed January 8, 2014.

7. Mammogram rate did not decline in U.S. after controversial recommendations. Science Daily. http://www.sciencedaily.com/releases/2013/04/130419080010.htm. Published April 19, 2013. Accessed January 8, 2014.

8. Burling S. New blood pressure guidelines raise concern among heart-health groups. Washington Post. December 26, 2013. http://www.washingtonpost.com/national/health-science/new-blood-pressure-guidelines-raise-concernamong-heart-health-groups/2013/12/26/eaa6c73e-6e3d-11e3-b405-7e360f7e9fd2_story.html. Accessed January 8, 2014.

9. Clinical guidelines we can trust. Institute of Medicine. http://www.iom.edu/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust/Report-Brief.aspx. Published March 23, 2011. Accessed January 8, 2014.