Fracture Risk Tool Validation in an Integrated Healthcare Delivery System

Objective:

To evaluate the utility of the Fracture Risk Calculator (FRC, Foundation for Osteoporosis Research and Education) for predicting 10-year hip fracture risk within a “real world” population.

Study Design:

Retrospective cohort study.

Methods:

We identified female members of Kaiser Permanente Northern California aged ≥50 years with bone mineral density (BMD) measured during 1997-2003. Hospitalization for hip fracture was ascertained up to 10 years following the BMD date, and 10-year observed hip fracture probabilities were calculated. Baseline data for fracture risk calculation were extracted from health plan databases, including age, race/ethnicity, smoking, body mass index, prior fracture, rheumatoid arthritis, glucocorticoid use, disorders associated with bone loss, and femoral neck BMD. Predicted 10-year FRC hip fracture probabilities were compared with observed 10-year hip fracture probabilities.

Results:

Among 94,489 women (mean age 62.8 ± 8.6 years, average femoral neck Z-score 0.1), the median duration of follow-up was 6.6 years, during which 1579 (1.7%) hip fractures occurred. Using the FRC, 23% met or exceeded the National Osteoporosis Foundation’s 3% hip fracture threshold. The FRC somewhat underestimated observed hip fracture probabilities; across 10-year risk categories <1%, 1% to 2.9%, and 3% to 4.9%, ratios of observed to median predicted probabilities ranged from 1.3 to 1.4.

Conclusions:

The FRC tool can be applied to assess fracture risk in large populations using data from administrative databases. Despite some underestimation, this relatively simple tool may assist targeting of at-risk populations for more complete fracture risk assessment.

(Am J Manag Care. 2011;17(3):188-194)

The utility of the Fracture Risk Calculator (FRC) for predicting 10-year hip fracture risk within a “real world” population was evaluated.

• The FRC provided rapid assessment of population fracture risk with some underestimation.

• Bone mineral density as an input parameter appeared to have little overall effect on the tool’s discrimination.

• While the FRC underestimated observed 10-year hip fracture probabilities by 30% to 40%, the ability to rapidly assess fracture risk using population data may be useful for osteoporosis programs in the initial identification of high-risk patients.

To better comply with current osteoporosis Healthcare Effectiveness Data and Information Set (HEDIS) measures,¹ many healthcare delivery systems have developed programs to identify and track women after fracture. Some health plans have also developed quality improvement programs that expand osteoporosis management beyond the HEDIS requirements to include all women and men at increased risk of fracture.^2,3 In large part, the stimulus for such targeting has been evidence that preventive osteoporosis treatment for those at increased risk of fracture is cost-effective⁴ and that comprehensive and widespread interventions can substantially reduce hip fracture rates and thereby produce savings for health plans.^2,3 However, adequate information technology resources must be available to capture information on patients, and management systems must be in place to mount large osteoporosis evaluation and treatment initiatives. Many health plans lack the organization for major fracture prevention outreach efforts, and more importantly, lack a simple, convenient system to identify those at increased risk.

Several Web-based fracture risk tools are available,^5-8 but these are designed for entry of individual patient-level information and currently do not function in an easily accessible, large-scale, batch mode where data on thousands of individuals can be instantly uploaded for fracture risk estimation. The most widely used fracture risk tool is FRAX, developed by the World Health Organization Collaborating Centre for Metabolic Bone Disease, University of Sheffield, UK^.6,9 This tool first solicits information on a number of key, independent clinical risk factors and then calculates 10-year probabilities of hip and of any 1 of 4 major osteoporotic fractures. While some of these risk variables are easily obtained from administrative data sets (eg, age, sex), others may be obtained with some effort (eg, race/ethnicity, height and weight, prior fracture, high-dose glucocorticoid exposure, history of rheumatoid arthritis, other conditions known to contribute to osteoporosis, current smoking). Certain variables (eg, family history of osteoporosis, heavy alcohol use) may not be obtained without direct patient questioning. In addition, although many patients have undergone bone mineral density (BMD) testing, numeric results may not be easily accessible; fortunately, fracture risk tools can produce results with or without BMD. Indeed, fracture risk tools can provide accurate probability estimates with less than complete input data.¹⁰ Especially for the purpose of population categorization and targeting osteoporosis outreach programs, results based on a limited data input tool would be considered adequate.

In this study, we modified an existing Web-based fracture risk tool (the Fracture Risk Calculator [FRC])⁷ to provide batch outputs using input values obtained from administrative data in a large population of women undergoing BMD testing. We evaluated the performance of the FRC batch tool against observed 10-year hip fracture rates. The FRC tool was chosen because it was highly accessible, efficient, and transparent. Furthermore, the FRC batch estimates were provided at no cost, and results were instantly available via a Web interface. We compared observed 10-year fracture rate estimates with average predicted rates in more than 90,000 female members over age 50 years who received a BMD test in Kaiser Permanente Northern California (KPNC) between 1997 and 2003. Our primary goals were to determine what proportion met or exceeded the National Osteoporosis Foundation (NOF) 3% 10-year hip fracture risk threshold^4,11 and to assess whether the tool overestimated or underestimated the true risk of hip fracture.

METHODS Population Cohort

Kaiser Permanente Northern California is a large integrated healthcare delivery system serving more than 3 million members in Northern California; approximately 0.5 million are women over the age of 50 years. Since 1991, bone densitometry has been available using Hologic dual-energy x-ray absorptiometry (DXA) scanners (Waltham, MA). By 1995, KPNC had a fully integrated, systemwide patient data collection system that included ambulatory visit diagnoses, radiologic records, and prescription drugs that complemented long-standing hospitalization databases. Therefore, we selected as our study cohort all women aged 50 to 85 years who underwent a hip bone density scan on a Hologic scanner (models QDR 2000, 4500, or Delphi), selecting the first scan during 1997-2003. We excluded those who did not have at least 1 year of continuous (<90-day gap) membership both prior to and following the DXA scan date, those for whom DXA data were not electronically accessible, and those with missing race/ethnicity. We also excluded women who had filled a prescription for a bisphosphonate in the year prior to the DXA test. The study was approved by the Institutional Review Board of KPNC.

Fracture Risk Calculator

Table 1

The fracture model estimate began with population-based 10-year fracture probability for age, sex, and race/ethnicity. Next, specific patient characteristics were compared with those of the base population, and relative risks were applied to factors that differed between the individual patient and the base population. In very simplistic terms, the product of base rate times the risk differences yielded the predicted absolute 10-year risk. The current base US 10-year fracture risks for men and for women used by the FRC are those calculated from the 2006 US National Inpatient Survey by Ettinger et al.¹² The FRC model’s relative risks for various clinical risk factors are shown in . A detailed manual of operations for the Web-based FRC batch tool is available through the Foundation for Research and Education (www.fore.org).

Data Input Variables

Age, race/ethnicity, and body mass index (BMI) were determined at the index BMD scan date. Those with missing BMI were assigned a null value of 25 kg/m2, which was the median value in our cohort. Using ambulatory care, hospitalization, and pharmacy databases, we obtained each patient’s data from the 1 year prior to the DXA measurement to secure the following exposures and diagnoses: glucocorticoid use ≥1825 mg of cumulative prednisone dose equivalent in the prior year (average 5 mg/day), rheumatoid arthritis diagnoses, and secondary causes of bone loss (diabetes mellitus with insulin use, malabsorption syndrome, chronic liver disease, and osteogenesis imperfecta). We determined prior history of fracture after age 45 years (up to 10 years prior to the DXA date) based on hospitalization and outpatient diagnoses of fracture (International Classification of Diseases,Ninth Revision [ICD-9] codes 800-829), excluding open fractures, fractures related to severe trauma (ICD-9 E-codes 800-845), and fractures of the fingers, toes, facial bones, and skull, since these are not generally considered to be osteoporotic. Information on alcohol consumption and parental history of hip fracture was not available, and smoking status was not uniformly available. We assumed that all missing input values were null.

Output From Fracture Tools

Under the auspices of institutional agreements for protection of proprietary interests, batch data without patient identifiers were securely uploaded to the Foundation for Osteoporosis Research and Education FRC Web site. Ten-year fracture probabilities were returned, calculated both with and without BMD inputs.

Follow-up and Observed Hip Fracture Outcome

Individual patient data were examined from the index BMD scan date until the earliest of the following: the fourth prescription for a bisphosphonate, likely indicating 1 year of exposure because typical prescriptions provide a 3-month supply (N = 19,440, 20.6%); a principal diagnosis of hip fracture (ICD-9 code 820.0X, 820.2X, and 820.8X, excluding open fractures and those associated with major trauma, ICD-9 E-codes 800- 845); death; disenrollment (>90-day gap in membership); or when 10 years had elapsed from the index date.

Statistical Analysis

Differences between women with and without subsequent hip fracture were compared using the X2 test for categorical variables and the Student t test for continuous variables. Incident hip fracture rates and 95% confidence intervals (CIs) were calculated per 1000 person-years up to 10 years followup, the censoring date, or December 2009 (whichever occurred first). We used Kaplan Meier product-limit estimates to calculate the 10-year hip fracture probabilities from observed events over time and compared these with the median predicted 10- year hip fracture risk for the subcategories <1%, 1% to 2.9%, and 3% to 4.9%, as well as 5% to 6.9%, 7% to 9.9%, and >10%. We also compared the proportions of women in each decade of age meeting or exceeding the NOF’s 3% cost-effectiveness hip fracture risk threshold^4,11 both with and without inclusion of BMD as an input parameter. We used the area under the receiver operating characteristic (ROC) curve (C statistic) to compare sensitivity and specificity of the FRC results, with and without BMD.¹³ The C statistic ranges from 0.5 (the predictions are no better than chance) to 1.0 (a perfect predictive model). Statistical analyses were performed using SAS version 9.1 (SAS Institute, Cary, NC) and Stata version 9.2 (Stata- Corp, College Station, TX).

RESULTS

Hologic BMD data were available for 94,489 women between the ages of 50 and 85 years, after excluding those not meeting health plan membership criteria (n = 19,178), those with missing race/ethnicity (n = 318), those with missing BMD data (n = 257), and those who received bisphosphonate drugs in the year prior to the index BMD date(n = 2730).

Table 2

Table 3

shows the characteristics of the cohort at baseline. The mean femoral neck Z-score was 0.14, suggesting that the study population overall was comparable to the Hologic reference range (National Health and Nutrition Examination Survey III).14 The median duration of follow-up was 6.6 years (interquartile range 3.6-8.3 years), during which 1579 hip fractures were observed. Those who subsequently suffered a hip fracture were older, more likely to be white, had lower BMI, had other risk factors for osteoporosis, and were less likely to be users of hormone therapy. The mean age at hip fracture was 78 years and hip fracture incidence rates increased markedly with age (). The incidence of hip fracture in nonwhites (Asians, Blacks, and Hispanics) was approximately half the incidence in whites.

Figure 1

When applying the NOF’s 3% 10-year hip fracture probability threshold, the FRC yielded an overall population proportion of 23% with a strong relationship to age (). Thus, the proportion of women qualifying under this criterion for osteoporosis management increased from about 1 in 25 at age 50 to 59 years to about 4 in 5 women at 75 years and older. The overall proportion identified as high risk showed no change when BMD was removed from the calculator input; however, when BMD was excluded, the proportion was lower among women aged 60 to 69 years (7.2% vs 17.1%) and higher among women aged 70 to 79 years (81.1% vs 59.4%).

Table 4

Figure 2

shows the predicted 10-year hip fracture probabilities compared with the observed 10-year hip fracture probabilities. The FRC underestimated the true fracture probabilities by 30% to 40%, particularly in the higher predicted risk categories (). Although 40% of our cohort used hormone therapy at baseline, a factor not included in the FRC model, this did not explain the model’s tendency to underestimate the observed fracture probability (data not shown). The area under the ROC performances of the FRC tool were similar regardless of whether BMD was included (C statistic = 0.85; 95% CI 0.84, 0.86) or not (C statistic = 0.83; 95% CI 0.82, 0.84).

DISCUSSION

We found that the FRC, a relatively simple, transparent fracture risk tool, provided rapid assessment of population fracture risk with some underestimation. When applied to our large population of women 50 years and older, approximately 1 in 4 were found to potentially qualify for osteoporosis treatment based on the NOF threshold of 3% 10-year hip fracture risk. Although the FRC somewhat underestimated the observed fracture probabilities in our cohort, similar underestimates have been reported by Sornay-Rendu et al when using FRAX to assess the risk of major osteoporotic fracture in a cohort of French women.¹⁵ Both the FRC and FRAX use the same base rate for 10-year fracture risk in women as this rate is provided by the same source¹²; the same risk input variables are also used, although the relative risks differ^.9,16 The FRAX model is also more complex mathematically, because it uses Poisson regressions to accommodate interaction terms and to calculate fracture probability offset by expected mortality.

The inclusion or exclusion of BMD as an input parameter for the FRC batch calculator appeared to have little effect overall on discrimination, as measured by the C statistic (C statistic 0.85 and 0.83 with and without BMD, respectively). Several studies have also shown that FRAX yields similar C statistic estimates in the range of 0.7 to 0.8,^10,15,17,18 including comparison between different models or with different input variables.¹⁰ It should be recognized, however, that there may be limitations to the utility of ROC curves in assessing clinical risk prediction since very large relative risks of included (or removed) variables are necessary to change the statistical outcome.¹³

We believe that both the calibration and discrimination of the FRC tool would be improved with more complete data input, particularly with inclusion of family history datain the model. Epidemiologic studies indicate that parental history of hip fracture increases from about 5% in women in their 50s to about 15% to 20% for women 70 years and older¹⁹; the model’s fracture prediction would nearly double for women with such a history. In addition, our ascertainment of prior fractures, limited to 10 years prior to baseline, yielded a prior fracture prevalence of about half that reported in epidemiologic studies^19,20; here too, the model predictions would nearly double for women with such a history. Other missing data (eg, alcohol intake missing for all, BMI missing for 30% of the cohort) are less important in the model’s calibration since thinness and heavy alcohol intake are not common in aging women.^15,20 These factors, along with more complete ascertainment of smoking status and other conditions contributing to bone loss, could be important when calculating risk for an individual.

Our analyses did not examine thresholds for clinical actions and further did not consider cost-effectiveness of such actions. The threshold for treatment depends on available resources; Kanis et al have suggested for the United Kingdom a changing threshold dependent on age,²¹ while in the United States, the NOF thresholds of 3% hip fracture and 20% major osteoporotic fracture probabilities are applied to women irrespective of age.⁴ Recently, the US Preventive Services Task Force has suggested thresholds for densitometry screening based on 10-year FRAX fracture risk thresholds of 1.2% for hip fracture and 9.3% for any 1 of 4 major osteoporotic fractures (probabilities equal to the 10-year risks for a 65-year-old white woman without risk factors).²² We note that FRC is reasonably accurate within this risk range.

The strengths of our study include the large cohort size with more than 1500 hip fracture outcomes and the fact that comprehensive data on both risk factors and outcome were available in the context of an integrated health plan system. Given the high membership retention rates in this population, we had a long follow-up period for hip fracture, accruing more than 570,000 patient-years.

Our study had the following limitations. First, our study population included only women undergoing BMD testing and thus may have included a somewhat higher proportion of younger midlife women. However, our cohort had 34,529 women older than 65 years, and the mean Z-scores of our population suggest our cohort was similar to the National Health and Nutrition Examination Survey III reference populations used by Hologic, Inc.¹⁴ A second limitation was missing or incomplete data for the fracture tool risk factors (discussed above). Others have also shown that less-than-complete risk data input appears to detract only a little from fracture risk prediction models,¹⁰ and a comprehensive assessment of clinical risk factors would be part of the appropriate next step prior to making a clinical treatment decision. Third, we ascertained only hip fracture outcomes and cannot comment on the tool’s ability to predict any 1 of 4 major osteoporotic fractures in our population. Determining incident spine fractures from administrative data is extremely difficult without individual chart review. Given the close relationship between hip and any 1 of 4 major osteoporotic fractures,¹² we predicted similar performance of the tool for this outcome. In fact, Leslie and colleagues have shown that one can accurately impute major osteoporotic fracture rates from hip fracture rates.²³ Finally, we did not have complete 10-year follow-up on all patients and thus the observed 10-year fracture rates were estimated.

A major barrier to use of any new clinical tool is its acceptance by providers and support by their institutions. The tool should also be easy to use, require little effort to run, and provide data to patients and providers that are easily grasped and interpreted. After initial targeting of at-risk patients, fracture probability should be assessed at the individual level, whereby decisions regarding aggressive osteoporosis management can be made once patients are properly evaluated and counseled. The FRC batch tool is meant to be used as a population stratification tool and is not intended to replace individual risk assessment. This approach may be helpful not only for systemwide quality improvement efforts but also for assessing potential thresholds for large-scale outreach efforts based on available resources.

In summary, it is now possible to apply a relatively simple risk tool using secure Web-based uploading of batched data in order to acquire aggregate fracture risk information. This population information may be useful for resource planning and integration into osteoporosis quality improvement programs to assist in rapid initial identification of those at risk for fracture.

Acknowledgments

These data were presented in part at the 92nd Annual Meeting of the Endocrine Society; June 19-22, 2010; San Diego, CA. The authors would like to thank Ryan Navarro, BA, and Mohammad Hararah, BA, for their assistance in assembling the bone mineral density data.

Author Affiliations: From Division of Research (JCL, ARP, MC, BE), Kaiser Permanente Northern California, Oakland, CA.

Funding Source: This study was supported in part by funding from the Division of Research, Kaiser Permanente Northern California.

Author Disclosures: Dr Lo reports being a member of the Foundation for Osteoporosis Research and Education Professional Education Committee. Dr Ettinger reports serving on the scientific advisory board and being a paid consultant for FORE. The other authors (ARP, MC) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (JCL, ARP, BE); acquisition of data (JCL, ARP, MC); analysis and interpretation of data (JCL, ARP, MC, BE); drafting of the manuscript (JCL, ARP, BE); critical revision of the manuscript for important intellectual content (JCL, ARP, BE); statistical analysis (ARP, MC); and supervision (JCL).

Address correspondence to: Joan C. Lo, MD, Division of Research, Kaiser Permanente Northern California, 2000 Broadway, Oakland, CA 94612. E-mail: joan.c.lo@kp.org.

1. Agency for Healthcare Research and Quality. Osteoporosis management in women who had a fracture: percentage of women 67 years of age and older who suffered a fracture, and who had either a bone mineral density (BMD) test or prescription for a drug to treat or prevent osteoporosis in the six months after the fracture. National Quality Measures Clearinghouse. http://qualitymeasures.ahrq.gov/content.aspx?id=14958. Accessed July 18, 2010.

2. Dell R, Greene D, Schelkun SR, Williams K. Osteoporosis disease management: the role of the orthopaedic surgeon. J Bone Joint Surg Am. 2008;90(suppl 4):188-194.

3. Newman ED, Ayoub WT, Starkey RH, Diehl JM, Wood GC. Osteoporosis disease management in a rural health care population: hip fracture reduction and reduced costs in postmenopausal women after 5 years. Osteoporos Int. 2003;14(2):146-151.

4. Tosteson AN, Melton LJ 3rd, Dawson-Hughes B, et al; National Osteoporosis Foundation Guide Committee. Cost-effective osteoporosis treatment thresholds: the United States perspective. Osteoporos Int. 2008;19(4):437-447.

5. Women’s Health Initiative (WHI). Hip Fracture Risk Calculator. http://hipcalculator.fhcrc.org/. Accessed July 18, 2010.

6. World Health Organization. WHO Fracture Risk Assessment Tool. http://www.shef.ac.uk/FRAX/. Accessed July 18, 2010.

7. Foundation for Osteoporosis Research and Education. 10-Year Fracture Risk Calculator. http://riskcalculator.fore.org/. Accessed July 18, 2010.

8. Hôpital St-Boniface Hospital. SBGH Fracture Risk Calculator. http://apps.sbgh.mb.ca/bmd-web-calculator/calculator.action. Accessed September 5, 2010.

9. Kanis JA, on Behalf of the World Health Organization Scientific Group. Assessment of Osteoporosis at the Primary Health Care Level. Technical Report. University of Sheffield, UK: World Health Organization Collaborating Centre for Metabolic Bone Diseases; 2007.

10. Ensrud KE, Lui LY, Taylor BC, et al; Study of Osteoporotic Fractures Research Group. A comparison of prediction models for fractures in older women: is more better? Arch Intern Med. 2009;169(22): 2087-2094.

11. Dawson-Hughes B; National Osteoporosis Foundation Guide Committee. A revised clinician’s guide to the prevention and treatment of osteoporosis. J Clin Endocrinol Metab. 2008;93(7):2463-2465.

12. Ettinger B, Black DM, Dawson-Hughes B, Pressman AR, Melton LJ 3rd. Updated fracture incidence rates for the US version of FRAX. Osteoporos Int. 2010;21(1):25-33.

13. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928-935.

14. Looker AC, Wahner HW, Dunn WL, et al. Updated data on proximal femur bone mineral levels of US adults. Osteoporos Int. 1998;8(5): 468-489.

15. Sornay-Rendu E, Munoz F, Delmas P, Chapurlat R. The FRAX® tool in French women: how well does it describe the real incidence of fracture in the OFELY cohort? J Bone Miner Res. 2010;25(10):2101-2107.

16. Watts NB, Ettinger B, LeBoff MS. FRAX facts. J Bone Miner Res. 2009;24(6):975-979.

17. Hippisley-Cox J, Coupland C. Predicting risk of osteoporotic fracture in men and women in England and Wales: prospective derivation and validation of QFractureScores. BMJ. 2009;339:b4229.

18. Sandhu SK, Nguyen ND, Center JR, Pocock NA, Eisman JA, Nguyen TV. Prognosis of fracture: evaluation of predictive accuracy of the FRAX algorithm and Garvan nomogram. Osteoporos Int. 2010;21(5):863-871.

19. Donaldson MG, Cawthon PM, Lui LY, et al; Study of Osteoporotic Fractures. Estimates of the proportion of older white women who would be recommended for pharmacologic treatment by the new U.S. National Osteoporosis Foundation Guidelines. J Bone Miner Res. 2009;24(4):675-680.

20. Berry SD, Kiel DP, Donaldson MG, et al. Application of the National Osteoporosis Foundation Guidelines to postmenopausal women and men: the Framingham Osteoporosis Study. Osteoporos Int. 2010;21(1):53-60.

21. Kanis JA, McCloskey EV, Johansson H, Strom O, Borgstrom F, Oden A; National Osteoporosis Guideline Group. Case finding for the management of osteoporosis with FRAX—assessment and intervention thresholds for the UK [published correction appears in Osteoporos Int. 2009;20(3):499-502]. Osteoporos Int. 2008;19(10):1395-1408.

22. Nelson HD, Haney EM, Dana T, Bougatsos C, Chou R. Screening for osteoporosis: an update for the U.S. Preventive Services Task Force. Ann Intern Med. 2010;153(2):99-111.

23. Leslie WD, Lix LM; Manitoba Bone Density Program. Imputation of 10-year osteoporotic fracture rates from hip fractures: a clinical validation study. J Bone Miner Res. 2010;25(2):388-392.

Download PDF: Fracture Risk Tool Validation in an Integrated Healthcare Delivery System