Quality of Clinical and Economic Evidence in Dossier Formulary Submissions

Objective: To investigate the quality and complete-ness of clinical and economic data in dossiers submitted by drug companies to a health plan using Academy of Managed Care Pharmacy guidelines (the Format) for formulary submissions.

Study Design: We reviewed the quality of economic analyses in dossiers submitted to Premera Blue Cross Health Plan (Mountlake Terrace, Washington; enrollment 1.6 million) between January 2002 and September 2005. For dossiers submitted in 2003, we examined the clinical studies included.

Methods: Dossiers were audited with a data collection form to judge the types of clinical studies used to support labeled and off-label indications, and the quality and transparency of economic analyses. We compared economic analyses for high-cost (30-day treatment cost > $1000) versus low-cost products, and for "innovative" versus "me-too" drugs.

Results: Evidence to support off-label indications often was included in 2003 dossiers, but the information was less extensive and of poorer quality than data for labeled indications. Of 115 dossiers submitted between 2002 and 2005, 53 (46%) included economic analyses. The economic analyses had low levels of compliance with standards: only 43% performed sensitivity analysis; 38% stated the study perspective; 37% discussed relevant treatment alternatives; 20% stated assumptions clearly; and 18% mentioned caveats to conclusions. Economic analyses of high-cost products and innovative products had higher compliance with recommended practices.

Conclusions: Drug companies are submitting dossiers of evidence to formulary committees. Dossiers often included clinical data to support off-label indications, but concerns persist about their quality. About half of dossiers included economic analyses, but these analyses had relatively low levels of compliance with recommended practices.

(Am J Manag Care. 2007;13:401-407)

Dossiers being submitted under AMCP Format provide an opportunity for health plans to evaluate a drug’s clinical and economic evidence. However, concerns persist about the quality of the clinical data and the economic analyses in dossiers; therefore, plans should:

• Be vigilant and cautious when reviewing clinical and economic evidence in dossiers, especially with lower-cost and “me-too” drugs.
• Train formulary committee members regarding the potential value of the Academy of Managed Care Pharmacy guidelines (the Format).
• Formalize a process for providing feedback to pharmaceutical companies on dossier quality and transparency.

In recent years, health plan officials have worked to standardize and improve the processes by which they decide whether and how to list new drugs for formularies, with the goal of grounding decisions in stronger clinical and economic evidence. The trend reflects 2 broader movements in healthcare, one toward evidence-based medicine and the other toward explicit consideration of cost-effectiveness or "value for money" arguments.^1-3 The movement toward formulary guidelines has evolved over the past decade. Although formulary committees have used clinical and pharma-coeconomic data to aid their decision-making processes for some time, the information was used informally, and great variation existed across plans.⁴ With rapid increases in drug spending in the 1990s, public and private health plans as well as pharmacy benefit management (PBM) companies began to use formularies more aggressively in an attempt to contain costs.⁵ This growing reliance on formularies, combined with the increase in the availability and acceptance of cost-effectiveness analyses, led to the issuance of new guidelines that called for a standardized process and format for the submission of clinical and economic evidence to support consideration of formulary listing in private health plans.⁶

In 2000, the Academy of Managed Care Pharmacy (AMCP), a national professional society of pharmacists in managed care environments, adapted, revised, and then endorsed guidelines called the AMCP Format for Formulary Submission (hereafter, the Format), and began actively encouraging health plans nationwide to implement them.⁷ The Format urges health plans to request formally that drug companies present a standardized dossier, which contains detailed information not only on the drug's effectiveness and safety for indications approved by the US Food and Drug Administration (FDA), but also on off-label indications and on the drug's economic value relative to alternative therapies. The Format also recommends that formulary committees establish a process for the transparent and efficient communication between drug companies and health plans (eg, sending an unsolicited request letter to drug firms 6 months before the product is placed on the pharmacy and therapeutics [P&T] committee agenda for a decision).

Many health plans, PBMs, and some hospitals have adopted the Format or a Format-like process.² Reports have indicated that most of these organizations use the clinical sections of the dossier and that some are beginning to consider meaningful cost-effectiveness analyses when these analyses are provided.^8,9 Little is known, however, about the information being submitted. Our objective was to evaluate the quality and completeness of clinical and economic information contained in dossiers adhering to the AMCP Format that were submitted to a health plan.

DATA AND METHODS

We audited all dossiers submitted to the Premera Blue Cross Health Plan in Mountlake Terrace, Washington (enrollment 1.6 million) between January 2002 and Septem-ber 2005. Each dossier audit was conducted by a physician researcher with graduate training in biostatistics, decision analysis, and cost-effectiveness analysis. Complete audits required approximately 4 hours per dossier. Dossiers were analyzed in either paper or password-protected CD-ROM form as submitted by pharmaceutical manufacturers to Premera.

We developed a data collection form with more than 80 variables. The form was pilot-tested several times for clarity. The form included numerous items related to the characteristics of clinical studies used as evidence to support labeled indications (consistent with the FDA's regulatory approval) and off-label claims in dossiers, and variables related to the quality and transparency of economic analyses.

Clinical studies included published or unpublished abstracts, posters, book chapters or sections, edited summaries, and full-text reprints of articles related to therapeutic uses of pharmaceutical products. We limited the detailed clinical analyses to the subsample of dossiers submitted in 2003 because of the large number of clinical studies included in dossiers (almost 400 studies in 2003 alone). We collected data on variables such as the research design used in the supporting studies, the sample size in those studies, and whether the study was published in a peer-reviewed journal.

We defined economic analyses to include research studies or mathematical models that combined clinical and economic data to estimate the economic value of a drug for an indication. We excluded budget impact models, which examine the impact of a drug on a health plan's budget rather than the economic value of a drug. We investigated the extent to which economic analyses adhered to key recommendations for good practices in the field.¹⁰

We hypothesized that clinical studies included or referenced in dossiers to support FDA-approved indications would be more extensive and have more rigorous study designs than studies supporting off-label claims. We also hypothesized that economic analyses in dossiers to support higher-cost and more innovative products would be of higher quality, because they would attract more resources and attention from drug companies seeking favorable formulary listing. We defined "high cost" as a cost per 30-day treatment higher than $1000 (based on the drug's price for that dossier's year).¹¹ The value $1000 was chosen arbitrarily as a round number that would reflect particularly expensive products (eg, biologics).

An "innovative" drug was defined by our researchers as a drug in a new therapeutic class or a drug with a clear-cut, relevant therapeutic advantage compared with competitors in its class (as opposed to a "me-too" drug). We considered therapeutic edge based on our judgments about clinical, quality-of-life, safety, or dosing advantages. "Me-too" products were judged to have no substantial advantages on these dimensions. We hypothesized that the quality of economic analyses would improve over time, as drug industry and health plan officials gained expertise. We examined whether adherence to various quality indicators improved over the years covered by the study period time using logistic regression analyses with adherence as the dependent variable, adjusting for the presence of high-cost and innovative products.

All researchers involved in data collection signed confidentiality agreements before being granted access to dossiers. In addition, signature of confidentiality agreements compliant with Health Insurance Portability and Accountability Act of 1996 guidelines was required before the data collector was allowed to enter Premera's premises. No data are reported on specific manufacturers or brands.

RESULTS

P

P

P

P

P

P

P

Table 1 shows the characteristics of 396 clinical studies contained in 35 dossiers submitted to Premera in 2003. Considerably more studies were submitted to support FDA-approved indications than off-label indications (314 vs 82). Proportionately more studies supporting on-label claims than off-label claims had random treatment allocation (50% vs 33%; = .01), were double-blinded (43% vs 27%; = .01), or were meta-analyses (8% vs 1%; = .02). They also contained many more full-text reprints (19% vs 6%; = .00) and larger sample sizes (492 vs 280; = .00); these differences were significant. In contrast, more of the studies supporting off-label claims were open-label clinical trials (27% vs 16%; = .02) or uncontrolled clinical trials (24% vs 6%; = .00).

Between January 2002 and September 2005, 115 dossiers were submitted to Premera, of which 53 dossiers (46%) contained economic data. The 53 economic-content analyses represented 28 drug manufacturers. These dossiers included 106 separate economic analyses (some dossiers contained more than 1 economic analysis, each pertaining to different clinical indications or including different types of studies). Within the 106 economic analyses, there was notable variation in the methods used (Table 2). For example, 82% stated the research question and 77% stated the primary outcome measure in the evaluation. Relatively few analyses reported sensitivity analyses (43%), discussed all relevant comparators (37%), clearly stated all assumptions (20%), accompanied conclusions with appropriate and specific caveats (18%), reported productivity changes (13%), or reported that comparators might be superior given changes in assumptions (8%).

Methods for conducting sensitivity analyses also varied considerably (Table 3). Although 63% included a statement about the range over which parameters were varied, less than half varied the cost of the drug (44%) or the direct medical costs (39%), and few varied the time horizon (7%).

P

P

P

P

P

P

P

P

Adherence to methodologic standards was generally higher in economic analyses of high-cost products (Table 4). Analyses of high-cost products were more likely to discuss all relevant comparators (75% vs 32%; = .01), state the form of economic evaluation used (92% vs 54%; = .01), provide details about the analysis/model design (100% vs 65%; = .02), and clearly state the model assumptions (42% vs 17%; = .06). Economic analyses of innovative products had better compliance to standards than economic analyses of me-too products, for example, in discussing all relevant comparators (81% vs 29%; = .00), stating the form of economic evaluation used (88% vs 53%; = .01), clearly stating all assumptions (44% vs 16%; = .02), and reporting the use of sensitivity analyses (75% vs 38%; = .01).

We found no evidence of improvement in the quality of economic analyses in dossiers over time. Of the 23 items listed in Tables 2 and 3, only 2 (statement of ranges over which parameters were varied in sensitivity analyses and cost of drugs varied in sensitivity analyses) showed significant improvement over the study period after adjusting for the presence of high-cost and innovative drugs. For 7 of the items, the percentage of analyses adhering to good methodologic practices showed significant declines over time, including statement of the viewpoint of the analysis, statement of the rationale behind the choice of comparators, and report of sensitivity analyses being performed (data not shown).

DISCUSSION

The advent of formulary submission guidelines, which call on pharmaceutical companies to generate and disseminate evidence of product value to health plans, represents a potentially powerful shift for consumers and producers of evidence. For formulary committees, guidelines mean more formal internal processes for judging whether a new drug is worthwhile. For pharmaceutical companies, the guidelines mean assembling and communicating standardized packages of evidence, including data supporting off-label claims and evidence of their product's value relative to alternative therapies.

One might expect that formulary guidelines would lead to increased efficiencies for the system and better health for patients. However, these expectations are based on an assumption that dossiers contain useful information. The findings here shed some new light on the issue and raise some concerns.

The clinical data reveal the type and breadth of the many clinical studies that are being submitted to support products. The data demonstrate that pharmaceutical and biotechnolo-gy companies use dossiers as a channel to support off-label claims with pharmacoeconomic evidence. Because such claims are unapproved, they cannot be supported by traditional marketing funds. The studies included to support these indications are generally less extensive and of lower quality than studies included to support FDA-approved indications. Although this finding is not surprising (by definition, on-label claims are required by the FDA to have substantiating evidence in the form of well-controlled trials), the data do raise questions about the extent to which plans may be basing decisions for off-label drugs on a relatively weak evidentiary base.

Only half of the dossiers we reviewed contained economic analyses. The methods used in the economic analyses that were included varied substantially, and frequently did not adhere to recommend protocols for the field. Many analyses lacked basic elements, such as stating the form of economic evaluation, stating the perspective of the study, or including sensitivity analyses, despite procedurally explicit guidance from the Format.

Why the low level of compliance? In part it may simply reflect the nature of the field: researchers continue to detect variations in methods in published economic evaluations as well as failure to meet standards.¹²

The poor quality of economic analyses in dossiers may also reflect the fact that drug companies are still climbing the learning curve in conducting analyses and communicating such information. That would be somewhat surprising, however, because pharmacoeconomic guidelines are not new–Australia, for example, has had them in place since 1992. Moreover, almost all large and many small pharmaceutical companies maintain internal health outcomes or economic departments, and all can contract out for the work to many research organizations with expertise.¹³ Still, it is likely that some drug companies, particularly smaller ones, continue to develop expertise in assembling dossiers for formulary submission. Hill et al,¹⁴ who analyzed economic analyses in submissions to the Australian reimbursement authorities, suggested that there is a long learning curve even where such submissions are mandated. It is also possible that dossiers are developed by marketing departments (rather than health economics and outcomes research departments), which lack expertise in conducting economic analysis or have different aims for its use.

The data also indicate that economic analyses frequently contain inappropriate comparators and overly optimistic assumptions, and rarely mention caveats to their conclusions. Perhaps that, too, is not unexpected: pharmaceutical companies clearly endeavor to put products in the best light possible. However, it suggests that drug companies may have low regard for health plans' ability to distinguish good studies from poor ones. Moreover, poor-quality data likely contribute to ongoing suspicion about the usefulness of economic models.

We also did not detect improvement in dossier quality over time, which could reflect a lack of learning among companies or possibly an increase in the number of smaller, inexperienced companies submitting dossiers. It also suggests the need for health plans to provide more feedback to drug companies about the quality of dossiers.

Our study did contain some positive findings: namely, that dossiers devoted to higher-cost and innovative products contained higher-quality economic analyses, suggesting that pharmaceutical companies put more resources into dossiers when the stakes are higher.

Because dossiers are proactively requested by health plans, they are considered unsolicited requests by the FDA and thus occupy a safe harbor against usual regulatory oversight by the agency against potentially false and misleading claims.¹⁵ That is, for unsolicited requests, the FDA permits drug companies to communicate information such as economic analyses and data supporting off-label claims, even though the information extends beyond the FDA-approved indications for drugs in question. The AMCP Format has always been premised on a belief that formulary committees are capable of judging for themselves the totality of evidence supporting drugs, whether or not the FDA has sanctioned the data. Our results suggest that formulary committees must be vigilant and cautious about the information they are receiving in dossiers, given this lack of FDA oversight.

It is important to note a few limitations in our analysis. First, our study focused on a single health plan, and it is not clear whether our findings are generalizable to other plans, although in all likelihood drug companies use similar dossiers across plans. Indeed, the Premera Health Plan has been a leader in dossier review, and, if anything, one might expect dossier quality to be lower in other plans. Still, it will be important for researchers to audit dossiers submitted to other plans to draw more definitive conclusions. Second, some of the items we analyzed involve subjective judgments, and other reviewers may have come to different conclusions. Third, despite AMCP Format guidance about disclosures, it was not clear from many of the dossiers whether economic analyses were conducted by drug company employees or were contracted out to private research organizations.

In the future, analysts should continue to monitor the data in dossiers. It would be useful to examine the effectiveness and efficiency of the AMCP Format process. For example, of the dossiers requested from manufacturers, how many are actually submitted? How much time elapses between the request for a dossier and the P&T committee meeting? It also will be useful to explore the relationship between dossier submissions and actual formulary placement: a recent study found that receipt of a dossier itself did not appear to influence the likelihood of a product attaining favorable formulary status.¹⁶

The impetus for future improvement in dossiers will ideally come from health plans themselves. Plan managers' insistence on high-quality submissions would create pressure on drug companies to provide better analyses. Further training and education of formulary committee personnel also would help. One problem nationwide may be that health plans do not possess the expertise necessary to judge the information in dossiers, particularly evidence contained in the economic models featured prominently in the guidelines. In the future, managed care plans themselves could educate formulary committee members about the potential value and how to make independent assessments of the contents of the dossiers that conform to the Format. The Foundation of Managed Care Pharmacy, which oversees research, educational, and other activities for the AMCP, has undertaken a series of initiatives to educate pharmacists, pharmaceutical company executives, and other interested professionals about the guidelines. It will be important to continue to monitor the progress of the field in the years ahead.

We acknowledge the Academy of Managed Care Pharmacy for support during early phases of this project. We are grateful to Alison Timm, BS, for research assistance.

Author Affiliations: Harvard School of Public Health, Boston, Mass (FC); Pharmaceutical Outcomes Research and Policy Program, University of Washington, Seattle, Wash (SDS); Center for the Evaluation of Value and Risk in Health, Institute for Clinical Research and Health Policy Studies, Tufts-New England Medical Center, Boston, Mass (JAP, KB, PJN); Division of Paediatric Orthopaedic Surgery, BC Children’s Hospital, The University of British Columbia, Vancouver, Canada (CAB); and Premera Blue Cross, Mountlake Terrace, Wash (JW).

Funding Sources: Funding was provided by the Agency for Healthcare Research and Quality under grant HHSP233200400483A. Carmen A. Brauer also was supported by a grant from the Alberta Heritage Foundation for Medical Research. Peter J. Neumann, ScD, and Sean D. Sullivan, PhD, had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the analysis.

Address correspondence to: Peter J. Neumann, ScD, Tufts-New England Medical Center, 750 Washington St, #063, Boston, MA 02111. E-mail: pneumann@tufts-nemc.org.

Author Disclosure: The authors (FC, SDS, JAP, CAB, KB, JW, PJN) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter discussed in this manuscript.

Authorship Information: Concept and design (FC, SDS, JW, PJN); acquisition of data (FC, SDS, CAB, JW); analysis and interpretation of data (FC, SDS, JAP, CAB, KB, PJN); drafting of the manuscript (SDS, JAP, KB, PJN); critical revision of the manuscript for important intellectual content (SDS, JAP, CAB, KB, JW, PJN); statistical analysis (FC, SDS, KB); provision of study materials or patients (JW); obtaining funding (FC, SDS, PJN); administrative, technical, or logistic support (JAP, JW); supervision (PJN).

1. Sullivan SD, Lyles A, Luce B, Gricar J. AMCP guidance for submission of clinical and economic evaluation data to support formulary listing in U.S. health plans and pharmacy benefits management organizations. J Manag Care Pharm. 2001;7:272-282.

2. Neumann PJ. Evidence-based and value-based formulary guidelines. Health Aff (Millwood). 2004;23:124-134.

3. Simon GE, Psaty BM, Hrachovec JB, Mora M. Principles for evidence-based drug formulary policy. J Gen Intern Med. 2005;21:964-968.

4. Luce BR, Lyles CA, Rentz AM.The view from managed care pharmacy. Health Aff (Millwood). 1996;15:168-176.

5. Navarro RP, Blackburn SS. Pharmacy benefit management companies. In: Navarro RP, ed. Managed Care Pharmacy Practice. Gaithers-burg, Md: Aspen Publications; 1999:221-242.

6. Mather DB, Sullivan SD, Augenstein D, et al. Incorporating clinical outcomes into drug formulary decisions: a practical approach. Am J Manag Care. 1999;5:277-285.

7. Academy of Managed Care Pharmacy.The AMCP Format for Formulary Submissions, Version 2.1. April 2005. http://www.fmcpnet.org/data/resource/Format~version2_1~Final_Final.pdf. Accessed September 9, 2006.

8. Cross M. Formulary submission process catches on … slowly. Manag Care. 2002;11:32-36.

9. Watkins JB, Minshall ME, Sullivan SD. Application of economic analyses in US managed care formulary decisions: a private payer’s experience. J Manag Care Pharm. 2006;12:726-735.

10. Gold MR, Siegel JE, Russell LB, Weinstein MC, eds. Cost-effectiveness in Health and Medicine. Oxford, England: Oxford University Press; 1996.

11. Medical Letter, Inc.The Medical Letter. http://www.medletter.com. Accessed December 31, 2003.

12. Neumann PJ, Greenberg D, Olchanski NV, Stone PW, Rosen AB. Growth and quality of the cost-utility literature, 1976-2001. Value Health. 2005;8:3-9.

13. DiMasi JA, Caglarcan E, Wood-Armny MW. Emerging role of phar-macoeconomics in the research and development decision-making process. Pharmacoeconomics. 2001;19:753-766.

14. Hill SR, Mitchell AS, Henry DA. Problems with the interpretation of pharmacoeconomic analyses: a review of submissions to the Australian Pharmaceutical Benefits Scheme. JAMA. 2000;283: 2116-2121.

15. Neumann PJ, Claxton K, Weinstein MC. The FDA’s regulation of health economic information. Health Aff (Millwood). 2000;19: 129-137.

16. Spooner JJ, Gandhi PK, Connelly SB. AMCP Format dossier requests: manufacturer response and formulary implications for one large health plan. J Manag Care Pharm. 2007;13:37-43.

Download PDF: Quality of Clinical and Economic Evidence in Dossier Formulary Submissions