Study Finds Moderate to Good Performance in Externally Validated HIV Risk Models

HIV risk assessment is a major global priority, and assessment tools can improve awareness and subsequent health-seeking actions, especially among men who have sex with men (MSM) at high risk.

While less than half of HIV infection risk assessment models have been externally validated, those that are validated demonstrate moderate-to-good discrimination ability for HIV infection prediction, according to research published in Archives of Sexual Behavior.

Specifically, the Amsterdam Score was shown to perform the best.

To come to these findings, researchers identified 18 HIV infection risk assessment models, including a total of 151,422 participants and 3643 HIV cases. Of these, 8 models were externally validated by at least 1 study: HIRI-MSM, Menza Score, SDET Score, Li Model, DHRS, Amsterdam Score, SexPro model, and UMRSS.

Clinical HIV prediction models combine multiple prognostic factors from an individual to estimate their risk of HIV. The 8 validated models each used between 3 and 12 predictor variables. Critical scoring variables included age, number of male sexual partners, unprotected receptive anal intercourse, recreational drug usage, and sexually transmitted infections (STIs).

The researchers found that all 8 externally validated models performed well in terms of discrimination, which quantifies how able the model is to correctly distinguish individuals with HIV from individuals without HIV. Discrimination is often quantified by the value of the area under the receiver operating characteristic curve (AUC), and the closer the AUC value is to 1, the better the model’s discrimination ability is.

In these 8 models, the pooled AUC ranged from 0.62 (95% CI, 0.51-0.73) for the SDET Score to 0.83 (95% CI, 0.48-0.99) for the Amsterdam Score. Only 10 of 28 included studies reported calibration performance, which limits summarizing pooled model calibration.

These findings raised concerns among the researchers.

With only 8 of 18 HIV infection risk assessment models being externally validated and because of general poorer performance of prediction models in new participants than in development studies, they said models should not be recommended for clinical use prior to external validation.

Additionally, predictors in some validation studies were not in the same format as they were in development studies, and some development studies had an inappropriately long time interval between predictor assessment and outcome determination, both of which implies high risk of bias.

Further, few studies reported the use of these HIV risk assessment models to identify candidates for targeted HIV prevention in the real-world clinical setting, which may be due to trade-offs in using these tools among MSM.

“Using these tools to reduce health service in these contexts risks missing some people who need it, even health providers trust their clinical judgment based on patients’ health history over model predictions, and MSM themselves also question the ability of a tool used in a single point to produce an accurate estimate of HIV risk,” the researchers said. “Some studies tried to use more sophisticated techniques, such as artificial intelligence (AI) with routine health record data to develop HIV prediction models to improve model predictive ability.”

They concluded that additional research is needed to validate different HIV risk models in different countries, and to determine how to apply these models in real-world health care practice.

Reference

Luo Q, Luo Y, Cui T, Li T. Performance of HIV infection prediction models in men who have sex with men: a systematic review and meta-analysis. Arch Sex Behav. Published online March 8, 2023. doi:10.1007/s10508-023-02574-x