Balancing Risks and Opportunities of Utilizing AI in Health Data: Carrie Kozlowski

In this interview, chief operating officer and cofounder of Upfront Healthcare, Carrie Kozlowski, OT, MBA, discusses the integration of artificial intelligence (AI) in health care, emphasizing both opportunities and risks. AI is being used in disease detection and scheduling, but it can initially increase costs and resource burden. She also explains key barriers, including model collapse and communication risks due to health literacy and cultural biases.

This transcript has been lightly edited for clarity.

Transcript

How do you see the role of AI in health data impacting the future of health equity as the trend evolves?

I would say I'm fairly conservative in my personal opinion and some of the folks that I'm talking to. When it comes to linking AI and equity, I think there's a lot of opportunity, and I think there's also some risk, which is where I generally take that more conservative approach. When we talk to a lot of the health systems that are starting to use AI in some areas, a lot of it is in detection of disease, like sepsis, for example, and some scheduling efficiencies. What I'm hearing from most of them right now is the overall cost is actually higher. The resourcing is higher. The burden is higher because they're putting so much oversight into managing it, which, in a way, maybe isn't the initial goal, but long-term feels like we're putting the right effort around making sure that we're thoughtfully approaching how we might leverage AI going forward. So, I think I'm optimistic, and I think that the approach when I talk to a lot of folks that are starting to use AI to think about opportunities across the health care system, I feel like people are taking an intentional approach that hopefully will help to hedge out any risk that can be associated with there being unintentional bias or mistrained models. And those kinds of things I think we're all generally concerned about when we have the conversation of equity in health care.

What are some barriers or challenges to utilizing AI efficiently in health care data?

I think cost is one. I think if there's an error, the cost is high, just from, like, a human-quality experience, etc. I also think there are some initial conversations that we've had—again, with different health systems in the work that they're doing—around the concern of model collapse as well. So when we think about training these models, and who we're training the models on, and what we're trying to do in terms of just how we're communicating, I think there's a high risk of using AI right now in communication with patients in particular. Because, I think that when we think about not only just health literacy but we think about language, we think about cultural bias, we think about things like what motivates and influences a person's behavior, it's really not easy to train like a ChatGPT, or a model, to have the breadth of sensitivity. It takes a significant amount of human intervention to get there.

I think where we're starting in some of the detection opportunities with significant clinical oversight makes sense to me. We should probably be further away than maybe people want to be when it comes to using AI in terms of engagement and communications, in order to kind of avoid developing greater mistrust. That's always my personal passion; when we lose trust, it's really hard to gain it back. And when we lose trust, we negatively impact health outcomes. So anything that we do that risks that, to me, is the thing we have to be really thoughtful and sensitive towards.