"No one should blindly follow AI"

Prof. Jakob Nikolas Kather is convinced that artificial intelligence will profoundly transform medicine in the coming years. The physician and AI researcher at the Else Kröner Fresenius Center for Digital Health at TU Dresden discusses modern cancer diagnostics and chatbots in the exam room in this interview. He also explains where the technology reaches its limits and what happens when it gets it wrong. 

Professor Kather, when you weigh everything you know today about artificial intelligence, does this development give you more hope or more cause for concern?
More hope, without a doubt. Especially in medicine and biology, there are many unsolved problems and, at the same time, strong indications that AI can help us. It can enable breakthroughs in medical research and practice – and in some cases, it’s already doing so today. In my view, we shouldn’t slow down this development; instead, we should keep moving full steam ahead.

Are there already medical tasks today where you trust AI just as much – or even more – than a human?
Yes. On the one hand, there are specialized AI systems for specific tasks, such as detecting breast cancer in mammograms. Today, they can match the level of human experts and, in some cases, are already approved as medical devices. On the other hand, there are large language models and chatbots. These have now reached an expert level for many medical questions, even regarding rare diseases. That’s why many patients and doctors are already using them today.

You yourself are researching the use of AI in cancer. What can AI detect in medical images that doctors might not see?
In my research group, we have developed AI systems that analyze radiological images and tissue samples from colorectal cancer. Not only can they derive a diagnosis from these, but they can also predict which genetic alterations are present in the tumor and which medication might be suitable. To do this, the AI identifies features that are very difficult or nearly impossible for humans to see.

Has this already been incorporated into everyday clinical practice?
Our systems are not yet in routine use. Several companies are currently developing such approaches into medical devices. However, the process from research to widespread application takes years.

Will doctors in the future need to understand why an AI arrived at a particular result? Or is it enough to prove that it works?
A basic understanding of how the method works is important. But even more important is proof that it works reliably. A doctor doesn’t need to be able to build an X-ray machine either. What matters is that the device is safe, approved, and delivers valid results. AI is no exception.

You’ve studied how AI – and language models in particular – are already being used in everyday medical practice. Has the reality outpaced regulation?
Yes, you could say that. For specialized AI systems, such as those used for breast cancer detection, the traditional approval process generally works well. With chatbots, however, the current regulatory framework isn’t necessarily a good fit. I see a need for action here. After all, such a system wasn’t developed for just one specific application. The same chatbot can be asked for a recipe or for advice on a persistent cough. A great many people are already using such systems. You can’t just ban them. That’s why we need to educate patients and colleagues on when they can trust these systems and when they can’t. Because, of course, things can go wrong and cause harm.

Kather and his team are examining a tissue section. AI can detect tumor patterns in such images that are difficult for humans to see. © Anja Stübner/EKFZ

So you wouldn’t have a problem if your doctor said during a treatment: “Let me ask the chatbot about this”?
No. I’d want my doctor to be able to use such tools effectively. Before a doctor’s visit, I’d probably have already entered the topic into a language model myself. After all, we also accept it when a doctor says, “I’ll have to look that up in the textbook.” Today, a chatbot can take the place of the textbook. But of course, I wouldn’t want a doctor who simply types every question into a chatbot and then reads the answer aloud.

Have there been moments in your own work when you’ve been surprised by what AI is capable of these days?
Yes. We now rely heavily on AI as a tool, for example in software development. A year ago, we were still writing every line of code ourselves. Now, 99 percent of it comes from AI. We manage the process, review the results, and take responsibility for them. This has dramatically increased our scientific productivity. Projects that used to take six months, we can now sometimes complete in two weeks. AI has completely transformed our work.

Do you also encounter situations where the AI doesn’t do what you expect?
Of course. We certainly encounter situations where an AI system strays off course and does something we didn’t intend. That’s why we only allow such systems to operate in environments where they can’t cause any harm. And we make sure that patient data or other sensitive data never ends up in any cloud.

You’ve spoken in the past about how difficult it is in Germany to obtain data for medical AI research. Has the situation improved?
No. This remains a major problem in Germany. Many hospitals run outdated software. As a result, it’s difficult to consolidate data, and the necessary infrastructure is often lacking. On top of that, there are lengthy legal reviews. European and German legal frameworks sometimes overlap or are interpreted differently. We therefore need both a more modern infrastructure and simpler legal procedures.

Could we achieve better medical care if we were more pragmatic in our handling of health data?
Yes. This does not mean giving up data sovereignty. We should comply with all laws but take a more pragmatic approach. Other European countries show that this is possible. Spain, for example, has a significantly more digital healthcare system, even though European rules apply there as well. Germany also faces structural problems: many hospitals and health insurance providers, significant fragmentation, and numerous barriers to data flow. These structures also pose a major obstacle to the use of AI.

You and your team have investigated how large language models could be operated directly in hospitals. What is the advantage of such local AI?
In our current study, we ran large language models on our own servers. This ensures that the data never leaves the hospital. We were able to demonstrate that such systems can be put to good use in this way – for example, to automate complex processes in the hospital. I consider this a sensible strategy for the future. However, hospitals must invest in the necessary infrastructure to make this happen.

But that still doesn’t solve the problem that a language model can formulate a wrong answer just as convincingly as a correct one.
Exactly. That’s why we’ve developed a new method for quantifying the uncertainty of an answer. In doing so, we observed that when the AI system displayed a high level of uncertainty, its result was often incorrect. In practice, this could mean that we should only trust an answer if the system indicates a high level of certainty. If it is uncertain, we must be especially cautious. We have demonstrated for the first time that this problem can be addressed in this way.

What happens if a doctor trusts an AI and a patient is harmed as a result? Or if the doctor disagrees with the AI and that turns out to be the wrong decision?
Both scenarios are possible. Doctors already synthesize a great deal of information today: the patient’s wishes, clinical guidelines, test results, and their own experience. In the future, AI will be added as another voice in this process. Ultimately, the doctor must make a decision based on all this information and bear responsibility for it. No one should blindly follow AI.

What will AI be capable of in medicine ten years from now that might still seem like science fiction to us today?
I believe that AI can solve many of the problems we face in medicine today. Documentation and other administrative tasks could already be largely automated using today’s technology. To achieve this, we need to adapt our workflows accordingly. A second major issue is access to medical care. AI could support some aspects of basic primary care. Many people already use chatbots for medical questions today.

If you could draw a red line for AI in medicine, where would it be?
I don’t see a specific red line just for AI. The same fundamental principles should apply to it as to medicine today: patient autonomy, beneficence, non-maleficence, and justice. We don’t need to invent entirely new principles; rather, we need to ensure that existing ones are upheld even when AI is used.