- Study based on 157,000 inpatient cases proposes an AI evaluation framework reflecting real-world clinical environments
- Demonstrates the importance of evaluating clinical applicability by considering temporal performance stability and alert burden beyond predictive accuracy

[July 28, 2026] AITRICS, a medical artificial intelligence(AI) company, announced that its research on a deep learning model for predicting the risk of Acute Kidney Injury (AKI) has been published in npj Digital Medicine, an internationally renowned journal in the field of digital healthcare.

Published by Nature Portfolio, npj Digital Medicine is a prestigious international journal that features cutting-edge research in medical AI and digital health.

The study developed multiple AI models for predicting the risk of AKI and compared their performance in a simulated clinical environment that reflects continuous patient data streams, with the aim of evaluating their applicability in real-world clinical practice.

AKI is one of the most common complications among hospitalized patients and is associated with high morbidity and mortality, making early prediction and timely intervention critical. Although AI-based early prediction technologies have been actively studied, most existing models have been evaluated primarily using single-point performance metrics such as the Area Under the Receiver Operating Characteristic Curve (AUROC), limiting their ability to accurately reflect the continuous monitoring environment of actual clinical settings.

To address this limitation, the AITRICS research team developed three deep learning models and two conventional machine learning models using data from 157,323 hospitalized patients across three medical institutions in Korea and overseas. The models were then evaluated in a simulated hospital environment designed to replicate continuous patient monitoring.

The results showed that the deep learning models outperformed the conventional machine learning models in predictive performance. More importantly, the study found that the model achieving the highest predictive accuracy at a single time point was not necessarily the one that performed best in real-world clinical practice.

Instead, the researchers demonstrated that evaluating models based on both temporal prediction stability and alert burden enabled the identification of models that are more suitable for practical clinical deployment.

The findings highlight the need to move beyond accuracy-focused evaluation metrics when assessing medical AI systems and instead consider long-term usability and operational burden in real clinical environments. By incorporating both temporal model stability and alert burden into the evaluation framework, the study presents a more clinically relevant approach to assessing AI models for healthcare applications.

Kyunghyun Lee, Researcher at AITRICS, said, "This study demonstrates that an AI model with the highest predictive accuracy is not necessarily the most useful in real clinical practice. To identify models that are truly deployable, it is essential to evaluate not only predictive accuracy but also the consistency of predictions over time and the volume of alerts clinicians are expected to manage. Moving forward, AITRICS will continue advancing deep learning technologies to develop AI models that deliver meaningful value in real-world healthcare settings."