Skip to main content
Download PDF
  • Research article
  • Open access
  • Published:

Using artificial intelligence to predict adverse outcomes in emergency department patients with hyperglycemic crises in real time

BMC Endocrine Disorders volume 23, Article number: 234 (2023) Cite this article

Abstract

Background

Hyperglycemic crises are associated with high morbidity and mortality. Previous studies have proposed methods to predict adverse outcomes of patients in hyperglycemic crises; however, artificial intelligence (AI) has never been used to predict adverse outcomes. We implemented an AI model integrated with the hospital information system (HIS) to clarify whether AI could predict adverse outcomes.

Methods

We included 2,666 patients with hyperglycemic crises from emergency departments (ED) between 2009 and 2018. The patients were randomized into a 70%/30% split for AI model training and testing. Twenty-two feature variables from the electronic medical records were collected. The performance of the multilayer perceptron (MLP), logistic regression, random forest, Light Gradient Boosting Machine (LightGBM), support vector machine (SVM), and K-nearest neighbor (KNN) algorithms was compared. We selected the best algorithm to construct an AI model to predict sepsis or septic shock, intensive care unit (ICU) admission, and all-cause mortality within 1 month. The outcomes between the non-AI and AI groups were compared after implementing the HIS and predicting the hyperglycemic crisis death (PHD) score.

Results

The MLP had the best performance in predicting the three adverse outcomes, compared with the random forest, logistic regression, SVM, KNN, and LightGBM models. The areas under the curves (AUCs) using the MLP model were 0.852 for sepsis or septic shock, 0.743 for ICU admission, and 0.796 for all-cause mortality. Furthermore, we integrated the AI predictive model with the HIS to assist decision making in real time. No significant differences in ICU admission or all-cause mortality were detected between the non-AI and AI groups. The AI model performed better than the PHD score for predicting all-cause mortality (AUC 0.796 vs. 0.693).

Conclusions

A real-time AI predictive model is a promising method for predicting adverse outcomes in ED patients with hyperglycemic crises. Further studies recruiting more patients are warranted.

Peer Review reports

Background

Diabetic ketoacidosis (DKA) and hyperosmolar hyperglycemic state (HHS) are severe acute complications of diabetes [1]. Precipitating factors include uncontrolled type 1 and 2 diabetes, infection, new-onset diabetes, pancreatitis, acute coronary syndrome, stroke, and medications [2, 3]. Visits to the emergency department (ED) for DKA and HHS have been increasing annually in the United States. In 2015, there were 3.1 visits for DKA and 2.9 visits for HHS per 10,000 adults with diabetes [1]. Although treatment includes hydration, insulin therapy, and electrolyte replacement, the mortality rate for hyperglycemic crises remains high [4, 5] and can also increase the risk for subsequent adverse cardiovascular events, end-stage renal disease, and long-term mortality [6,7,8]. Risk stratification (e.g., sepsis, intensive care unit [ICU] admission, and mortality) may improve outcomes in hyperglycemic crises [2, 3]. Prior studies identified mortality predictors, such as age, mental status, severe coexisting diseases, serum pH < 7.0, high insulin dose within the first 12 h, and serum glucose > 16.7 mmol after 12 h [4, 5, 8], but a clinical prediction rule may be more practical.

In 2013, the predicting the hyperglycemic crisis death (PHD) score was proposed as a tool to help ED physicians stratify the mortality risk and make decisions regarding patients in hyperglycemic crises [7]. It consists of six predictors and stratifies patients into low, intermediate, and high-risk groups. While the area under the curve (AUC) for the rule was 0.925 in the validation set, the PHD score was limited by a small derivation sample and manual calculation [7]. In recent years, artificial intelligence (AI) techniques have become a promising method to assist in medical decisions, and several AI predictions for adverse outcomes have been implemented in ED [6, 9,10,11]. However, no study has yet evaluated the feasibility and accuracy of AI predictions of adverse outcomes in ED patients with hyperglycemic crises in real time [12, 13]. Therefore, we carried out this study to clarify it.

Methods

Study design, setting, and participants

We established a multi-disciplinary team at the Chi Mei Medical Center (CMMC), including emergency physicians, data scientists, information engineers, nurse practitioners, and quality managers to implement big data and AI. Adults (age ≥ 20 years) with hyperglycemic crises who visited the EDs of three hospitals (CMMC, Chi Mei Liouying Hospital, and Chi Mei Chiali Hospital) between 2009 and 2018 were recruited (Fig. 1). The rationale that we used to select patients aged ≥ 20 years is that a criterion for an adult in Taiwan is “ ≥ 20 years”, and it has been adopted in many studies [6, 11]. The criteria for hyperglycemic crises were defined as the final diagnosis of DKA or HHS in the ED, using the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes 250.1 or 250.2 and ICD-10 codes E11.1 or E11.0. Patients who did not have a record of subsequent follow-up and those who visited the ED for multiple hyperglycemic crises were excluded.

Fig. 1
figure 1

Study flow chart. CMMC, Chi Mei Medical Center; ED, emergency department; AI, artificial intelligence; HIS, hospital information system

Full size image

Definition of feature variables

The 22 feature variables retained for analysis were age, sex, body mass index (BMI), vital signs at triage (Glasgow Coma Scale [GCS], systolic blood pressure, heart rate, respiratory rate, and body temperature), bedridden, nasogastric tube feeding, history of hypertension (ICD-9-CM: 401–405 or ICD-10: I10–I16), hyperlipidemia (ICD-9-CM: 272.0–272.5, 277.7 or ICD-10: E78.0–E78.5, E88.81), malignancy (ICD-9-CM: 140–208 or ICD-10: C00–C69), chronic kidney disease (ICD-9-CM: 585 or ICD-10: N18), and laboratory data, including blood urea nitrogen, serum creatinine, white blood cell count, hemoglobin, glucose, and high sensitive C-reactive protein (hs-CRP), as well as concomitant infection (ICD-9-CM: 001–139, 320–326, 390–392, 480–488, 540–543, 555–558, 566–567, 599.0, 601, 604, 614–616, 680–686, 730 or ICD-10: A00–B99, G00–G09, I00–I02, J09–J18, K35–K38, K50–K52, K61, K65, N39.0, N41, N45, N70–N77, L00–L08, M86, R65). The feature variables were suggested predictors of adverse outcomes in previous studies, and possible risk factors for adverse outcomes in clinical practice [7, 14,15,16,17]. History was pre-existing at the time of presentation as diagnosed by the physician in the electronic medical records (EMRs). Age was divided into four subgroups of 20–34, 35–49, 50–64, 64–74, and ≥ 75 years according to previous studies [6, 11, 18]. BMI was divided into four subgroups according to the Asian BMI levels: < 18.5, 18.5–22.9, 23–24.9, and ≥ 25 kg/m2 [19, 20].

Outcome measurements

We defined three adverse outcomes, including sepsis or septic shock < 1 month (ICD-9-CM: 038, 790.7 or ICD-10: A40–A41, R65, R7881), ICU admission < 1 month, and all-cause mortality < 1 month following the time of presentation in the ED. The general “ICU admission” criteria in the study hospital were unstable vital signs and the need for intensive monitoring and treatment. “All-cause mortality” was defined as a record of death certification or discharge against medical advice in a patient in critical condition in the EMRs. We defined “ < 1 month” for outcomes according to previous studies of hyperglycemic crises and AI [7, 11].

Ethical statement

This study was approved by the Institutional Review Board of the CMMC and was conducted according to the Declaration of Helsinki. Informed consent from the patients was waived because this study was retrospective and contained de-identified information, which did not affect the rights or welfare of the patients.

Data processing, comparison, and application

The study had two phases: pre- and post-implementation. The pre-implementation phase developed an AI predictive model and integrated it with the HIS. The post-implementation phase compared outcomes between the non-AI and AI groups. The feature of sex was transformed into 1 (male) or 0 (female). Missing or ambiguous data were defined by a team comprising emergency physicians, data scientists, information engineers, nurse practitioners, and quality managers. Data with missing feature variables were deleted or estimated with an average value. Second, we divided the data into training (70%) and test (30%) datasets according to previous studies [6, 11, 21]. There were fewer outcomes, particularly ICU admissions, which may have caused an imbalance in the data. Therefore, we used the synthetic minority over-sampling technique to improve the data imbalance in the training dataset [22]. Machine learning (ML) and deep learning (DL) are the two major fields of AI [23]. ML, including random forest, logistic regression, support vector machine (SVM), K-nearest neighbor (KNN), and Light Gradient Boosting Machine (LightGBM), is the ability that a computer system uses to automatically improve their function or to “learn” with continuous data [23]. DL, as the multilayer perceptron (MLP) in this study, has a more complex network of nodes between the inputs and outputs for solving complex problems more accurately [23]. Because the case number was small, we used MLP, a classical neural network method, to represent the DL method. The MLP has been adopted successfully in our studies [6, 9, 11, 24, 25]. We used fivefold cross validation technique to build all models. We compared the ML algorithms, including random forest, logistic regression, SVM, KNN, LightGBM, and MLP for accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F1, and AUC. Accuracy was defined as the fraction of cases that the model correctly predicted [26]. Sensitivity was the fraction of positive cases predicted as positive [26]. Specificity was the fraction of negative cases predicted as negative [26]. PPV was the fraction of true positive cases from all cases that the model predicted to be positive [26]. NPV was the fraction of negative cases from all cases that the model predicted to be negative [26]. F1 was the harmonic mean of PPV and sensitivity [26]. Accuracy, PPV, NPV, and F1 depend on the prevalence of adverse outcomes [26]. We used the AUC to determine the best model for further implementation [13,14,15] because the AUC considers the predictive performance of the positive and negative outcomes. An AUC of 0.5 suggests no discrimination, 0.7–0.8 suggests acceptable, 0.8–0.9 suggests excellent, and > 0.9 suggests outstanding [26]. The tuning parameters we used to refine our models are shown in Supplementary Table 1. We performed the DeLong test to assess overfitting of the training and test models and plotted the learning curves for our model (best model) [27]. The p-value of the DeLong test for the best model (MLP model) was > 0.05, indicating no significant difference between the training and test models. Therefore, no significant overfitting existed. Using the learning curve [28] (Supplementary Fig. 1), we observed no significant overfitting as the number of samples increased, with the training score (F1 score) curve gradually approaching and overlapping the testing score curve. Subsequently, we integrated the AI predictive model into the HIS, deployed it at the AI web service, and launched it for real-time decision-making assistance by ED physicians. To reveal the real-time prediction result, a physician simply needed to press the AI button set up in the HIS. We then conducted a retrospective impact study between December 1, 2019, and April 30, 2021, in which all ED patients with hyperglycemic crises were identified and divided into non-AI and AI groups to compare outcomes. The use of AI was an aid to decision-making and depended on the physician's discretion.

ML algorithms used in this study

MLP is an artificial neural network that maps input data to appropriate outputs using an input layer, hidden layer, and output layer, each connected by a synaptic weight matrix and with nonlinear activation functions and trained via backpropagation [29]. Its multiple layers and activation functions enable it to distinguish non-linearly separable data [29]. In a study predicting adverse outcomes from pneumonia, MLP had AUCs of 0.749, 0.792, and 0.802 for sepsis or septic shock, respiratory failure, and mortality, respectively [6].

Random forest is an efficient ensemble technique that contains multiple decision trees generated from combined optimization decision trees, useful for classification and regression, and preventing overfitting with high accuracy even for incomplete datasets [30]. Random forest has been widely used in AI medical studies for prediction [31], including a study of predicting outcomes in older ED patients with influenza, where their random forest model achieved an AUC of 0.840 for hospitalization, 0.765 for pneumonia, 0.857 for sepsis or septic shock, 0.885 for ICU admission, and 0.875 for in-hospital mortality [9].

Logistic regression is a statistical approach and supervised ML algorithm used for classification problems by mapping features to categorical targets and predicting the probability of a new case belonging to a target class [32]. In a recent study of predicting major adverse cardiac events in ED patients with chest pain, logistic regression was used to achieve AUCs of 0.868 for acute myocardial infarction at < 1 month and 0.716 for all-cause mortality at < 1 month [11].

LightGBM is a high-performing gradient boosting framework that utilizes tree-based learning algorithms and includes Gradient-based One-Side Sampling and Exclusive Feature Bundling methods for selective sampling and reduced dimensionality [33]. A study using LightGBM as an algorithm reported AUCs of 0.774 for sepsis or septic shock, 0.847 for respiratory failure, and 0.835 for mortality prediction [6].

SVM is a versatile algorithm that can address regression, binary, and multi-class classification problems by identifying a hyperplane that maximizes the distance between classes in the feature space [34]. In cases where the classes are not linearly separable, the kernel trick is used to project the feature vectors to a higher-dimensional space [34]. SVM is widely used in medicine, with a study reporting AUCs of 0.840 for hospitalization, 0.733 for pneumonia, 0.806 for sepsis or septic shock, 0.778 for ICU admission, and 0.762 for in-hospital mortality in older patients with influenza [9].

KNN is a non-parametric, supervised learning classifier that predicts the grouping of an individual data point using proximity to other data points [35]. A study using KNN to predict major adverse cardiac events in ED patients with chest pain reported AUCs for acute myocardial infarction at < 1 month and all-cause mortality at < 1 month of 0.865 and 0.969, respectively [11].

Results

A total of 2,666 ED patients with hyperglycemic crises were recruited into the study at the three hospitals between 2009 and 2018 (Table 1). Their mean age was 65.3 ± 16.9 years, and the percentage of females was 45.7%. The four age subgroups were 20–34 years (5.8%), 35–49 years (11.9%), 50–64 years (25.8%), 65–74 years (20.2%), and ≥ 75 years (36.3%). The mean BMI was 23.0 ± 4.8 kg/m2. There were 60.2% of bedridden patients and 8.0% of patients being fed by nasogastric tube. A history of hypertension (53.0%), hyperlipidemia (26.2%), cerebrovascular accident (29.8%), malignancy (14.2%), and chronic kidney disease (11.4%) were found. Concomitant infection was found in 46.8% of the patients. Within 1 month, 31.7% of patients had sepsis or septic shock, 6.0% required ICU admission, and 12.8% died. Missing data were assigned values according to decisions made at a multi-disciplinary team meeting (Supplementary Table 2).

Table 1 Characteristics of all ED patients with hyperglycemic crises in the three hospitals
Full size table

The MLP model outperformed other algorithms with AUCs of 0.852 for sepsis or septic shock, 0.743 for ICU admission, and 0.796 for all-cause mortality in the testing dataset (Table 2 and Supplementary Fig. 2) [36]. After a consensus was reached, MLP was chosen for AI implementation. SHapley Additive exPlanations (SHAP) values were used to identify feature associations and importance (Supplementary Fig. 3). A model was developed for predicting ICU admissions < 48 h with an AUC of 0.780 in the test dataset, outperforming other algorithms. A DeLong test was used to compare AUC values between algorithms (Supplementary Table 4).

Table 2 Comparison of performance among the random forest, logistic regression, SVM, KNN, LightGBM, and MLP algorithms for adverse outcomes in ED patients with hyperglycemic crises
Full size table

Meanwhile, it is crucial for models to be well calibrated when used in real-world patient-level scenarios, as inaccuracies in individual predicted probabilities may lead to inappropriate decisions by physicians. To assess the calibration of our models, we generated calibration plots that depict the distribution of observed and predicted case states across absolute probability subgroups or bins. A calibration curve that closely aligns with the diagonal indicates a higher level of calibration for the corresponding model. Our evaluation, as demonstrated in Figs. 2, 3 and 4, reveals that the calibration guideline for all MLP models was not significantly violated. Therefore, these models can be considered suitable for implementing a prediction system.

Fig. 2
figure 2

Calibration plot: predicted and true probability results for sepsis and septic shock

Full size image
Fig. 3
figure 3

Calibration plot: predicted and true probability results for ICU admission. ICU, intensive care unit

Full size image
Fig. 4
figure 4

Calibration plot: predicted and true probability results for all-cause mortality

Full size image

The HIS of the ED had an AI button (Supplementary Fig. 4) that displayed the prediction within 1 s after being pressed by the clinician (Supplementary Fig. 5). AI predictions were personalized and presented as percentages, with risks categorized as low (0%–33%), moderate (33%–66%), or high (66%–100%).

Patients with hyperglycemic crises (n = 271) between December 1, 2019 and April 30, 2021 were identified to compare the adverse outcomes between the non-AI and AI groups (Table 3). The AI group tended to have a lower ICU admission rate (11.1% vs. 19.8%) and all-cause mortality (11.1% vs. 15.0%) than the non-AI group; however, the differences were not significant. In addition, we used the same data to validate the PHD score and found that the AI model using MLP for predicting all-cause mortality performed better than the PHD score (Table 4).

Table 3 Comparison of clinical characteristics and adverse outcomes between the non-AI and AI groups in new ED patients with hyperglycemic crises between December 1, 2019 and April 30, 2021
Full size table
Table 4 Comparison of predicting the ICU admission and all-cause mortality rates between the AI model using MLP and the PHD score
Full size table

Discussion

We developed an AI prediction model using MLP for ED patients with hyperglycemic crises that provided real-time decision-making assistance to physicians. The AUC of the model was 0.852 for sepsis or septic shock, 0.743 for ICU admissions, and 0.796 for all-cause mortality within 1 month. The impact study showed that the AI group tended to have lower ICU admissions and all-cause mortality than the non-AI group, but the differences were not significant.

Clinical decision rules (CDRs) like the PHD score can help with critical decision-making regarding patient health [37,38,39], but they have limitations. CDRs are designed to simplify complexity, and they should be externally validated in diverse settings to ensure applicability [37, 38]. They may not be applicable to a user’s clinical setting or a targeted population, and they require manual calculation, which can be inconvenient in a busy ED [37, 38].

AI is a breakthrough in healthcare that has the potential to improve the system. MLP, a significant model in the artificial neural network, is preferred for solving nonlinear problems. It consists of the input, hidden, and output layers and mimics the human brain [40]. Unlike other computerized tools, AI learns, tests, and generates autonomously by analyzing big data [23, 41]. AI offers various opportunities for ED care, including image interpretation, predicting patient outcomes, monitoring vital signs, reducing documentation burden with natural-language-processing, home monitoring systems, and outbreak prediction tools [41,42,43,44].

We integrated an AI prediction model into the HIS, which overcame barriers between AI research and clinical practice, but there were implementation barriers. Hospital policies and cooperation from the hospital information department were crucial for successful implementation. Additionally, incorporating AI into the HIS was technically challenging and may require overhauling existing information technology systems. Finally, concerns regarding malpractice, accuracy, and physician replacement by AI may affect physician acceptance of AI implementation [23].

Based on the same dataset, the AUC of all-cause mortality of the best model in our study was superior to that of the PHD score (0.796 vs. 0.693), suggesting that our AI model may be a better tool for predicting adverse outcomes in ED patients with hyperglycemic crises than the conventional PHD score.

We used the AUC, a recognized and comprehensive metric, to select the algorithm for our study [6, 9,10,11]. A major advantage of AUC is that it measures the ranking of predictions, rather than their absolute values, and is classification-threshold-invariant [45]. However, the choice of metric depends on the study’s aim [10]. For instance, if high sensitivity to predict sepsis or septic shock was the aim, we may have chosen LightGBM since it had the best sensitivity of 0.803 in our study.

We used the SHAP value, a new method to increase the transparency of AI prediction, to identify the importance of each feature variable for determining adverse outcomes [36]. In the SHAP summary plot, red and blue indicate high and low associations, respectively, between the feature variable and an adverse outcome [36].

The study implemented a real-time AI prediction model integrated in the HIS to predict adverse outcomes in ED patients with hyperglycemic crises, which was a major strength. However, there were some limitations. The AUC for predicting ICU admission was lower than that for sepsis or septic shock and all-cause mortality, possibly due to the subjective nature of ICU admission decision-making [46]. The results of the DeLong test (Table 2) indicate that, except for MLP models, there is a potential for overfitting in most models, which should be approached with caution. It is worth considering increasing the size of the data to potentially mitigate this issue and improve the performance of the models. The “black box” phenomenon remained a problem [23], but using the SHAP value may help increase transparency [36]. The impact of AI prediction on clinical practice was not fully evaluated, and further studies are needed. The AI prediction model may not be generalizable to other hospitals, and ethical and legislative issues may arise from using AI predictions as a tool. There were also limitations in the ICD measures [47, 48]. Lastly, the sample size of new patients was small, warranting more patients to be recruited to delineate this issue.

Conclusions

We developed the first AI model to predict adverse outcomes in ED patients with hyperglycemic crises and integrated it into the HIS to provide real-time decision assistance. ED physicians obtained a second opinion from big data in real time using AI, which helped them in their decision making. The impact study showed no significant difference in the ICU admission or all-cause mortality rates between the non-AI and AI groups; however, further studies recruiting more patients will clarify this issue.

Availability of data and materials

The datasets analyzed for this study are available from the corresponding author upon reasonable request.

Abbreviations

DKA:

Diabetic ketoacidosis

HHS:

Hyperosmolar hyperglycemic state

ED:

Emergency department

ICU:

Intensive care unit

PHD:

Predicting the hyperglycemic crisis death

AUC:

Area under the curve

AI:

Artificial intelligence

CMMC:

Chi Mei Medical Center

ICD-9-CM:

International Classification of Diseases, Ninth Revision, Clinical Modification

BMI:

Body mass index

GCS:

Glasgow Coma Scale

hs-CRP:

High-sensitivity C-reactive protein

EMRs:

Electronic medical records

HIS:

Hospital information system

ML:

Machine learning

DL:

Deep learning

SVM:

Support vector machine

KNN:

K-nearest neighbor

LightGBM:

Light Gradient Boosting Machine

MLP:

Multilayer perceptron

PPV:

Positive predictive value

NPV:

Negative predictive value

SHAP:

SHapley Additive exPlanations

CDRs:

Clinical decision rules

References

  1. Benoit SR, Hora I, Pasquel FJ, Gregg EW, Albright AL, Imperatore G. Trends in Emergency Department Visits and Inpatient Admissions for Hyperglycemic Crises in Adults With Diabetes in the U.S., 2006–2015. Diabetes Care. 2020;43(5):1057–64.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  2. Van Ness-Otunnu R, Hack JB. Hyperglycemic crisis. J Emerg Med. 2013;45(5):797–805.

    Article  PubMed  Google Scholar 

  3. Kitabchi AE, Umpierrez GE, Miles JM, Fisher JN. Hyperglycemic crises in adult patients with diabetes. Diabetes Care. 2009;32(7):1335–43.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  4. Chung ST, Perue GG, Johnson A, Younger N, Hoo CS, Pascoe RW, Boyne MS. Predictors of hyperglycaemic crises and their associated mortality in Jamaica. Diabetes Res Clin Pract. 2006;73(2):184–90.

    Article  PubMed  Google Scholar 

  5. Efstathiou SP, Tsiakou AG, Tsioulos DI, Zacharos ID, Mitromaras AG, Mastorantonakis SE, Panagiotou TN, Mountokalakis TD. A mortality prediction model in diabetic ketoacidosis. Clin Endocrinol (Oxf). 2002;57(5):595–601.

    Article  PubMed  Google Scholar 

  6. Chen YM, Kao Y, Hsu CC, Chen CJ, Ma YS, Shen YT, Liu TL, Hsu SL, Lin HJ, Wang JJ, et al. Real-time interactive artificial intelligence of things-based prediction for adverse outcomes in adult patients with pneumonia in the emergency department. Acad Emerg Med. 2021;28(11):1277–85.

    Article  PubMed  Google Scholar 

  7. Huang CC, Kuo SC, Chien TW, Lin HJ, Guo HR, Chen WL, Chen JH, Chang SH, Su SB. Predicting the hyperglycemic crisis death (PHD) score: a new decision rule for emergency and critical care. Am J Emerg Med. 2013;31(5):830–4.

    Article  PubMed  Google Scholar 

  8. MacIsaac RJ, Lee LY, McNeil KJ, Tsalamandris C, Jerums G. Influence of age on the presentation and outcome of acidotic and hyperosmolar diabetic emergencies. Intern Med J. 2002;32(8):379–85.

    Article  PubMed  CAS  Google Scholar 

  9. Tan TH, Hsu CC, Chen CJ, Hsu SL, Liu TL, Lin HJ, Wang JJ, Liu CF, Huang CC. Predicting outcomes in older ED patients with influenza in real time using a big data-driven and machine learning approach to the hospital information system. BMC Geriatr. 2021;21(1):280.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Taylor RA, Pare JR, Venkatesh AK, Mowafi H, Melnick ER, Fleischman W, Hall MK. Prediction of In-hospital Mortality in Emergency Department Patients With Sepsis: A Local Big Data-Driven Machine Learning Approach. Acad Emerg Med. 2016;23(3):269–78.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Zhang PI, Hsu CC, Kao Y, Chen CJ, Kuo YW, Hsu SL, Liu TL, Lin HJ, Wang JJ, Liu CF, et al. Real-time AI prediction for major adverse cardiac events in emergency department patients with chest pain. Scand J Trauma Resusc Emerg Med. 2020;28(1):93.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Agrawal VSP, Sneha S. Hyperglycemia Prediction Using Machine Learning: A Probabilistic Approach. In: International Conference on Advances in Computing and Data Sciences. 2019. p. 304–12.

    Chapter  Google Scholar 

  13. Ramyea RPS, Keerthana K, Keerthana R, Kavivarman J. An Intellectual Supervised Machine Learning Algorithm for the Early Prediction of Hyperglycemia. In: 2021 Innovations in Power and Advanced Computing Technologies (i-PACT). 2021. p. 1–7.

    Google Scholar 

  14. Huang CC, Weng SF, Tsai KT, Chen PJ, Lin HJ, Wang JJ, Su SB, Chou W, Guo HR, Hsu CC. Long-term Mortality Risk After Hyperglycemic Crisis Episodes in Geriatric Patients With Diabetes: A National Population-Based Cohort Study. Diabetes Care. 2015;38(5):746–51.

    Article  PubMed  CAS  Google Scholar 

  15. Kao Y, Hsu CC, Weng SF, Lin HJ, Wang JJ, Su SB, Huang CC, Guo HR. Subsequent mortality after hyperglycemic crisis episode in the non-elderly: a national population-based cohort study. Endocrine. 2016;51(1):72–82.

    Article  PubMed  CAS  Google Scholar 

  16. Huang CC, Chou W, Lin HJ, Chen SC, Kuo SC, Chen WL, Chen JH, Wang HY, Guo HR. Cancer history, bandemia, and serum creatinine are independent mortality predictors in patients with infection-precipitated hyperglycemic crises. BMC Endocr Disord. 2013;13:23.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  17. Huang CC, Chien TW, Su SB, Guo HR, Chen WL, Chen JH, Chang SH, Lin HJ, Wang YF. Infection, absent tachycardia, cancer history, and severe coma are independent mortality predictors in geriatric patients with hyperglycemic crises. Diabetes Care. 2013;36(9):e151-152.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Kirkman MS, Briscoe VJ, Clark N, Florez H, Haas LB, Halter JB, Huang ES, Korytkowski MT, Munshi MN, Odegard PS, et al. Diabetes in older adults. Diabetes Care. 2012;35(12):2650–64.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Bae YJ, Shin SJ, Kang HT. Body mass index at baseline directly predicts new-onset diabetes and to a lesser extent incident cardio-cerebrovascular events, but has a J-shaped relationship to all-cause mortality. BMC Endocr Disord. 2022;22(1):123.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Shukohifar M, Mozafari Z, Rahmanian M, Mirzaei M. Performance of body mass index and body fat percentage in predicting metabolic syndrome risk factors in diabetic patients of Yazd. Iran BMC Endocr Disord. 2022;22(1):216.

    Article  PubMed  CAS  Google Scholar 

  21. Gholamy A, Kreinovich V, Kosheleva O. Why 70/30 or 80/20 relation between training and testing sets: A pedagogical explanation. 2018.

    Google Scholar 

  22. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic Minority Over-sampling Technique. J Artif Int Res. 2002;16:321–57.

    Google Scholar 

  23. Grant K, McParland A, Mehta S, Ackery AD. Artificial intelligence in emergency medicine: surmountable barriers with revolutionary potential. Ann Emerg Med. 2020;75(6):721–6.

  24. Li YY, Wang JJ, Huang SH, Kuo CL, Chen JY, Liu CF, Chu CC. Implementation of a machine learning application in preoperative risk assessment for hip repair surgery. BMC Anesthesiol. 2022;22(1):116.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  25. Liao KM, Ko SC, Liu CF, Cheng KC, Chen CM, Sung MI, Hsing SC, Chen CJ. Development of an interactive AI system for the optimal timing prediction of successful weaning from mechanical ventilation for patients in respiratory care centers. Diagnostics (Basel). 2022;12(4):975.

  26. Erickson BJ, Kitamura F. Magician’s Corner: 9. Performance Metrics for Machine Learning Models. Radiol Artif Intell. 2021;3(3):e200126.

    Article  PubMed  PubMed Central  Google Scholar 

  27. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837–45.

    Article  PubMed  CAS  Google Scholar 

  28. Anzanello MJ, Fogliatto FS. Learning curve models and applications: Literature review and research directions. Int J Ind Ergon. 2011;41(5):573–83.

    Article  Google Scholar 

  29. Pal SK, Mitra S. Multilayer perceptron, fuzzy sets, and classification. IEEE Trans Neural Netw. 1992;3(5):683–97.

    Article  PubMed  CAS  Google Scholar 

  30. Breiman L. Random Forests. Mach Learn. 2001;45:5–32.

    Article  Google Scholar 

  31. Tsai WC, Liu CF, Lin HJ, Hsu CC, Ma YS, Chen CJ, Huang CC, Chen CC. Design and implementation of a comprehensive AI dashboard for real-time prediction of adverse prognosis of ED patients. Healthcare (Basel). 2022;10(8):1498.

  32. Bisong E. Building Machine Learning and Deep Learning Models on Google Cloud Platform: Apress. 2019.

    Book  Google Scholar 

  33. Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, Ye Q, Liu TY. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In: 31st Conference on Neural Information Processing Systems (NIPS 2017). Long Beach. 2017.

  34. Schölkopf BS, AJ. Learning with Kernels - Support Vector Machines, Regularization, Optimization and Beyond. Cambridge, MA, USA: MIT Press; 2001.

    Google Scholar 

  35. Keller JM, Gray MR, Givens JA. A fuzzy K-nearest neighbor algorithm. IEEE Transactions on Systems, Man, and Cybernetics. 1985;SMC-15(4):580–5.

    Article  Google Scholar 

  36. Lundberg SML, S.I. A Unified Approach to Interpreting Model Predictions. In: Advances in neural information processing systems 30: 2017; 2017.

  37. Chung JY, Hsu CC, Chen JH, Chen WL, Lin HJ, Guo HR, Huang CC. Geriatric influenza death (GID) score: a new tool for predicting mortality in older people with influenza in the emergency department. Sci Rep. 2018;8(1):9312.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Green SM. When do clinical decision rules improve patient care? Ann Emerg Med. 2013;62(2):132–5.

    Article  PubMed  Google Scholar 

  39. Lim SH. Clinical decision rules in emergency care. Singapore Med J. 2018;59(4):169.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Singh P, Singh S, Pandi-Jain GS. Effective heart disease prediction system using data mining techniques. Int J Nanomed. 2018;13(T-NANO 2014 Abstracts):121–4.

    Article  Google Scholar 

  41. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44–56.

    Article  PubMed  CAS  Google Scholar 

  42. Berlyand Y, Raja AS, Dorner SC, Prabhakar AM, Sonis JD, Gottumukkala RV, Succi MD, Yun BJ. How artificial intelligence could transform emergency department operations. Am J Emerg Med. 2018;36(8):1515–7.

    Article  PubMed  Google Scholar 

  43. Crampton NH. Ambient virtual scribes: Mutuo Health’s AutoScribe as a case study of artificial intelligence-based technology. Healthc Manage Forum. 2020;33(1):34–8.

    Article  PubMed  Google Scholar 

  44. Ramesh AN, Kambhampati C, Monson JR, Drew PJ. Artificial intelligence in medicine. Ann R Coll Surg Engl. 2004;86(5):334–8.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  45. Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol. 2010;5(9):1315–6.

    Article  PubMed  Google Scholar 

  46. Nates JL, Nunnally M, Kleinpell R, Blosser S, Goldner J, Birriel B, Fowler CS, Byrum D, Miles WS, Bailey H, et al. ICU Admission, Discharge, and Triage Guidelines: A Framework to Enhance Clinical Operations, Development of Institutional Policies, and Further Research. Crit Care Med. 2016;44(8):1553–602.

    Article  PubMed  Google Scholar 

  47. Garvin JH, Redd A, Bolton D, Graham P, Roche D, Groeneveld P, Leecaster M, Shen S, Weiner MG. Exploration of ICD-9-CM coding of chronic disease within the Elixhauser Comorbidity Measure in patients with chronic heart failure. Perspect Health Inf Manag. 2013;10:1b.

    PubMed  PubMed Central  Google Scholar 

  48. Hsieh CY, Su CC, Shao SC, Sung SF, Lin SJ, Kao Yang YH, Lai EC. Taiwan’s National Health Insurance Research Database: past and future. Clin Epidemiol. 2019;11:349–58.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank Miss Yu-Shan Ma and Miss Yu-Ting Shen for their assistance with the statistics and algorithms. We thank Enago for the English revision.

Funding

This work was supported by grant CMFHR108124 from the Chi Mei Medical Center. The funder had no role in study design, data collection, and analysis, decision to publish, or preparation of the manuscript.

Author information

Author notes

  1. Chin-Chuan Hsu and Yuan Kao contributed equally to this work.

Authors and Affiliations

  1. Department of Emergency Medicine, Chi Mei Medical Center, 901 Zhonghua Road, Yongkang District, Tainan City, 710, Taiwan

    Chin-Chuan Hsu, Yuan Kao, Chien-Chin Hsu, Hung-Jung Lin & Chien-Cheng Huang

  2. Graduate Institute of Medical Sciences, College of Health Sciences, Chang Jung Christian University, Tainan, Taiwan

    Yuan Kao

  3. School of Medicine, College of Medicine, National Sun Yat-sen university, Kaohsiung, Taiwan

    Chien-Chin Hsu, Hung-Jung Lin & Chien-Cheng Huang

  4. Information Systems, Chi Mei Medical Center, Tainan, Taiwan

    Chia-Jung Chen & Tzu-Lan Liu

  5. Department of Nursing, Chi Mei Medical Center, Tainan, Taiwan

    Shu-Lien Hsu

  6. Department of Emergency Medicine, Taipei Medical University, Taipei, Taiwan

    Hung-Jung Lin

  7. Department of Anesthesiology, Chi Mei Medical Center, Tainan, Taiwan

    Jhi-Joung Wang

  8. Department of Anesthesiology, National Defense Medical Center, Taipei, Taiwan

    Jhi-Joung Wang

  9. Department of Medical Research, Chi Mei Medical Center, 901 Zhonghua Road, Yongkang District, Tainan City, 710, Taiwan

    Chung-Feng Liu

  10. Department of Emergency Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan

    Chien-Cheng Huang

  11. Department of Environmental and Occupational Health, College of Medicine, National Cheng Kung University, Tainan, Taiwan

    Chien-Cheng Huang

Authors
  1. Chin-Chuan Hsu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuan Kao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chien-Chin Hsu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chia-Jung Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shu-Lien Hsu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tzu-Lan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Hung-Jung Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jhi-Joung Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Chung-Feng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Chien-Cheng Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

CCH (first author), YK, CFL, and CCH (tenth author) designed and conceived this study. CJC and TLL performed the data processing, deployed the AI web service, integrated the HIS, tested the application, and launched the application in the HIS. CFL performed model training and testing and statistical analysis. CCH (third author), SLH, HJL, and JJW provided professional suggestions. All authors have read and approved the final manuscript.

Corresponding authors

Correspondence to Chung-Feng Liu or Chien-Cheng Huang.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Institutional Review Board of the Chi Mei Medical Center and was conducted according to the Declaration of Helsinki. Informed consent from the patients was waived because this study was retrospective and contained de-identified information, which did not affect the rights or welfare of the patients.

Consent for publication

Not applicable.

Competing interests

All authors deny any competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Supplementary Table 1.

Hyper-parameters range for experiments. Supplementary Table 2. Statistics of missing value and given value for model training. Supplementary Table 3. The AI models for predicting ICU admission <48 hours in the ED patients with hyperglycemic crises. Supplementary Table 4. The p-value from the DeLong test to compare the model AUC.

Additional file 2: Supplementary Figure 1.

Learning Curve for MLP in three adverse outcomes. Supplementary Figure 2. The AUC for three adverse outcomes in different algorithms. Supplementary Figure 3. SHAP values for the MP model. Supplementary Figure 4. AI button was integrated in the main screen of existing emergency department system. Supplementary Figure 5. A snapshot of the AI prediction result.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hsu, CC., Kao, Y., Hsu, CC. et al. Using artificial intelligence to predict adverse outcomes in emergency department patients with hyperglycemic crises in real time. BMC Endocr Disord 23, 234 (2023). https://doi.org/10.1186/s12902-023-01437-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12902-023-01437-9

Keywords

BMC Endocrine Disorders

ISSN: 1472-6823

Contact us