Classifying Clinical Work Settings Using EHR Audit Logs: A Machine Learning Approach

ABSTRACTObjectives: We used digital well being file (EHR)–primarily based uncooked audit logs to categorise the work settings of anesthesiology physicians offering care in each surgical intensive care models (ICUs) and working rooms.Study Design: Observational examine.Methods: Attending anesthesiologists who labored not less than 1 shift in 1 of 4 surgical ICUs in calendar 12 months 2019 have been included. Time-stamped EHR-based audit log occasions for every week have been used to create occasion frequencies and represented as a time period frequency–inverse doc frequency matrix. Primary classification final result of curiosity was a doctor’s medical work setting. Performance of a number of supervised machine studying classifiers have been evaluated.Results: A whole of 24 attending physicians have been included; physicians carried out a median (IQR) of 2545 (906-5071) EHR-based actions per week and labored a median (IQR) of 5 (3-7) weeks in a surgical ICU. A random forest classifier yielded the most effective discriminative efficiency (imply [SD] space beneath receiver working attribute curve, 0.88 [0.05]; imply [SD] space beneath precision-recall curve, 0.72 [0.13]). Model explanations illustrated that medical actions associated to signing of medical notes, printing handoff knowledge, and updating prognosis info have been related to the constructive prediction of working in a surgical ICU setting.Conclusions: A random forest classifier utilizing a frequency-based function engineering method efficiently predicted work settings of physicians with a number of medical tasks with excessive accuracy. These findings spotlight alternatives for utilizing audit logs for automated evaluation of clinician actions and their work settings, thereby affording the flexibility to precisely assess context-specific work traits (eg, workload).Am J Manag Care. 2023;29(1):In Press_____Takeaway PointsPhysicians usually carry out completely different roles in a number of settings; routinely classifying doctor actions from uncooked digital well being file (EHR)–primarily based audit logs may help precisely assess their work-related behaviors related to every setting (eg, an intensive care unit vs an working room). We developed an information pipeline and related machine studying algorithms for computerized classification of medical work settings.A random forest classifier had a excessive discriminative efficiency (accuracy = 0.92), and mannequin explanations confirmed discriminant validity.Automatically classifying work settings helps precisely assess setting-specific work patterns; this has implications for correct workload measurements, focused doctor help, and EHR design aligned with work-related contexts._____Modern medical work is documented utilizing digital well being information (EHRs).1 Given the complexities of medical apply, physicians usually work in several roles throughout a number of settings (eg, a trainee working in inpatient, emergency care, and outpatient settings over the course of their rotations). The duties of monitoring the sequence of medical workflows and measuring clinician workload related to completely different medical work settings and tasks are difficult. Although observational strategies have been used for evaluating workflows up to now,2,3 these approaches are time- and effort-intensive and liable to observational biases.4Audit log recordsdata—trails of clinician interactions with an EHR—have offered appreciable alternatives for unobtrusively monitoring clinician actions, related workloads, and downstream outcomes.5-9 Although a lot of the prior analysis on audit logs has centered on descriptive characterization of clinician workload and its influence,10,11 latest analysis has used machine studying approaches to position clinician work actions inside particular contextual conditions. For instance, in a latest examine, Mai et al12 developed a predictive mannequin to categorise affected person encounters related to a cohort of pediatric residents. This mannequin relied on audit log knowledge to precisely classify physician-patient encounters in inpatient, main care, and emergency division settings for trainees who labored throughout these 3 settings. Moreover, the algorithm was used to develop and monitor medical work metrics (eg, variety of sufferers seen) throughout numerous time intervals.13 Several comparable research utilized audit logs to establish medical work practices and behaviors, together with figuring out precise resident obligation hours compared with self-reported obligation hours,14 assessing time spent on main care affected person exams,15 and investigating medical be aware writing practices and their influence on be aware writing effectivity.16A comparable, however unexplored and necessary, class of workflow identification drawback with audit logs is ascertaining the medical work settings and related work tasks of clinicians who apply in a number of roles. For instance, figuring out the medical service and apply setting of a clinician with a number of practices and experience (eg, an anesthesiologist seeing sufferers within the surgical intensive care unit [ICU] and in an working room) usually requires complicated proxy metrics and heuristics.The variations in medical work settings introduces diverging patterns of labor tasks and actions. These variations in work patterns can probably be ascertained by means of the evaluation of actions which might be carried out within the EHR. Such an method has relevance within the fashionable medical work atmosphere, the place EHR-based actions are sometimes tracked to evaluate clinician efficiency metrics to handle each their workload and their wellness (eg, Epic’s Signal platform offering time spent on the EHR). Appropriately classifying the medical service setting throughout medical care is vital to assessing medical apply patterns (eg, documentation), behavioral patterns, workload (eg, time spent on the EHR), and different downstream results (eg, errors) related to medical service (eg, for a dually licensed doctor within the emergency and ICU settings).In this examine, we developed an information pipeline and an analytical method to trace the work actions of physicians to categorise their related medical work settings utilizing uncooked audit log recordsdata. Our main speculation was that EHR-based uncooked medical exercise logs—a proxy for the medical actions related to setting-specific work tasks—might be used for discerning medical work settings. Specifically, we investigated using supervised machine studying classifiers to establish the medical work settings primarily based on a sequence of EHR-based work actions of anesthesiology physicians who offered care in each surgical ICUs and working rooms.METHODSParticipants and Study DesignThis examine was performed at Barnes-Jewish Hospital in St Louis, Missouri, a tertiary care hospital that’s a part of the tutorial medical heart related to the Washington University School of Medicine. The examine inhabitants included attending anesthesiology physicians who labored not less than a single shift in 1 of 4 surgical ICUs between January 1 and December 31, 2019.Clinical tasks for vital care–educated anesthesiologists are usually cut up between work weeks serving because the supervising attending doctor in a surgical ICU and weeks supervising the availability of anesthesia in working rooms. Based on our casual discussions with anesthesiologists (and coauthor S.S.L., a practising anesthesiologist), an attending doctor’s job tasks usually differ in these 2 roles. For instance, within the surgical ICU, attending physicians look after as much as 35 critically ailing sufferers directly; typical EHR work actions embody reviewing affected person info and cosigning or creating addenda to notes written by resident physicians or superior apply suppliers. In the working room, attending physicians supervise as much as 4 working rooms concurrently, every staffed by a nurse anesthetist or a resident doctor; typical EHR actions embody reviewing affected person info, writing preoperative and postoperative assessments, documenting intraoperative occasions, and inserting medicine and laboratory orders. These are examples of anticipated medical work actions and will not be a complete checklist of all attainable actions in both of those settings.The knowledge for this examine have been half of a bigger examine evaluating the work practices of anesthesia clinicians in perioperative settings.6 This examine was authorised by the institutional evaluate board of Washington University (IRB #202009032) with a waiver of knowledgeable consent.Data AssortmentAs mandated by the Health Insurance Portability and Accountability Act, all EHR-based actions are recorded to watch entry to protected affected person well being info. These knowledge—generally known as audit logs—are trails of clinician exercise and are saved in structured databases. Each person motion on the EHR to export, modify, or view creates corresponding audit log occasions that embody knowledge on the person performing the motion, a time stamp of the occasion, accessed EHR part (eg, reviewing affected person chart, ordering medicine), and related affected person identifiers. We extracted EHR-based uncooked audit logs for every attending doctor within the examine from Epic’s Clarity database (Epic Systems) for calendar 12 months 2019. We didn’t make the most of vendor-derived audit log aggregated metrics (eg, Epic Signal) as a result of these measures are topic to alter and should not have an related anesthesia knowledge mannequin, making them unsuitable for week-level classification.Primary ConsequenceThe main final result of curiosity was the identification of whether or not an attending doctor was working in a surgical ICU on a given week. A grasp schedule from the institutional billing workplace was used because the “floor reality” to establish weeks throughout which a doctor was working on this setting. Based on the billing knowledge, a doctor’s work week within the ICU was assigned a constructive binary label and work weeks within the non-ICU setting (ie, working room) have been assigned a detrimental binary label.Feature EngineeringAudit log augmentation. Raw audit logs lack particulars relating to EHR parts associated to every entry occasion. Therefore, we retrieved extra metadata from Epic’s Clarity database for actions associated to notes and studies, to populate a “report title” area with granular info on the kind of report or be aware that the doctor accessed (eg, affected person chart advisories report, progress notes).17 The “metric title” area extracted with the uncooked audit log recordsdata, which represents an motion carried out within the EHR, was mixed with the report title area to signify distinct EHR actions. We refer to those as EHR motion pairs.Frequency encoding. The uncooked audit log knowledge set was segmented into distinctive clinician work weeks (within the surgical ICU vs working room, the place the identical anesthesiologist offered care), matching our main final result definition of per week in a medical setting. We created representations of labor weeks by computing the relative frequency of every EHR motion pair (ie, metric title and report title mixture) for every week. This method was analogous to a bag-of-words method utilized in pure language processing (NLP). The aim of this method was to encode the looks of tokens (ie, phrases for NLP, EHR motion pairs for uncooked audit logs) by assigning numerical values that measured the frequency of look inside a sequence of tokens (ie, paperwork for NLP, EHR motion pairs recorded over every work week for uncooked audit logs).We thought of every week of medical work as a bag-of-words, and the frequency of every distinctive EHR motion pair (ie, a phrase) in that week was computed. Next, we computed the time period frequency–inverse doc frequency (TF-IDF) statistic, generally utilized in info retrieval and NLP, for all weeks such that uncommon EHR motion pairs got higher weight and thus assigned a better relative frequency worth. The ensuing matrix was then used as a frequency-based function matrix, with every row representing a singular clinician’s work week and every column a singular EHR motion pair.Machine Learning Model Development and TrainingLogistic regression, multinomial naïve Bayes, and the random forest classifiers have been used for supervised classification. For all supervised classification fashions, we used a nested, stratified cross-validation method for coaching, hyperparameter tuning, mannequin choice, and mannequin analysis. The stratification of cross-validation splits ensured that class imbalance was preserved. Within the nested cross-validation construction, the outer cross-validation consisted of 10 folds. Each of the ten folds was used as a held-back check set whereas all different folds collectively have been used for coaching for a complete of 10 outer iterations. During every outer cross-validation iteration, for every coaching knowledge set (ie, all folds aside from a single fold used because the held-back check knowledge set), a 5-fold inside cross-validation was carried out to pick out the most effective performing mannequin hyperparameters recognized from grid search. This method due to this fact resulted in 10 greatest fashions (ie, 1 for every of the 10-fold cross validation iterations).As a baseline, a logistic regression mannequin was used. For this mannequin, the next hyperparameters have been tuned: regularization technique, stopping standards, regularization energy, and most variety of iterations till convergence. Then, a multinomial naïve Bayes classifier with a tuned additive smoothing hyperparameter and realized class prior possibilities was additionally educated and examined, as it’s recognized to be appropriate for textual content classification with discrete options and works for the fractional counts produced by TF-IDF. Finally, a random forest classifier, a complicated ensemble of choice tree learners, was used to suit a bunch of choice timber on the information set to manage overfitting and enhance classification accuracy. The following hyperparameter units have been tuned for the random forest mannequin: variety of choice timber and most variety of options thought of for every cut up. Ranges of hyperparameter areas searched are listed within the eAppendix Table (accessible at ajmc.com).Supervised machine studying classifiers usually carry out poorly when the variety of options exceeds the variety of coaching examples.18 Therefore, principal part evaluation (PCA) was used for dimensionality discount. PCA is an orthogonal linear transformation of the unique knowledge onto a lower-dimension linear area that maximizes variance of projected knowledge and minimizes info loss.19,20 We utilized PCA to our function matrix by together with the principal parts which have a cumulative defined variance above the 80% threshold. All supervised studying fashions have been utilized to each the unique function matrix and the PCA-reduced function matrix, aside from the multinomial naïve Bayes, which can not deal with detrimental function values produced by the PCA.All analyses have been carried out utilizing customized Python code with Python 3.8.13 and sci-kit study 1.0.2 variations.Model AnalysisEach mannequin was evaluated utilizing efficiency metrics—accuracy rating, space beneath the receiver working attribute curve (AUROC), space beneath precision-recall curve (AUPRC), precision or constructive predictive worth, recall (sensitivity), and F1 rating—and averaged throughout 10-fold cross validation loops. For the most effective performing mannequin, we used the Shapley additive explanations (SHAP)21 to establish the highest 20 EHR motion pairs that contributed to the general classification final result. SHAP is a model-agnostic clarification approach that helps in figuring out the contribution of every function to the mannequin’s prediction utilizing a game-theoretic method.In addition, we recognized the highest 5 EHR motion classes related to every medical work setting. The rank of contribution was decided utilizing the magnitude of the imply SHAP worth, and the medical motion classes have been manually produced by grouping actions that have been related to a single medical job (eg, cosigning medical be aware, cosigning medical be aware with attestation, and signing a medical be aware have been grouped right into a class known as “signing medical notes”).RESULTSGeneral CharacteristicsA whole of 24 attending anesthesiology physicians who labored not less than 1 shift within the surgical ICU in 2019 have been included. Each doctor labored a median (IQR) of 47 (38-50) weeks per 12 months, with a median (IQR) of 5 (3-7) weeks within the ICU. A whole of 1042 work weeks have been noticed, with every week containing a median (IQR) of 2545 (906-5071) audit log actions. A whole of 1177 distinctive EHR motion pairs have been recognized within the audit log knowledge.Model PerformanceThe random forest classifier on the unique TF-IDF matrix yielded the best discriminative efficiency, with a imply (SD) AUROC of 0.88 (0.05) and a imply (SD) AUPRC of 0.72 (0.13), and was chosen as the most effective performing mannequin (Figure 1). Baseline logistic regression mannequin additionally yielded a excessive predictive efficiency, with a imply (SD) AUROC of 0.85 (0.06) and a imply (SD) AUPRC of 0.55 (0.11). Multinomial naïve Bayes classifier efficiency on the unique function matrix yielded a imply (SD) AUROC of 0.78 (0.05) and a imply (SD) AUPRC of 0.33 (0.10). Multinomial naïve Bayes classifier was not utilized to the PCA-reduced matrix because of its lack of ability to course of detrimental function values created from PCA. All analysis metrics have been averaged throughout 10 cross-validation loops, for which every cross-validation iteration had a greatest mannequin chosen by means of a nested 5-fold cross-validation of the coaching folds. Performance of all supervised classifiers are summarized in Table 1.For the random forest mannequin utilized to the unique TF-IDF function matrix, 20 options with the best influence on the mannequin’s prediction have been recognized (Figure 2). In any given week, larger adjusted frequencies of actions associated to signing medical notes elevated the mannequin’s chance of predicting per week within the ICU (ie, constructive label). Similarly, larger frequencies of actions associated to printing handoff knowledge and updating prognosis associations (see Figure 2 for an in depth checklist) additionally elevated the mannequin’s chance of predicting a piece week within the ICU. In distinction, larger frequencies of actions associated to accessing order units, modifying orders, and loading the navigator template had probably the most influence on predicting a piece week within the working room). The prime 5 motion classes contributing to predicting every setting are offered in Table 2.DISCUSSIONIn this examine, we developed an automatic course of for classifying the medical work settings of anesthesiologists who had twin tasks of working in each surgical ICUs and working rooms. Our main speculation was that EHR-based uncooked medical exercise logs—a proxy for the medical actions related to setting-specific medical work tasks—might be used for discerning medical work settings. Toward this finish, we developed an automatic prediction pipeline utilizing uncooked audit logs and utilized numerous supervised machine studying algorithms. We discovered {that a} random forest classifier may establish physicians’ work setting (ie, surgical ICU) with 92% accuracy on common. This highlights the flexibility of fashions that depend on unobtrusively collected sequences of medical work actions to routinely classify actions of the identical doctor throughout completely different medical work settings (ie, surgical ICU, working room).For the best-performing random forest fashions throughout nested cross-validation, explanations derived utilizing SHAP values confirmed discriminant validity: differentiating between duties carried out within the surgical ICU and the working room; for instance, physicians usually signed residents’ medical notes within the ICU. Other options of surgical ICU work have been additionally distinguished and included printing handoff knowledge, viewing affected person chart advisories report, viewing be aware exercise associated to procedures, and updating prognosis info.There are a restricted variety of research making use of machine studying strategies to audit logs; a majority of those research have relied on unsupervised clustering strategies to generate exploratory groupings of audit log actions.12-16 Moreover, to the most effective of our data, there are not any recognized research of automated work setting classification utilizing audit log occasions. These findings have important implications for burgeoning analysis on using uncooked audit logs for unobtrusively ascertaining medical work actions and settings. Several teams of physicians work throughout a number of settings and modalities; for instance, physicians work in each outpatient and inpatient settings, usually on the identical day, and trainees usually transition amongst completely different settings primarily based on their medical rotations. Identifying the suitable settings and context of labor is necessary for correct measure computation, given the appreciable consideration on standardized metrics associated to EHR-based work,22 equivalent to time spent on the EHR, documentation time, and inbox time.From a methodological standpoint, our method relied on “engineered” options from uncooked sequences of audit log occasions. This function engineering method was adopted from the phrase tokenization and frequency computation strategies of textual content evaluation strategies; such strategies have been used to signify the frequency of EHR-based occasions inside quick time frames. The use of phrase tokenization and frequency measurements was significantly appropriate for uncooked audit log knowledge as a result of similar to phrases in a sentence, every EHR motion pair was recorded sequentially to kind a longitudinal set of motion occasions.This examine highlights the potential of audit logs to establish medical actions and settings related to these actions, which is usually troublesome to perform utilizing medical knowledge alone. For occasion, inside a piece week, a doctor within the working room can entry a number of sufferers, and a few of these could also be sufferers they cared for throughout earlier encounters (eg, holdover from the earlier week). EHR-based knowledge are additionally hampered by incomplete knowledge relating to doctor schedules or modifications of their schedules, which are sometimes managed exterior to an EHR. As such, approaches counting on alternate knowledge sources, equivalent to EHR-based audit logs, present potential instructions for future analysis.LimitationsThis examine has a number of limitations. This was a single-center examine with a small variety of physicians working in a selected set of medical settings (ie, surgical ICU and the working room), and as such the findings might not be generalizable to different settings or doctor teams. Temporality of audit log sequences was not thought of; incorporation of such info could higher signify contextual info and enhance predictive efficiency. Furthermore, there’s a must validate the highest contributive EHR-based actions from the SHAP evaluation to raised interpret their which means within the medical workflow context. Finally, there’s a massive class imbalance between the category labels; oversampling strategies could cut back the category imbalance drawback and enhance predictive efficiency.The use of the TF-IDF statistic in audit log–primarily based function creation could introduce an unintended bias by upweighting uncommon actions that aren’t consultant of labor tasks (eg, holdover actions from the earlier week). However, usually there are frequent work actions no matter work settings, and the markers of setting-specific distinctive actions may be uncommon. Thus, we prevented masking the impact of discriminative capacity of these setting-specific distinctive actions. As a end result, we didn’t incorporate any extra weighting schemes to the actions (eg, weighting primarily based on the motion content material).CONCLUSIONSFor clinicians who’ve numerous duties throughout a number of medical settings, figuring out medical work actions and roles is necessary to check medical apply patterns and behaviors, workload, and different downstream results equivalent to errors pertaining to every setting. Toward this finish, our focus was on creating an automatic course of to categorise medical work settings primarily based on uncooked audit log knowledge. We discovered that supervised machine studying strategies together with a frequency-based function engineering method can efficiently predict medical work settings with fairly excessive accuracy and face validity. These findings present a possibility for an automatic method to characterize clinician actions and work settings in different domains.AcknowledgmentsThe authors wish to thank Derek Harford for his help in acquiring the information used on this examine.Author Affiliations: Institute for Informatics (SK, SSL, LRB, TK), Division of Biology and Biomedical Sciences (SK, LRB, TK), and Department of Anesthesiology (SSL, TK), School of Medicine, Washington University in St. Louis, St Louis, MO; Department of Computer Science and Engineering, McKelvey School of Engineering (TK), Washington University in St. Louis, St Louis, MO.Source of Funding: None.Author Disclosures: The authors report no relationship or monetary curiosity with any entity that might pose a battle of curiosity with the subject material of this text.Authorship Information: Concept and design (SK, SSL, TK); acquisition of information (SSL, TK); evaluation and interpretation of information (SK, SSL, TK); drafting of the manuscript (SK, LRB, TK); vital revision of the manuscript for necessary mental content material (SK, SSL, LRB, TK); statistical evaluation (SK); acquiring funding (SSL); administrative, technical, or logistic help (SK, LRB, TK); and supervision (SK, SSL, TK).Address Correspondence to: Thomas Kannampallil, PhD, Washington University in St. Louis, 660 S Euclid Ave, Campus Box 8054, St Louis, MO 63110. Email: [email protected]. Jha AK. Meaningful use of digital well being information: the street forward. JAMA. 2010;304(15):1709-1710. doi:10.1001/jama.2010.1497.2. Abraham J, Reddy MC. Challenges to inter-departmental coordination of affected person transfers: a workflow perspective. Int J Med Inform. 2010;79(2):112-122. doi:10.1016/j.ijmedinf.2009.11.0013. Abraham J, Reddy MC. Re-coordinating actions: an investigation of articulation work in affected person transfers. In: CSCW ’13: Proceedings of the 2013 Conference on Computer Supported Cooperative Work. Association for Computing Machinery; 2013:67-78. doi:10.1145/2441776.24417874. Sedgwick P, Greenwood N. Understanding the Hawthorne impact. BMJ. 2015;351:h4672. doi:10.1136/bmj.h46725. Adler-Milstein J, Adelman JS, Tai-Seale M, Patel VL, Dymek C. EHR audit logs: a brand new goldmine for well being companies analysis? J Biomed Inform. 2020;101:103343. doi:10.1016/j.jbi.2019.1033436. Lou SS, Kim S, Harford D, et al. Effect of clinician consideration switching on workload and wrong-patient errors. Br J Anaesth. 2022;129(1):e22-e24. doi:10.1016/j.bja.2022.04.0127. Lou SS, Lew D, Harford DR, et al. Temporal associations between EHR-derived workload, burnout, and errors: a potential cohort examine. J Gen Intern Med. 2022;37(9):2165-2172. doi:10.1007/s11606-022-07620-38. Lou SS, Liu H, Warner BC, Harford D, Lu C, Kannampallil T. Predicting doctor burnout utilizing medical exercise logs: mannequin efficiency and classes realized. J Biomed Inform. 2022;127:104015. doi:10.1016/j.jbi.2022.1040159. Kannampallil T, Abraham J, Lou SS, Payne PR. Conceptual issues for utilizing EHR-based exercise logs to measure clinician burnout and its results. J Am Med Inform Assoc. 2021;28(5):1032-1037. doi:10.1093/jamia/ocaa30510. Rule A, Chiang MF, Hribar MR. Using digital well being file audit logs to check medical exercise: a systematic evaluate of goals, measures, and strategies. J Am Med Inform Assoc. 2020;27(3):480-490. doi:10.1093/jamia/ocz19611. Rule A, Melnick ER, Apathy NC. Using occasion logs to look at interactions with digital well being information: an up to date scoping evaluate exhibits growing use of vendor-derived measures. J Am Med Inform Assoc. Published on-line September 29, 2022. doi:10.1093/jamia/ocac17712. Mai MV, Orenstein EW, Manning JD, Luberti AA, Dziorny AC. Attributing sufferers to pediatric residents utilizing digital well being file options augmented with audit logs. Appl Clin Inform. 2020;11(3):442-451. doi:10.1055/s-0040-171313313. Mai MV, Muthu N, Carroll B, Costello A, West DC, Dziorny AC. Measuring coaching disruptions utilizing an informatics primarily based software. Acad Pediatr. Published on-line March 16, 2022. doi:10.1016/j.acap.2022.03.00614. Lin JA, Pierce L, Murray SG, et al. Estimation of surgical resident obligation hours and workload in actual time utilizing digital well being file knowledge. J Surg Educ. 2021;78(6):e232-e238. doi:10.1016/j.jsurg.2021.08.01115. Neprash HT, Everhart A, McAlpine D, Smith LB, Sheridan B, Cross DA. Measuring main care examination size utilizing digital well being file knowledge. Med Care. 2021;59(1):62-66. doi:10.1097/MLR.000000000000145016. Gong JJ, Soleimani H, Murray SG, Adler-Milstein J. Characterizing types of medical be aware manufacturing and relationship to medical work hours amongst first-year residents. J Am Med Inform Assoc. 2022;29(1):120-127. doi:10.1093/jamia/ocab25317. Lou SS, Liu H, Harford D, Lu C, Kannampallil T. Characterizing the macrostructure of digital well being file work utilizing uncooked audit logs: an unsupervised motion embeddings method. J Am Med Inform Assoc. Published on-line December 8, 2022. doi:10.1093/jamia/ocac23918. Hughes G. On the imply accuracy of statistical sample recognition. IEEE Trans Inform Theory. 1968;14(1):55-63. doi:10.1109/TIT.1968.105410219. Hotelling H. Analysis of a fancy of statistical variables into principal parts. J Educ Psychol. 1933;24(6):417-441. doi:10.1037/h007132520. Jolliffe IT. Principal part evaluation for particular sorts of knowledge. In: Jolliffe IT. Principal Component Analysis. Springer; 2002:338-372.21. Lundberg SM, Lee SI. A unified method to decoding mannequin predictions. In: Advances in Neural Information Processing Systems. Neural Information Processing Systems; 2017.22. Sinsky CA, Rule A, Cohen G, et al. Metrics for assessing doctor exercise utilizing digital well being file log knowledge. J Am Med Inform Assoc. 2020;27(4):639-643. doi:10.1093/jamia/ocz223

https://news.google.com/__i/rss/rd/articles/CBMibWh0dHBzOi8vd3d3LmFqbWMuY29tL3ZpZXcvY2xhc3NpZnlpbmctY2xpbmljYWwtd29yay1zZXR0aW5ncy11c2luZy1laHItYXVkaXQtbG9ncy1hLW1hY2hpbmUtbGVhcm5pbmctYXBwcm9hY2jSAQA?oc=5