"Predictive Analytics By Ruben Amarasingham, Rachel E. Patzer, Marco Huesch, Nam Q. Nguyen, and Bin Xie doi: 10.1377/hlthaff.2014.0352 HEALTH AFFAIRS 33, NO. 7 (2014): 1148–1154 Implementing Electronic Health ©2014 Project HOPE— The People-to-People Health Foundation, Inc. Care Predictive Analytics: Considerations And Challenges Ruben Amarasingham (ruben ABSTRACT The use of predictive modeling for real-time clinical decision [email protected]) is president and CEO of PCCI, a making is increasingly recognized as a way to achieve the Triple Aim of nonprofit research and development corporation and improving outcomes, enhancing patients’ experiences, and reducing an associate professor in the health care costs. The development and validation of predictive models Departments of Internal Medicine and Clinical Sciences for clinical practice is only the initial step in the journey toward at the University of Texas mainstream implementation of real-time point-of-care predictions. Southwestern Medical Center, both in Dallas. Integrating electronic health care predictive analytics (e-HPA) into the clinical work flow, testing e-HPA in a patient population, and Rachel E. Patzer is an assistant professor in the subsequently disseminating e-HPA across US health care systems on a Department of Surgery, broad scale require thoughtful planning. Input is needed from policy School of Medicine, and the Department of Epidemiology, makers, health care executives, researchers, and practitioners as the field Rollins School of Public Health, both at Emory evolves. This article describes some of the considerations and challenges University, in Atlanta, Georgia. of implementing e-HPA, including the need to ensure patients’ privacy, Marco Huesch is an assistant establish a health system monitoring team to oversee implementation, professor in the Leonard D. incorporate predictive analytics into medical education, and make sure Schaeffer Center for Health Policy and Economics, Sol that electronic systems do not replace or crowd out decision making by Price School of Public Policy, physicians and patients. University of Southern California, in Los Angeles. Nam Q. Nguyen is a business operations associate at PCCI. heUShealth caresystem facessig- following capabilities: data retrieval from elec- Bin Xie is a health services nificantchallenges,includinghigh tronic data repositories, such as electronic manager at PCCI. costs, poor quality, and variable healthrecords(EHRs),medicaldevices,orwear- 1–3 performance. Technology sys- abletechnologies;datacleaningandharmoniza- T tems designed to predict and pre- tion; risk calculation; updating or resetting of vent poor clinical outcomes could help. risk prediction models given new data; and the At the start of a discussion of these technolo- activationofclinicalorotherpathways,perhaps gies,itisworthwhiletodistinguishbetweenrisk indirectly through the EHR or directly through predictionmodelsandthesoftwaresystemsthat alerts to patients’ or providers’ devices (see the 5 canexecuteandemploythem.Traditionally,clin- online Appendix). ical risk prediction models have been defined as Risk-prediction models as decision-making models that “combine a number of characteris- toolshavelongplayedimportantrolesinclinical tics (e.g. related to the patient, the disease, or practice. Well-known examples include the treatment)topredictadiagnosticorprognostic Framingham risk model for cardiovascular 4 6 7 outcome.” We define the technologies or soft- events andmortality andtheAcutePhysiology ware systems that can autonomously employ— and Chronic Health Examination (APACHE) II 8 andsometimesreengineer,modify,orupdate— score for intensive care unit (ICU) mortality. these models as electronic health care predictive However, the implementation of e-HPA on a analytics (e-HPA). widescaletoaidinreal-time,point-of-caredeci- These systems may have some or all of the sion making is still in its earliest stages. A 2011 1148 Health Affairs July 2014 33:7 Downloaded from content.healthaffairs.org by Health Affairs on July 10, 2014 by Rachel McCartneyJAMA review of twenty-six risk-prediction mod- a real-world setting and testing it for the first elsforhospitalreadmissionidentifiedonlythree time in practice through a technology system; peer-reviewed models that were integrated into and scaling the model for broader implementa- an EHR and designed to identify high-risk pa- tionacrosshealthcaresystems.Inthisarticlewe 9 tients in real time. focus on the latter two phases because substan- An important limitation of basic risk-predic- tiveliteratureexiststhataddressesdataacquisi- tionmodelsisthattheymaynotbecustomizedto tionandtheconstructionandvalidationofmod- 4,10,16–22 thelocalpopulationorhealthcaresystem;donot els. change or “learn” over time in response to un- Toilluminatesomeofthepertinentchallenges derlying population changes; and may not be inthelattertwophasesofe-HPA,weoccasionally easilyautomatedwithinanEHR,thusrequiring describesomeofthechallengesinthecontextof significantstafftimetomanuallycuratethedata predictingtheriskofhospitalreadmissionwith- 10 tocalculateandthencomputetheriskscore. In in thirty days. This may be one of the most contrast,e-HPAhasthepotentialtoapplymod- straightforward and established uses of e-HPA. els throughout a health care system with mini- 11 mal additional staff time. The emergence of e-HPA has been well docu- Practical Challenges In mented in mainstream media and press re- Implementing e-HPA 12–14 leases. However, a formal regulatory frame- Thescaleandscopeofimplementinge-HPAvary, worktoguidethefieldinitsearliestphasesdoes dependingonthenatureofthemodels,thetech- not exist. Little is known about how best to in- nologiesused,thepopulation,andtheoutcome corporate e-HPA into the work flow of a health ofinterest.Practicalchallengesintestinge-HPA care system; how to evaluate success or protect in a real-world setting for the first time may against error; or how such systems should be include ensuring appropriate oversight, the en- scaledbroadlyacrossclinicalorganizationswith gagementofkeystakeholderstoensuresuccess- varying administrative,scientific, and technical ful and sustainable implementation of e-HPA 15 capabilities. Duringthenextdecade,organiza- into clinical work flow,establishment of appro- tions that implement e-HPA will undoubtedly priate patient privacy and consent policy encounter fresh questions across a range of do- and procedures, and data quality assurance mains within health care delivery. (Exhibit 1). The development and implementation of e- Intheexampleofe-HPAforhospitalreadmis- HPA can be broadly divided into four phases: sionforpatientswithheartfailure,amodelpre- acquiring data to builda risk-prediction model; dictingthirty-dayreadmissionriskmustfirstbe buildingandvalidatingthemodel;applyingitin developed and validated using electronic data Exhibit 1 Key Challenges And Recommended Actions To Integrate, Test, And Disseminate Electronic Health Care Predictive Analytics (E-HPA) Challenge Recommended action Testing the model in a real-world setting under appropriate supervision Appropriate approval and oversight of implementation Establish a health care system operations team to oversee implementation Stakeholder engagement Work with key stakeholders to develop and implement relevant clinical protocols Human subjects research protection Ensure that the health care system operations team assesses the need for human subjects Protection of patients’ privacy protection and IRB approval Data assurance IRB determine the need for patient consent, if applicable Conduct pilot implementation of e-HPA Broad implementation of the model in a health care setting Patient privacy protection, patient consent, approval and Apply lessons learned from above oversight, stakeholder engagement Follow standards to ensure interoperability within and across health systems Interoperability of health systems Ensure open sharing of HPA methods to foster collaboration across health systems Transparency of HPA within health systems Long-term challenges Impact on doctor-patient relationships Ensure that shared patient-provider decision making is not replaced by e-HPA Medical education and training Implement medical school education and clinical workforce training in e-HPA Sustainability of e-HPA in health care systems Align patient care quality and population health management goals; have stakeholders (including payers, vendors, and health systems) advocate for reimbursement incentives for HPA in care processes SOURCE Authors’ analysis. NOTE IRB is Institutional Review Board. July 2014 33:7 Health Affairs 1149 Downloaded from content.healthaffairs.org by Health Affairs on July 10, 2014 by Rachel McCartneyPredictive Analytics from a hospital EHR on demographic, clinical, Stakeholder andsocioeconomicriskfactors.Thenthemodel mustbeautomatedandintegratedintotheclini- engagement may be cal work flow of the hospital. Several steps are necessarytoensureitseffectiveimplementation. important to ensuring Appropriate Oversight Of Implementation Health system oversight is recommended when support of e-HPA implementinge-HPAforthefirsttimeinclinical practice to ensure broad stakeholder support, operations within the smooth integration of e-HPA into the exist- ingworkflow,andtheappropriateprotectionof clinical settings. patients’ privacy and autonomy. There are no widely accepted guidelines for ongoing over- sight of e-HPA within and across health care institutions. E-HPA implementation requires careful over- sightofmultiplefactors.Forexample,thescope and deter its widespread use. and complexity of the e-HPA algorithms, the Onepotentialapproachthatinstitutionscould numberofsimultaneousmodelsbeingdeployed use to ensure appropriate oversight of e-HPA in a given institution over time, and the antici- implementation is to create a multidisciplinary patedmodificationrateofamodelinresponseto clinical and operational oversight committee to changing conditions at the institution could all oversee the implementation. The committee’s affect the scale of oversight. Other considera- primary role could be to conduct a rigorous as- tionsmightincludetheexpertiserequiredifar- sessmentofwhethere-HPAresultedinearlysuc- tificial intelligence or machine learning ap- cessorfailure.Thecomposition,size,anddegree proaches were used for model tuning, and the of oversight of this committee would likely de- costandpersonnelrequirementsneededtoeval- pendontheseverityofthepotentialimpactofe- uate the complex integration of e-HPA into ex- HPAandthelevelofexistingevidencetosupport isting clinical workflows. e-HPA efficacy and safety. The cost of ownership associated with over- For example, if e-HPA output affects only re- sight may be a significant barrier to the proper source allocation, such as the intensity of care and safe use of e-HPA for many institutions, transition services given to patients with heart particularlysmallerfreestandingorganizations. failureathighriskofhospitalreadmission,and Clinicalinstitutionsincludinghospitalsystems, thereisevidencetosupportitsefficacyandsafe- academic institutions, and even individual ty, then less rigorous review of the e-HPA may stand-alone clinics will also need to consider suffice.Incontrast,implementationofe-HPAto thepotentiallydeleteriousresultsofimplement- predictwhichpatientsareatriskforacardiopul- inge-HPAwithoutproperoversight.Inextreme monary arrest event outside the ICU— which cases, these results could include enrolling pa- would require their immediate transfer to the tientsinthewrongtherapeuticorinterventional ICUforpotentiallylife-savingtreatment—might pathway, misallocating clinical resources in the requirecreationofamultidisciplinaryoversight event of software failure, and decisional paraly- committeeconsistingofclinicalexperts,admin- sis when e-HPA systems are down. istrators, and e-HPA developers. Intheexampleofusinge-HPAtopreventhos- Inaddition,managementoftheday-to-dayop- pital readmission, a high-risk patient who war- erationof e-HPAwithin a specific set of clinical rantsintensivecasemanagementmayfailtore- work flows is needed. Institutions should have ceivesuchhelpiftheunderlyingriskmodelfails flexibility to define their own operations man- toaccuratelycapturehisorhercompleterisk.To agement structure. However, stakeholder en- mitigateagainstanyharmsinsuchanexample,a gagementmaybeimportanttoensuringsupport thoughtfuloversightprocessisclearlydesirable. of e-HPA operations within clinical settings. Ontheotherhand,theoversightprocessmustbe In the case of preventing readmissions, ad- nuancedandflexible;oversightthatrequiresap- vance planning of the clinical and operational provalfromacommitteeoneveryaspectofmod- activitiesthatmustoccurwhenthemodelflagsa elrefinementandclinicalwork-flowreengineer- patient as being at high risk for readmission is ing is likely to be unsustainable and to impede neededtomaximizeefficiencyandminimizeop- theuseofpowerful,semi-autonomouspredictive erational setbacks. Organizations will need to modeling systems—and reduce theirbenefits. A evaluate unforeseen but related consequences, poorly structured review and approval process including limited availability of resources for may slow down the implementation of a model othertypesofpatientsorconditions,unexpected 1150 Health Affairs July 2014 33:7 Downloaded from content.healthaffairs.org by Health Affairs on July 10, 2014 by Rachel McCartneycedures are needed to safeguard the data and We are not aware of obtainconsentfrompatientsfortheuseoftheir datawithout slowingdowntheimplementation any framework for ofe-HPA,makingitlessaccurate,orreducingthe lead time available to make a prediction. patient consent that Wearenotawareofanyframeworkforpatient consent that is specific to e-HPA. Until there is is specific to e-HPA. national consensus or established guidance or regulation,institutionswillneedtodeveloptheir own policies to address a number of questions. These include how a patient would be properly informed when a risk-prediction model did not clinical adverse events, failure to adhere to the recommendtreatment;howanindividualphysi- prescribedpathways,andsituationsinwhichthe cianwouldknowwhatresource allocations(for interventions are ineffective. example,allocationsofcasemanagementcapac- Stakeholder Engagement E-HPA imple- ityorICUcapacity)areoccurringatthesystemor mentationinvariablyaffectsmanystakeholders, enterprise level that might affect his or her pa- includinghospitalstaff,clinicians,andpatients tients;andwhethertherewouldbeamechanism and their families. Health care systems should for patients to dispute a model’s recommenda- work with key stakeholders as well as e-HPA tion that no treatment be given. developers to determine how e-HPA should be Thesequestionsbecomeparticularlychalleng- integrated into clinical work flow. ing in large population settings. For example, In some cases, e-HPA might identify patients would an organization have to notify each indi- whoseriskofaneventissohighorwhosechar- vidual in a population of 30,000 patients with acteristics are so unmodifiable that no suitable chronic kidney disease who was not recom- interventionalpathwayexiststopreventtheout- mended to receive one of fifteen available pre- come;predictionsthatarrivetoolatemayalsobe ventiveappointmentslotswithanephrologistin impossible to act upon. At that point, the goal thefollowingsevendays?Themodelmightselect maybethemitigationofharmortheprovisionof thefifteenpatientsoutofthe30,000whowould palliativetherapy.Muchofthisdecisionmaking be most likely to benefit. Numerousethical and should be locally governed. legal issues emerge in this scenario, and frame- In the example of using e-HPA to predict the worksneedtobedevelopedtoguideinstitutions. risk of readmission, stakeholder engagement Whene-HPAisbeingtestedforthefirsttimein may determine which staff member—for exam- areal-worldsetting,patients’consentshouldbe ple, a nurse, physician, or social worker—is an absolute requirement if e-HPA implementa- alerted when e-HPA flags a patient as being at tioncouldexposepatientstoseriousrisks.This high risk for readmission, when or at what in- mightbeespeciallypertinentwhentheinterven- tervals the staff member should be alerted, and tionbeingallocatedisparticularlyconsequential what mode of communication (for example, an and can be received only through the determi- automatic page generated by the EHR, a secure nations of the predictive model. textmessage,ortheactivationofasetoforders) In the example of e-HPA designed to reduce should be used. thirty-dayreadmissionforheartfailure,aninsti- An institution could allow e-HPA to identify tutionmightusethepredictionstoallocatecase several groups of patients at high risk for read- managementcapacity at thepopulation level. If mission, but the interventions could be patient onlythepatientswiththehighestriskscoresgot specificasaresultofpatients’specificriskphe- an intervention—such as a consultation with a notypes.Forexample,somepatientsathighrisk cardiac specialist—it is conceivable that if the forreadmissionmayrequireintensivecaseman- model fails or makes a misprediction, some de- agement with the use of remote monitoring de- servingpatientsthusfailtoreceivetheconsulta- vicesforclosefollow-up.Othersmayneedacar- tion.Anumberoftechnological,procedural,and diology consultation for a potential transplant, clinical checks and balances would need to be inwhichcasearepeatedreadmissionisjustified. instituted to mitigate this risk. Still others may need palliative care. High- Wewouldnotadvocateusinge-HPAastheonly performanceclinicalengineeringteamsarecrit- mechanismfortriggeringanintervention.How- ical to these efforts. ever,ifthatwerethecase,aninstitutionshould Appropriate Patient Privacy And Consent discloseandinformpatientsthate-HPAisbeing Policies To make predictions, e-HPA relies on usedsolelytoallocatespecificresources,suchas detailed data about individual patients, which obtainingoneofalimitedsetofavailablefollow- areoftenobtainedcontinuallyandinbulk.Pro- up appointments, and should provide a mecha- July 2014 33:7 Health Affairs 1151 Downloaded from content.healthaffairs.org by Health Affairs on July 10, 2014 by Rachel McCartneyPredictive Analytics nism to inform patients when these resources To help equip are not recommended for them. Data Quality Assurance Manytimes,mod- physicians for the els designed for real-time clinical use are first built and tested on retrospective, non-real-time e-HPA era, medical datasets.Althoughitmaybeobvious,itisworth statingthatsuchmodelsneedtobesubsequently education and training validated using real-time data, which often be- have differently from retrospective data. Input may need to be data for predictive analytic models to help deci- sion making may change over time. Changes in modified. data quality or the storage location of the input data might also degrade the model’s per- formance. For example, if a risk-prediction model for readmission was built using health insurance payer data, what would happen if there were temcanhelpinformbroaderimplementation,or subtlechangestothedata?Whowouldmonitor scalability,ofe-HPAacrosshealthcaresystems. thosedata?Whenandhowoftenwouldthemod- However,thestrategiesforoversightandstake- el be updated? Who would approve the update? holder engagement may be different in this These are crucial considerations when dynamic phase, as the focus shifts to a level of oversight updating of a model occurs within e-HPA. that can be more easily replicated on a large Itisalsoimportanttodeterminewhethersuf- scale. ficient high-quality outcome data are accessible We are unaware of any published reports on withintheEHRorelectronicdataframeworkto the widespread dissemination of e-HPA across evaluate e-HPA over time. This may be particu- the health care system. Evidence-based stand- larlyrelevantforlonger-termoutcomesorevents ards for all aspects of e-HPA areworth develop- 15,23 that occur outside the health system. Forexam- ingasthescalingupofthetechnologybegins. ple,iflong-termoutcomessuchasmortalityare Ideally, e-HPA vendors, institutions that use e- the outcome of interest in e-HPA, assessments HPA, or independent researchers should per- might not be valid if mortality data on some form and publish a prospective evaluation of e- patients are unavailable because they are lost HPAthroughmultipleimpactandcost-effective to follow-up or die outside the health system. analysesacrossvariedsettingsandpopulations. Simulatingorpilotinge-HPAbeforeactivating Appropriatelyconstructedincentivestoencour- the model and integrating it into the clinical ageadoptionofbestpredictivemodelsorEHRs work flow may be a reasonable way to address mayalsohelpsustaine-HPAbestpractices.Stud- such data quality issues. In the readmission ex- ieshavesuggestedthatperverseincentiveshave ample, e-HPA could be simulated in real time sometimes resulted in overuse of expensive in- withoutits predictions’being actedon initially, terventions,underuseofinterventionsthatpro- then be piloted on one or two ward units. After videprimaryorsecondaryprevention,distorted that,e-HPAcouldbeappliedtotheentirehealth fee structures, or an undersupply of general 24,25 care system. practitioners. An operations management team, including In contrast, the federal government’s mean- clinical and administrative staff as well as the ingful-use incentives for EHR adoption and its data analytics team or vendor, could oversee readmission reduction program have accelerat- the pilot implementation. An ongoing process ed the development and adoption of EHRs and of failure modes and effects analysis—that is, even the use of predictive models for readmis- 26,27 theprocessofidentifyingpotentialfailurepoints sion. Similarincentiveframeworkscouldas- in the system, understanding their effects, sistwiththedisseminationofsounde-HPAprac- and managing the work flow to mitigate those tices. In addition, the government and payers, effects—couldthenbeinstitutedtoexamineeach for example, could insist on acceptable stand- element of software, modeling, and work-flow ards or certifications—analogous to Under- performance. writers Laboratories’“UL” mark—when e-HPA methods, technologies, or even institutions comply with established best practices. Challenges In The Broader Inadditiontoadequatevalidation,impactan- Implementation Of e-HPA alyses,andestablishingbestpracticesfore-HPA, Lessons learned from pilot analyses and e-HPA other barriers stand in the way of widespread implementation within a single health care sys- scalability.Theserevolvearoundissuesofinter- 1152 Health Affairs July 2014 33:7 Downloaded from content.healthaffairs.org by Health Affairs on July 10, 2014 by Rachel McCartney"