In "
2009 a Pivotal Year In Healthcare IT" I wrote that:
2009 is proving to be a pivotal year in healthcare IT. Recent authoritative articles and reports on health IT problems largely validate the issues presented at this blog and others focusing on health IT issues, and at my academic website on HIT difficulties started over a decade ago, in 1998, freely available to the industry.
I then listed some of those articles and reports.
2009 is not over, however, and it has yielded another article, a systematic literature review and somewhat novel analytical approach, that supports the observations of numerous independent-minded investigators that health IT is very, very much harder than it looks, and that irrational exuberance unsupported by robust evidence (but perhaps -- in my own opinion -- supported by the color of money) is the prevalent health IT driver today.
The new article is entitled "
Tensions and Paradoxes in Electronic Patient Record Research: A Systematic Literature Review Using the Meta-narrative Method" by Greenhalgh, Potts, Wong, Bark and Swinglehurst at University College London. The article appeared in the Dec. 2009
Milbank Quarterly, a multidisciplinary journal of population health and health policy published for over eighty years. The study was based on findings from hundreds of previous studies from all over the world.
The article is summarized at the Oct. 14, 2009 Healthcare IT News item "
Electronic health records not a panacea, researchers say." The summary includes these key points:
- Large-scale electronic health record projects promise much, but sometimes deliver little, according to a new study.
- Researchers said their findings have implications for President Barack Obama’s election promise to establish electronic health records for every American by 2014, and for other large-scale EHR initiatives around the world.
- EHRs are often depicted as the cornerstone of a modern healthcare, capable of making care better, safer and cheaper. Yet, clinicians and managers the world over struggle to implement EHRs.
- outside the world of the carefully-controlled trial, between 50 and 80 per cent of electronic health record projects fail – and the larger the project, the more likely it is to fail."
I have reviewed the full article, available free as of this writing at
this link (PDF). I note the following. The authors observed that the HIT literature is heterogeneous and at times conflicting (e.g., about benefits, risks and implementation challenges of HIT), not least because it covers multiple research traditions with different underlying philosophical assumptions and methodological approaches.
The authors used a somewhat novel meta-narrative method of systematic review, and identified over 500 full-text sources. They used ‘conflicting’ findings to address higher-order questions about how the electronic patient record (EPR) and its implementation were differently conceptualized and studied by different communities of researchers.
They noted a number of "tensions" in relation to and among:
- the EPR itself - is it a mere data container or a work-altering and defining ‘itinerary’?; the EPR user; the organizational context of the EPR (‘the setting within which the EPR is implemented’ or ‘the EPR-in-use’); the process of change including the 'politics' that accompany IT-mediated change; and other factors.
In other words (mine), health IT tools are virtual clinical tools for use in complex medical/social environments that happen to reside on a computer, not computer applications that happen to be used by clinicians.
The authors found that (emphases mine):
... while secondary work (audit, research, billing) may be made more efficient by the EPR, primary clinical work may be made less efficient; that paper, far from being technologically obsolete, currently offers greater ecological flexibility than most forms of electronic record; and that smaller systems may sometimes be more efficient and effective than larger ones.
They concluded that
The tensions and paradoxes revealed in this study extend and challenge previous reviews and suggest that the evidence base for some EPR programs is more limited than is often assumed.
The relevance of that statement towards national HIT initiatives cannot be understated.I am going to highlight some other key points in the paper:
... We previously developed the meta-narrative method as a way of systematically making sense of complex, heterogeneous and conflicting bodies of literature (Greenhalgh, Robert, Macfarlane, Bate, and Kyriakidou 2004). We recommend that those unfamiliar with this approach access our methodological paper (Greenhalgh, Robert, Macfarlane, Bate, Kyriakidou, and Peacock 2005) ...
... The meta-narrative should be thought of not as the unified voice of a community of scholars but as the unfolding of what they are currently disagreeing on ... In a synthesis phase, we compared and contrasted the different meta-narratives and exposed tensions and paradoxes; and we sought explanations for these in terms of how researchers had conceptualized the world and chosen to explore it. [See full paper for further details -ed.]
While some might consider such analytical methods "soft", unorthodox, and therefore not robust, I counter that traditional methods of analytics and validation in biomedicine right up to and including today's peer review, especially related to issues of power and profit, have come under suspicion of corporate
manipulation and other forms of taint. While the methodology will surely be subject to criticism and debate, which I will follow, I welcome possibly disruptively innovative methods that are probably clear of such pecuniary influences.
We found a complex and heterogeneous literature characterized by diverse philosophical assumptions about the nature of reality (ontology), how that reality might be known (epistemology), and the preferred research approaches and study designs (methodology) ... Our exploratory reading identified a number of historical roots which informed later research on the EPR, including human-computer interaction (HCI), evidence-based medicine (EBM), symbolic interactionism and ethnomethodology, workplace redesign, safety-critical systems research, the social practice view of knowledge, complexity theory, and science and technology studies (previously known as philosophy of science).
This only underscores the true complexity at the intersection of IT and medicine, a complexity I've repeatedly written has been glossed over by (and indeed, due to often narrow and limited backgrounds, may not be comprehensible to) HIT industry pundits and operatives.
... The data suggested a significant difference in the likelihood of success between local ‘home grown’ EPR systems (developed in an ad hoc way by clinicians close to the operational detail of key work practices [such as in this personal example I often cite- ed.]), and ‘off the shelf’ systems (developed either as commercial products or as public-sector systems of choice).
In other words, local projects led by experts were far more likely to provide major benefits than the extant commercial and industrial models of "shrink wrapped" (and massively expensive) HIT.
Studies consistently showed that introducing the EPR in an organization or across organizations is a complex task. It requires a well-articulated vision and strategy, strong leadership, adequate resources, good project management, an enabling organizational culture, effective communication, and attention to human resource issues [e.g., appropriate talent management and organizational structures - ed.]. Even when these preconditions were present, success was not guaranteed...
In other words, hospitals, being generally in strained economic conditions and generally being an IT backwaters are highly risky environments for implementing health IT, which are still largely experimental technologies to start with.
Information systems (IS) research is a heterogeneous tradition that emerged in business schools to consider the role of technology in business and management ... But very few such studies have been published on the EPR, perhaps because of the complexity and unpredictability of healthcare work and the highly institutionalized nature of the healthcare sector ... We found only three empirical studies ... all of which demonstrated that model-based analyses of the determinants of EPR success left much of the observed variance unexplained.
In other words, as much is unknown about "doing HIT right" as is known and further cross disciplinary research is urgently needed.
...there are multiple and conflicting framings of the EPR by users (assumptions about it, expectations of it, versions of the problem to which it is seen as a solution), some of which are explained by deeply-held institutional values (e.g. what counts as ‘professionalism’ amongst doctors or what is seen as ‘good nursing care’); these contrasts partly explain the low adoption and slow spread of the EPR in many healthcare settings.
In other words, HIT is primarily about people, not technology and grandiose promises of cybernetic miracles.
... individuals, working collectively around common tasks in organizations, actively and explicitly shape both technologies and work routines in a way that is mutually adaptive ... It would appear that in relation to the EPR, this adaptation is not happening – or at least, not happening smoothly or unproblematically ... healthcare work is uniquely complex and dependent on the coordinated practice of multiple actors. Research to date has barely scratched the surface of what the introduction of the EPR means, at the level of fine-grained detail, for a healthcare organization and the staff and patients who practice and interact in that setting – and still less so when the EPR is part of a large-scale regional or national program.
In other words, HIT is still largely a social experiment, and we as a society really don't know what we're doing or what the outcomes are likely to be. It raises the issue of slowing down the current locomotive of national HIT diffusion in five years.
In ‘failed’ EPR projects, technical designers typically missed these subtleties and produced artefacts that fitted poorly with the situated nature of knowledge and the micro-detail of clinical work practices. Paper records, being flexible, portable and tolerant of ambiguity, support the complex work of clinical practice remarkably well. CSCW [computer supported cooperative work - ed.] studies have highlighted a telling paradox – that high-tech healthcare environments such as intensive care units often make extensive use of paper charts, white boards, sticky notes, and oral communication.
In other words, paper has its place in healthcare, always will, and promises of "the paperless hospital" or office are utopian and not pratical.
... technologies reflect the interests and values of those who produce them, hence power struggles between bosses and workers, clinicians and managers, men and women, and the state and the citizen are played out partly through the design and use (or, indeed, non-use) of technology (Zuboff 1988). The EPR may be a focal point around which disputes of professional jurisdiction are fought.
In other words, HIT projects are intensely political, especially with regard to medical affairs. (IMO management information systems IT personnel are far out of their league in attempting to lead such projects.)
The synthesis in part concludes:
Much of the literature covered in this review suggests, conversely, that (a) the EPR may be alternatively conceptualized as an ‘itinerary’, ‘organizer’ or ‘actor’ [in the latter concepts HIT is a medical device -- that should probably be regulated -- not a simple passive tool - ed.]; (b) seamless integration between different EPR systems is unlikely ever to happen because human work will always be needed to bridge the model-reality gap and re-contextualize knowledge for different uses [so much for dreams of 100% interoperability - ed.]; (c) whilst secondary work (audit, research, billing) may be made more efficient by the EPR, primary clinical work is often made less efficient; (d) the EPR may support, but it will not drive, changes in the social order of the workplace; (e) paper will not necessarily disappear as it offers a unique level of ecological flexibility (though workable paperless systems have been developed in one or two centres); and (f) smaller, more local EPR systems may often (though perhaps not always) be more efficient and effective than larger ones.
Each item on this list contradicts nearly every precept of the national HIT initiative now underway in this country, at the expense of tens of billions of dollars.
Two of the recommendations made by the authors are of special note:
... as a cross-cutting theme in all the above areas, the realpolitik of EPR projects within and between organizations and interest groups should be more explicitly explored ... Orlikowski and Yates have called for more research on the “messy, dynamic, contested, contingent, negotiated, improvised, heterogeneous, and multi-level character of ICTs in organizations” (page 132) (Orlikowski and Yates 2006).
We suggest that sponsors and publishers eschew sanitized accounts of successful projects and instead invite studies of the EPR in organizations that “tell it like it is” – perhaps using the critical fiction technique to ensure anonymity (Winter 1986).
This recommendation is particularly intriguing to me.
I am perhaps one of the few people in the HIT world to have undertaken that approach in the de-identified
"tell it like it is" cases at my Drexel U. website "
Common Examples of Health IT Failure" begun over ten years ago. (I've made the lead author aware of the site and of the new multi author book "
HIT or Miss: Lessons Learned from Failed HIT Implementations" that takes the same approach, to her approval.)
... Finally, given the mismatch between what is known about the EPR in organizations and what many policymakers assume is known, there is also scope for research that addresses this mismatch ... The role of the systematic reviewer in this process is itself worth studying, since very little research on knowledge translation to date has addressed such turbulent waters.
Yet, our government (and that of the UK as well) feel confident that HIT is such a perfected technology, investment of tens of billions of dollars on a frenetic timeframe will create massive quality improvements and cost savings. This is perhaps the height of magical thinking and political hubris.
Finally, the authors identify what does NOT need be done:
This review has also identified some areas where more research does not appear to be needed ... [including] simplified experimental studies based on functionalist and determinist assumptions of the general format “what is the impact of technology X on outcome Y?” or variations thereof ... the circumstances in which they add value are more limited than is often assumed.
We [also] believe that surveys of attitudes of patients or staff towards ‘the EPR’ or ‘computerization’ which are not adequately contextualized have almost no enduring value.
I note that these are the types of studies that seem to commonly appear in the soft "throwaway" literature aimed at the hospital governance board, CIO and IT worker.
In conclusion, I believe this literature review supports the notion expressed in other studies and opinion pieces here and elsewhere that we really need to SLOW DOWN the current HIT stampede, largely promoted by the HIT industry lobby. We need to take the appropriate time to better understand how to "do HIT well" before plunging in as if we actually know what we're doing.
In the authors' own words:
"It is time for researchers and policymakers to move beyond simplistic, technology-push models [for HIT] and consider how to capture the messiness and unpredictability of the real world."
Indeed.
They report, you decide.
-- SS
Addendum: this paper led me to the massive, near-600 page report "
The Impact of eHealth on the Quality & Safety of Healthcare", A Report for the NHS Connecting for Health Evaluation Programme, Car et al., Imperial College, London, March 2008. This report was unknown to me but also validates much of the work of those studying the problems, risks and downsides of HIT.
I will post on the report in the near future. It is available (warning - 7 Mb PDF)
here. The executive summary is quite informative.
