433

The evidence blind spot in healthcare

Medicine prides itself on being evidence-based.  The United Kingdom's National Health Service (NHS) makes this explicit:  the opening chapter of the NHS Long Term Plan says that the service has "a strong scientific tradition of evidence-based decisions about care". [1] 

A quick look at medical culture and practice seems to confirm this.  Clinicians have access to huge databases of medical research. Their training and professional development are informed by that research, and their clinical guidelines, practice protocols and diagnostic aids are evidence-based.  Conferences and journals help busy practitioners to keep abreast of a non-stop flow of new findings.

Medical research is progressing all the time, but even when older evidence becomes outdated, it is not abandoned.  There is recognition of the importance of organisational memory, and the need to understand where current knowledge comes from.  So historical evidence is cherished and preserved in well-maintained archives. 

Clearly, medicine respects evidence.  Or rather, it respects medical evidence. 

There is another strand of evidence in healthcare that is less well respected.  It comes from patient feedback, and it describes people's experiences of care.

The NHS works to a principle established by the 2008 Darzi review [2]: that high quality healthcare is built on three pillars:  clinical effectiveness, patient safety and patient experience.  There is no hierarchy - all are of equal importance.  Like a three-legged stool, the weakening of any one component weakens the structure as a whole. In spite of this, the NHS repeatedly fails to accord the same respect to patient feedback as it does to other forms of evidence.

Language reveals culture

The terms that healthcare systems use to describe different types of evidence can be revealing.  We can start with the language of complaints.

When a health professional flags up something that has gone wrong, it is called an incident report.  But when a patient does the same, it is called a complaint.  The word “complaint” is synonymous with words like “objection”, “grievance,” and “criticism”. In contrast with the calm objectivity of "incident report", the term "complaint" sets a negative tone.

It is perhaps unsurprising, then, that research finds health professionals having difficulty in seeing complaints as evidence.  Some see complaints as “a breach in fundamental relationships involving patients’ trust or patients’ recognition of their work efforts“ [3].  Consequently, it is “rare for [professionals] to describe complaints raised by patients as grounds for improving the quality of care“ [4].

A similar language problem affects wider patient feedback—from the Friends and Family Test, patient surveys, focus groups, social media posts, and so on. This is frequently described as “anecdotal evidence.” The term indicates a tendency to see patient feedback as subjective, irrational, and potentially unreliable. The term “soft evidence” is also used to distinguish patient feedback from the “hard evidence” of statistics—seen as objective, rational, and reliable.

Research, however, has shown that healthcare statistics are easily manipulated—for example through intimidating staff to achieve performance targets, or by distorting the process of care to misrepresent actual performance [5].  This was amply illustrated during the 2020 Covid-19 crisis, with the UK government's release of questionable statistics [6] on the quantities of personal protective equipment distributed, on the scale and effectiveness of "test and trace", and on infection and death rates.

In yet another verbal twist, comments from patients and service users are often described as "patient stories".  But health professionals who give their own accounts of care processes are never described as "telling stories".  They are credited with the much more serious work of "keeping records".  The difficulty here is that record-keeping, too, can be abused.

Numerous avoidable harm inquiries have shown how dysfunctional professional and organisational cultures can put reputation before truth. At Gosport [7], Mid Staffordshire [8], Morecambe Bay [9] and elsewhere, defensiveness, collusion and cover-up were common factors. Inquiry reports describe "denial that any problem existed" [10], and tell of families who "had their experiences denied and their motives questioned" [11].  Investigators have stated that the "reticence of some clinicians and healthcare professionals to concede error or identify the underperformance of colleagues was frustrating and depressing" [12] and that "The obfuscation by those in authority has often made the relatives of those who died angry and disillusioned" [13].

So patient stories may indeed be unreliable at times. But, sometimes, the same can apply to professionals’ stories.

These terms - "incident reports" versus "complaints", "hard" evidence versus "soft" or "anecdotal" evidence, professional "records" versus patient "stories" reveal a culture in which some forms of evidence (in spite of Darzi) are seen as more equal than others. When it comes to "evidence-based decisions about care", there seems to be a double standard.  

If you can't find the evidence you can't use it

 On the clinical side, the NHS is well supplied with medical research databases.  Evidence is extensively catalogued and archived, ensuring that healthcare has ready access to the raw material it needs for policymaking, professional training and clinical guidance.  In patient experience, the situation is somewhat different.

 Evidence gathering is carried out via the Friends and Family Test, the CQC national patient surveys (adult inpatients, children and young people, community mental health services, maternity services, and urgent and emergency care) and the NHS England national patient surveys (cancer services and GPs). Alongside these are thousands of reports emanating from 150 local Healthwatch organisations, as well as from health charities and other patient voice organisations. Academic studies add to the mountain of literature, while NHS Trusts and Clinical Commissioning Groups carry out their own surveys and focus group work.  One recent study [14] identified no fewer than 37 different types of patient feedback on offer to staff within UK hospitals.

 The output from this welter of activity is published across hundreds of different websites. Some are poorly maintained, so links get broken, pages go out of date, and evidence gets lost. Even dedicated patient experience staff can find it hard to know where all the evidence is held. One is reported as saying “…you’re flying blind with your service and you’re just picking out bits of data from everywhere“ [15].  Another said, “We are data rich, but we don’t bring it all together… It’s a nightmare to see what’s going on“ [16].

The loss of evidence has been relentless.  Government has invested heavily in “patient voice” initiatives, starting with the establishment of Community Health Councils in the 1970’s. Those were succeeded by Patient and Public Involvement Forums, then by Local Involvement Networks and latterly by Healthwatch. All gathered large volumes of evidence on patient experience, but none was ever archived. So as each initiative shut down, its entire body of knowledge went with it. While medical evidence is cherished, patient experience evidence has been treated as disposable.

In 2000, the Department of Health published "An organisation with a memory" [17] - a report on learning from adverse events.  In the foreword, the then Health Secretary said "Too often in the past we have witnessed tragedies which could have been avoided had the lessons of past experience been properly learned".  But twenty years on, with its continuing failure to preserve testimony from patients, the NHS remains unable to learn from past experience.   It is, in fact, an organisation with an incomplete memory.

The evidence-practice gap

Evidence on patient experience is gathered in large volume by many different organisations.  The scale of the effort can be illustrated by the fact that the Friends and Family Test alone can bring in over one million comments per month [18]. The 2020 GP patient survey had nearly three quarters of a million respondents [19].  Other national surveys have tens of thousands of respondents.

Unfortunately, the evidence is not always well used.  Studies have shown the following:

      "None of the minutes of board meetings and nobody who spoke at board meetings explicitly stated that patient feedback provided assurance of the quality of care. The discussion at board meetings about patient surveys did not translate into explicit statements of assurance about quality." [20]

     "Patients are increasingly being asked for feedback about their healthcare experiences. However, healthcare staff often find it difficult to act on this feedback in order to make improvements to services." [21]

     "…the eagerness for collecting [data] dissipates into confusion as busy staff struggle to transform reams of patient comments into useful information. The inevitable result is that, despite the best efforts of staff, information which patients share in good faith is wasted." [22]

So what are the barriers to making sense and making use of patient experience evidence?  As shown above, some barriers are cultural, as revealed by use of language.  Some arise from the difficulty in tracking down patient experience evidence in the first place, and from the failure to preserve it.  But some relate to a lack of skills and capacity among patient experience staff.  The following quotes illustrate the problem.

     "...gleaning information from experience data requires the same analytical capability as interpreting clinical data; however, that capability is often unavailable. Staff across health systems consider patient feedback to be valuable but have neither the time nor the expertise to use it." [23]

       "...it has been known about for some time that many members of ward staff find interpretation of [patient experience] data sets difficult or impossible as they have minimal or no training in analytics or quality improvement... but there was no strategy in place or forthcoming at any of the three organizations we studied to address this issue." [24]

      "...only around one-quarter [of patient experience leads] were able to collect, analyse and use patient experience data in inpatient settings to support change."[25]

 Once again we can see a difference between the use of medical evidence as against the use of patient experience evidence. 

 Clinicians are faced with vast quantities of medical evidence.  It would be entirely unreasonable to expect them to read and assimilate it unaided.  So medical evidence is embedded in clinical practice via professional training, clinical guidelines and diagnostic aids.  The work of knowledge translation is done for clinicians so that they do not have to do it for themselves.

There is, by contrast, no standard training for patient experience staff, and no professional development pathway.  Analytical tools are few and far between, and those that do exist, like the Patient Experience Headlines Tool, and the FFT analysis tool, are hard to use.  There was, until very recently, no professional journal for patient experience staff, and the one that now exists is run on a voluntary basis, with no funding support. 

While clinicians are actively helped to deliver evidence-based practice, patient experience staff are expected to muddle through. 

When the patient voice goes unheard, people die

Patient feedback is sometimes seen as trivial - involving grumbles about appointments, car parking and hospital food.  These matters are seen as less important than clinical care, where mistakes could lead to the death of a patient.

But patient experience work is, in this respect, exactly the same as clinical work.  Ask any GP about their caseload and they too will describe the minor complaints - headaches, earaches, stomach aches - that they have to deal with day in and day out.

Doctors know that attending to apparently trivial matters is important.  Because minor complaints could be the incipient signs of much more serious illness.  The same is true in patient experience work.  The routine gripes of dissatisfied patients are what Sir Robert Francis called "the early warning signs that something requires correction" [26].  It was the failure to take patient feedback seriously that led, at the Mid Staffordshire NHS Foundation Trust, to what Francis described as the "appalling suffering of many patients" [27]. Large scale avoidable harm and death was, he said, "primarily caused by a serious failure on the part of a provider Trust Board. It did not listen sufficiently to its patients" [28].

Two years after Francis, the Kirkup report into avoidable harm and death at the Morecambe Bay Trust described "repeated failures to be honest and open with patients, relatives and others raising concerns" [29].

Since then, further reports with similar observations have been coming in at the rate of more than one per year - from Southern Health [30], Gosport [31], the Northern Ireland Hyponatraemia Inquiry [32], and Cwm Taf in Wales [33].  Ombudsman's reports such as Learning from Mistakes [34], Ignoring the Alarms [35] and Missed Opportunities [36] have added to the pile, as has the independent inquiry into Paterson [37]. 

The most recent is the Cumberlege review, which had this to say: "It became all too clear that those who have been affected have been dismissed, overlooked, and ignored for far too long. The issue here is not one of a single or a few rogue medical practitioners, or differences in regional practice. It is system-wide." [38]

Different times, different places, different providers and services. But running all the way through is a failure to take seriously the concerns raised by patients and relatives.  Their testimony is not seen as evidence.  It is not deemed worthy of serious investigation.  It is not taken as a prompt for action.  It is swept aside - met with disdain or outright denial.  Consequently, patients suffer and die - needlessly, and in large numbers.

It's not a conspiracy - it's a culture

There is an evidence blind spot in healthcare.  Medical evidence is taken seriously and embedded in policy and practice.  Patient experience evidence - meant to have equal standing - is looked at askance.

The disparity is not accidental.  But neither is it deliberate, in a conscious or conspiratorial kind of way.  It is, in fact, cultural. 

The Care Quality Commission spotted this in 2019, when it reported on progress within NHS Trusts on implementing Learning from Deaths guidance.  In his foreword, the Chief Inspector of Hospitals said, "Issues such as fear of engaging with bereaved families, lack of staff training, and concerns about repercussions on professional careers, suggest that problems with the culture of organisations may be holding people back from making the progress needed" [39]. A year later, Cumberlege described "a culture of dismissive and arrogant attitudes that only serve to intimidate and confuse" [40].

Culture is hard to change - not least because when certain practices are just “the way we do things around here,” staff can become oblivious to the impact of their own behaviour and attitudes.  Hence the evidence blind spot:  if you don't see patient feedback as evidence, you won't use it as evidence. If you see it as a lesser form of evidence, you'll pay less heed to it. If you don't consider it worth preserving, you won't learn from past mistakes. 

But we have to recognise that organisational culture does not arise in isolation.  It is not enough to set up inquiry after inquiry, looking at culpable organisations one after another, and pursuing a never-ending "bad apple" theory.  NHS Trusts operate within an overarching national culture which - for all the reasons set out above - gives implicit messages that patient experience evidence is not all that important.  

And this is why every single year in healthcare, we relive the Groundhog Day of Mid Staffs and Morecambe Bay and Gosport and so on, all the way through to the forthcoming Shrewsbury and Telford investigation - and every single report tells the same story of patient experience evidence being ignored or suppressed. 

Removing the blind spot

In 2015, frustrated by the lack of a national evidence base for patient experience, we - the Patient Experience Library - decided to build one.  When we asked for help, we were told by national health sector organisations that the task would be too difficult or too expensive.  We were not convinced.  Big medical research databases are deemed both feasible and affordable, so why should a research database for patient experience be any different?

Undeterred, we pressed on, and now the Library holds over 60,000 documents on patient experience and patient/public involvement.  In 2019, there were around 70,000 visits to the website and 100,000 document downloads.  There is clearly - among individuals at least - an appetite for patient experience evidence. 

Determined to stem the loss of historic evidence, we have created an archive.  Among its contents are thousands of reports from the Healthwatch network, dating back to its inception in 2013.  Next time government decides it needs a new patient voice initiative, we will not see the legacy of the current one being scrapped.

We know that NHS staff struggle to keep track of the many sources of patient feedback.  So we have given every NHS Trust in England a "one-click" access point for their key patient experience datasets. 

New surveys and studies are published all the time, and it can be hard to keep up with the flow of information.  So we produce a series of publications including weekly summaries of new research, along with a quarterly journal and an annual Patient Experience in England overview.

We are on a mission to get patient experience evidence taken as seriously as medical evidence.  We have, so far, received no funding from national NHS bodies, and no practical support.  We believe that is because they are still not ready to treat patient experience evidence with the respect it deserves and the seriousness it needs. 

But we also believe that the healthcare system cannot continue with the distressing and repetitive pattern of harm, cover-up, and official inquiry.  So we have faith that sooner or later, the NHS nationally will join us in our mission.  In the meantime, we are propelled by comments, like this, from the Cumberlege review:

"Patients often know when something has gone wrong with their treatment. All too often they are the first to know. Their experience must no longer be considered anecdotal and weighted least in the hierarchy of evidence-based medicine." [41]

Miles Sibley

[1] The NHS Long Term Plan, January 2019. Page 11. 

[2] Dept of Health, 2008.  High Quality Care For All:  NHS Next Stage Review Final Report

[3] Adams et al, 2017. 'It's sometimes hard to tell what patients are playing at': How healthcare professionals make sense of why patients and families complain about care. Sage Journals: Health.

[4] Ibid.

[5] Dr. Foster, April 2015.  Uses & Abuses of Performance Data in Healthcare. Page 19.

[6] Letter from the Chair of the UK Statistics Authority, Sir David Norgrove, to the Secretary of State for Health and Social Care, 2nd June 2020.  https://uksa.statisticsauthority.gov.uk/wp-content/uploads/2020/06/02.06.2020_SDN_Matt_Hancock_MP.pdf

[7] Gosport War Memorial Hospital. The Report of the Gosport Independent Panel, June 2018.

[8] Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry, 2013.

[9] Dr Bill Kirkup CBE, March 2015. The Report of the Morecambe Bay Investigation.

[10] Dr Bill Kirkup CBE, March 2015. The Report of the Morecambe Bay Investigation. Page 5.

[11] Care Quality Commission, December 2016. Learning, Candour and Accountability. Page 4.

[12] The Inquiry into Hyponatraemia-related Deaths, 2018. Page 25.

[13] Ibid.

[14] Marsh, C. et al., 2019. Patient experience feedback in UK hospitals: What types are available and what are their potential roles in quality improvement (QI)? Wiley Health Expectations.

[15] MES and InHealth Associates, June 2015. Making Sense and Making Use of Patient Experience Data. Page 15 

[16] Ibid.

[17] Department of Health, 2000. An organisation with a memory. Report of an expert group on learning from adverse events in the NHS chaired by the Chief Medical Officer.

[18] There were 1,319,638 responses to the Friends and Family Test in February 2020.

https://www.england.nhs.uk/wp-content/uploads/2020/04/fft-summary-february-20.pdf

[19] Around 740,000 patients completed and returned a questionnaire.  https://www.england.nhs.uk/statistics/2020/07/09/gp-patient-survey-2020/

[20] Lee R, et al, 2017. The use of patient feedback by hospital boards of directors: a qualitative study of two NHS hospitals in England BMJ Qual Saf. doi:10.1136/bmjqs-2016-006312

[21] Sheard et al, 2017. The Patient Feedback Response Framework - Understanding why UK hospital staff find it difficult to make improvements based on patient feedback: A qualitative study. Social Science and Medicine Journal.

[22] Gibbons, C., Greaves, F., 2017. Lending a hand: could machine learning help hospital staff make better use of patient feedback? BMJ Quality and Safety.

[23] Flott, K. et al., 2018. Improving the Usefulness and Use of Patient Survey Programs: National Health Service Interview Study. Journal of Medical Internet Research Volume 20, Issue 4

[24] Sheard L, et al, 2018. What's the problem with patient experience feedback? A macro and micro understanding, based on findings from a three-site UK qualitative study. Health Expect. 2018;00:1-8

[25] University of Birmingham, October 2019. Evaluating the Use of Patient Experience Data to Improve the Quality of Inpatient Mental Health Care (Euripides).

[26] Public Inquiry into the Mid Staffordshire NHS Foundation Trust, Volume 1, Chapter 3 pp 245 

[27] Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry House of Commons, 2013. Executive Summary, Page 3.

[28] Ibid.

[29] Dr Bill Kirkup CBE, March 2015. The Report of the Morecambe Bay Investigation. Page 11.

[30] Mazars LLP, December 2015.  Independent review of deaths of people with a Learning Disability or Mental Health problem in contact with Southern Health NHS Foundation Trust April 2011 to March 2015. 

[31] Gosport War Memorial Hospital. The Report of the Gosport Independent Panel, June 2018.

[32] The Inquiry into Hyponatraemia-related Deaths, 2018.

[33] Broderick, C. 2019. Listening to women and families about Maternity Care in Cwm Taf.

[34] Parliamentary and Health Service Ombudsman,2016.  Learning from mistakes. An investigation report by the Parliamentary and Health Service Ombudsman into how the NHS failed to properly investigate the death of a three-year old child.

[35] Parliamentary and Health Service Ombudsman, 2017. Ignoring the alarms: How NHS eating disorder services are failing patients.

[36] Parliamentary and Health Service Ombudsman, 2019.  Missed opportunities: What lessons can be learned from failings at the North Essex Partnership University NHS Foundation Trust. 

[37] Report of the Independent Inquiry into the Issues raised by Paterson. February 2020.

[38] First Do No Harm. The report of the Independent Medicines and Medical Devices Safety Review. July 2020. Page 3

[39] Care Quality Commission, 2019. Learning from deaths. A review of the first year of NHS trusts implementing the national guidance. Page 4.

[40] First Do No Harm. The report of the Independent Medicines and Medical Devices Safety Review. July 2020. Page 7

[41] First Do No Harm. The report of the Independent Medicines and Medical Devices Safety Review. July 2020. Page 10