‘It doesn’t have to be this way’

Currently in the NHS there are 103,000 vacancies. That's one in 11 of all staff, including close to 40,000 nursing vacancies. There are very high levels of sickness absence compared with the rest of industry, and the level of turnover is similarly high. Lots of people are leaving the NHS for various reasons at every level.

Added to that we know that 50 per cent more staff in the NHS suffer from debilitating levels of work stress compared with the general working population as a whole. Every year in the staff NHS survey, 38 to 40 per cent of staff report being unwell as a result of work stress in the previous year. The level of work stress in the NHS is of real concern. We know chronic work stress – and it is chronic in the NHS – is associated with cardiovascular disease, hypertension, heart attacks, addictions, cancer, diabetes, and more severe mental health problems including depression.

It is a paradox that the NHS is a service focused on promoting health and wellbeing but in the process it's damaging the health and wellbeing of a large proportion of its workers. It's worth noting that the NHS employs one in 20 of the workforce in the NHS in England, and if we take social care into account it's probably around one in 10. We know it impacts on the quality of care. We know that higher levels of stress are associated with errors that can harm patient care, and in the acute sector it's associated with higher levels of patient mortality. It doesn't have to be this way.

From coal face to coal face

I was drawn into occupational psychology and working with the NHS for a few reasons. When I finished my PhD on the psychology of meditation back in the 1970s, I worked in a coal mine in South Wales for a year as a labourer. What I observed underground was just how important teamworking was to safety and wellbeing at work. When I started to work in the NHS it became clear that the quality of teamworking in the NHS is often quite poor.

I worked with colleagues to design the NHS National Staff Survey and we ran it for nine years. In the 15 years since the survey has been implemented we've collected data from around a quarter of a million people per year. We estimate from that data that about 40 per cent of staff in the NHS work in what we'd call real teams – those with clear objectives that meet regularly to review their performance. But around 96 per cent of people say they work in teams in the NHS. We also know that the more people who work in what we'd call 'pseudo-teams' – the 55 or 60 per cent not working in real teams – the higher the levels of stress, injuries and patient mortality. We estimate that if we could increase the percentage of staff in real teams in the acute sector in hospitals from 40 per cent to just 65 per cent that would be associated, at a minimum, with the prevention of 5,000 avoidable patient deaths annually. We must extend and build the quality of teamworking within the NHS.

Creating compassionate cultures

I've come to believe that the core value of our health services, and the core work value of the vast majority of those who work within healthcare, is compassion. All of those who work within the NHS have a vocational commitment to helping others. The challenge for us is to create compassionate cultures in the NHS where they can effectively provide the high-quality compassionate care they want to deliver. We have to ensure that leaders lead with compassion rather than being directive, punitive and blaming which is a particular problem within the NHS at times.

I'm currently involved in three projects in the NHS that are seeking to implement these principles, including working with the regulator NHS Improvement and the King's Fund in developing a culture and leadership programme for trusts. I'm also co-chairing a General Medical Council (GMC) inquiry, with Dame Denise Coia, into the mental health of doctors. The third project is part of the NHS Long Term Plan, Workforce Implementation Initiative called Best Place to Work.

The culture and leadership programme stemmed from a research programme that I, and colleagues in a number of universities, worked on following the Mid Staffs inquiry into poor quality care and avoidable patient deaths in that hospital. It was a four-year (£1 million) study funded by the Department of Health looking at the extent to which there were cultures of high-quality care across the NHS in England.

We identified five key cultural factors which were present in high-quality care organisations. One was a strong shared vision or narrative around delivering high-quality care, which was embodied by all or most of the leaders. These were organisations that had simplified what the strategic priorities were, and ensured there were a limited number of clear objectives at every level in every team and for every individual within their organisations. People were not overwhelmed by initiatives or priorities. Leaders also created a feedback-rich environment by providing data for teams, so they could be clear about how well they were doing.

These were organisations that had an enlightened approach to people management. They had developed their leadership internally (not just sending them away on training courses) to create more collective and compassionate leadership. And leaders were generally authentic, open, optimistic and appreciative.

These organisations had also given staff the training, freedom and resources to make improvements in their areas of work. They were striving to become flatter, less hierarchical organisations. There was a very high level of trust in these organisations, along with lower levels of discrimination against minority groups. This created a sense of justice, fairness and autonomy or control. We also found that these were organisations that generally had high levels of engagement, which proved to be the best predictor of NHS hospital/trust performance. There was a strong commitment to quality improvement, learning, and innovation also. They empowered staff at every level to make improvements rather than creating cultures of blame and fear, or climates that were underpinned by very strong hierarchies and excessive bureaucracy.

These organisations also were more likely to have well-developed team and inter-teamworking across the organisation, so that teams were functioning effectively. At a minimum, teams had clear objectives and met regularly to review their performance and how to improve it. Overall, we found that, by far, the best predictor of NHS Trust performance was staff experience as measured in the National Staff Survey.

The importance of leadership

Having identified those key cultural characteristics in the best performing trusts, I had conversations with colleagues in the King's Fund and NHS Improvement focused on how we could help organisations nurture these cultures. The most important factor influencing culture is the leadership at every level of the organisation. Yet there were problems with NHS leadership: many people in interim leadership positions, many leadership vacancies, and no pipelines of people being prepared for future leadership roles. The very senior leaders of organisations, such as Chief Executives, were in post, on average, for less than two years. In some cases, clinicians were being pressured to take on senior leadership roles such as medical director.

We worked with the Centre for Creative Leadership, a US not-for-profit international organisation, along with the King's Fund and NHS Improvement to develop a culture and leadership programme focused on those five areas. The programme involves firstly helping organisations in the NHS assess their existing culture and leadership through a variety of methods. The second step is designing a leadership strategy in relation to the five areas of culture, but also developing everybody in organisations to ensure widely held compassionate leadership skills (attending, understanding, empathising and helping). The third stage is implementing the strategy.

There are now 80 NHS trusts on the culture and leadership journey. Some of those are high performing, but some are more challenged in their performance, because we want to ensure that the programme is applicable across the range. The intent was that all the tools we developed would be evidence-based and open-source. We have drawn primarily on evidence from psychology and work and organisational psychology to develop the 60+ tools and these are free to download. Each individual tool (such as compassion-based recruitment) includes a description of what it is, why it is important, what the research evidence is, how it is done. We provide case studies from NHS organisations wherever possible.

Towards the best place to work

We're just over a year into the GMC inquiry into doctors' mental health. We've heard lots of evidence from doctors, had roundtables with the Royal Colleges, we've been interrogating various data sets and reviewing all the literature both nationally and internationally, to get a sense of the prevalence and incidence of mental health difficulties for doctors. Our work is looking also at the consequences for doctors and for health care quality. We are aiming to have an initial draft of our report produced by the summer, probably to be published in September.

The intent is that we will identify the factors in doctors' workplaces that need to be changed in order to deal with the problems that they are facing. We'll design or provide examples of interventions that are most likely to make a difference to the work lives of doctors and trainees. Those interventions will not be focused on health and wellbeing programmes such as mindfulness and exercise, good though those things are; they'll be focused on changing the workplace factors that damage doctors' health and wellbeing. Some of those will be familiar such as quality of teamworking, supervision, bullying and harassment, discrimination, justice and fairness, autonomy and control.

There are also hygiene factors to do with rotas, long hours, not having places to sleep at night when you're on call, not having anywhere to get a hot drink or something to eat. For junior doctors there is also the problem that they are being bounced from organisation to organisation during their training and not establishing any continuity regarding work relationships and supervision. The major problem almost all NHS staff face is excessive workload. That is having a huge impact on staff turnover, sickness absence and major recruitment problems. We will also look at the experience of the sub-groups of doctors and the particular challenges they face, for example doctors in emergency medicine who are working in very pressured environments, and both UK and international BME doctors who face high levels of discrimination, disciplinary processes and differential promotion.

There's a fortuitous alignment with the NHS Long Term Plan and the Workforce implementation Group – they're urgently addressing similar kinds of issues for all groups of NHS staff. As part of the implementation of the NHS Long Term Plan, the NHS has also set up a programme to look at what changes needed to be made to address the workforce issues I have described. The aim is that the Workforce Implementation Plan will form part of the implementation of the 10-year NHS Long Term Plan. There are workstreams on leadership development and talent management, technical skills and enablement, and the workstream on making the NHS a 'Best Place to Work' which I am contributing to.

The Best Place to Work stream is chaired by Dr Navina Evans, who is the Chief Executive of East London Foundation Trust. At the start of the process we adopted a model from psychology that describes the needs underpinning the wellbeing and motivation of staff – called self-determination theory (developed by Deci and Ryan). It describes the three core needs people have at work: the need for belonging, conferred by being a valued member of the team and organisation; the need for competence or effectiveness (this is to do with people growing and developing their skills in response to the challenges they face at work); the need for autonomy and control in the workplace, where people can control their working environment and ensure the work they are doing is consistent with their values and professional integrity.

What the evidence suggests is that if any of those three workplace needs is not met, it will have a damaging impact on health, wellbeing and motivation at work. In other words, it's no good just having two that are really good and one that isn't… you have to make sure all three needs are being met in the workplace.

It feels a real privilege to be involved in these initiatives in the NHS. The plan is that there will be significant interventions at a national level to change the workplaces people are experiencing. And that in turn will have an impact on the health and wellbeing of the communities the NHS serves.