#ChangeChallenge Methodology

#ChangeChallenge Methodology

#ChangeChallenge Methodology

"The geeky bit" - how did we get 3,595 people from across the world to discuss how to create bottom-up change in the NHS?

Since January Clever Together have been working with NHSIQ, HSJ and Nursing Times on the Change Challenge. A campaign to challenge top-down change in the NHS and help NHS staff to co-create workable solutions to deliver change from the bottom-up. Overall 3,595 people in 45 countries shared 13,895 ideas, comments and votes exploring how we can create better bottom-up change in the NHS. A voting system and expert panels whittled down the thousands of contributions received into a simple toolkit, containing the barriers and building blocks of bottom-up change, and summaries of solutions – shared by the crowd. You can access the toolkit, and find out more about the campaign on HSJ's Change Challenge channel.

The Change Challenge was delivered using one of Clever Together's crowdsourcing methodologies, this method is often referred to as a hackathon (although at Clever Together we feel this term is a bit overused). The method involved four phases of crowdsourcing activity, the first three used the same technology-enabled five-step cycle, in the last, super interested stakeholders were supported, using google-docs, to further develop the most compelling ideas into workable solutions for inclusion in this guide.

Clever Together's "hackathon" method:


Clever Together's five-step crowdsourcing cycle:


Phase 1.Understand - the barriers to and the building blocks of bottom-up change

Phase one of our Change Challenge campaign ran from 9 January to 26 January. It resulted in a list of 10 barriers and 11 building blocks for bottom-up change. Each and every one of them was defined directly from the 7,519 contributions of the 1,455 people who took part. Here's how...

An announcement regarding the Change Challenge campaign was issued via the comms channels of the HSJ, Nursing times and NHS IQ’s virtual hub 'The Edge'. A follow-up email was sent to their subscribers to introduce the campaign. A week later on 9 January, the Change Challenge was formally launched with a personal invite that issued the following challenge question:

What things help or block you when you try to create bottom-up change:

  • within your organisation, to improve or transform services?
  • across organisations, in order to improve the health and wellbeing of a local population?

Invitees were also invited to share the opportunity to get involved with the Change Challenge with their colleges across the health and care economy.

The dedicated crowdsourcing platform deployed for this campaign – a Clever Together technology - acts like an online workshop. Clever Together designed it with health and care professionals as a method for having hundreds, even thousands of people in one "virtual" room, at a time that suits them, within the time window that's been defined for the campaign. People can visit as many times as they like, they can anonymously share thoughts and ideas, comment on the ideas of others, and vote the best ideas to the top. In this environment, ideas are judge on their merit and not the role or the reputation of their author – a critical, but often overlooked condition for many forms of crowdsourcing.

To analyse the data and facilitate the online conversation, we assembled two independent research teams - one from HSJ and one from Clever Together. They worked separately, coordinated by a steering group made up of leaders from both organisations.

To analyse the contributions and votes of the crowd, the teams deployed one of Clever Together’s research methods enabled through our technology. This method uses what the research trade calls a 'grounded theory approach'.

The Clever Together team reviewed the first 4,000 contributions, looking for frequently occurring themes with regard to where in the process of change, as defined by academic literature, people say they get blocked. In doing so, they identified 7 common 'barriers to change'. They then looked for frequently occurring themes that point to what is needed to support bottom-up change. This revealed 16 common 'building blocks'.

The steering group then presented these potential barriers and building blocks to the HSJ analysis team, who were trained in Clever Together’s approach. They tested the validity of these themes by trying to 'code' the ideas within the first 5,000 contributions. This second analysis revealed that the ideas falling into two of the barriers should be further divided into subsets. For example, the project management theme was separated into (a) selecting & funding projects and (b) managing projects. Likewise, the analysis revealed that some of the building blocks could be merged (e.g. collaboration inside and between organisations into peer collaboration). Ultimately, the HSJ team suggested there were 11 common barriers and 13 building blocks.

The Clever Together and HSJ teams iterated this process as the data continued to flow in from the online workshop.

To ensure validity of the data both within each analysis team and also between the two, every piece of data was reviewed by at least two pairs of people from each team, and the teams worked separately and only together when examining and challenging each other’s analyses. Once all the data were analysed and the two teams had arrived at shared conclusions, the project team was happy that the crowd perceived the existence of 10 barriers and 11 building blocks.

Clever Together then re-analysed the most supported contributions within each theme, both the barriers and building blocks, to enhance the draft definitions. The themes and definitions were presented to an extended steering group, consisting of former front line staff and the 'Horizons Team' from NHS England's Improving Quality unit. Two workshops later, the definitions were either accepted or refined, based upon the direct quotes of those who took part in the online workshop.

Although the analysis process was intensive and complex, our technology and methods enabled us to focus the energy and efforts of both teams on the work that really mattered. Imagine the months of work it would have taken to develop the same insight by running face to face workshops and collating 7,500 post-it notes.

With these voices from the frontline heard, the next stage of the Change Challenge was a facilitated brain storm designed to capture ideas and examples of how we can overcome the barriers to bottom-up change and best establish the supportive building blocks.

Phase 2. Answer - brainstorm solutions that can overcome the barriers and create the building blocks of bottom-up change

Phase two of our Change Challenge campaign ran from 10 February to 23 February. It resulted in a list of 291 solutions to support bottom-up change. Each one of them was defined directly from the 4,230 contributions of 2,612 people who took part. Here's how...

The conclusions of the analysis in phase 1 were disseminated via the HSJ, Nursing Times and NHS IQ's comms channels. Links to these conclusions were also placed on the dedicated Change Challenge platform.

Via an email invite, the following questions were posed to the crowd:

  • What solutions work?
    What approaches have you used or observed that others could learn from and copy? Please share your stories and learning about what did and didn't work.
  • What solutions should we test?
    What ideas do you have that we could test out? Share ideas (these can be completely new) that you believe could support bottom-up change in the NHS if we gave them a chance.

As the solutions and insights came rolling in, the HSJ and Clever Together research teams facilitated the crowd by offering simple prompts to get people to expand upon their ideas.

Once the 291 ideas were collected, the research teams worked in the same format as before; however, this time, they coded every proposed solution by:

  • the themes developed in phase 1 – barriers and building blocks;
  • whether they had evidence of being used in the past or were untested experiments;
  • whether they were more suited to bottom-up change within an organisation or between organisations, and
  • the number of votes and comments they had attracted.

Every proposed solution was assessed as making a primary contribution to at least one barrier and one building block. Once again, there was fierce debate between the research teams but eventually they arrived at shared conclusions regarding the codification of the solutions.


The graphs above show that the crowd generated solutions to tackle every barrier and to create each building block.

Phase 3. Prioritise - nurture a shortlist of compelling solutions that can overcome the barriers and create the building blocks of bottom-up change

Phase three of our Change Challenge campaign ran from 26 February to 5 March. It resulted in the project team prioritising 47 ideas and the "crowd" being asked to nurture these ideas through comments alone. By the end of this phase a total of, 3,595 people had taken part in the campaign. This phase alone attracted 722 comments to build and nurture the shortlist and led to the selection of 17 ideas for a deep dive development in phase 4. Here's how...

Using the results of the coding in phase 2, the HSJ and Clever Together research teams prioritised 32 ideas from the 291 as high potential solutions. To do this, the ideas categorised within each building block were assessed by:

  • the number of votes they attracted from the crowd, and
  • the two research team’s perception of potential.

In parallel, a panel from NHS IQ identified 15 additional ideas as 'potential gems'. Whilst these had not been selected by the research teams, NHS IQ's experience was not to be ignored.

The conclusions of the analysis in phase 2 and this shortlisting were disseminated via the HSJ, Nursing Times and NHS IQ's comms channels. Again, links to these conclusions were also placed on the dedicated Change Challenge platform.

Via an email invite, the following statement and questions were posed to the crowd:

Can you help us develop the shortlisted ideas into real solutions?

The Change Challenge platform listed the solutions beneath the following statement:
Below are solutions to challenge top-down change, which have been prioritised by you the crowd and a team of experts. We now need you to develop these into workable tools; tangible and practical enough to be used widely across health and care organisations to deliver bottom up change for patients, staff and the public. To do this, please tell us:

  • what steps need to be taken,
  • who needs to be involved, and
  • what commitment need to be made?

Following the comments from the crowd, the HSJ and Clever Together teams create a smaller short list for deep dive development and publication. To do this, the most powerful and compelling solutions from each building block were selected based upon five criteria:

  1. Is it supported? Is there a community (or emerging community) supporting the solution (i.e. did it attract comments)?
  2. Is it specific? Does the idea present a clear solution, even if at a high level?
  3. Is it doable? Has the solution been proven somewhere or could someone easily take this idea forward with a low-cost prototype or experiment, or could it be rapidly implemented with a bit of 'senior' support?
  4. Is it brave? Does the solution clearly help to establish one or more building blocks of bottom up change?
  5. Is it innovative? Does the solution offer a clever approach compared to existing practices or processes (for at least a meaningful group)?

By applying these method the Change Challenge project team selected 17 ideas for phase 4.

Phase 4. Action - turn compelling solutions into guides for action and then use them

Phase four of our Change Challenge campaign ran from 9 March to 11 March. It resulted in the project team publishing 15 of the most powerful and compelling solutions, at least one for each building block. Invited members of the "crowd", who had authored or developed the selected solutions, were asked to further nurture these ideas with a simple template. By the end of this phase, 3,595 people had taken part in the campaign. This phase created 15 guides. Here's how...

Using the results of the prioritisation in phase 3, Clever Together’s technology enabled the project team to create anonymous 'design teams' to develop 'guides' for each solution.

First, specific individuals were invited; these included:

  • the key author of each solution,
  • the people who made the most influential contributions to comments, and
  • the Change Challenge project team.

The invitees were asked to review and edit a collaborative document created for each shortlisted solutions. Each document was set out with the following template:

  1. Idea title
  2. Summary – 50 words
  3. Quotes from key contributors
  4. Steps you need to take to make it happen
  5. Impact – barriers it will help you overcome and the building blocks you will create
  6. Optional: key contributors – names of key contributors, job titles and organisations

Of the 17 initial solutions selected for 'deep dive', 15 were selected for publication based upon quality of the contribution and the time permitting. The result is a cutting-edge interactive guide that can inspire staff from every level in all health and care organisations to overcome the barriers to bottom-up change and to create the conditions that support this critical process with the UK's health economy.