Each submission provided through ‘Give feedback on care’ (GFC) is linked directly to a specific service. The quality of the search function is fundamental in allowing users to locate their service with ease. Whilst search success continues to hover below 60% for the service, how can we help those users who are unable to find their service?
Unsuccessful search — a user has entered a search query but been unable to select a service from the search results
Successful search — a user has entered a search query and has been able to find their service from the search results
As explored in earlier analysis, we took a deeper dive into the types of searches GFC users were making which were unsuccessful in order to find trends and commonalities in their queries. Findings showed that users were searching for provider names, archived services, as well as health and social services not in England. However, the biggest grouping were searches deemed not being specific enough. Essentially — a search term where the query was too vague to decipher when compared against GFC search results, the CQC main website search and an external search engine (Google). From our sample of data, we found that 45% of unsuccessful searches were not specific enough — but what can we do with this information?
Back in April 2020, Google amended their search results section in a way which begins to address this issue. Google will now let users know if the search results provided aren’t a close match to the original query.
The idea is not that this message will appear all the time, but the intention is that they can let the user know when the listed search results aren’t a close enough match and provide advice and tips that could help to generate a better result.
See: https://www.theverge.com/2020/4/23/21231876/google-search-results-matches-notification-message
Inspired by how this feature could help boost successful searches, the UX team began to explore how this thinking could be applied to GFC.
After discussions with the development team, they explained how the search functions for GFC.
Users are encouraged to search by service name or address and provided hint text as a suggestion.
The search term queries the index of potential services and provides a list of any of the services which have generated a search score.
The system used is additive, so the user could find that providing both the location name and the full address boosts the service they are look for to the top of the list, rather than just the location name. Note that some services operating in different areas of the country may be registered with similar names.
The potential services are then ordered from highest score to lowest score. If a service has a search score of zero, they are not displayed on the results page.
If all of the services have a score of zero, then no results would display and an message would appear to the user to let them know — ‘there are no results matching your search’
We could in theory set a baseline to let the user know that there aren’t any great matches from the list below, then encouraging the user with tips to review their query and try again.
As searches are ordered by highest to lowest scores, we can look at frequency of score values for both successful searches and unsuccessful searches. We can use this information to find a threshold, where if the highest score falls below a certain value, a message appears to let the user know that ‘there are no great matches’.
The balancing act now means that we need affect more unsuccessful searches than successful searches.
Unsuccessful searches
The graph above displays the frequency of search scores by intervals of 0.15 for unsuccessful searches. The average unsuccessful score has a search score of 1.73 and the largest interval is 1.05–1.2, which doesn’t mean much in this context without looking at successful searches.
Successful Searches
This graph highlights the frequency of search scores for successful searches at an interval of 0.3. The average being 2.17 — this is reassuring as it should be higher than the unsuccessful list. The largest interval is 1.81–2.11 — also higher.
Note that the scores are arbitrary. Getting closer to 1, doesn’t mean a perfect score. The frequency charts show us where to start looking for a good threshold.
We want to affect the largest number of unsuccessful searches, whilst not deterring the number of successful searches, hence we also want to consider the greatest impact factor (% of unsuccessful searches / % of successful searches) in addition to total number of unsuccessful searches.
A score of less than 0.3 would affect no successful searches, however it only affects 31 searches in total, hardly a radical number — 2.3% of unsuccessful searches.
As we begin to increase by 0.1 we see that the number of successful searches affected increases.
The orange highlights in the table show intervals with large impact factor. We’re beginning to build up a picture of the sweet spot.
- 0.4 would have the greatest impact factor, however this is largely due to a small number of users on both unsuccessful and successful searches
- 0.6–0.7 has a strong impact factor of around 5 times as many unsuccessful searches effected.
- 0.65 would have an impact factor of 9.3/1.61 = 5.7.
Next steps… work on the content, discuss through user research, implement with the team, review the impact, adjust accordingly.
- With a week to go before live assessment, we begin to reflect back on the public beta journey and fine tune those story highlights
- Digital Take-up continues to exceed expectation with the last 4 months showcasing Give feedback on care as the main service channel. Q3 looks likely to finish around 40% of concerns by channels.
- Form starts begin to rise likely as a knock-on effect of the latest spike in activity for the ‘Because we all care’ campaign
- Submissions have also risen at faster rate than forms starts, meaning that the completion rate has jumped to 48% — highest level we’ve seen in public beta.
- Further investigation is required to pin point why and if the campaign is having the opposite effect than the initial launch which saw a dramatic dip in completion rate to as low as 28%.
