Why User Research is always worth it

When I was working in the GDS transformation team, one of the questions I got asked often was “why should we spend money and time on user research?”. This case study is the answer I usually gave. I still talk about it today, despite the fact that it dates back to 2014. [With the agreement of all involved, I used the screenshots and details in this post to describe this case study for external audiences at the time. This is the first time I’ve written it down.]

Why do I come back to this example? It’s partly because it’s so clear: a single factor that none of us spotted and which could have meant significant problems for an entire transactional service. I mainly come back to it because the sheer scale of the service still makes me shiver when I think about what would have happened if I hadn’t stuck my neck out to make sure that some user testing happened. 

The service in question was Electronic Vehicle Licence, or EVL, better and colloquially known as “road tax”. The performance platform screenshot shows you the volume of usage at around the time of the following events. 

1. We can’t do it here

Whilst the existing road tax service had existed for some time, it was undergoing a fairly major change as a result of a ministerial commitment to introduce the option for users to pay with a direct debit option. It was also going through a service standard assessment for the first time, and – lucky me – I was the person tasked with making sure that this happened. 

When I first raised the idea of carrying out some User Research on the new service with the teams working on EVL and EVL direct debit, the response was consistent. It’s not possible. I asked why, and there were two main reasons. Firstly, there was no resource available. Secondly, it wasn’t possible to test any interface outside of the building. 

As with almost every other government department in 2014, user research in the sense of testing prototypes of new services with representatives of user groups wasn’t well established at that point [1]. This meant that there was some reluctance to consider user research being as critical as some other activities that were part of the established release procedure. The main issue was the two practical problems detailed above. 

2. How we managed to do it

Like a good transformation manager, I looked into the two problems. They were definitely both barriers. User Researchers in DVLA at that point were supplied by a central team, and they were maxed out on other pieces of work and had no spare resource. 

All environments with any EVL screens or flows or mock-ups were tightly restricted and only accessible from within DVLA premises. This was for stringent and sensible security reasons: the thought of what might happen if anyone got access to any bit of a system generating that much money is enough to make anyone lie awake at night for a long time.  

Cutting a long story of negotiation, bribery and pleading short, I managed to persuade the User Research team to free up one researcher to run a day’s worth of research sessions. I did this partly by promising to a) write the scripts and questions for the sessions and then b) by promising that if they got me the videos of the sessions, I would review the sessions and take responsibility for all of the write-ups and reporting that would otherwise fall to them (and usually take longer than the sessions themselves would).

I can’t take credit for getting around the second problem. I had realised that the restrictions on access were insurmountable. What I also realised was that we didn’t need access to the system to test the proposed approach. To check whether a proposed flow worked, all we needed was screenshots or process diagram flows of the new parts of the service. I could then recreate these in bare wireframes, or – absolute worst case scenario – clickable Powerpoint slides. 

What happened at this point was that I asked my colleague Dafydd Vaughan, our GDS Tech Arch for DVLA at the time, for help getting access to the screen designs. Dai being the generous and technically skilled human being that he is, he not only got hold of some screen designs but – some time in his hotel room later – he had a fully working prototype for use in testing.  

3. The user research session 

These screenshots show the flow that we asked our cross-section of users to test on the day. It’s around ten screens, so not the longest or most complex government service. The first two screens confirmed the registration number of the vehicle: I’m not including these here as they’re not important, but I will include the rest of the screens, in order.

The first screen shown here was the start of the changed approach, asking users to choose which way of paying for their road tax [4] they wanted, with – you will notice – the ministerial commitment to include direct debit options included:  

The rest of the flow asked for some personal details: 

Then details of the bank account where the money you were paying was coming from: 

If you were paying by the new direct debit option, you’d see a summary of payments: 

And then you got the legal text: 

Before finally getting a confirmation: 

The date that we had to run the testing was dependent on the availability of a researcher to run the session, so I wasn’t able to be there in person. But I can tell you that it only took until part way through the second session for the observer from the team to head straight back up the hill [2] to the main DVLA building to phone me and to instruct the team that an urgent change had to be made. 

One and a half research sessions, less than 90 minutes, and a major change: that’s quite an impact. Did you spot the problem in the screens above that caused this? No? If you didn’t, that’s the point. None of us had, either. You carry out user research for a number of reasons but the biggest one is that your users will always use the service you have designed in ways that you don’t expect. Or, as I sometimes say, no service survives first contact with a user. The only way to find out about this is to test. 

4. What had gone wrong? 

The user behaviour that had caused the concern related to the last two screens. Users were getting to the long screen with all of the legal text on it, weren’t getting to the end of it [3], and were then closing the browser screen because they thought that they’d finished. The problem was that they hadn’t. 

Users were half right in their assumption: by the time they were reading the legal text, they’d completed the setup of a direct debit payment. The problem was that there was another step, which only took place once you reached the next screen. This final step was attaching the payment to the vehicle specified at the beginning of the flow, so that the text on the confirmation confirms a payment for a vehicle

Making a payment and not attaching it to a vehicle could end quite badly: it would leave the vehicle without tax. This was fairly fundamental: the service that we were testing was to allow users to tax their vehicle, and most users that we tested with didn’t end up doing that. That’s not just suboptimal in terms of outcome; it could also leave users who thought that they’d complied with the law in the difficult position of not being legally compliant and potentially suffering consequences for this (the Sun has a guide to what might happen in these circumstances, and there’s even a separate service for it).

5. How did we fix it? 

To start with, we fixed it in the most straightforward way possible: we added another green button that said “continue” to the top of the screen, so that you couldn’t miss it if you didn’t scroll down. We tested it and it worked OK. 

To be fair to everyone involved, I should point out that the team had originally challenged the need to have all of that text in the long screen on the long screen but had been asked to keep it there for legal reasons. The team eventually managed to win that particular debate and, a little time after initial launch, that long screen changed to one that looked like this: 

6. What would have happened if we hadn’t tested? 

Any problem on a service that runs at the scale of EVL is undesirable. It’s clear that if we hadn’t tested then this particular problem was unlikely to have been picked up before launch, and that it would have caused some issues. 

I did talk to some experienced ops people about how those issues would have been managed if we hadn’t uncovered it before the service went live. The consensus view was that the obvious solution would have been to rush the “extra green button” fix that we created through as quickly as possible, but that would have taken some time to go through all of the test, security and release protocols. 

Whilst that was happening, the unavoidable task would have been to manually check through all of the EVL transactions to find any that hadn’t been matched with a vehicle, and then try to contact the person who had made the transaction and alert them to their error. That’s a lot of transactions, and would probably have taken a fair bit of work (particularly as there was no guarantee that we there was an easy way of contacting the person who had made the transaction, as they weren’t necessarily the owner of the vehicle).  

7. The end

This was a tricky piece of work: I’d told some people that they needed to do something and then they, not unfairly, pointed out why they couldn’t do it. Finding a way through involved the efforts of a lot of people. I was hoping that it worked out and we got some usable insight; it’s fair to say I had no idea that we’d uncover something so fundamental. 

What was brilliant was that when the team called me back some time later to talk about another small issue that they were having, they showed the results of the latest round of user research. Seeing the value of user research was better than an abstract discussion about it. Once they’d seen the value of it, they’d kept doing it. 

User research can seem like extra hassle, one extra thing in the way of a quick launch, one more problem, until you realise that it saves you time in the long run. Sometimes – as with this example – it saves you a lot of time and embarrassment. Cutting user research is like cutting the number of lifeboats you include in a ship design: you’ll save some time and money in the immediate short-term, but the chance of a disaster after that increases significantly. 

[1] That’s absolutely not the case now. Whilst I haven’t worked at or with DVLA for a while, I have led some service standard assessments with their excellent design and user research people on the panel. 

[2] The testing took place in the Richard Ley development centre, or the RLDC, on Swansea Enterprise Zone. This is at the bottom of the large hill on which the DVLA main building that you can see from the M4 is located. Trust me, it’s a large hill: I’m 40-something and my mum still complains about having to push me up it in a pushchair when I was little. 

[3] Let anyone who has read their iTunes terms and conditions to the end throw the first criticism here.  

Leave a Reply:

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.