Imagine this scenario: you have built a short form to get information from your user about how much they are willing to donate to a cause and you see that the majority of your users are triggering a validation error and some leave because of this error. You think to yourself how hard could it be to enter a number? It strikes you as a “user error”, a term that is uncomfortably putting the blame on the user about interactions with your tool.
Do users make errors? Yes absolutely, and it’s not always about the usability of your website. But when they make errors and are left with a frustrating experience using your tool that’s your problem and you need to figure out what’s happening.
If you have the right tools and the right attitude however, user mistakes are gifts of insight that help you build better products. But that’s only true if you are listening. This is where tools come in. Observing the behavior of your users require a robust instrumentation strategy about things that matter. When designing your instrumentation, you need to ask yourself if you are taking into account and analyzing what’s going on.
Years ago I was listening to a presentation by Daphne Koller, one of the co-founders of Coursera. She gave an example of a math question with open ended text input and showed a scatter plot of numerical responses to the question (I’m showing my representation of that example in the chart below). The correct calculation has a large number of responses and there were some other random responses. But an interesting point was that there was another cluster where many students were finding the same wrong answer. The instructors went back to understand how a large number of students could end up in this wrong answer and they found that it was due to a simple mistake in one of the steps. So they brought this up in the teaching and used it as a resource to address a calculation problem many students were facing.
This insight was useful in their teaching and following the same methods has the potential to provide insights to your application as well. However very few developers would be inclined to collect input data like this let alone analyze it with a scatter plot. But it could provide you with insights.
In the scenario above, if you knew what exact numbers people were entering and especially those that left the website, you can have a better sense of what’s going on. One scenario could be that they are trying to enter a ‘$’ where your validation is limiting it to only numbers. You are maybe including the ‘$’ in your design but people enter it anyway and you slap them with a validation error. In this case you should not force validation error but gracefully accept input and adjust it accordingly in your application. You are asking for money amount after all and you can’t be cross with your users for not seeing your dollar sign which your designer says should be very clear.
Another possibility is that you are asking people cent figures for their donations and when your users don’t enter the cent figures you give an error that says the number they enter all the digits. You can take a look at the number that people entered and if you see that in most cases cents are not typed in, then you can be pretty sure that people don’t want to be bothered by this detail. It looks like this scenario is not one where cents matter. And if you really think that cent figures matter you can give sensible default like .00 and trust that users will enter the cents if it’s important to them.
Another outcome of the analysis of this figure is to get a sense of what the most likely number your users are giving and try to add that as a default. You can build a scatterplot of all the figures entered and see for yourself at which numbers they cluster and offer these as options.
Other ways of checking mistakes are important as well in your interactions. Let’s take a look at more examples:
Your users are complaining that they are receiving notifications when they were the ones that set it
This is a sure sign that your users don’t understand what they are doing with your notifications UI. It’s unlikely that they are idiots, you need to find out what part of your UI does not explain things. If you add instrumentation you can check when they are making contradicting choices like creating a notification but not filling out the method to notify. Or you can check for scenarios that would be problematic, for instance setting notification to get an email every time someone visits their website when they have 3K visitors a day.