Recently a dear friend of mine asked,

“Do you measure Usability using SUS?” 

“I have read about it but have not measured it that way yet.”

“Why not? It’s so famous and used by many UX experts”

“You’re right. I don’t know why I haven’t tried it yet but we’ll do one soon”

This conversation was the reason we decided to try and understand for ourself the feasibility and usefulness of this famous usability measure. 

If you are wondering what is SUS, it is the System Usability Scale. SUS is a set of 10 questions to measure the usability of websites, devices and other applications. It is more like a survey which uses the Likert scale.

The SUS was created by John Brooke in 1986 to measure the usability of electronic office systems, but it can be applied to any web or mobile app as well.

The 10 questions are as follows: 

• I think that I would like to use this system frequently.
• I found the system unnecessarily complex.
• I thought the system was easy to use.
• I think that I would need the support of a technical person to be able to use this system.
• I found the various functions in this system were well integrated.
• I thought there was too much inconsistency in this system.
• I would imagine that most people would learn to use this system very quickly.
• I found the system very cumbersome to use.
• I felt very confident using the system.
• I needed to learn a lot of things before I could get going with this system.

If you try replacing the word system with website or mobile application you can relate more in terms of measuring usability. 

To test this out we chose Amazon mobile app (from the insights of a previous user research done on ecommerce apps). The survey was conducted with a sample size of 20. All 20 users had Amazon app installed in their mobile for long and were frequent users. 

We called each of these users to give an introduction about the study we were doing and sent them the survey over google forms. They had to rate each of these questions on a scale of 1 to 5 from Strongly disagree to Strongly agree. 

The data was then collected and analysed by UX experts.

Our survey questions:

• I think that I would like to use Amazon mobile application frequently.
• I found the application unnecessarily complex.
• I thought the application was easy to use.
• I think that I would need the support of a technical person to be able to use this application.
• I found the various functions in this application were well integrated.
• I thought there was too much inconsistency in this application.
• I would imagine that most people would learn to use this application very quickly.
• I found the application very cumbersome to use.
• I felt very confident using the application.
• I needed to learn a lot of things before I could get going with this application.

We also gave our users a comment section where they could optionally write anything about the application they wanted.

Study results:

• SUS scored ranged between 65 to 92.5 among 20 participants.
• Average SUS score for the application is 75.375 which means the site is above average in terms of usability.

Our observations on the method

1. This method is simple, but the interpretation of the score is complex
2. It is best to automate the excel while doing the calculations
3. Never interpret the score as percentages, it is similar to a percentile ranking ie a score below 68 is below average and score above 68 is above average
4. We may use this method to compare 2 designs and see which design scores more in terms of usability. 
5. This method will not pinpoint the exact usability problems in your website/app but it can tell you where your website stands in terms of usability (above average or below average)

The tricky part  of this process was the interpretation of the scores. Usability.gov explains the interpretation of scores as follows:

“The participant’s scores for each question are converted to a new number, added together and then multiplied by 2.5 to convert the original scores of 0-40 to 0-100.  Though the scores are 0-100, these are not percentages and should be considered only in terms of their percentile ranking.

Based on research, a SUS score above a 68 would be considered above average and anything below 68 is below average, however the best way to interpret your results involves “normalizing” the scores to produce a percentile ranking”.

This definitely looks like a complex calculation. uxuitrend.com explains it in a more intuitive way:

“Step 1: Convert the scale into number for each of the 10 questions

• Strongly Disagree: 1 point
• Disagree: 2 points
• Neutral: 3 points
• Agree: 4 points
• Strongly Agree: 5 points

Step 2: Calculate

• X = Sum of the points for all odd-numbered questions – 5
• Y = 25 – Sum of the points for all even-numbered questions
• SUS Score = (X + Y) x 2.5

The rationale behind the calculation is very intuitive. The total score is 100 and each of the questions has a weight of 10 points.

As odd-numbered questions are all in a positive tone, if the response is strongly agree, you will want to give them the maximum point which is 10 for each question. If the response is strongly disagree, you will want to give them the minimum point which is 0. By subtracting 1 from each of the odd-numbered questions, you ensure that minimum is 0. After which, by multiplying by 2.5, you ensure that the maximum is 10 for each of the questions.

Vice versa, for the even-numbered questions in a negative tone,  if the response is strongly agree, you will want to give them the minimum point which is 0 for each question. If the response is strongly disagree, you will want to give them the minimum point which is 0. As such, by subtracting the points of each question from 5, you ensure that minimum is 0. After which, by multiplying by 2.5, you ensure that the maximum is 10 for each of the questions”.

You can also check their SUS calculator here: https://uiuxtrend.com/sus-calculator/ 

Have you tried this method to measure the Usability of any websites or applications? Do let us know via uxshots@gmail.com.