David Alfaro: Scrum Costa Rica

Yesterday I heard someone saying proudly that he was able to conduct successfully usability test for non-internal-use product using his employees because they fit the target user profile, so recruiting users for the “usability exercises” was easy. That comment made me feel impelled to state clearly why you should look for REAL users to conduct usability test, and BTW what does REAL mean?

In Aggiorno we are totally concern about letting our users to shape the product, to give them value in the way the want. In Aggiorno we have a rigorous process to find the appropriate users to participate in our usability tests, so I can tell you we are able to observe high quality behavior during the usability tests, why?

Carolyn Snyder states it clearly in her book Paper Prototyping: The Fast and Easy Way to Design and Refine User Interfaces (Interactive Technologies) when talking about recruiting in general Usability Tests (that is, not only applies for paper-prototyping):

“Co-workers aren’t real users. Co-workers are usually not representative of real users, even if they talk to users every day or used to be users themselves. Your co-workers know stuff that real users don’t, such as your company’s business strategy, acronyms, brand, other products, and so on. Very likely, your co-workers know more about your interface—even if they haven’t seen it—than real users do. Therefore, the behavior of a co-worker usually isn’t a good proxy for what real users will do. (Possible exception: You’re designing an intranet or other interface specifically intended for employees of your company.) ”

Anyone who has assisted to any Usability Week will recall (as I do) the guidelines about recruiting once you have done a user profile that fit your persona description:

1. Look for people who never had seen the product you are going to test. Your target market has never seen your product, you need to watch the first impression the final users is going to get, the pains that someone you want to convince will go through.
2. Look for people who had never been in a usability test. Once someone participates in one Usability Test, that person, consciously or not, will learn what makes a facilitator taking notes, will try please or annoy the facilitator. Neutrality and the effort to emulate a real setting are lost. At least, allow six months between tests of different products for the same user.
3. Look for people unfamiliar with you. Any feeling linked with the familiarity with you easily mares the objectivity of the test. Consider possible user’s thoughts: “My friend/girlfriend/father/husband/boss has worked so hard in this product! I would be uplifted during the test, I’m sure this is a great product”, “I don’t want to be fired” “I don’t want to hurt my friend’s feelings” “I hate this guy, I will be sure to let him know that his product sucks”.

All the companies that charges you for recruiting users for usability tests set good part of their quality by finding users that comply with those guidelines.

Any “tropicalization” of such guidelines of methodologies doesn’t make them better. The realities behind them are still valid everywhere. Even when the product is created in a “tropical” country, there is no reason to loose those guidelines, specially when your target market is not “tropical”.

Still, I am not against reviewing the UX with co-workers, even so remember this piece of advice of Carolyn Snyder:

“You don’t want more opinions. I’ve yet to work at a company where there’s a shortage of opinions. The whole purpose of usability testing is to gather data from real users, not yet another internal opinion. It may not be wise to redesign an interface solely on the basis of internal feedback. If a co-worker gives you feedback that you don’t want to act upon (at least not yet), then you’re put in the awkward position of explaining why you’re apparently ignoring the valuable advice you asked him or her to give.”

Remember, a usability test is getting a REALITY CHECK, not yet another co-workers discussion. Is that easy Not at all! In the case of Aggiorno, we had had to go the places where our real users are, we had had to move our usability lab in order to get them. We want to listen, to watch as real as possible the Aggiorno use.

Feel free to let know any other useful guideline in recruiting users for usability tests.

I just finished the round #2 of Usability Tests of Aggiorno. It was amazing! I found out many important issues regarding how the user expects Aggiorno to behave given the Interaction Design it presents. And because I assume you are here to receive value I will tell you the generalities of doing a usability test and the general lessons.

I can’t show you specific details about the detected issues neither clips because of privacy matters outside of my control. However, in a near future post I can talk a bit about the design principles you always need to remember when designing User Interface.

The Process

The Usablity Test was conducted in a web development company called Cis-Solutions. From that company, five users were picked for this test. The selection process was based on a questionnaire where we were sure each user fits our target persona. The main aspects in this sample are:

1. College’s Bachelor Degree.

2. 20 to 30 years old.

3. 1 to 3 years working in ASP and/or ASP.NET

4. Uses Visual Studio Environment usually.

5. Develops sites of 16 to 50 pages

6. Maintains sites of +50 pages

Due technical problems during test executions, I wasn’t able to gather data to do a System Usability Scale (SUS). Instead of SUS, I resorted to ask the users to score (from a scale from 1 to 7) the satisfaction level using Aggiorno. Additionally, I asked them to score how useful Aggiorno is.

After a short but complete explanation of what Aggiorno is, does and how it is invoked, each user was asked to do some tasks with Aggiorno in a Web site already opened in Visual Studio 2005. Those tasks were carefully selected in order for us to see user’s behavior in our intended User Interface.

Conducting a Usability Test

Every time I conduct a usability test, I learn a lot and become more aware of more details. “User A is tapping his feet” “User B is humming” “User C is yawning” That’s behavior!! That together with “Think aloud” exercise allow me to collect the user is feeling (i.e. usability). I always ask the users: “Please pretend I am hot here (neither is any observer) so can swear freely in the case you want to.” I collected a big amount of notes in my chronological log. This is an art!

Finding patterns in raw data and reporting

Reviewing collected data is like drinking water from a hydrant. My preferred method for organizing that data into themes is Affinity Diagrams where each sticky note has a color correspondent to a user: User A has sticky notes in green and User B in yellow and so on. So I and observers (it would be great if any) organize in themes all the sticky notes. If a theme has all the colors available it means that that is a common issue, worthy to be noted.

Because the sticky notes are OBSERVATIONS, that is, OBJECTIVE NOTES, you have to describe consistently the behavior (crystallize the pattern) and make it an issue for the report.

What to do with that?

Soon I’ll let you know how to manage the Debriefing meeting.

There are so many details I am skipping in doing this post, so feel free of contacting me.

Why is User Studies so important for me? Because it means getting closer to user, being iterative, being open to the issues you are going to find and the potential changes they imply, learning to watch as an alternative to listen, inspecting over the collected data, adjust your course because of the lessons your learn from the tests. Do you follow me? For me THAT’S BEING AGILE!