David Alfaro: Scrum Costa Rica

Yesterday I heard someone saying proudly that he was able to conduct successfully usability test for non-internal-use product using his employees because they fit the target user profile, so recruiting users for the “usability exercises” was easy. That comment made me feel impelled to state clearly why you should look for REAL users to conduct usability test, and BTW what does REAL mean?

In Aggiorno we are totally concern about letting our users to shape the product, to give them value in the way the want. In Aggiorno we have a rigorous process to find the appropriate users to participate in our usability tests, so I can tell you we are able to observe high quality behavior during the usability tests, why?

Carolyn Snyder states it clearly in her book Paper Prototyping: The Fast and Easy Way to Design and Refine User Interfaces (Interactive Technologies) when talking about recruiting in general Usability Tests (that is, not only applies for paper-prototyping):

“Co-workers aren’t real users. Co-workers are usually not representative of real users, even if they talk to users every day or used to be users themselves. Your co-workers know stuff that real users don’t, such as your company’s business strategy, acronyms, brand, other products, and so on. Very likely, your co-workers know more about your interface—even if they haven’t seen it—than real users do. Therefore, the behavior of a co-worker usually isn’t a good proxy for what real users will do. (Possible exception: You’re designing an intranet or other interface specifically intended for employees of your company.) ”

Anyone who has assisted to any Usability Week will recall (as I do) the guidelines about recruiting once you have done a user profile that fit your persona description:

1. Look for people who never had seen the product you are going to test. Your target market has never seen your product, you need to watch the first impression the final users is going to get, the pains that someone you want to convince will go through.
2. Look for people who had never been in a usability test. Once someone participates in one Usability Test, that person, consciously or not, will learn what makes a facilitator taking notes, will try please or annoy the facilitator. Neutrality and the effort to emulate a real setting are lost. At least, allow six months between tests of different products for the same user.
3. Look for people unfamiliar with you. Any feeling linked with the familiarity with you easily mares the objectivity of the test. Consider possible user’s thoughts: “My friend/girlfriend/father/husband/boss has worked so hard in this product! I would be uplifted during the test, I’m sure this is a great product”, “I don’t want to be fired” “I don’t want to hurt my friend’s feelings” “I hate this guy, I will be sure to let him know that his product sucks”.

All the companies that charges you for recruiting users for usability tests set good part of their quality by finding users that comply with those guidelines.

Any “tropicalization” of such guidelines of methodologies doesn’t make them better. The realities behind them are still valid everywhere. Even when the product is created in a “tropical” country, there is no reason to loose those guidelines, specially when your target market is not “tropical”.

Still, I am not against reviewing the UX with co-workers, even so remember this piece of advice of Carolyn Snyder:

“You don’t want more opinions. I’ve yet to work at a company where there’s a shortage of opinions. The whole purpose of usability testing is to gather data from real users, not yet another internal opinion. It may not be wise to redesign an interface solely on the basis of internal feedback. If a co-worker gives you feedback that you don’t want to act upon (at least not yet), then you’re put in the awkward position of explaining why you’re apparently ignoring the valuable advice you asked him or her to give.”

Remember, a usability test is getting a REALITY CHECK, not yet another co-workers discussion. Is that easy Not at all! In the case of Aggiorno, we had had to go the places where our real users are, we had had to move our usability lab in order to get them. We want to listen, to watch as real as possible the Aggiorno use.

Feel free to let know any other useful guideline in recruiting users for usability tests.

I just finished the round #2 of Usability Tests of Aggiorno. It was amazing! I found out many important issues regarding how the user expects Aggiorno to behave given the Interaction Design it presents. And because I assume you are here to receive value I will tell you the generalities of doing a usability test and the general lessons.

I can’t show you specific details about the detected issues neither clips because of privacy matters outside of my control. However, in a near future post I can talk a bit about the design principles you always need to remember when designing User Interface.

The Process

The Usablity Test was conducted in a web development company called Cis-Solutions. From that company, five users were picked for this test. The selection process was based on a questionnaire where we were sure each user fits our target persona. The main aspects in this sample are:

1. College’s Bachelor Degree.

2. 20 to 30 years old.

3. 1 to 3 years working in ASP and/or ASP.NET

4. Uses Visual Studio Environment usually.

5. Develops sites of 16 to 50 pages

6. Maintains sites of +50 pages

Due technical problems during test executions, I wasn’t able to gather data to do a System Usability Scale (SUS). Instead of SUS, I resorted to ask the users to score (from a scale from 1 to 7) the satisfaction level using Aggiorno. Additionally, I asked them to score how useful Aggiorno is.

After a short but complete explanation of what Aggiorno is, does and how it is invoked, each user was asked to do some tasks with Aggiorno in a Web site already opened in Visual Studio 2005. Those tasks were carefully selected in order for us to see user’s behavior in our intended User Interface.

Conducting a Usability Test

Every time I conduct a usability test, I learn a lot and become more aware of more details. “User A is tapping his feet” “User B is humming” “User C is yawning” That’s behavior!! That together with “Think aloud” exercise allow me to collect the user is feeling (i.e. usability). I always ask the users: “Please pretend I am hot here (neither is any observer) so can swear freely in the case you want to.” I collected a big amount of notes in my chronological log. This is an art!

Finding patterns in raw data and reporting

Reviewing collected data is like drinking water from a hydrant. My preferred method for organizing that data into themes is Affinity Diagrams where each sticky note has a color correspondent to a user: User A has sticky notes in green and User B in yellow and so on. So I and observers (it would be great if any) organize in themes all the sticky notes. If a theme has all the colors available it means that that is a common issue, worthy to be noted.

Because the sticky notes are OBSERVATIONS, that is, OBJECTIVE NOTES, you have to describe consistently the behavior (crystallize the pattern) and make it an issue for the report.

What to do with that?

Soon I’ll let you know how to manage the Debriefing meeting.

There are so many details I am skipping in doing this post, so feel free of contacting me.

Why is User Studies so important for me? Because it means getting closer to user, being iterative, being open to the issues you are going to find and the potential changes they imply, learning to watch as an alternative to listen, inspecting over the collected data, adjust your course because of the lessons your learn from the tests. Do you follow me? For me THAT’S BEING AGILE!

Finally I am back!! I apologize for my temporal aloofness!!
I was reading interesting postings at ScrumDevelopment group about Taskboard and how the progress is visualized so I want to share some ideas from our particular experience.

A very engaging aspect of Scrum (and Agile in general) is to find innovative ways for solving problems found in the Retrospective meetings. By the way, the Retrospectives are are rich well of requests and improvement opportunities. By pandering to those requests (and inspecting at the solutions) you will get a tailored improvement to the process.

How do we at Artinsoft do for keeping track of our Sprint progress using the Task Board? Let me explaining it to you by telling you the story.

We have acknowledged that the following phases are needed in order to set a done criteria for the User Stories we committed:

• Design: Meetings to discusses architectural issues,
• Development: Coding
• Application Quality Improvement: Pair Programming and/or Code Review. We prefer Pair Programming.
• Testing: Quality Assurance in the way of Developer or Integration Tests
• Usability Tests

Additionally, a sixth kind of tasks emerged: Environment, as a mean to say that it is a task that encompasses all the environment setup (technology setup) to accomplish the User Story.

Well, in one Retrospective long time ago we pointed out that some User Stories were lacking of enough quality. Why? …. Got it! one or more aforementioned phases were being skipped when planning and implementing features. Why? We EASILY forget to get sure we get them while planning and even implementing. At the second level of “Why?” we realized we needed something catchy to remember those phases and to avoid the constant tendency of realizing them when the deadline is looming, or even worse: never!

We devised a solution: Use a specific color for each kind of task when doing the Sprint Planning and do a Color Distribution Assessment constantly: Do we have enough of all colors for all User Stories? If not, is there a unanimous and intentional awareness of it?

Our Taskboard looks like this:

We always have a good supply of sticky notes next to the TaskBoard, one color stack per phase or kind.

Along the Sticky notes we also have a supply of red round labels. Why so?

In the diagram, I pictured an hypothetical second day of the Sprint, in the Stand Up. The team member David (hence the “D” in the sticky note) delayed more than the recommended (and estimated) one day for that pink task (Usability) he committed to finished. He sticks a red round label to the task and in that way the Task Board is irradiating to the whole team that we are behind of the Sprint schedule.

Again, this has worked for us and fits to our specific circumstances. What do you think about it? Feel free to give me your opinions, all your comments are needed and welcomed!!

We had an exciting week at Artinsoft. We are very close to an alpha state of our product named Aggiorno. What Aggiorno is about? Well,l I can’t tell you yet until release. But I can tell you I don’t know any other product claiming to do the same thing.

After several Sprints, we are ready to follow all the sacred guidelines for conducting Usability Tests. How do we know about those? Artinsoft foresaw this necessity and I was sent to the Usability Week 2007, Washington D.C. on April 22 through 27.

Roughly, I am leading the following steps

• Recruiting people based on a Screening definition.
• Scheduling the usability tests.
• Executing the usability tests
• Analyzing the usability tests

Recruiting people based on a Screening definition

To make a long story short, we found a way to form partnerships with high quality companies whose employees can be benefited with product. They found users (that fit our Aggiorno User Profile) able to willingly participate in our usability tests.

I am especially grateful to those companies for helping us to make a very usable product. I would let you know who they are very soon.

In this matter I just want to point out how important was to recruit just five users for this product stage. The reason is clearly explained in the following article Why You Only Need to Test With 5 Users of Jakob Nielsen. Nielsen was one of the speakers at Usability Week. I had the chance to talk with him after the training in order to discussing how to fit agile processes with Usability Studies.

Next steps

We setup our Usability Lab to run the tests at after work hours. I improve considerably my facilitator skills with each usability test, I love it!! Currently, we are evaluating the Morae software of Techsmith and in a future post I will let you know my opinion about it.

In this picture you can see me facilitating the test and explaining the setting of the usability lab to one of our users.

Keep tuned!