User Research Participants as Experts

Listening to Whitney Quesenbery’s interview of medical anthropologist Francine Harris on the UIE Brain Sparks podcast, I was struck by the suggestion that Harris made that UX researchers and designers should move away from thinking about the people who participate in the research as “subjects” and instead consider them to be “experts.”

Subjects reflect an inherently passive role in a research project. Researchers administer questionnaires that are designed by the research. They analyze results.

They do lots of things that confirm the expertise of the researcher. But, in reality, subjects are the experts. They have the knowledge that we want to understand and use. The question that researchers and designers need to ask themselves is, “How do we find out what they know?” It’s as much an attitude towards research and people. It has implications for how we define our research goals.

The interviewer, Whitney Quesenbery, then suggests to Harris that “it sounds like a big shift in your attitude about your research” from a position of ‘I am the person in the white coat studying things’ to ‘I’m engaging with people as I work with them.’” As someone who has done a lot usability testing, I find the mindset that Harris advocates compelling. So how do I think this might change my approach to designing research projects?

For one thing, I think remembering that the participant is an expert keeps my hubris in check. It’s easy for me to get carried away in thinking I’m the expert, I’m the person who knows this and that about user behavior and best practices, etc. But if I was such an expert, then I wouldn’t really need to do much testing. I’d already know everything I need to know.

My users are the experts, and it is up to me to continually be going back to them to understand how they use our systems. Yes, as a librarian, I know that there are more efficient ways of doing things in our systems and that sometimes there are even ways of doing thing that are objectively “right.. But that really doesn’t matter to me so much as a UX researcher. I’m not a user of our systems in the same way our primary audience is, or if I am, I’m a very special case (as are all of my fellow librarians). The “experts” at using our systems are the people these systems are primarily designed for (at my library, that would be the students and faculty of the college where I work). These experts are using our systems in their own way, not necessarily the way a librarian would (but that doesn’t matter). My job is to consider them as experts and then try to understand how they approach our systems, how they interact with them, what mental models they have in mind as they use our systems.

Keeping this perspective–that our users are our “experts” that we need to learn from and understand–will prevent me from designing research questions for projects that are flawed because I’ve fooled myself into thinking that I already “know” what I’m going to find. When it comes to analyzing usability tests, this perspective can help me maintain a more open mind about what I’m seeing.

And, finally, I think it will allow me to be more open to design suggestions from our experts. We should be fitting our systems when possible to their way of doing things, not strictly our preconceived models of what’s best for them. This last point speaks to what Harris was getting at when she mentioned that “subjects reflect an inherently passive role in a research project.” Later in the interview she talks about “participatory action research” as a method whereby the people being studied can play an active role in developing the research agenda, in analyzing the results, or designing solutions. I’m going to have think about that some more to see if there’s a way to bring that kind of collaboration into the UX design work I’d like to pursue. Stay tuned.

Using GoToAssist for Mobile Screencapture in Usability Testing

This year, some colleagues and I are doing a somewhat involved research project involving usability tests that will compare searching in the mobile catalog interface to the traditional web interface. One of our great challenges is finding a good way to capture what happens on the screens of the mobile devices (Android and iOS) of our test participants. For the Android users, we’ll be providing them one of our own phones to use during the test. I’ve been looking at using the GoToAssist app as a way to capture what’s on the screen of our Android (actually, it’ll likely be my own Samsung Galaxy S3) phone.

I thought I’d share what’s involved by making a recording on my desktop (using SnagIt) of what the screensharing looks like when viewed on a PC. This setup may work for testing we have in mind, although the video transmission from phone to my PC seems to be delayed by about 10-15 seconds. Also, if I scroll really fast on the phone, the screencapture from GoToAssist doesn’t completely capture that; instead you get kind of a snapshot effect in the transmission and not a smooth video. But this may work well enough.


For the iPhone we’ll be also using for testing, we are hopeful that the Reflector app will do nicely.

How the Mobile Web Affects Perceptions of the Traditional Web

This fall, I’m going to be working with some colleagues from other parts of CUNY to do some research that will allow us to explore the how perceptions of the web on mobile devices and on desktop/laptop browsers affect the user experience. Specifically, we want to look at an interface where there is a mobile web version that is notably different from the traditional web interface. How distinct can the interfaces be before the user experience is notably degraded? And which interface will suffer more in the way it’s perceived?

I realize that one way to get around the problem of forked interfaces (one for desktop/laptop browser dimensions and one for mobile browser dimensions) is to focus instead on a responsive design that shifts content around depending on the device being used. But I still wonder where the break point is for responsive design, too. How much shifting of content on a page in a responsive design can be done before the user who visits a site on different devices regularly gets thrown off by the altered layouts?

It may be the case that most users have minimal issues in adjusting themselves as they use different devices to visit the same site. I’m interested now in finding literature that addresses these questions and hopeful that the research my colleagues and I are about to  undertake will offer evidence with respect to interfaces in library resources.

Redesigning a Faculty Services Page

Today, we start the process of redesigning a page on our library’s website that details various services available to faculty members. I thought it might useful to document that process a bit.

Some History

The library here at Baruch College launched a redesigned website on December 26, 2012. Most of the work that went into the redesign focused on student needs. Now, we’d like to take some time to rethink how the services and resources of interest to the faculty are presented on the site. The text on the current “Faculty Services” page is mostly the same text we had on the old site.

The Plan

The short version: needs assessment, redesign, usability tests, tweak design as needed.

The long version: Today, I meet with a five-person Committee on the Library, a body whose members are elected from departments across the three schools at Baruch (a business school, a school of arts and sciences, and a school of public affairs). To prepare for today’s meetings, I asked members to complete a three-question survey:

  1. What are the top three tasks that you come to the library website for?
  2. What other reasons or tasks bring you to the library website?
  3. What brings you to the library website more: your own research needs or your teaching needs?

At the meeting, I intend to:

  • review the overall plan for redesigning the page
  • review the results of my survey and a survey administered last fall to the faculty that focused on the value of the library
  • do a card sorting exercise where I ask them to arrange cards featuring services and resources into piles that make sense to them (from this, I hope to discern a useful way to chunk the content on the page, to find better wording for that content, and to learn if there are any things I forgot or that I can forget)

After the meeting is over, I expect to do the following this semester:

  • work with our web design team to come up with a new layout and new text for the page
  • make individual appointments with members of the Committee on the Library to have them do a traditional usability test that will likely take them to the redesigned faculty services page as well as other places on the website
  • after analyzing the results of the usability tests, I’ll work with the web design team to further refine the faculty page

I expect that the process will be completed this spring semester. As I complete steps in the plan, I’ll try to return to my blog to write up posts about how it’s been going.

Using Qualtrics for Usability Testing

At the marvelously helpful [email protected] event I was at yesterday, I learned about a great way to use survey software (Qualtrics) for usability testing. Since we have the same software here at Baruch College, I spent part of today setting up a few sandbox surveys so that I could try out different question types and get a sense of how survey data would be recorded and displayed. I’ve found three question types so far that look like they’ll be useful. All of them involved uploading screenshots to be part of the question.

Question Type: Heat Maps

Looking at a screenshot, the user gets to click somewhere on the screen in response to some question posed in the survey. The data then gets recorded in a heat map of click data; if you mouse over different parts of the heat map report, you can see how many clicks were done in that one spot. Another way you can set up the screenshot is to predefine regions that you want to name so that the heat map report not only offers the traditional heat map display but also a table below showing all the regions you defined and how many clicked in one of those special regions.

Question Type: Hot Spots

As with the heat map question type, the hot spot question presents the user with a screenshot to click on. But this type of question requires that the person setting up the survey predefine regions on the screenshot. When the test participant is viewing the screenshot, they are are again being asked to click somewhere based on the question being posed. The survey designer can either make those predefined regions have borders that are visible only on mouse over or that are always visible. By making the region borders visible to the test participant, you can draw the participant’s eye to the choices you want him/her to focus on.

Question Type: Multiple Choice

Although multiple choice questions are the most lowly of question types here–no razzle dazzle–it wasn’t until today that I realized how easy it is to upload an image (such as a screenshot) to be part of the answer choice. This seems like a great way to present 2 or more design ideas you are toying with.

Many Uses for a Survey

As a one-person UX group at my library, I find running tests a challenge sometimes if I can’t find a colleague or two to rope into lending a hand with the test. Now I feel like I’ve got a new option for getting feedback, one that can be used in conjunction with a formal usability test or that can be used in lots of different ways:

  • Load the survey in the browser of a tablet and go up to students in the library, the cafeteria, etc. and ask for quick feedback\
  • Bring up the survey at the reference desk at the close of a reference interaction when it seems like the student might be open to helping us out for a minute or two
  • Distribute the survey link through various communication channels we’ve got (library home page, email blast to all students, on a flyer, etc.)

Sample Survey

I made a sample survey here in Qualtrics that you can try out. It’s designed to show off some of the features of questions in Qualtrics, not to answer any real usability questions we currently have here at Baruch. At the close of the session, I set it up so that it offers you a summary of your response (only I can see all the responses aggregated together in a report. It’s likely that when I use Qualtrics surveys for usability, I’ll set them up so they end either by looping back to the first question (useful when I’m going up to people with my iPad in hand and survey loaded in the browser) or by giving them some thank you message. If I get enough responses in this sample survey, I’ll write a new post to show what the report for the survey looks like. In the meanwhile, I’d be interested in hearing from anyone that is already using Qualtrics for usability testing or another survey tool.

Usability Testing Basics

The kind folks who run the Carterette Series Webinars for the Georgia Library Association invited me to do a presentation on usability testing basics. I just finished up an hour ago and wanted to share my slides as soon as possible. The webinar recording will be archived and freely available soon (check the archived sessions page). In the meanwhile, here are my slides:

If you want to see my slides with my notes, you can get the original PowerPoint slides, too.

During the presentation, I read aloud from a script we used this past January when we were testing a draft of the library website. Here’s that script:

Test Script

First of all, we’d like to thank you for coming. Before we get started, I’m going to start a recording here so that we can document this session. Please don’t worry, as this session will be kept private and you’ll remain anonymous.

[Test moderator hits CTRL-F8 on the laptop keyboard to start the audio and screen recording]

As I mentioned earlier, we’re in the process of redesigning the library web site. In order to make it as easy to use as possible, we’d like to get some input from the people who will be using it. And that’s where you come in. We’re going to ask you to perform a very simple exercise that will give us some great insight into how we can make this web site easier to use.

I want to make it clear that we’re testing the site, not you. You can’t do anything wrong here. We want to hear exactly what you think, so please don’t worry that you’re going to hurt our feelings. We want to improve it, so we need to know honestly what you think.

As we go along, I’m going to ask you to think out loud, to tell me what’s going through your mind.

If you have questions, just ask. I may not be able to answer them right away, since we’re interested in how people do when they don’t have someone sitting next to them, but I will try to answer any questions you still have when we’re done.

Do you have any questions before we begin?

[initial questions for the test subject]

Before we begin the exercise, I’d like to ask you a few quick questions:

What year are you in school?
Approximately how many times have you used the library web site? (sample responses: several times a day, once a day, once a week, once a month, less than once a month)
Can you give me a list of 3-4 things you would expect to find or be able to do on the library’s website?
What type of information or services have you looked for or used on the library web site?
Is there any information or services you have had trouble finding on the library site?

OK, great. We’re done with the questions and we can begin the exercise. Here’s how it works. First, I’m just going to ask you to look at the home page of the test library website and tell me what you think it is, what strikes you about it, and what you think you would click on first.

And again, as much as possible, it will help us if you can try to think out loud so we know what you’re thinking about.

[Test moderator opens browser to test page]

OK. Is there anything that interests you on this page that you might click on?

Before you click, can you tell me what you expect to find when you click on the link?

[after clicking] Did you find what you expected?

[Three main tasks that test subject will complete]

I’m going to ask you to try to complete some tasks using the test library site. Please keep in mind that some of the interior pages of the library site don’t have all the text or links that ultimately will. And as you can see from the library home page, there are some open spaces that we haven’t put content into yet.

[First task; make sure the browser is back to the home page]

OK, beginning at the library home page, pretend that you want to know what the hours are for the library next week. Where would you go to find that information?

[Second task]

Great. OK, now let’s say that you’ve checked out a book that is due back soon. You’d like to extend the loan period. Can you see a way to use the library site to help with that?

[Third task]

Great. Now let’s say that you want to find a textbook titled “Brief Calculus.” Can you see a way to do that?

Thank you so much for your time. Your help today is going to be fed right back into our redesign efforts.

[Test moderator presses CTRL F9 on the laptop keyboard to stop the audio and screen recording]

Please feel free to reuse this script without attribution.

Usability Testing Our New Website

This past week, I’ve been working with with two colleagues from campus IT to run a first round of usability tests on library site redesign. Over the course of three days, we watched ten different undergraduate students perform tasks we had designed in advance (see our testing protocol if you’d like more details on what we did). We used CamStudio to record the screens and to capture audio from a USB mic and relied on one of us from the team to serve as an observer who would take notes during each test using this form.

I’m in the middle of re-reading the observer notes and watching the videos of the screen recordings as I try to write up a report summing what needs to be tweaked in the new site and what seems to be working. In the process of doing the tests this week, I learned a few things that will help us for the next round of usability tests on the redesign:

  • Make sure the testing situation is completely ready for the next test subject before they come in to the room. We asked students to run searches in a catalog search box on the home page. We realized after a while that we should have been clearing the browser cache after each subject was done; we noticed that the previous test subject’s search query was visible in the drop down list below the search box once the next test subject starting typing the same query.
  • Use more stable screen recording software. Although CamStudio offers two great features–it’s free and it works fairly well–it isn’t the most stable software. I think we’re going to want to look into getting Camtasia Studio or Contribute, which I am pretty sure my college has a site license for.

Tonight, I stumbled on a great post by Matthew Reidsma about how he does usability testing at the libraries at Grand Valley State University. He had a great idea of doing monthly tests with just three test subjects. Even more interesting was his way to having colleagues from the library watch the test in a room separate from the one where the test subject is (the test subject’s computer and the computer in the room where the librarians are watching are connected via screensharing software). This sounds like a great way for the staff doing usability testing to get buy in from colleagues about the changes being made. It also seems like it offers great evidence about how our students actually use our sites, thereby curtailing arguments over design issues where everyone is making suggestions based on theoretical ideas about user behavior.

One final note about usability and libraries. There is a new mailing list getting started that will focus on user experience and libraries. Subscribe? Check.

My Summer Projects

Personal Projects

  1. Port over the decade old content from my Teaching Librarian website into a new Drupal version.
  2. Finish editing of an article about McCarthyism and the central library of the New York Public Library. I’m hoping the journal editors will let me post a pre-peer reviewed draft online.
  3. Rework my syllabus for the LIB 1015 class that I taught for the first time this spring. Since my fall class will be part of the college’s learning communities program, my students (all first year students) will also be in a philosophy class (an intro to ethics course). The philosophy professor and I are looking for points of alignment between our courses.

Projects for My Co-workers and Me

  1. Expand my library’s collection of LibGuides.
  2. Work on ways to improve the answering percentage among the six CUNY libraries that share a group subscription to QuestionPoint (answering percentage is ratio that compares the number of questions your library puts into the cooperative service to the number of questions your library answers).
  3. Revise an old plagiarism tutorial.
  4. Write a proposal for a text message reference service.
  5. Rething the process of taking in requests for research consultations and for delegating them to librarians.
  6. Do usability testing of the union catalog shared by CUNY libraries.