Using Qualtrics for Usability Testing

At the marvelously helpful [email protected] event I was at yesterday, I learned about a great way to use survey software (Qualtrics) for usability testing. Since we have the same software here at Baruch College, I spent part of today setting up a few sandbox surveys so that I could try out different question types and get a sense of how survey data would be recorded and displayed. I’ve found three question types so far that look like they’ll be useful. All of them involved uploading screenshots to be part of the question.

Question Type: Heat Maps

Looking at a screenshot, the user gets to click somewhere on the screen in response to some question posed in the survey. The data then gets recorded in a heat map of click data; if you mouse over different parts of the heat map report, you can see how many clicks were done in that one spot. Another way you can set up the screenshot is to predefine regions that you want to name so that the heat map report not only offers the traditional heat map display but also a table below showing all the regions you defined and how many clicked in one of those special regions.

Question Type: Hot Spots

As with the heat map question type, the hot spot question presents the user with a screenshot to click on. But this type of question requires that the person setting up the survey predefine regions on the screenshot. When the test participant is viewing the screenshot, they are are again being asked to click somewhere based on the question being posed. The survey designer can either make those predefined regions have borders that are visible only on mouse over or that are always visible. By making the region borders visible to the test participant, you can draw the participant’s eye to the choices you want him/her to focus on.

Question Type: Multiple Choice

Although multiple choice questions are the most lowly of question types here–no razzle dazzle–it wasn’t until today that I realized how easy it is to upload an image (such as a screenshot) to be part of the answer choice. This seems like a great way to present 2 or more design ideas you are toying with.

Many Uses for a Survey

As a one-person UX group at my library, I find running tests a challenge sometimes if I can’t find a colleague or two to rope into lending a hand with the test. Now I feel like I’ve got a new option for getting feedback, one that can be used in conjunction with a formal usability test or that can be used in lots of different ways:

  • Load the survey in the browser of a tablet and go up to students in the library, the cafeteria, etc. and ask for quick feedback\
  • Bring up the survey at the reference desk at the close of a reference interaction when it seems like the student might be open to helping us out for a minute or two
  • Distribute the survey link through various communication channels we’ve got (library home page, email blast to all students, on a flyer, etc.)

Sample Survey

I made a sample survey here in Qualtrics that you can try out. It’s designed to show off some of the features of questions in Qualtrics, not to answer any real usability questions we currently have here at Baruch. At the close of the session, I set it up so that it offers you a summary of your response (only I can see all the responses aggregated together in a report. It’s likely that when I use Qualtrics surveys for usability, I’ll set them up so they end either by looping back to the first question (useful when I’m going up to people with my iPad in hand and survey loaded in the browser) or by giving them some thank you message. If I get enough responses in this sample survey, I’ll write a new post to show what the report for the survey looks like. In the meanwhile, I’d be interested in hearing from anyone that is already using Qualtrics for usability testing or another survey tool.

First Presentation on Summon

At the CUNY IT Conference last week, I was fortunate enough to be asked to be a panel about discovery services with a bunch of really great folks: Angela Sidman from the CUNY Office of Library Services, Nadaleen Templeman-Kluit from NYU, and Bruce Heterick from JSTOR. My presentation was focused on how our pilot of Summon has been going. This was the first time since we launched Summon in January of this year that I’ve been asked to do a presentation on it. It was really useful to take some time to think about what impact we’ve seen so far and what kind of an impact we hope to see in the coming years.

Here’s the presentation on Google Drive

And here are the notes for the slides:

Slide 1

  • I’m a user experience librarian at Baruch College; do a lot of usability testing of online resources and interface tweaking
  • Mike Waldman couldn’t be here today

Slide 2

  • Like all other CUNY schools, Baruch is a commuter school
  • we have a FTE of about 14,000
  • We’re primarily a business school
  • about 80% of our materials budget is spent on electronic resources (Serials, ebooks, datasets)

Slide 3

  • Like most colleges, Baruch saw the number of databases it subscribed to multiply quickly; reference and instruction required us to tell the students to first go here to search, then go here, then go here, etc.
  • In 2008, we tried to pull access to many of those databases together into a single search screen using a federated search service called 360 Search; we called the tool “Bearcat Search” and added it to our list of databases and gave it a special high visibility location with a large graphic; over the next few years, we found the interface slow, balky, wonky, and high maintenance
  • in 2012, we swapped out our 360 Search subscription for a Summon subscription (both are products from Serials Solutions); we kept the name and placement of the links to the service as before
  • As Angela noted earlier, discovery services like Summon let you add your own local metadata from things like your catalog, your institutional repository, your digital media collections, etc., to the central index provided by the vendor (that central index is pre-populated with a massive collection of records for articles and ebooks)
  • Because this is a Baruch-only pilot project, it didn’t make sense for us to add catalog records for Baruch items, as doing so would require large nightly exports from a catalog server that is shared across the whole CUNY library system
  • One interesting local set of records that we added are our LibGuides

Slide 4

  • before talking about the impact we’ve been seeing from Summon so far, let me just highlight some notable features of it; in general, the search we present is stripped down basic box, as unintimidating as your typical search engine

Slide 5

  • Results are returned very fast in Summon (maybe loading in only 1% of the time it would take a typical 360 Search to load)
  • Let’s take a closer look at the search results page for this search for “cognitive load theory”
  • You can see the articles found from our search here; the full text of these articles may be found in any one of our databases that offers full text, so a search here may lead you to a database from JSTOR, Oxford, EBSCO, ProQuest, Cambridge, Elsevier, etc.

Slide 6

  • One clever thing Summon does is recommend subject specific databases at the top of your search results pages
  • As of a few days ago, we can now tweak the way this database recommender system makes it suggestions
  • For those who worry that a discovery system might eclipse your specialized databases, this feature shows that it can complement and even spotlight resources your students and faculty didn’t even know about in the first place

Slide 7

  • On the left side of every search page is a way to filter by format type (articles, ebooks, etc.)

Slide 8

  • Also on the left is a way to filter by subject
  • One thing that we really like about Summon is speed with which results are returned after a facet is clicked
  • Usability testing I conducted earlier this year surprised me by showing me the opposite of something I’d had long assumed to be true. I’d always thought students ignored the facets and filters on the results page and focused exclusively on the list of results; instead, I saw that student instinctively used the facets to refine the search (no instruction was required!)

Slide 9

  • Another new feature this week is the “did you mean” feature that suggests a new query if it thinks you misspelled something

Slide 10

  • So lets look at that same search for “cognitive load theory” in a very popular database, one that many colleges have long had and that is intended to search across periodicals representing a wide spectrum of subjects: Academic Search Complete
  • Like most libraries, we’ve long gone with presenting the advanced search screen as the default; there is a basic one of course, but many librarians have long assumed that students need the advanced search screen even if those students didn’t know it

Slide 11

  • Summon’s search results page isn’t really that much of a departure from the typical library database
  • Summon’s interface is a bit cleaner, though; it would be interesting to test usage levels for the facets in Summon vs. those in a traditional database like this one
  • Note that in Summon, we found 56,000 items in our search; here in Academic Search Complete, only 206

Slide 12

  • So what are the key ways that Summon is affecting our library? Here is what we know
  • For reasons that are unclear, we’re seeing use of Bearcat Search much higher now that it’s powered by Summon and not 360 Search
  • On a monthly basis, we’re now seeing about 50% more search sessions in Summon than we had in 360 Search, and more than 200% searches being run
  • the speedier delivery of results in Summon mean users are more likely to do the kind of iterative searching they are used to doing in Google (average number of searches per session is 5 compared to 2 in 360 Search)
  • The redesign of our library that we are launching at the end of this year will feature a search box dead center on the home page and at the top of every internal page; we expect our stats will really explode after that

Slide 13

  • So we see the raw numbers going up but we don’t know yet who is using it and why
  • We hope that Summon will increase other things for us, too
  • Given the ease of using this tool, it serves underclassmen well and may make a better candidate for a database to use when teaching 1st year students how to search
  • Because the index in Summon is so huge and includes records in databases that we know rarely get used, it’s hoped that it’s leading students to e-content that had previously been little used
  • We also hope that the database recommender feature may be yet another way that we try to steer our students to specialized databases that they typically only think to use when a teacher or a friend recommends it
  • And finally, we hope that student satisfaction will go up as they find a tool that is easier and more pleasant to use that still taps deep in relevant content they need for their assignments

As I was digging into the statistics a bit while preparing my presentation, I realized I had a number of questions that I’d like to find answers for:

  • Do students use facets on the search results page more or less than they do in Summon? In my usability testing of Summon this spring, I was surprised by how often and easily students used the facets without any prompting from me. If they use facets more often in Summon than in other databases, why is that the case?
  • How can we find out if Summon is driving up access for full text journals that had previously been underutilized because the only way to find them previously was to use lesser known databases?
  • Do students find searching in Summon more or less satisfying than searching in our traditional article databases?
  • Is there a better way to present the recommended databases that frequently appear at the top of the search results pages?  Do students actually see these recommendations? What do they think of them? How often and when will they actually click through to the recommended database?
  • How do students feel that information from many of our business databases that feature specialized reports and data about companies, industries, etc. are unlikely to ever appear in search results pages of Summon (except as recommended databases)? If they are searching in Summon for data that is only found in specialized databases, are they more likely to give up and try their luck in Google or will they ask for help or see what other databases/tools we offer?

It looks like I could fill up the rest of my professional career as an academic librarian trying to answer all these questions. No time to get started like the present.