At the CUNY IT Conference last week, I was fortunate enough to be asked to be a panel about discovery services with a bunch of really great folks: Angela Sidman from the CUNY Office of Library Services, Nadaleen Templeman-Kluit from NYU, and Bruce Heterick from JSTOR. My presentation was focused on how our pilot of Summon has been going. This was the first time since we launched Summon in January of this year that I’ve been asked to do a presentation on it. It was really useful to take some time to think about what impact we’ve seen so far and what kind of an impact we hope to see in the coming years.

Here’s the presentation on Google Drive

And here are the notes for the slides:

Slide 1

• I’m a user experience librarian at Baruch College; do a lot of usability testing of online resources and interface tweaking
• Mike Waldman couldn’t be here today

Slide 2

• Like all other CUNY schools, Baruch is a commuter school
• we have a FTE of about 14,000
• We’re primarily a business school
• about 80% of our materials budget is spent on electronic resources (Serials, ebooks, datasets)

Slide 3

• Like most colleges, Baruch saw the number of databases it subscribed to multiply quickly; reference and instruction required us to tell the students to first go here to search, then go here, then go here, etc.
• In 2008, we tried to pull access to many of those databases together into a single search screen using a federated search service called 360 Search; we called the tool “Bearcat Search” and added it to our list of databases and gave it a special high visibility location with a large graphic; over the next few years, we found the interface slow, balky, wonky, and high maintenance
• in 2012, we swapped out our 360 Search subscription for a Summon subscription (both are products from Serials Solutions); we kept the name and placement of the links to the service as before
• As Angela noted earlier, discovery services like Summon let you add your own local metadata from things like your catalog, your institutional repository, your digital media collections, etc., to the central index provided by the vendor (that central index is pre-populated with a massive collection of records for articles and ebooks)
• Because this is a Baruch-only pilot project, it didn’t make sense for us to add catalog records for Baruch items, as doing so would require large nightly exports from a catalog server that is shared across the whole CUNY library system
• One interesting local set of records that we added are our LibGuides

Slide 4

• before talking about the impact we’ve been seeing from Summon so far, let me just highlight some notable features of it; in general, the search we present is stripped down basic box, as unintimidating as your typical search engine

Slide 5

• Results are returned very fast in Summon (maybe loading in only 1% of the time it would take a typical 360 Search to load)
• Let’s take a closer look at the search results page for this search for “cognitive load theory”
• You can see the articles found from our search here; the full text of these articles may be found in any one of our databases that offers full text, so a search here may lead you to a database from JSTOR, Oxford, EBSCO, ProQuest, Cambridge, Elsevier, etc.

Slide 6

• One clever thing Summon does is recommend subject specific databases at the top of your search results pages
• As of a few days ago, we can now tweak the way this database recommender system makes it suggestions
• For those who worry that a discovery system might eclipse your specialized databases, this feature shows that it can complement and even spotlight resources your students and faculty didn’t even know about in the first place

Slide 7

• On the left side of every search page is a way to filter by format type (articles, ebooks, etc.)

Slide 8

• Also on the left is a way to filter by subject
• One thing that we really like about Summon is speed with which results are returned after a facet is clicked
• Usability testing I conducted earlier this year surprised me by showing me the opposite of something I’d had long assumed to be true. I’d always thought students ignored the facets and filters on the results page and focused exclusively on the list of results; instead, I saw that student instinctively used the facets to refine the search (no instruction was required!)

Slide 9

• Another new feature this week is the “did you mean” feature that suggests a new query if it thinks you misspelled something

Slide 10

• So lets look at that same search for “cognitive load theory” in a very popular database, one that many colleges have long had and that is intended to search across periodicals representing a wide spectrum of subjects: Academic Search Complete
• Like most libraries, we’ve long gone with presenting the advanced search screen as the default; there is a basic one of course, but many librarians have long assumed that students need the advanced search screen even if those students didn’t know it

Slide 11

• Summon’s search results page isn’t really that much of a departure from the typical library database
• Summon’s interface is a bit cleaner, though; it would be interesting to test usage levels for the facets in Summon vs. those in a traditional database like this one
• Note that in Summon, we found 56,000 items in our search; here in Academic Search Complete, only 206

Slide 12

• So what are the key ways that Summon is affecting our library? Here is what we know
• For reasons that are unclear, we’re seeing use of Bearcat Search much higher now that it’s powered by Summon and not 360 Search
• On a monthly basis, we’re now seeing about 50% more search sessions in Summon than we had in 360 Search, and more than 200% searches being run
• the speedier delivery of results in Summon mean users are more likely to do the kind of iterative searching they are used to doing in Google (average number of searches per session is 5 compared to 2 in 360 Search)
• The redesign of our library that we are launching at the end of this year will feature a search box dead center on the home page and at the top of every internal page; we expect our stats will really explode after that

Slide 13

• So we see the raw numbers going up but we don’t know yet who is using it and why
• We hope that Summon will increase other things for us, too
• Given the ease of using this tool, it serves underclassmen well and may make a better candidate for a database to use when teaching 1st year students how to search
• Because the index in Summon is so huge and includes records in databases that we know rarely get used, it’s hoped that it’s leading students to e-content that had previously been little used
• We also hope that the database recommender feature may be yet another way that we try to steer our students to specialized databases that they typically only think to use when a teacher or a friend recommends it
• And finally, we hope that student satisfaction will go up as they find a tool that is easier and more pleasant to use that still taps deep in relevant content they need for their assignments

As I was digging into the statistics a bit while preparing my presentation, I realized I had a number of questions that I’d like to find answers for:

• Do students use facets on the search results page more or less than they do in Summon? In my usability testing of Summon this spring, I was surprised by how often and easily students used the facets without any prompting from me. If they use facets more often in Summon than in other databases, why is that the case?
• How can we find out if Summon is driving up access for full text journals that had previously been underutilized because the only way to find them previously was to use lesser known databases?
• Do students find searching in Summon more or less satisfying than searching in our traditional article databases?
• Is there a better way to present the recommended databases that frequently appear at the top of the search results pages?  Do students actually see these recommendations? What do they think of them? How often and when will they actually click through to the recommended database?
• How do students feel that information from many of our business databases that feature specialized reports and data about companies, industries, etc. are unlikely to ever appear in search results pages of Summon (except as recommended databases)? If they are searching in Summon for data that is only found in specialized databases, are they more likely to give up and try their luck in Google or will they ask for help or see what other databases/tools we offer?

It looks like I could fill up the rest of my professional career as an academic librarian trying to answer all these questions. No time to get started like the present.