Research: Delving the Deep Web

How much serious research can you accomplish on the Web? probably much more than results from major search engines like Google and Yahoo might lead you to expect. Net savvy librarians are working hard to make more “deep” information resources available through common Web searches.

Check out today’s New York Times: Old Search Engine, the Library, Tries to Fit Into a Google World, by Katie Hafner.

Excerpt: “For the last few years, librarians have increasingly seen people use online search sites not to supplement research libraries but to replace them. Yet only recently have librarians stopped lamenting the trend and started working to close the gap between traditional scholarly research and the incomplete, often random results of a Google search.”

I bump up against this issue quite often in my work as a journalist and all-around information and communication junkie. I used to haunt libraries, but now I usually only go to a library to consult or borrow a particular resource which I have already confirmed online or by phone exists at that library. I really hadn’t thought about that shift in my habits much until I read that article, but I can see the implications. Here are a few observations…

  1. When libraries don’t get visited and used as much, their funding tends to decrease. This is often but not always true. Municipal governments and educational or research institutions tend to have limited funds which they must distribute wisely among their facilities and servicies, including libraries. Popularity definitely counts in these funding decisions.

  2. Library fishing expeditions just don’t seem to be as rewarding as they once were. This is my personal opinion only, but I used to go to libraries and spend hours leaping from one resource to the next, following a breadcrumb trail of cross-references to intriguing discoveries. The last few times I did that at various libraries, I was disappointed. Periodical collections were notably thinner, with many missing or misfiled issues. Staff seemed less available or able to help. Reference books had gone missing. In some cases I’m sure that thoughtless patrons were as much to blame as budget cuts and unskilled staff. Still the result was that I am now much more inclined to kill time online than in a library.

  3. To survive, libraries must adapt to evolving research habits. The advent of the Internet has definitely changed the way most in-depth research is handled in many parts of the world. Most people – even scholars – have limited time and resources for conducting research. Therefore, understandably, they will use the mose convenient and time-efficient methods first. If the results of those methods satisfy their core needs, it’s unlikely these researchers will visit a library in order to delve further. Unfortunately, this means that many rich and vital resources that aren’t easily found through ordinary Web searches are more likely to get overlooked, regardless of significance or relevance.

Google doesn’t show you everything that’s online. Many people, including myself, forget this point from time to time. The more levels a site or resource contains, the more it relies on dynamically-generated pages, and the more complex the structure – the more likely it is that the most popular search engines will miss indexing much of its in-depth content.

The NYT article covers the efforts of many libraries and other institutions to work with the major search engines to make more of this deep, rich content accessible to the average Web user.

…All of this is great. However, I think one missing but important part of this discussion is the role of the publishers. It seems to me that publishers of books, textbooks, and other resources that get purchased by libraries (whether print, electronic, audio, whatever) could be doing a lot more to coordinate and cooperate in terms of metadata that would allow their materials to be indexed more effectively online. (More on that later…)


In that NYT article, Kate Wittenberg (director of Columbia University’s Electronic Publishing Initiative”) is quoted as saying: “We can’t pretend people will go back to walking into a library and talking to a reference librarian.”

True. In fact, according to the article, a new three-year EPIC study of research habits, which included surveys of 1,233 US students, concluded that electronic resources have become the main tool for information gathering, particularly among undergraduates. The study also surveyed faculty and conducted a focus groups for librarians. The results are well worth reading. Here are some highlights:

  • When going online to do work for a course, 46.5% of students reported being more likely to use an Internet search engine than a library-sponsored electronic resource (21.9%). However, this pattern depend upon the level of research being done. For general assignments, students are more likely to turn to the Internet. In contrast, for in-depth research assignments students are somewhat more likely to turn to library sponsored electronic resources.

  • Undergraduate students are more likely than graduate students to use non-library sponsored electronic resources, while graduate students are more likely to use library sponsored electronic and library sponsored print resources.

  • 31.5% of students surveyed learn about school-related electronic resources primarily through their library website (and another 27.1% get that information through their professors). Only 13.7% of students reported using Internet search engines to find academic electronic databases.

  • Many students reported that one or more of their professors use course management programs (such as Blackboard), and almost a third of the students report that at least one of their classes use online materials in place of textbooks.

  • Books and journals are still cited by most students when writing a term paper, however the number of students citing websites does not lag far behind. To put this in context, I’ve heard that some schools have policies forbidding students to cite Web references in their papers.

  • 40% of faculty respondents somewhat or strongly agreed that they would rather settle for what they can find online rather than making a trip to the library.

  • 61% of faculty reported that credibility/reliability is a key concern with online research resources. About half of faculty reported difficulties in assessing online source credibility.

  • Nearly all (97%) faculty respondents believe electronic resources have enhanced the quality of their research. Similarly, 93% believe electronic resources have enhanced the quality of their teaching. However, most faculty respondents reported concern over students’ difficulty in evaluating the quality of content of online information. Further, they expressed concern over students’ tendency to rely on online information too much, foregoing information in print form – which I find somewhat ironic, given the increasing faculty preference for electronic resources at the expense of print.
  • In the focus group, librarians confirmed that library users increasingly demand electronic resources. In most cases, when there is a choice, library users overwhelmingly prefer electronic resources to print. Some undergraduates use electronic sources exclusively. Relative ease of use, availability at all hours, and ease of repurposing text drive this demand.

  • Librarians reported these disadvantages of electronic resources: stressful new job responsibilities, the need to keep up with new electronic products, loss of control over the titles in the library’s collection, and loss of control over organized instruction of library materials.

  • Libraries often use multiple methods for alerting users of available resources. The most common include notification through e-mail and word of mouth. An academic library’s web site may also be an effective medium for alerting students and faculty. Interestingly, webfeeds (RSS or Atom) format were not mentioned here – but I wouldn’t be surprised if savvy libraries were exploring this communication channel as well.

Recommended Resource: If you’re interested in learning more about how libraries and librarians are adapting to the online age, read The Shifted Librarian, an excellent weblog by Jenny Levine.