Project Ethnography: An Anthropological Approach to Assessing Digital Library Services

OFTENLIBRARIES TRY TO ASSESS DIGITAL LIBRARY SERVICE for their user populations in comprehensive terms thatjudge its overall success or failure. This article’s key assumption is that the people involved must be understood before services can be assessed, especially if evaluators and developers intend to improve a digital library product. Its argument is simply that anthropology can provide the initial understanding, the intellectual basis, on which informed choices about sample population, survey design, or focus group selection can reasonably be made. As an example, this article analyzes the National Gallery of the Spoken Word (NGSW). It includes brief descriptions of nine NGSW micro-cultures and three pairs of dichotomies within these micro-cultures. INTRODUCTION Questions rained down, and continue to rain down, . . . . Questions about the coherence of life-ways, the degree to which they form connected wholes. Questions about their homogeneity, the degree to which everyone in a tribe, or even a family (to say nothing of a nation or civilization) shares similar beliefs, practices, habits, feelings. Questions about discreteness, the possibility of specifying where one culture, say the Hispanic, leaves off, and the next, say the Amerindian, begins. (Geertz, 1995,pp. 42-43) Geertz’s words are relevant here. Often libraries try to assess digital library service in comprehensive terms thatjudge its overall success or failure for their user populations. A variety of methods are used: surveys, usage statistics, standards, and occasionally even focus groups. All of these Michael Seadle, 100 Library, Michigan State University, East Lansing, MI 48824-1048 LIBRARY TRENDS, Vol. 49, No. 2, Fall 2000, pp. 370-385 02001 The Board of Trustees, University of Illinois SEADLE/PROJECT ETHNOGRAPHY 371 methods have their virtues and can offer valuable information as part of an assessment process, but misuse is common. It is tempting to assume coherence and homogeneity among the many populations that use any complex digital library, or at least to assume discrete boundaries between certain populations. Students, for example, from a population that could include people from high school through graduate school and English majors to engineers. A bright high school student may also be taking college classes, and an undergraduate could well have an English-engineering double major. It is also tempting to assume that the developers of a complex digital library project have agreed on common service goals rather than separate (though, it is hoped, symbiotic) priorities. A service goal of making material accessible might, for example, mean a search algorithm to one of the developers, index structure to another, and subject categories to a third. Treating them as indistinguishable parts of a common product loses key information which would help to identify problems and improve services. This article does not offer anthropology as a substitute methodology for evaluating digital library services. Anthropology as practiced today seeks to be relatively nonjudgmental, even in a good cause. It tries instead to observe accurately and to lay out the dynamics of interactions in ways that explain situations and behaviors. This article’s key assumption is that the people involved must be understood before services can be assessed. Its argument is simply that anthropology can provide the initial understanding, the intellectual basis, on which informed choices about sample population, survey design, or focus group selection can reasonably be made. It offers a first step, but one which, if ignored, can trip the most sophisticated evaluation scheme. METHODOLOGY The standard method for research in cultural anthropology is to find a set of people, learn their language and everything else known about them, and then live with them long enough to come away with new insight and understanding. At one time, the people tended to come from remote tribes, like Margaret Mead’s (1932) Samoans. Later the pool grew to include ex-colonial territories-people from complex, but non-western, civilizations, such as Clifford Geertz’s (1956) Indonesians. More recently, cultural anthropologists have taken an active interest in aspects of contemporary western society such as John Borneman’s (1992) Germans or Bonnie Nardi’s (1999) corporate librarians. The methodology for this article follows a similar pattern. Language is a particularly important aspect of the methodology, even when an English speaker is dealing with other English-speaking Americans. Words do not always convey a simple dictionary meaning, especially across cultural and disciplinary boundaries. “Research,” for example, means 372 LIBRARY ?’RENDS/FALL 2000 something with books, articles, archives, and footnotes to the historian, but generally implies mathematics, experimentation, hypothesis, and results to an engineer. The same word conjures up different approaches and different products to differently trained people. The nuances of meaning matter, especially in understanding what the real service goals of a project are. For a study like this, the language training consists not of foreign words but of foreign concepts, acronyms, even symbols (e.g., c). The human subjects in this case are university people, librarians, historians, engineers, education faculty, computer professionals, and others. Since I am a trained historian, a computer professional, and a librarian, I understand the language, the specialized words, the acronyms, and the implicit meanings of three of the subgroups. This is crucial in being able to describe their interests and intentions faithfully. I am also aware of how poorly I understand the meanings of, for example, the engineers, whose mathematical discourse far exceeds my last meager courses in calculus over thirty years ago. The participant-observer must play two roles simultaneously. It is not always easy. One example of this problem comes from Frank Hamilton Cushing. In 1879, that influential ethnologist went to live among the Zuni and became so completely one of them that he participated in their secret rituals. Ultimately, he became a “BowPriest,” and destroyed many of his notes rather than betray Zuni secrets (Schoumantoff, 1999, pp. 14344). Where does the participant leave off and the observer begin? There is no simple answer. Geertz (1995) writes: “It is a matter of living out your existence in two stories at once” (p. 94). Use of the first person in anthropological articles has always been fairly common. It reminds readers of the filter through which they are viewing the world. The references in the body of this article come mainly from my notes, my memory, my records of conversations. I deliberately avoid naming individuals and quoting conversations as I have normally done when writing oral histories because of the ongoing, active, ever-delicate work relationships that could easily be damaged. The rules of evidence in anthropology lack precision compared to some other social sciences. Clifford Geertz in particular has thought about this issue: The ability of anthropologists to get us to take what they say seriously has less to do with either a factual look or an air of conceptual elegance than it has with their capacity to convince us that what they say is a result of their having actually penetrated (or, if you prefer, been penetrated by) another form of life; of having, one way or another, truly “been there.” (Geertz, 1988,pp. 45) . . . .Such, such are the facts. Or, anyway so I say. The doubts that arise, whether in me or my audience, have only very partially to do with the empirical basis on which these accounts, or others like them, rest. The canons of anthropological “proof‘ being what they are SEADLE/PROJECT ETHNOGRAPHY 373 (mimicries of sterner enterprises like mechanics or physiology) that is, indeed, how such doubts are most often phrased and, to the degree they are, most often quieted. Footnotes help, verbatim texts help even more, detail impresses, numbers normally carry the day. But, in anthropology anyway, they remain somehow ancillary: necessary of course, but insufficient, not quite to the point. The problem-rightness, warrant; objectivity, truth-lies elsewhere, rather less accessible to dexterities of method. (Geertz, 1995,pp. 17-18) The examples cited in this article are not verifiable except to the limited extent that a few other people heard the same words (though not necessarily the same meanings). This imprecision may bother some readers, but it lies at the core of the evaluation issue. The most careful survey, the most rigorous statistical test, depends ultimately on the meanings that the words, categories, even the numbers, convey. The worst evaluation disasters occur when the respondent does not understand (or misunderstands) the questions, or when the people responsible for the service being evaluated throw the results into the life-imprisonment of a deep file drawer because the results miss all the issues they could use to make improvements. ABOUTCULTURES AND MICRO-CULTURES Culture represents a nexus of shared meaning. It can be used in a broad sense to refer to “western” or “Asian” culture or more narrowly to refer to “German” or “American” culture or still more narrowly to refer to “midwestern” or “Afro-American” culture. The number of possible distinctions has no obvious limit. The culture of a nuclear family can, in fact, differ from its neighbor: different holiday traditions, different vacation preferences, even private words loaded with special meaning (sometimes understood by the spouses alone). In this article, the word micro-culturerefers to units of shared meaning as small as professions, departments, and interest groups. The reason for this specialized word, instead of more standard descriptions, is that it evokes the range of anthropological discourse which, for all its flaws and imprecision, offers a theoretical framework for analyzing the social processes involved in service evaluation. Within the culture of an academic institution, this variety of microcultures is easily recognized. The boundaries have some sharp edges, but more that are indistinct. How would one classify a woman who, for example, is working on a library degree (and therefore is a graduate student), but has a doctorate in the history of science (therefore belongs both to the natural sciences and the humanities), and is concurrently teaching as an adjunct at a nearby peer institution (and therefore counts as faculty)? Such a person belongs to multiple micro-cultures and may react to a particular digital library service from any one of these identities or from another equally important identity altogether. 374 LIBRARY TRENDS/FALL 2000 Most people are, in fact, blends of micro-cultures. One of the important factors in establishing an evaluation system is to understand which cultures and micro-cultures matter. That the “end user” matters most is a common gut reaction, but it may not be true. For a grant-funded project, for example, the real value (in monetary terms) of the service may depend entirely on how the funding agency’s project officer perceives the work. That one person can make or break the next year’s allocation. The planned end-user may also differ from the actual end-user. My own office, the Digital Sources Center at Michigan State University, made some public-domain Ku Klux Klan pamphlets available on the Web for classes in American radicalism. Later we found that the Klan itself linked to these materials (P. Berg, personal communication, April 17, 1998).A user-based survey about this service could well involve enough Klan members to give unexpected results. They might, for example, find links to Black Panther pamphlets within the same collection offensive and ask that they be removed. For reasons which have to do with our own cultural values, we would not do so. We would also not knowingly include Klan representatives in a focus group for evaluating that particular digital library service, even if such a person were a student on our campus. Our reaction to members of that particular micro-culture affects our opinion of their responses so greatly that their evaluation of the service becomes irrelevant, and we exclude them intentionally from our definition of the end-user population, even though they actually use the materials. In the modern workplace, the micro-cultures interact more than they did in traditional hierarchical corporate, or even academic, organizations. In a recent article, Bonnie Nardi and her co-authors argue: that it is increasingly common for workers to replace the organizational backdrop and predetermined roles of old style corporate working with their own personal assemblages of people who come together to collaborate for short or long periods. These assemblages are recruited to meet the needs of the current particular work project. (Nardi, Whittaker, & Schwarz, 2000) Most of the second round of Digital Library Initiative grants, and many of the first, involve collaborations that cut across traditional fields-the old style organizational backdrop for corporate academe. Library and computing (or engineering) partnerships are particularly common. Examples can be seen in “Project Prism” at Cornell (http://www.prism.cornell.edu/), “Emulation Options for Digital Preservation” at the University of Michigan and University of Leeds, and the “National Gallery of the Spoken Word” (NGSW) at Michigan State University (www.ngsw.org). Most of the examples in this article will come from the NGSW, whose four-way partnership includes the university library, the College of Engineering, the College of Education, and MATRIX (the “Center for the Humane Arts, Letters, and Social Sciences Online”) in the College of Arts and Letters.’ SEADLE/PROJECT ETHNOGRAPHY 375 What follows is an ethnographic look at the culture of the NGSW and its many micro-cultures. The argument is simply that examining these is essential to understanding the developers’ intentions about both services and end-user populations. Special emphasis will be put on what these concepts mean for each of the micro-cultures involved. THENGSW WORLD Origins matter in defining a culture, and the origins of the NGSW lie as shrouded in myth as any ancient cult, even though the project is a scant two years old. At the official kick-off ceremony with an audience of provosts, deans, and visiting dignitaries, one of the co-principal investigators told the story of how a friend approached her at church and proposed a partnership to go after the grant. Another remembered its origin as stemming from a conversation with the vice-provost who recommended, as only a vice-provost can, a partnership with the Computer Science Department, which in its turn made a link to a signal-processing engineer. Yet another co-principal investigator suggested that the true origin dates from his vision to use materials from the Vincent Voice Library (WL) years before the grant was written. Friendship, institutional ties, and vision each played a role in the project’s origins, and each tends to define the project’s nature, purpose, and measures of success in a somewhat different way. The text of the original grant proposal could, and perhaps in theory should, provide a common basis for evaluation, but the text has thus far had little value as a common reference point. The co-principal investigators rarely refer to it or quote from it in their discussions, except occasionally regarding financial matters. An exception occurred during a discussion of applying Bayesian statistics to indexing. One person insisted that something like that had been included in the proposal, but a search of the digital copy of the final draft produced no references to Bayes or Bayesian. The reference probably existed in some version of the text but not the final one. The proposal text fails to provide a unifymg set of principles, in part because no individual could write the proposal as a whole. No person’s expertise ranged sufficiently widely to encompass all of the engineering, computing, library, and educational issues. The first draft was a simple composite of uneven texts from each of the co-principal investigators. One saw it as a prototype, a throw-away version to get some of the basic ideas on paper, another reacted in horror at it as amateurish and disorganized. The rewrite blended parts with a heavy hand. One author worked on it remotely by e-mail from Australia, another tried to talk through the ideas before writing them down. The process bogged down so badly that the group went to the vice-provost the Friday before the proposal was due to talk about waiting for the next round. The library director made the key comment: what is there to lose? Even if the text is bad, others might be 376 LIBRARY TRENDS/FALL 2000 worse. The submission went forward, butwith a strong sense that the project description was seriously flawed. Those who have worked in a complex grant culture will recognize both the Yariety of origin myths and the disdain for the defining text as ordinary, even predictable. A genuine partnership, with no single commanding leader, and participants with national standing in each of their disciplines, cannot work out the nuances of vocabulary and priority for a five-year project in the few months of intermittent effort prior to submission. Some modest dissonance may in fact represent the freshness of ideas and vigor of thought that were the true reasons for funding in the first place: a healthy red-cheeked bloom of active intellectual engagement. Unfortunately, this does not help to determine service objectives. Evaluation is an explicit part of the NGSW proposal, and an external evaluator has a budget line in the subcontracts section (National Gallery OJ the Spoken Word, 1998). Evaluation received little discussion during the proposal-writing period. It was almost an afterthought, a last-minute addition by those accustomed to the NEH requirements for project evaluation for teaching-related projects. The evaluator is himself a statistician who has strong ties to one (and only one) of the four co-principal investigators and brings substantial experience with educational but not engineering or library settings. He has met with the whole group only once, at the very start, and mainly discussed indexing schemes, not user populations. The general sense is that his work will come mainly at the end, though some murmurings of concern about how to define who and what gets evaluated have surfaced at the edges of meetings, those important periods just before and just after the formal agenda when friend collars friend and seatmate turns to seat-mate. Bonnie Nardi’s “intensional” (a combination of “intentional” and “tension”) networks have been discussing it, even if the project team as a whole has not (Nardi, Whittaker, & Schwarz, 2000). NGSW MICRO-CULTURES A cultural map of the NGSW world is difficult to draw. The first temptation is to accept the institutional boundaries as if they represent the borders between say, France, Germany, Belgium, and Holland. It is too simplistic but cannot entirely be discounted. Such a map would contain the following “lands.” Library The library’s collections are in the top quarter of the Association of Research Libraries in terms of sheer number of volumes. The number of professional staff, however, lies in the second quarter and ranks near the bottom of the fourth quarter in terms of the ratio of professional staff to full-time students (Association of Research Libraries, 2000).Although these figures suggest understaffing, the library has the advantage of an enerSEADLE/ PROJECT ETHNOGRAPHY 377 getic new director whose willingness to put resources into information technology has resulted in the establishment of aDigital Sources Center, plus a growing number of grant proposals and grant-funded projects. The library has significantly increased the number of Ph.D.s on staff in recent years and has recruited vigorously from the best library and information schools in the country. The new recruits are not necessarily young. They bring outside life and work experience and choose to work at Michigan State in part because they perceive it as (and it sells itself as) a place of opportunity. The librarians have faculty status. They are expected to publish. There is a strong sense that the local culture has changed. The three librarians directly involved with the NGSW at present all have doctorates in history, and all have their library degrees from the same school (University of Michigan), though from somewhat varying eras and specialties. The library has had two explicit goals for the NGSW. The first is to preserve its large collection of reel-to-reel tapes of speeches, oral histories, and other forms of spoken-word recordings. Since the collection is estimated to have as many as 50,000 hours of sound, this could be no small task. The second explicit goal is to bring those materials under better bibliographic control. Only a small portion of the works have been cataloged in MARC, using an approach that treated each segment as if it were a separate monograph. Another implicit goal is to establish copyright rules and do a systematic check of the collection since the copyright rules for sound are complex and not always well understood. The library’s understanding of NGSW service goals focuses mainly on access and preservation. Its traditional user populations include the broadest possible range of students, graduate students, faculty, and potentially all Michigan citizens as part of the university’s explicit “land grant philosophy.” The library is also concerned with standards setting, both for the preservation and the bibliographic control issues, which makes librarians at other institutions another user population for the NGSW work.