Why study users?
First Monday

Why study users? An
environmental scan of use and users of digital resources in humanities and
social sciences undergraduate education by Diane Harley



Abstract
This article presents an overview of a two-year study [1] that (1) mapped the universe of digital resources available to undergraduate educators in the humanities and social sciences (H/SS); and, (2) examined how a better understanding of the variation in use and users can benefit the integration of these resources into undergraduate teaching. Our results suggest that faculty use a vast array of online materials from both educational and “non-educational” sources, but many do not use digital resources for a host of reasons including the lack of direct relevance to their preferred pedagogical approaches, and insufficient time and classroom resources. Our discussions with digital resource providers confirmed that an understanding of the actual use of their resources in undergraduate settings is often murky. These discussions also made clear that resources created by higher education institutions will continue to proliferate despite a lack of formal knowledge about users and/or clear models for financial sustainability. A more precise understanding of the diversity of use and user behavior, and the ability to share findings from user studies, will require that the digital resource development community make typologies, standards of data and data collection, and results more transparent.

Contents

Introduction
Overview of methods
Results
Conclusions

 


 

Introduction

A “build it and they will come” approach to many university digitization initiatives, including open content initiatives, has precluded systematic investigations of the actual demand for these resources. Those who fund and develop digital resources have identified the general lack of knowledge about the level and quality of their use in educational settings as concerns. To address these issues, we conducted research over approximately 24 months that (1) mapped the universe of digital resources available to undergraduate educators in a subset of disciplines in the humanities and social sciences (H/SS); (2) surveyed faculty about their use of digital resources; and, (3) examined how understanding use and users can benefit the integration of these resources into undergraduate teaching. This article provides an overview of our findings. The entire report and associated data sets are available online (Harley, et al., 2006).

Background and rationale

The question, “Why study users?” was at the heart of our investigations. There are, in fact, myriad reasons cited for undertaking and conducting user studies. They range from product design and usability testing, to surveillance of user activity on Web sites, to facilitating policy and investment decisions. For our purposes, there were three interrelated rationales for conducting the present research, each of which is described more fully below:

  1. Addressing questions of strategic planning and investments in digital resource provision and use;
  2. Identifying the special needs of the humanities and social sciences, particularly as they relate to the future of liberal education in a digital age; and,
  3. Sharing effective strategies for understanding the array of uses and users across a wide variety of educational digital resource initiatives.

Strategic planning and investments

Strategic investment decisions by funders and institutions will undoubtedly hang on the question of how to pay for the significant costs of digital resource production and maintenance. The question of cost becomes more pressing in an era of shrinking institutional budgets and deflated expectations of profitable consumer markets for digital curricular materials (Matkin, 2002). We know very little, however, about how digital resources, such as those produced at research universities, are actually being used by the different tiers of higher education institutions both in the U.S. and abroad. There is an implicit assumption that faculty at a variety of institutions import digital content to enhance their undergraduate teaching, but we simply do not know if such importation occurs on a measurable scale. And if it does not occur, then why not? This question is of particular importance in California, where there is a presumption that digital assets will flow from the public University of Caifornia research university system to institutions with fewer resources, such as high schools and community colleges.

There is an implicit assumption that faculty at a variety of institutions import digital content to enhance their undergraduate teaching, but we simply do not know if such importation occurs on a measurable scale.

Determining the costs to institutions in creating and maintaining these digital assets is perhaps impossible given the mosaic of development and funding models that exist for any one set of assets. Funding sources are often cobbled together from a variety of institution and foundation budgets, and there are the frequently unpredictable, ongoing costs of maintenance and updating. Moreover, what of the significant costs incurred by digital resource developers to simultaneously meet the needs of audiences that range from scholars to school children, both internationally and domestically? In addition to institution-sponsored resources, there is the growing mass of “educational,” digitized rich media objects created by individual scholars and others. What is their value, and who will maintain and preserve them (Smith, 2003)? As decisions are made about financing resource design and scope, an understanding of the level and type of use of these, as well as more “organized” resources, will be needed in strategic planning.

Focus on humanities and social sciences

Within the academy, there is an abundance of good models for integrating online materials in science and technical courses such as chemistry, physics, biology, and computer science (see, for example, Twigg, 2003; Fisher and Nygren, 2000). The application of technical solutions to undergraduate teaching in the humanities and social sciences (H/SS), however, has been more elusive and less robustly funded. Indeed there appears to be a paucity of literature that has systematically examined these issues, especially as they relate to the integration of non-text, non-library resources that are valued frequently by faculty in the H/SS [2].

The humanities and social sciences are of course not a monolith, nor are user types. We contend that a disaggregation of users by discipline and institution type allow us to better understand the existing variation in user and non-user behavior. The American Council of Learned Societies (ACLS) Commission on Cyberinfrastructure for the Humanities and Social Sciences lays out the “grand challenge” of harnessing the potential of information and communication networks to serve the special and diverse needs of these scholars (ACLS, 2006). Understanding the technology needs of the H/SS community has particular relevance to the future of liberal arts or general education delivery and the increasing “vocationalization” of higher education (Rothblatt, 2003; Smelser and Schudsen, 2004).

Consolidation of effective strategies for understanding use

To date, there has been no coordinated conversation about user research that could apply across the many types of available digital resources and their sources. This is due in part to the immense variety of digital resources found on the Internet. Nationally and internationally, resources range from ambitious attempts to put up course Web pages or whole courses, to discipline-specific course materials, to clearinghouses of individual learning objects, to digital library/museum collections, to collections assembled by individual scholars. The available studies about the users of this vast array of resources are themselves fragmented by purpose, method, and context.

Lack of a clear picture about users of these resources makes coordination of user studies (methods, findings, business models, strategic planning) across projects challenging. What is the overall value of “user” studies? How can we begin to assess overall user demand, and what analytic methods are useful for the various phases of decision-making (e.g., start-up, site design, dissemination, maintenance, scaling, new audiences)? For example, usability studies or testing of pedagogical applications in the classroom are clearly useful for site and content design, but they will not yield the kinds of data needed to make decisions about initiating a new project, developing funding models, or assessing/targeting new audiences. There are a number of very good usability studies. Unfortunately, they tell us only about relatively enthusiastic users of a particular brand of content, but nothing about whether a brand may be valued or useable by a wider potential audience operating in varied educational contexts.

 

++++++++++

Overview of methods

Research proceeded on three parallel tracks. We used a suite of methods in our sampling of relevant populations: digital resource owners, faculty in three types of higher education institutions (California public research universities, liberal arts colleges, and community colleges), librarians, and other users. Our choice of methods was determined by the nature of our goals, which were to provide (1) a relatively quick scan of use across a wide range of unrestricted digital resources by a variety of user types; and, (2) a possible model that could yield comparable data across a variety of digital resources. Given these goals, we employed a combination of surveys, discussion groups, and interviews to get a broad look at both user and resource provider behaviors. Our definition of digital resources was intentionally broad and included rich media objects (e.g., maps, video, images, etc.) as well as text. These digital resources may reside in or outside of digital libraries and include those developed by individual scholars and by non-academic entities [3].

 

++++++++++

Results

The work summarized here is presented in three sections:

  1. Understanding the humanities and social science digital resource landscape: a literature review and discussions with various stakeholders to provide a scan of the digital resource universe, and where the user fits into that universe;
  2. Gaining a faculty perspective on use and non-use of digital resources: discussions with and surveys of faculty at three types of California higher education institutions as well as subscribers to humanities and social sciences listservs;
  3. Assessing how user study results might be shared more effectively: interviews of site owners, resource providers, and use researchers, and the organization of a symposium to explore how gathering comparative user and non-user data across a variety of digital resources might be achieved.

Track 1. Understanding the humanities and social science digital resource landscape

Literature review

To assess the landscape of user studies that target H/SS faculty in undergraduate settings, we conducted a literature review, which by necessity encompassed six broad and overlapping domains: (1) humanities and technology initiatives; (2) real and virtual classrooms; (3) undergraduate education reform; (4) information literacy initiatives; (5) tools for instructional use of digital resources; and, (6) specific research on digital resource users (electronic resource/digital library use studies; cultural heritage research; site-specific user studies; image-service studies; and complex new media such as interactive video, GIS, 3D applications, games, and social software).

In addition to pointing out the lack of common vocabulary available, we discovered that a description of this space entails complicated definitions about, and analyses of (1) the scope, variety, and origins of the available rich media resources; (2) how the resources are actually used (or not used); and, (3) the variation that exists between and among a diverse group of “users” and “owners” (and a recognition that users and owners are often embodied in the same person).

There is an especially complex set of stakeholder interests and agendas when it comes to defining the value of user studies. There are policy-makers, funders, and administrators who oversee educational reform or digital library efforts, developers who create resources, and technicians or designers who develop tools for the integration of resources into undergraduate settings. This diversity of perspectives and agendas complicates the understanding of how an exceptionally diverse set of digital resources is actually used.

Creating a resource typology from multiple perspectives

In an effort to tackle the problem of common vocabulary, and to create a typology of what digital resources are available to undergraduate educators, we began by simply describing resource types (e.g., curriculum, video, maps, electronic journals) based on actual faculty discussion group data (below), thereby generating a useful map for describing digital resources from a faculty perspective.

... we soon discovered that digital resources of all kinds are proliferating in many different environments and are created by many different kinds of developers.

As we attempted to categorize resource types systematically by their origin and other characteristics, we soon discovered that digital resources of all kinds are proliferating in many different environments and are created by many different kinds of developers. Users, when compared to resource providers, often employ a different level of granularity in defining a resource (e.g., whether they can find on the Web a format, a photo, a picture, or a passage). Furthermore, categories of users often comprise diverse individuals with varying and idiosyncratic needs, perceptions, and ways of finding and using digital resources.

We convened a number of groups to discuss and to assess the digital resource provider or site owner perspective of our work. A colleague suggested that the set of roles under the designation “owner” (and the individuals in those roles) ordinarily have different interests, values, and, especially, different levels of access to traces of user behavior. These roles are: aggregators, who select which digital resources are to be available in what combinations, and try to bring them to the attention of users; developers of tools, who shape user interactions, export mechanisms, and access paths; and content creators and owners, who conceive, assemble, describe, and digitize content.

Track 2. Focusing on the faculty perspective: Discussion groups and online faculty survey

To determine how, how much, and even if digital resources are being used in targeted H/SS teaching and learning contexts among diverse higher education communities, we conducted discussion groups and a survey of full-time and part-time faculty and graduate students from California research universities (UCs), liberal arts colleges, and community colleges.

Faculty discussion groups and surveys

Methods. We hosted four sessions of discussion groups with 31 instructors from three institutions [4]. The discussions informed the development and creation of the faculty survey instrument. In order to elicit responses as unbiased as possible, we assiduously avoided judgments about the “value” of specific resources in discussion groups and surveys. Instead, we asked instructors to tell us what resources they found more and less useful, why or why not, and for what purposes. The survey instrument delved into eight domains:

  • Teaching background,
  • Types and sources of digital resources used,
  • Personal digital collections,
  • How digital resources are used in teaching,
  • Motivations for using digital resources,
  • Motivations for not using digital resources,
  • Barriers and frustrations to use,
  • Support and assistance needs.

The survey targeted 4,500 faculty from specific disciplines at a stratified random sample of community colleges, University of California campuses, and liberal arts colleges in California; the survey was administered both online and on paper [5].

Survey results. Results of the survey reinforced our early impressions from discussion groups, and only highlights are summarized here. (Detailed analyses, raw data, and associated graphics are available in the final report referenced earlier and its associated appendices.)

User “types.” The degree to which personal teaching style and philosophy influence resource use was striking. There is a broad spectrum of user types, ranging from the non-user, to the inexperienced-novice user, to the highly proficient and advanced user of digital resources. Non-users were themselves diverse. They included those who were passionately opposed to the use of technologies in their classroom for a variety of valid pedagogical reasons (e.g., these technologies cannot substitute for a faculty member’s preferred teaching approaches; they undermine learning). Non-users also included self-described enthusiasts frustrated by technical and non-technical barriers, and those simply without time to think about, let alone use, technology in teaching.

Google-type searches were the most frequent way in which faculty found resources.

What digital resources faculty use. Respondents used an exceptionally wide range of resource types for a variety of reasons. Images and visual materials were the most frequently used resources, and were often used for classroom presentation or posting on the Web. News and other media resources, video, and online reference sources were also heavily used. Google-type searches were the most frequent way in which faculty found resources. A faculty member’s own “collection” of digital resources was the second most frequent source of material. Curricular materials were relatively low on the list of what faculty said they used, although community college faculty were the heaviest users within that relatively small group. Instructors in foreign language, writing, and art and architecture were the heaviest users of curricular materials; instructors of anthropology, language and literature, history, and political science were the lowest users.

Why faculty use digital resources. Faculty respondents used digital resources to improve their students’ learning, to integrate primary source materials into their teaching, to provide students with a context for a topic, to include materials or teaching methods that would otherwise be unavailable, and/or to integrate faculty research interests into a course. Some said they used digital resources to teach critical thinking, because it increased convenience for themselves and/or students, and/or because it was expected by their students or their colleagues. Very few said it would help their promotion and tenure prospects.

Why faculty do not use digital resources. The foremost reason for not using digital resources was that they simply did not support faculty’s teaching approaches. Lack of time was a major constraint, regardless of institution. It was not at all easy for most of our respondents to use the plethora of digital resources available to them. Faculty — including those active and enthusiastic in their use of digital resources — identified many obstacles to using these resources for teaching, including how to find, manage, maintain, and reuse them in new contexts. One of the most-cited obstacles to the effective use of digital resources was the availability, reliability, and expense of the necessary equipment, both in the classroom and for personal use. Almost all faculty need support for a variety of tasks. Both novices and advanced users face challenges when integrating digital resources into their teaching, but they experience somewhat different needs and barriers; thus, support systems that are helpful to one group may not be for another.

More than 70 percent of faculty said they maintain their own collections, although fewer of them (~35 percent) make their resources available to others on the Web.

Personal collections. As noted above, responses also emphasized the importance of personal digital collections in faculty work practices. More than 70 percent of faculty said they maintain their own collections, although fewer of them (~35 percent) make their resources available to others on the Web. It was clear from our discussions and from comments on the surveys that many faculty want the ability to build their own collections, which are often composed of a variety of materials, including those that are copyright protected. How to manage this potpourri of resources and integrate them into teaching practice is the challenge. Although there may be an array of tools available to faculty for collecting, developing, and managing resources, the efficacy and interoperability of these tools for the immediate tasks that faculty need supported often fall short.

Discipline. An analysis by discipline revealed variation among scholarly fields. Faculty who use texts extensively depended on different kinds of sources for different pedagogical goals than faculty in art, architecture, history, and anthropology, who rely more heavily on images. Faculty in political science were the heaviest users of datasets, and faculty who teach writing had special needs around information literacy and the use of reference materials. Not only do faculty in different disciplines require different types of resources, they use them in different ways and for different reasons.

Demographics. When the data were analyzed by age, the oldest instructors (age 62 and up) were the lightest users. A multiple regression analysis demonstrated, however, that age alone is a very weak predictor of a person’s overall level of digital resource use. Regression and path analysis further showed that individual opinions and attitudes have a greater effect on a person’s total level of digital resource use than do institutional, disciplinary, or demographic characteristics.

Track 3. Why study users? Summary of interviews and meeting with digital resource providers and user researchers

In the third track of our research, we devoted considerable time to talking with digital resource providers about why they studied users, what they knew about users, and what more they would like to learn. Our sample included sites that provided online educational resources and that had at least some freely available resources. The goals of these interviews were: (1) to test our initial sort of digital resource characteristics; (2) to collect opinions on the importance of user research to digital resource providers; and, (3) to determine if certain factors and their attendant digital resource characteristics (e.g., histories, funding models, architectures) are associated with successful strategies for integrating an understanding of users into development and maintenance activities. Where possible, data on cost and collaborative development strategies were collected. In addition to conducting formal interviews, we convened a group of resource providers, funders, and user researchers for a two-day meeting in 2005. The interviewees and meeting participants represented a variety of perspectives in the field of online educational resources [6]. The focus of our interviews and the meeting were “generic” online educational resources (OER) and the subset of “open” online educational resources (OOER).

Interviews with digital resource providers

In-depth interviews with 13 digital resource providers and two other stakeholders in the field underscored the diversity of projects, tools, and services available to the H/SS community and the difficulty of making comparisons among them. The interview analyses suggested that there were no common terms, metrics, methods, or values for defining use or users among the targeted projects. Yet digital resource providers shared the desire to measure how and for what purpose materials were being used once accessed; few providers, if any, however, had concrete plans for undertaking this measurement in a systematic way.

Our research revealed that community building is important to digital resource providers ...

Many digital resource providers targeted faculty as their primary audience. Several sites, however, were exploring expansion to new audiences either through targeted planning or in a more serendipitous fashion. Our research revealed that community building is important to digital resource providers, and many were exploring tools to enable the development or support of user “communities.” Some also suggested that community contributions might hold a key to sustainability challenges.

Sustainability for the initiatives we studied is a pressing, if elusive, question for most sites. Formal agreements or plans to determine long-term financial, technical, and organizational sustainability are practically nonexistent. Success and value were slippery topics, though it was apparent that high-quality projects often bring advantages to their institution’s resident students and faculty, and can also bring some level of prestige to the larger organization. This recognition of value by the sponsoring institution provides a potential route to long-term support and funding.

Site owner and user researcher workshop

Sixteen experts were convened over two days for a discussion of “online educational resources” (OER) to explore how and if questions about user behavior are tightly linked to questions of policy and planning. A majority of participants had been interviewed before the meeting. The following summarizes the meeting’s discussions, which covered four broad topics:

  • Codifying content and contexts;
  • Appropriate questions and methods for understanding users;
  • Users, user demand, and sustainability;
  • Larger research questions and agendas to be addressed.

Finding a common framework: codifying categories of content, users, uses, and user studies. Comparing data among OER projects poses a significant challenge to those who conduct or rely on user studies for decision-making. Specific suggestions to improve current data collection and research included the need to:

  • Differentiate among types of OER content; the category of OER needs to be refined. For example, is the content sophisticated, carefully crafted curricula such as Carnegie Mellon University OLI; syllabi and associated materials such as MIT Opencourseware; modular learning objects such as MERLOT; a repository that can be added to in chunks as with Connexions at Rice University; or a digital library such as the Museum Online Archive of California (MOAC) or Research Libraries Group (RLG) Cultural Materials?
  • Differentiate among OER users and the contexts in which OER can be potentially used. Are users students, faculty, or self-learners, and what is the importance of distinguishing among these users? Because these groups use OER materials in very different ways, developing unique strategies for both the site and content design to meet those user needs may be essential for success.
  • Differentiate users with different skill levels and learning objectives. Some users seek a particular digital object for a specific purpose; some seek a completely stand-alone course; some seek supplemental material for a research project, assignment, or their own course; some seek to create their own online course or resource site; and others may just be engaged in lifelong learning. Different types of users also have different skill levels and technological sophistication.
  • Study non-users. One key group to study is individuals who do not currently use OER, especially if the aim is to increase OER’s audience. If we better understood the myriad reasons for non-use, including social and economic barriers, perhaps we could redesign OER in ways that would be useful to targeted audiences (and that would potentially counteract existing disincentives and barriers).

In addition to understanding the nature of users and content, it was agreed that some common questions and methods for conducting user studies should be adopted to allow for better comparisons among studies. Any conversation about users must first establish a common framework and vocabulary in which both terms and methods are codified to ensure successful comparisons among projects. Similarly, it would be helpful to clarify which different types of studies — degree of formality, scale, rigor of data analysis — are best suited to particular objectives and particular contexts. It is only when general principles and standards are articulated as a community and results shared openly that discussions of project results will be meaningful. Some pointed out that the purpose and objectives of user studies need clarification. While many OER sites may use similar tools for collecting data (transaction logs, online surveys, focus groups/interviews), the questions asked and the related metrics used are often unclear.

... in-house evaluations can result in a “self-fulfilling prophecy” where studies frequently favor relationships and products that already exist.

The practicalities of conducting user studies present some obstacles, however: high-quality research is resource intensive, and in-house evaluations can result in a “self-fulfilling prophecy” where studies frequently favor relationships and products that already exist. Another stumbling block to collecting good data is the difficulty of understanding the full range of an OER site’s users (e.g., users who do not register, users who do not log in from recognizable institutions, the difficulty of interpreting transaction log data, etc.). Informal, and often inexpensive, user studies should not be discounted entirely. Feedback from individual users that have been chosen non-randomly, especially in-depth feedback, may be especially useful during the early stages of development, when a site is determining direction, and later when the usability of existing tools needs assessment. Informal studies may be less useful if decisions are being made about investments in new and expensive initiatives.

Value and sustainability. Because discussions of users and user demand lead to the subject of value and sustainability, we spent some time discussing these topics. Definitions of value and approaches to sustainability vary according to each OER’s context and goals. The only way to understand the value of an OER — for individuals, communities, and institutions — is to measure its impact and its outcomes. Moreover, disaggregating the ingredients of sustainability is essential. Four types of sustainability were identified: curricular, technical/infrastructural, organizational, and financial.

  • Curricular sustainability. Creating and sustaining high quality curricular resources can be costly. Real concerns were voiced about the potential for rampant propagation of misinformation and poor quality educational materials on the Internet. While producers can actively control quality by strictly enforcing their own pedagogical and production standards, doing so can make the material difficult to reuse outside the context originally envisioned by the producers. Alternatively, the user community itself can take the place of institutional or individual authority over quality, although fears about diminution of quality are an especial concern among content experts with this model (enthusiasm about Wikipedia in some circles notwithstanding).

    The development of user communities among OER sites, as a measure of curricular sustainability, was discussed at length. There are ongoing problems with community reuse, however. Currently there is no common set of standard tools or practices to help achieve interactive community on a large scale in educational contexts, though emerging social computing models were noted by some as promising.

    One question that arose was whether OER sites could or should adapt their content or services to unintended users. To some participants, unintended use is an opportunity for creative reuse, while many believed that an OER site should not or could not change course to serve an unintended audience. How a site accommodates unintended use may require a complicated calculus taking into account the site’s mission, scope, financial model, desired impact, quality control, and targeted constituencies. It was agreed that studying unintended users is exceptionally difficult and that accommodating them in site development can incur potentially significant costs.

  • Technical/infrastructural sustainability. It was argued that OERs, and especially open-access OERs, need a common place where they can be reliably housed, organized, searched, and preserved, perhaps in one or more centralized OER repositories. How a centralized repository would be organized was open to debate, however, and several possible solutions were discussed. Several participants agreed that the development of federated searching among all OER sites would be desirable and most “user friendly.”

  • Organizational sustainability. Organizational value is related to how OER fits into the organization that supports it. To what degree does the host institution value the OER site, and to what degree does the site’s value drive institutional support? In many cases, there is an ad hoc approach in which a faculty member cobbles together local support. If she leaves the institution or runs out of funding, the OER can potentially be compromised. Long-term commitment for OER is often unclear.

  • Financial sustainability. At this stage, many OERs depend on a mix of institutional, foundation, and corporate funding, and few have concrete plans for financial sustainability. Various "business" and financial models were discussed, including endowment models, subscription models, and others. There was discussion about the hesitation in academic circles to endorse the concepts of business models, market research, and sustainability. For those OERs that wish to remain non-commercial entities, a combination of foundation, institutional, and corporate funding nonetheless remains the only source of financial sustainability. The questions remain: how do we define and measure a "critical mass" of users relative to a specific OER, and how do we measure, then demonstrate, successful outcomes to funders?

 

++++++++++

Conclusions

There were three interrelated questions we sought to explore through the examination of different stakeholder perspectives. We elucidate them in turn below.

How do we begin to assess whether the unique modes of scholarship and pedagogy that characterize the humanities and social sciences require different educational technology solutions than those employed in science, technical, and vocational fields?

What have we learned from the humanities and social science faculty with whom we talked? Faculty use almost every conceivable type of resource, many of which fall outside of what are formally called “collections” or “educational.” Faculty from different disciplines often have different needs with regard to the types of resources they want and how they ultimately use resources in educational contexts. In addition to these disciplinary differences, where a faculty member teaches and his or her personal needs and experiences can influence specific choices and challenges.

We should not expect faculty, who we can assume know more about teaching their subject than non-specialists, to shoehorn their approaches into a technical developer’s ideas of what is valuable or the correct pedagogical approach.

The fact that the most-cited reasons for not using digital resources was that they simply do not mesh with faculty members’ pedagogies is worth noting and has implications for those who wish to increase technology adoption by faculty. We should not expect faculty, who we can assume know more about teaching their subject than non-specialists, to shoehorn their approaches into a technical developer’s ideas of what is valuable or the correct pedagogical approach. Tools and resources must be developed to support what faculty want to do.

Our work also indicates that faculty use a variety of strategies for negotiating the digital morass. For most, the path of least resistance is the one usually taken — a Google search, a walk down the hall or an e-mail to a colleague, a visit to the Web site of a trusted archive, or often a personal and eclectic “collection” of digital stuff. What is deemed “good enough” for users will depend on the problem at hand; a single individual may have different standards and strategies that are determined by the immediate objective, time constraints, budgets, personal and institutional equipment, and support staff, among other variables. Related to this issue is the large majority of faculty who maintain their own personal digital “collections” for teaching. Where does this material originate? What format is it in? How is it stored and preserved? This wealth of material is off the radar of most institutional or commercial support providers, but it apparently represents a large percentage of what faculty value.

Many faculty want to build their own re-aggregated resources, using their own materials and mixing them with resources they have collected along the way. They are concerned about the significant inadequacy of the classroom technologies available to them. How to first manage the array of available resources, and then integrate them into teaching practice is a concern for those who are involved in supporting faculty pedagogies and developing useful technical tools. For faculty, there may be an array of tools available for collecting, developing, managing, and actually using resources, but the efficacy and interoperability of these tools for the immediate tasks that faculty need supported are questionable. A related issue is the integration of learning management systems (LMSs) with library resources and other course content. Current LMSs appear to have limited overall functionality, especially since they may not allow easy integration with the diversity of digital resources that faculty want to use.

The challenges faced by those charged with building the future tools to reaggregate varied resources for easier use include:

  • The difficulty, if not current impossibility, of reaggregating objects that are bundled and “locked” into fixed, often proprietary resources;
  • Managing and interpreting digital rights, which may include pulling data from one resource for integration into another;
  • The unevenness of interface usability and aesthetics (in some disciplines, such as art history, faculty may care a lot about resolution quality, while in other disciplines, faculty may create “hodgepodge” resources, often not caring about varying resolution quality from one record to the next);
  • The growing demand from users for granularity (e.g., the ability to search and find the one particular image or piece of text they need within an entire resource);
  • The issue of knowing about and finding digital objects — simply put, many faculty have no idea about the existence of local and non-local resources, especially licensed resources, that may be available to them.

As Borgman (2003) suggested, most users will at one time or another need to create personal digital libraries that allow the integration of resources from diverse sources for reuse. The possibility of a tool with these capabilities in the near future is not clear, however, despite efforts to construct prototypes of such spaces. Although there are many development efforts in the pipeline, we have been struck by the fact that most faculty may be adrift until these technical promises can be fulfilled.

By focusing our work on faculty in the humanities and social sciences, we have begun to develop a baseline understanding of their needs and how these needs vary by discipline. Such a baseline will facilitate future comparisons with the needs of faculty in scientific, vocational, and technical fields. Until comparable work is undertaken in these fields, however, direct comparisons will not be possible [7]. We contend that comparable research design and protocols will move us closer to understanding how solutions can be targeted more precisely to the varying needs of the full array of disciplines represented in higher education.

Are investments in digital resource production, management, and maintenance worth it?

The short answer is, of course, “yes,” simply because we now live in a world where these resources are expected to be there, and many users expect them to be free for the taking. At this stage, however, many academic and other non-commercial providers depend on a mix of institutional, foundation, and corporate funding, but few have concrete plans for financial sustainability nor are they completely clear on the current or prospective use of their resources in undergraduate educational settings. How do the vast universe and diversity of resources, the great variety of users, and the ubiquity of faculty personal collections influence thinking about sustainability and economics of educational resources in the H/SS?

Talking to digital resource providers, we came to understand that what they provide, and for whom, may represent different things to different types of users in different contexts. If we are to begin a productive conversation around users and use in varied educational environments, and about whether specific resources can be sustained in these environments, distinctions need to be made among types of digital resources and among types of users. A user’s institutional affiliation, skill level, and educational goals are among the variables that must be codified.

The only way to understand the value of a digital resource — for individuals, communities, and institutions — is to measure its impact and its outcomes.

Definitions of value and approaches to resource sustainability vary according to each provider’s context and goals. The only way to understand the value of a digital resource — for individuals, communities, and institutions — is to measure its impact and its outcomes. When we convened a group of experts interested in use and users, we all agreed that focusing on financial sustainability alone is overly simplistic. It is necessary to break down sustainability further so that curricular, technical/infrastructural, and organizational sustainability can be factored into any calculus for assessing value to institutions or individuals.

The development of user communities around open digital resources is a hot topic, with some suggesting that sustainability can be approached when communities contribute to and organize content, primarily through new social software tools and associated practices. But can this vision be realized while ensuring high quality (which is itself achieved only at considerable cost)? On one hand, if providers control quality actively by strictly enforcing their own pedagogical and production standards, it may become more difficult to reuse the material outside the educational context originally envisioned. On the other hand, reuse introduces its own set of tensions, most notably real concerns about user communities propagating misinformation and poor-quality educational materials. A related issue is how or if developers can afford the costs of customizing their resources for audiences that include students of all ages, secondary teachers, university faculty, and lifelong learners (among others), many of whom are simply impossible to study because of the informal way in which they access resources on the Web.

No discussion about investments can ignore how the growing mass of “educational” digitized rich media objects meshes with established scholarly research publication models. These objects range from personal collections to commercial image services to traditional library collections and beyond. Where do personal faculty digital “collections,” which cross the boundaries between the teaching and research realms, fit into traditional ideas about scholarly communication? Smith (2003) describes some of the challenges to preserving new media scholarship and the related economic and sustainability issues. We suspect that scholarly practice may be linked inextricably to pedagogical approaches in various sectors of H/SS teaching, and that many faculty indeed place high value on integrating their own resources and research into their teaching practice. If this is so, what are the implications for the learning object repository movement and reuse of digital resources, especially curricular materials that are developed for specific pedagogical goals and carry the weight of the developers’ preconceived ideas of value?

Future planning cannot ignore the new cohort of “always on” students that is poised to enter higher education institutions. We simply do not understand enough about these students ...

All of these issues become more complex when the fast pace of technological change and the unpredictable introduction of new products, presentation modes, and licensing agreements, among other things, are factored in. Future planning cannot ignore the new cohort of “always on” students that is poised to enter higher education institutions. We simply do not understand enough about these students, who will have been weaned on peer-to-peer file swapping, Google searches, MySpace, and wireless instant messaging, nor about how new software and portable devices will influence the configuration of future learning environments (Harley, 2002; D. Oblinger and J. Oblinger, 2005; Kvavik and Caruso, 2005; Jenkins, 2006).

How can we leverage the knowledge that providers and researchers have about users so that it can be shared effectively?

Comparing data among digital resource projects is a significant challenge facing those who conduct and/or rely on user studies for decision-making. Communication among resource providers is the key to leveraging results and knowledge. In addition to gathering more and better user data, the data should be widely disseminated to help producers expand their sense of what is possible and what is valuable. Developing a common approach to user studies would allow the articulation of general principles and standards as a community.

Before user statistics can be shared, they have to be transparent, because comparison requires some approximation of standardization, which is now almost non-existent in user studies. How do you compare a transaction log analysis from a digital library site to the same from a curricular materials site? How do we assess if visitors to a site are finding any value in the materials, and compare that to other sites? We suggest that we cannot make these comparisons effectively until the categories of OER content and OER users are codified so that common questions and protocols are agreed on. All relevant variables of study — mission, users, producers, and sustainability — are pertinent to such comparisons.

The unanswered question is how or if such analyses can help those involved in resource and tool development build more useful systems and supportive environments. We and our colleagues underscore the importance of sharing usage statistics and user results openly and the importance of the digital resource community articulating general principles and standards around user studies.

Finally, as we think about users and potential users of online educational resources, Martin Trow’s (1997) caution that we must disaggregate the concept of higher education is particularly relevant. To understand the diffusion and uptake of educational resources in higher education, it will be crucial to understand how “the distinctions between elite, mass, and universal access to higher education point to different forms of teaching and learning, to differences in their contexts and uses.” We believe that acknowledging the many diverse, complex, and overlapping “ecosystems” within which open and other educational resources will or will not be integrated is an essential first step to understanding users and non-users of the many high quality digital resources available. End of article

 

About the author

Diane Harley, Ph.D., the Principal Investigator of this project, is an anthropologist and senior researcher at the Center for Studies in Higher Education (CSHE) at the University of California, Berkeley. She directs the Higher Education in the Digital Age (HEDA) Project.
Web: http://cshe.berkeley.edu/people/dharley.htm
E-mail: dianeh [at] berkeley [dot] edu

 

Acknowledgements

This work was made possible by generous funding from the William and Flora Hewlett Foundation and the Andrew W. Mellon Foundation. Additional support was provided by the Hewlett-Packard Company, the Center for Information Technology Research in the Interest of Society (CITRIS), the California Digital Library (CDL), and the Vice Chancellor of Research, UC Berkeley. This work would not have been possible without the significant contributions of Jonathan Henke and Shannon Lawrence. Contributions to data collection and analysis were also made by Ian Miller, David Nasatir, Cara Bautista, and Irene Perciali, among others. Finally, I am grateful to the many participants in formal and informal interviews, discussion groups, and those who contributed their valuable time and opinions to our survey.

 

Notes

1. The full report is available at the Center for Studies in Higher Education Web site at http://cshe.berkeley.edu/research/digitalresourcestudy/.

2. But note Green’s (2006) recent study of digital images in postsecondary teaching.

3. Detailed descriptions of our methods, survey instruments, datasets, and other materials are available online at http://cshe.berkeley.edu/research/digitalresourcestudy/report/.

4. Discussion groups were conducted in Fall 2003 and Winter 2004, and are summarized in a separate publication that can be found at http://cshe.berkeley.edu/research/digitalresourcestudy/documents/faculty_discussion_group_june05.pdf.

5. Surveys were conducted in 2004 and early 2005. We received 831 valid responses (a response rate of 19 percent) to the large faculty survey. A follow-up telephone survey of selected non-responders found no convincing evidence of response bias in the survey. We also conducted a second, parallel survey of instructors from a broader range of institutions, disciplines, and geographic areas, recruited through online discussion groups; we received 452 responses. The results from this second survey corresponded closely with the main faculty survey on most dimensions.

6. The following organizations were represented in interviews and or participation in the meeting: Curricula: CMU OLI, Connexions, MERLOT, UC Irvine, MIT OCW; Digital libraries: JSTOR, ARTstor, NSDL; Tools and reuse: Carnegie Foundation, MIT, IKSME; Foundation and society perspectives: ACLS, Hewlett Foundation.

7. A project funded by the NSDL, and run by Alan Wolf and Flora McMartin, is using some of our research design to address issues of digital resource use in science, technology, engineering, and mathematics (STEM) disciplines. See http://serc.carleton.edu/facultypart/index.html.

 

References

American Council of Learned Societies (ACLS) Commission on Cyberinfrastructure for Humanities and Social Sciences, 2006. Our cultural commonwealth: The report of the American Council of Learned Societies Commission on Cyberinfrastructure for the Humanities and Social Sciences, at http://www.acls.org/cyberinfrastructure/OurCulturalCommonwealth.pdf, accessed 3 January 2007.

Christine L. Borgman, 2003. “Personal digital libraries: Creating individual spaces for innovation,” paper presented at NSF Workshop on Post-Digital Libraries Initiative Directions (4 June), Chatham, Mass., and at http://www.sis.pitt.edu/~dlwkshop/paper_borgman.pdf, accessed 20 October 2006.

Saul Fisher and Thomas I. Nygren, 2000. “Experiments in the cost-effective uses of technology in teaching: Lessons from the Mellon program so far,” Mellon Cost-Effective Uses of Technology in Teaching (CEUTT) Initiative, Andrew W. Mellon Foundation, New York, at http://www.ceutt.org/ICLT%20CEUTT.pdf, accessed 20 October 2006.

David Green, 2006. “Using digital images in teaching and learning: Perspectives from liberal arts institutions,” Academic Commons (30 October), at http://www.academiccommons.org/imagereport, accessed 07 November 2006.

Diane Harley, 2002. “Planning for an uncertain future: A U.S. perspective on why accurate predictions about ICTs may be difficult,” Journal of Studies in International Education, volume 6, number 2 (Summer), pp. 172-187.

Diane Harley, Jonathan Henke, Shannon Lawrence, Ian Miller, Irene Perciali, and David Nasatir, 2006. Use and users of digital resources: A focus on undergraduate education in the humanities and social sciences. Center for Studies in Higher Education, University of California, Berkeley, at http://cshe.berkeley.edu/research/digitalresourcestudy/index.htm, accessed 20 October 2006.

Henry Jenkins, 2006. Confronting the challenges of participatory culture: Media education for the 21st century. MacArthur Foundation, at http://www.digitallearning.macfound.org/site/c.enJLKQNlFiG/b.2108773/apps/nl/content2.asp?content_id=%7BCD911571-0240-4714-A93B-1D0C07C7B6C1%7D&notoc=1, accessed 27 October 2006.

Robert B. Kvavik and Judith B. Caruso, 2005. ECAR study of students and information technology, 2005: Convenience, connection, control, and learning. EDUCAUSE Center for Applied Research (ECAR), at http://www.educause.edu/ir/library/pdf/ers0506/rs/ers0506w.pdf, accessed 20 October 2006.

Gary Matkin, 2002. Learning object repositories: Problems and promise. William and Flora Hewlett Foundation, Menlo Park, Calif., at http://www.hewlett.org/NR/rdonlyres/18867B66‑5E37‑4626‑A2CB‑EF6544F608C7/0/LearningObject.pdf, accessed 20 October 2006.

Diana G. Oblinger and James L. Oblinger (editors), 2005. Educating the Net generation. Washington, D.C.: EDUCAUSE, at http://www.educause.edu/educatingthenetgen, accessed 20 October 2006.

Sheldon Rothblatt, 2003. The living arts: Comparative and historical reflections on liberal education. Washington, D.C.: Association of American Colleges and Universities.

Neil Smelser and Michael Schudsen, 2004. Proposal for a Commission on General Education in the Twenty-first Century. Center for Studies in Higher Education (CSHE), University of California, Berkeley, at http://cshe.berkeley.edu/research/gec/documents/gen_ed_proposal.pdf, accessed 20 October 2006.

Abby Smith, 2003. New-model scholarship: How will it survive? Washington D.C.: Council on Library and Information Resources (CLIR), at http://www.clir.org/pubs/reports/pub114/contents.html, accessed 20 October 2006.

Martin Trow, 1997. “The development of information technology in American higher education,” Daedalus, volume 126, issue 4 (Fall), pp. 293-314.

Carol A. Twigg, 2003. “Improving learning and reducing costs: New models for online learning,” EDUCAUSE Review, volume 38, number 5 (September/October), pp. 28-38, and at http://www.educause.edu/ir/library/pdf/erm0352.pdf, accessed 20 October 2006.


Editorial history

Paper received 8 November 2006; accepted 19 December 2006.


Contents Index

Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 2.5 License.

Why study users? An environmental scan of use and users of digital resources in humanities and social sciences undergraduate education by Diane Harley
First Monday, volume 12, number 1 (January 2007),
URL: http://firstmonday.org/issues/issue12_1/harley/index.html





A Great Cities Initiative of the University of Illinois at Chicago University Library.

© First Monday, 1995-2017. ISSN 1396-0466.