OJPHI: Vol. 4 Issue 1:
Journal Information
Journal ID (publisher-id): OJPHI
ISSN: 1947-2579
Publisher: University of Illinois at Chicago Library
Article Information
©2012 the author(s)
open-access: This is an Open Access article. Authors own copyright of their articles appearing in the Online Journal of Public Health Informatics. Readers may copy articles without permission of the copyright owner(s), as long as the author and OJPHI are acknowledged in the copy and the copy is used for educational, not-for-profit purposes.
collection publication date: Year: 2012
Electronic publication date: Day: 17 Month: 5 Year: 2012
Volume: 4 Issue: 1
E-location ID: ojphi.v4i1.3849
DOI: 10.5210/ojphi.v4i1.3849
Publisher Id: ojphi-04-2

Improving Access to HIV and AIDS Information Resources for Patients, Caregivers, and Clinicians: Results from the SHINE Project
Brian E. Dixon1
Kellie Kaneshiro2
1Regenstrief Institute, Indiana, Indiana University School of Informatics, Dept. of Veteran Affairs, Veterans Health Administration
2Indiana University School of Medicine
Correspondence: Correspondence: Brian E. Dixon, MPA, PhD, Research Scientist, Regenstrief Institute, Inc., 410 West 10th Street, Suite 2000, Indianapolis, IN 46202-3012, 317-423-5582 (phone), 317-423-5695 (fax), Email: bdixon@regenstrief.org


Human immunodeficiency virus and acquired immunodeficiency syndrome (HIV/AIDS) remains a significant international public health challenge. The Statewide HIV/AIDS Information Network (SHINE) Project was created to improve HIV/AIDS health information use and access for health care professionals, patients, and affected communities in Indiana.


Our objective was to assess the information-seeking behaviors of health care professionals and consumers who seek information on the testing, treatment, and management of HIV/AIDS and the usability of the SHINE Project’s resources in meeting end user needs. The feedback was designed to help SHINE Project members improve and expand the SHINE Project’s online resources.


A convenience sample of health care professionals and consumers participated in a usability study. Participants were asked to complete typical HIV/AIDS information-seeking tasks using the SHINE Project website. Feedback was provided in the form of standardized questionnaire and usability “think-aloud” responses.


Thirteen participants took part in the usability study. Clinicians generally reported the site to be “very good,” while consumers generally found it to be “good.” Health care professionals commented that they lack access to comprehensive resources for treating patients with HIV/AIDS. They requested new electronic resources that could be integrated in clinical practice and existing information technology infrastructures. Consumers found the SHINE website and its collected information resources overwhelming and difficult to navigate. They requested simpler, multimedia-content rich resources to deliver information on HIV/AIDS testing, treatment, and disease management.


Accessibility, usability, and user education remain important challenges that public health and information specialists must address when developing and deploying interventions intended to empower consumers and support coordinated, patient-centric care.


Human immunodeficiency virus and acquired immunodeficiency syndrome (HIV/AIDS), a progressive disease that dismantles the human immune system and has no cure, remains one of the greatest international public health and health care challenges. The United Nations estimates that 33 million people worldwide are living with HIV/AIDS (1). In the United States, while the number of new diagnoses of and deaths from AIDS are decreasing, the U.S. Centers for Disease Control and Prevention (CDC) reports that the number of people living with HIV/AIDS (PLWHA) is increasing (2). HIV/AIDS disproportionally affects minorities and men who have sex with men (1, 2). Furthermore, PLWHA are more likely to be unemployed and/or poor and much more likely to be uninsured or dependent on public insurance programs such as Medicaid (3).

Diagnosing, treating, and managing HIV/AIDS in the U.S. occurs in an increasingly fragmented health care ecosystem, presenting challenges for both health care providers and PLWHA. Diagnosing patients remains problematic as our nation’s testing strategy often fails to reach those at most risk. HIV testing occurs primarily (44 percent) in private settings (e.g., physician practice), yet hospitals and emergency rooms (22 percent) and community clinics (9 percent) are the most likely places to find positive results (3). Only five percent of HIV tests are performed in public health clinics, correctional settings, sexually transmitted diseases clinics, and drug treatment clinics. Thirty nine US states offer anonymous testing, so even if testing occurs, it does not guarantee that a patient’s status will be known to his or her treating provider (4). Even if a patient’s status is known, strict federal and state laws regarding disclosure of HIV/AIDS status (5) and the lack of a sophisticated electronic infrastructure for sharing test results (6, 7) make it difficult to share that knowledge with a treating physician in another practice.

The result of a poor testing and restricted knowledge sharing is an estimated 500,000 patients who are not receiving care for their disease, of which approximately 250,000 do not know they are HIV positive (8). Additionally, many individuals who receive a diagnosis of HIV infection late during the course of the disease are less likely to receive standard-of-care antiretroviral therapy (3).

This challenge is compounded by a shortage of infectious disease specialists and primary care doctors. Broad consensus exists that the care of PLWHA should be guided by an HIV expert. Expert care has been associated with reduced morbidity, mortality, and cost of care (912). However, in a recent survey of HIV specialty clinics, only 15% indicated they could absorb a significant increase in patient load (13). These clinics reported a 30–70 percent increase in patient volume, yet only a handful had financial resources to hire more staff. While adept at providing a patchwork of services, state and local health departments often struggle to provide HIV prevention and care services due to inadequate funding, fragmented systems, and a host of federal and state regulations (14). These factors, combined with a general aging of the HIV-infected population, have resulted in increased “mainstreaming” of PLWHA into primary care and generalist care practices, which places a greater knowledge burden on clinicians in primary and general care (15).

The goal of the Statewide HIV/AIDS Information Network (SHINE) Project is to improve HIV/AIDS health information use and access for HIV/AIDS health care professionals, patients, and affected communities in Indiana. A national study of HIV care continuing education and consultation needs of health care professionals revealed that 64% of respondents cited quality of care or difficulties staying up-to-date as their main reason for their “unwillingness” to accept new patients with HIV or AIDS (16). The SHINE Project attempts to make information resources available to clinicians (e.g., HIV experts, primary care, and generalists) to support increasingly complex clinical decision-making, including but not limited to the diagnosis of acute HIV infection, initiation of antiretroviral therapy (ART), antiretroviral side effect awareness and monitoring, toxicity of antiretroviral agents, as well as routine primary care screening for diseases such as breast and prostate cancers for PLWHA.

The SHINE Project further attempts to make resources available directly to healthcare consumers, including PLWHA, and caregivers of PLWHA (e.g., family members, social workers, case managers). Previous studies have documented the various methods by which PLWHA and their caregivers access information related HIV infection and AIDS. The previously identified methods include healthcare providers, print media, the Internet, other PLWHA, education programs, and social networks (1720). Personal contacts, including healthcare providers, professional caregivers (e.g., social workers, case managers), and peers, have been cited as a preferred source of information for PLWHA. The Internet is often a secondary source of information for PLWHA, yet professional caregivers are twice as likely to provide information from the Internet or print something from the Internet and give it to their patients (20). Furthermore, PLWHA have reported that the myriad of sources provide an overwhelming, often confusing, and sometimes unscrupulous array of information that can be just as frustrating as it might be helpful (17, 18). Finally, it has been documented that the information needs of PLWHA changes over time (21). PLWHA, often when first diagnosed, begin as sponges and seek to absorb as much information as possible about the disease, coping with life as someone infected, and available treatments and management of HIV/AIDS. As the disease progresses and/or the PLWHA advance in age, information needs shift away from absorption towards experiential knowledge exchange (21).

To meet the needs of providers and consumers in Indiana, the SHINE Project created at Website (http://library.medicine.iu.edu/shine) to serve as a statewide information and training resource. The site provides access to electronic information resources (e.g., journal articles, up-to-date guidelines, MedlinePlus®) that providers and consumers need to make informed HIV/AIDS testing, treatment, and quality of life decisions. The site further links users to INHealthConnect, a statewide database of educational and social services available to providers and consumers who desire to either learn more about HIV/AIDS treatment and care or gain access to counseling, shelter, and food. The SHINE Project involved a public-private collaborative consisting of core team members from the Indiana University School of Medicine and Regenstrief Institute and an informal advisory board composed of individuals from HIV care providers, public health agencies, and HIV community organizations.

To ensure that the site is both user-friendly and meets users’ needs, the SHINE Team conducts periodic site evaluation. This article focuses on the results of recent usability testing, a common and recommended strategy for assessing a Web site (22). From December 2009 through February 2010, the SHINE Team held a series of usability testing sessions to gather detailed knowledge on the information needs of key, prioritized audiences: Clinicians (e.g., those who treat patients with HIV/AIDS) and Consumers (e.g., those at risk or living with HIV/AIDS). In this paper, we present the lessons from the usability testing sessions as they have implications for the SHINE Project and other public health initiatives that develop and deploy information resources for use by clinicians and consumers.


An equal number of participants from each prioritized population were invited to participate. For usability testing, a population of 6–8 in each audience segment is a recommended best practice. The SHINE Team attempted to recruit 8 clinicians and 8 consumers.

Clinicians were recruited primarily by email invitations sent through professional organizations, including the Midwest AIDS Training and Education Center (MATEC) Indiana, a provider of HIV/AIDS education to physicians and nurses throughout the state, and the Indiana Rural Health Association (IRHA), a federation of nurses, physicians, social workers, and administrators working in Indiana-based rural clinical settings. Physicians practicing at sites near the Indiana University School of Medicine were also targeted. However, the SHINE Team initially focused efforts on recruiting clinicians that were based outside of the Indianapolis metropolitan area to ensure a representative mix of clinicians from around Indiana. Clinicians were not incentivized to participate in the usability testing.

Consumers were primarily recruited on site at HIV/AIDS service providers in Indianapolis. One consumer approached the SHINE Team staff via email in response to the call for clinician participation described above. Consumer participants were selected through a convenience approach, and randomization was not utilized. These participants were offered a $10 grocery store gift certificate for participation.

Participants ranged between 23–62 years of age, and the mean participant age was 40.5 (N=13). Nearly all of the participants felt they were both experienced Web users and knowledgeable about the latest trends and research in HIV/AIDS.

Half of the clinicians who participated (N=3) were physicians. The remaining clinicians included nurses (N=2) and a licensed clinical social worker (LCSW). One of the physicians was an Infectious Disease specialist while the other two physicians were Internists.


Participants completed the testing from a comfortable setting where they normally access the computer and Internet. Health care providers participated from their office workstation. The physician participants were in the same room with the facilitator at their normal workstation. The nurses and social worker were remotely tested using methods described previously by Dixon (23). Consumers were tested on a workstation at the service provider location. The consumer who approached us also participated remotely, but in this instance the facilitator followed the procedures as described by Dixon (23) and the user simply interacted with the facilitator via speakerphone instead of an online conferencing tool.

Study Design

This study was approved by the IUPUI-Clarian Institutional Review Board (IRB), Study #EX0911-15. Each participant was provided with a one-page overview of the study that explained the methods and process used to capture their feedback regarding the SHINE Website. Upon reviewing the instructions, a study facilitator answered any participant questions then proceeded to assign the participants a series of tasks to complete using the SHINE Website. Participants were given tasks until either they completed all available tasks or 15 minutes had elapsed, whichever came first. The tasks were considered “normal” use cases for why an individual might come to the SHINE Website. Upon completing the assigned tasks, users were asked to complete a questionnaire about their experience interacting with the site.

The primary instrument for measuring the user experience was the system usability scale (SUS). The SUS is a widely used and scientifically validated usability instrument (24). The SUS produces a usability score between 0–100, based on respondents’ component scores. Users were also asked about their general impressions of the site, recommended changes they had for the site developers, and other open-ended questions intended to capture qualitative feedback.

The study facilitator took detailed notes while the users interacted with the site to complete their assigned tasks. Users were encouraged to “think-aloud” while completing each task. The facilitator recorded participant’s comments, and the facilitator asked and recorded responses to follow-up questions. If a participant appeared to be stymied, facilitators were allowed to assist users, but facilitators were not allowed to “give away” the answer to the task at-hand or suggest where the user should click on the site to reach a destination.

Data Analysis

The quantitative SUS scores were averaged to derive statistical mean scores which describe the general usability of the SHINE site. Participants’ qualitative feedback included notes from the facilitator and open-ended responses from a post-test questionnaire. Analysis involved a comparative method of continuously comparing important concepts within the collected data through a process where the authors collated, annotated, and discussed the data and their meaning. The authors looked for salient and recurring topics related to users’ information needs as well as satisfaction with the SHINE site.

Usability Scores

The results from the usability testing provide data on how well the site supports a positive user experience. The SUS values from the usability tests ranged from 40 – 95 with a mean SUS score of 75.77 (N=13) – a “very good” score – and a standard deviation of 15.86.

A comparison of providers to consumers reveals a clear distinction in SUS scores. Providers favored the site more than consumers. The mean SUS score for providers (N=6) was 85.42 – an “excellent” score – with a standard deviation of 10.54. The mean SUS score for consumers (N=7) was 67.50 – a “good” score – with a standard deviation of 15.41.

Information Seeking Behaviors

All of the participants reported spending a significant amount of time on the Internet. The top named site of choice was Google. The majority of participants were frequent visitors to online news sites, such as CNN, New York Times, MSN, and the Indianapolis Star. Slightly more than half of the participants (N=8) were frequent visitors to social networking sites such as Facebook and YouTube.

The clinicians varied widely in their readership of peer-reviewed journals and trade publications. Journals specifically named by participants included the New England Journal of Medicine, Journal of the American Medical Informatics Association, Annals of Internal Medicine, Clinical Infectious Diseases, Emergency Nurses Association Journal, Journal of Forensic Nursing, Forensic Examiner, and Nurse Management. However, clinicians reported reading journals on a sporadic basis due to time constraints. Fewer than half of clinicians were regular readers of trade publications. For those clinicians who did read trade publications, e-newsletters were mentioned more frequently than paper-based periodicals.

Participants were asked to identify current sources they regularly consult when looking for information on HIV/AIDS. Providers were evenly split between those who preferred to learn from a human being (e.g., ask a peer, attend an educational session) and those who were willing to search for information using an electronic resource such as MEDLINE. Consumers were similarly divided. Slightly more than half (N=4) of the consumers reported they turned to a human being (e.g., care coordinator, their doctor), while others said they learned by reading (e.g., brochures in their doctor’s office, the public health department’s website, email discussion lists for those living with HIV/AIDS).

User Satisfaction

In addition to differences in the SUS scores assigned by providers and consumers, there were differences in opinions between these groups when asked about their satisfaction with the site. Clinicians

The qualitative responses from clinicians complemented their quantitative scores. Clinicians generally liked the site. Most commented verbally that they would visit the site in the future. One of the clinicians said she was excited to share the site’s URL with a colleague. None of the clinicians had used or knew about the site prior to the usability test. This made them a novice site user even though all but one labeled himself or herself an experienced Internet user.

Clinicians quickly navigated to the “For Clinicians” section, or Clinician Toolkit, when completing their assigned tasks. Physicians in particular preferred this section of the site. The clinicians found the section easy to navigate. One clinician had trouble with the anchors/bookmarks, and the other clinicians tended to use the back button to move from a given section of the toolkit to the main list of categories. One clinician said verbally she felt that the toolkit allowed her to get close to what she needed in one-to-two clicks, which was ideal in her opinion.

Several clinicians commented that the site, and the toolkit in particular, contained a “comprehensive” list of resources. One clinicians said she often receives emails from other sites about their resources but found this site to have several resources of which she was unaware.

Clinicians said they were likely to visit the site in the future. However, they cautioned that their return would be to perform a select list of actions. First, several clinicians expressed a desire for resources to pass along to their patients. Physicians and nurses indicated the resources they would want for their patients include information on where they could go for testing, treatment, and support groups. One resource of particular interest was a site where the physician could find information on who to refer their patient when the patient moves out of the area. This included resources inside and outside the state. The social worker said she also desired resources for her patients that included follow-up services, including financial assistance, medical assistance, and help with nutrition.

Clinicians further indicated a desire for up-to-date access to information on post-exposure prophylaxis (PEP). PEP for exposure in the workplace has been a CDC recommended practice since the 1990s (2527), and in 2005 the CDC expanded its recommendations to include non-work related exposures (28). While the CDC recommendations are available online through Google, PubMed, and other public websites, the guidelines are dense and complex which does not lend itself to quick decision-making in clinical practice. Furthermore, the guidelines do not reflect the latest in research and development of antiviral medications, side effects, or the challenges and interactions with co-morbidities and other medications. Clinicians who participated in this study requested an easy, quick, and up-to-date reference online for making decisions for their patients.

Finally, clinicians said they desired information on prophylaxis, including what clinicians should do when treating other sexually transmitted infections (STIs) and common reactions between STIs and retroviral medications. One physician mentioned that he currently uses a printed resource regarding the appropriate time to initiate prophylaxis treatment. He suggested that this resource would be more useful if electronic and available “on the Web” so he could access from wherever he might be when making clinical decisions.

Beyond these specific actions, clinicians indicated they generally discuss HIV/AIDS prevention, treatment, and post-exposure “best practices” with their colleagues and hospital administrators. They were unsure that the site would be used by clinicians to get a lot of information about guidelines because these are often provided by resources integrated into electronic health record (EHR) systems.


Overall, consumers found useful resources on the site. However, the majority of consumers criticized the site for lacking organization and having a boring appearance.

Like the clinicians, consumers seemed to find the “For Consumers” section, or Consumer Toolkit, quickly. Two consumers, however, suggested that the SHINE Team could improve the site to more easily direct consumers to the toolkit. Once at the toolkit, consumers found a lot of information – sometimes an overwhelming amount of “technical” information (two users said the site was too technical). Consumers felt the site had credibility and useful information, even if it was difficult to read through pages of text. Several consumers requested the site add more graphics or pictures to help draw users into the various parts of the Consumer Toolkit.

Consumers overwhelmingly indicated they would come back to the site to access the “latest information” on HIV/AIDS. Most of the consumers specifically mentioned “medications” or “drugs” as the type of information they felt would be provided on the site. Results of new studies and research were a close second choice. Only one consumer pointed out that the site featured testing location information. This indicates that several of the consumers were likely HIV positive and would use the site to access information they could use for treatment and management of the disease.

Terminology was an issue for some consumers. One consumer didn’t understand the term “clinician.” Another consumer didn’t understand the term “consumer.” Two users said “medical encyclopedia” sounded like it was for physicians and not patients.

Two of the consumers felt uncomfortable using a computer. One of these consumers stated he only accesses the Internet at the library and does not own a personal computer.

In general, consumers felt the site was often too busy and required “too much reading.” They strongly preferred sites that used a lot of graphics. This is evident in their preferred websites. Most of them simply used the Web to access news and general information or engage in social networking. They expected a clean user interface similar to those of sites like CNN and Facebook.


Usability testing was performed with a convenience sample of providers and PLWHA to assess user information needs and satisfaction with a particular online resource. The resource was generally perceived as a valuable source of information, but participants’ information seeking needs and behaviors varied widely. The feedback from participants provided useful feedback that the SHINE Project will use to improve its Web site in the future. The feedback further suggests several lessons that may be useful for public health, informatics, and computing professionals who desire to create, enhance, or support the use of similar information systems and resources. Below we discuss the implications of our findings for the public health informatics community, and we outline planned changes to the SHINE site.

Analysis of participants’ feedback validates previous work that has demonstrated the following aspects of information resource development are critical to meet the information needs of users: 1) accessibility; 2) usability; and 3) user education.


Clinicians and consumers reported that they often lack access to useful information resources when making decisions about the testing for, treatment of, and maintenance of HIV/AIDS. In one case, a consumer indicated he or she lacked physical access to a computer except through the public library. In another instance, a clinician reported having access to the Internet but his or her institution did not have access to a specific information resource.

Accessibility challenges, however, are not a new phenomenon. It is well known that PLWHA are more likely to be socio-economically disadvantaged, and there exists a significant digital divide with respect to personal computers and Internet access (29, 30). Furthermore, it is known that many small and rural hospitals as well as physician practices often lack access to journals and other medical information resources (3133). Participant feedback in this study demonstrates that these kinds of accessibility issues were present in our sample, and they continue to be a challenge for clinicians and PLWHA who are most at-risk.

In addition to general accessibility issues, clinicians in this study reported a lack of access to a specific resource they felt would be useful for treating PLWHA. These clinicians expressed a desire for an electronic referral tool to support coordinated care for PLWHA. The infectious disease specialist indicated an electronic referral tool would help her refer patients to primary care physicians who could support HIV/AIDS management. The internists indicated they desired a referral tool for help in finding the limited number of specialists who could best serve their patients.

Given the general trend towards mainstreaming PLWHA to primary care and general practice clinicians, the feedback suggests that electronic tools for supporting bi-directional referrals and communications for PLWHA may warrant further informatics investigation. The comments confirm earlier research that shows general and specialty practices are limited in the number of HIV/AIDS patients they can handle, and mechanisms for coordinating care across a fragmented health system are necessary to support caring for HIV/AIDS patients that are living longer and managing a complex disease (13, 15). Given that the participating physicians in this study practice in advanced, urban health systems that utilize highly integrated information systems suggests that additional research and development may be necessary to support the national aims of better care coordination and patient-centered outcomes. Electronic referral tools, for example, have been shown to improve the transfer of administrative and clinical information (34), reduce duplicate test-ordering (35), and improve both the referring and subspecialty physician’s ability to make treatment decisions (36). This is true not only in urban areas but also in safety net providers (37).

Beyond reporting a lack of accessibility to certain resources, participant feedback demonstrates that clinicians and PLWHA generally have access to a wide range of information sources that might contain knowledge about HIV/AIDS testing, treatment, or management. Clinicians indicated using human and technology resources to find answers to their questions about HIV/AIDS. Larger studies have suggested that clinicians use a wide range of sources but generally prefer to rely on colleagues to answer most of their questions about patient care (38, 39). This is due, in part, to access their peers’ tacit knowledge, although there is evidence that the most likely reason is that of practicality. Asking a colleague in the office is quicker than searching for an answer in a book, journal, or information system. This is true even when clinicians are provided access to resources with higher quality information, suggesting that accessibility trumps quality in real-world, busy clinical practice settings (40).

Consumers in this study also identified a wide range of sources used to access HIV/AIDS information, something demonstrated in several other studies on consumer access to HIV/AIDS information (41, 42). Many of the sources indicated by consumers were part of the mass media, the predominant source of HIV/AIDS information for consumers in other studies (43, 44). These data suggest that consumers, like clinicians, generally choose sources based on their accessibility. Mass media and human beings are more available than specialized, targeted online resources focused on HIV/AIDS testing, treatment, and disease management. Therefore information resources need to be made available when and where clinicians and consumers are asking questions and making decisions. Resources for clinicians integrated in applications already in use by clinicians, such as electronic health record (EHR) systems, might make more sense than a standalone resource available only through a Google search. Resources for consumers available as web sites, mobile phone applications, and accessible through personal health record (PHR) systems may be a better approach than a traditional standalone Internet site.


Common feedback from patients and caregivers in this study included statements like the SHINE site is “overwhelming,” “intimidating,” and “boring.” Furthermore, consumers, on average, scored the site a 67.5 on the SUS. The comments and ratings suggest that the SHINE site should be enhanced to better meet users’ information needs. The feedback is also in line with previous studies that have examined end user information needs. These studies found that patients and caregivers generally express feeling overwhelmed by the volume of information available on the Web (45, 46). The studies’ authors suggest creating a resource that identifies the important Web links or “quick tips” for end users, reducing their need to scan pages of search results. This is what the SHINE site attempts to do, index available, high quality information on HIV/AIDS for quick reference by providers, consumers, and caregivers. Therefore similar feedback obtained in this study suggests that a usable Web site is much more than just a resource that can distill thousands of search results into a convenient index.

Usable Web sites are those that can connect users to the information they seek in efficient ways that support user workflows, preferences, and cognitive understanding of the information. The feedback from clinicians in this study suggests that the site supports clinician’s cognitive models of HIV and care delivery, but the site fails to meet clinician workflow or the reality of a busy practice setting. Consumer and caregiver, on the other hand, workflows were supported by the site, but the site did not present information in a way that met their cognitive understanding of HIV/AIDS or resources for PLWHA. Nor did the site support consumers’ preferences for splashy landing pages and highly graphical interfaces often found on mass media sites.

Specifically consumers complained that the site contains “too much text” and requires “a lot of reading.” When asked how to improve the site, consumers responded that it needed more “color,” “graphics,” “multimedia” content, “Web 2.0” applications, and words that “pop.” These comments suggest that users with limited health literacy had difficulty approaching the site’s content and interpreting the resources indexed on the site.

Approximately one-third of patients in the United States have limited health literacy.(47) These patients have difficulties correctly reading basic items commonly encountered in the health care setting, such as prescription drug warning labels, appointment slips, and health education materials. Despite readability concerns, materials created for patients continue to be written at too high a level for the average American, and they are incomprehensible to those with limited health literacy (48, 49). This includes patient handouts, materials provided through an EHR system, and material generally available on the Internet (4952).

Information resources for consumers and caregivers need to be accurate, accessible, and actionable. Currently the SHINE Team includes a medical librarian and a graduate student in public health who evaluate and select information links on the consumer portion of the site. The team further performs periodic examinations, like the one detailed in this report, of the site to ensure accessibility and ease of use. The results suggest that the team also needs to screen the site and its contents for readability. Otherwise, consumers and caregivers may be able to access the site and find resources that will support their information needs, but they may not be able to fully understand the content or act upon it (e.g., make informed decisions about their health). Currently there are no widely accepted methods for determining whether health information resources are accurate, accessible, and actionable. Such methods should be developed by the research community, so information system designers and implementers can put them to use for consumer materials on Web sites and other health information systems. Meanwhile, system developers and implementers can apply best practices from the printed material world to their products (53).

User Education

In addition to strengthening accessibility and usabilty, user feedback demonstrates that the SHINE Team needs to provide better education with respect to the availability of the site and how the site can be used to improve health care decision-making and quality of life for PLWHA. The clinicians and consumers participating in this study had never heard of the SHINE site prior to the study despite their recruitment through key partner organizations that have helped inform the development of the site and its contents. The SHINE Team therefore needs to do more outreach with its partners and the community to make PLWHA, clinicians, and the general public more aware of resources to support the prevention, testing, treatment, and maintenance of HIV/AIDS.

Educating clinicians about the existence of SHINE is especiall important, because clinicians are generally unaware or unfamiliar with health information tools available for use by their patients (33). By educating clinicians about resources available for use by PLWHA, we may be able to increase clinician recommendation for their HIV/AIDS patients to access the site and benefit from the resources contained therein. Several clinicians, especially nurses and social workers, in this study commented that they would indeed be willing to share the SHINE site information with their patients. In combination with presentations at HIV/AIDS shelters and treatment facilities, educating the wide specture of care providers for PLWHA might improve adoption and use of the SHINE site by those who have a need for the information on the site.

Outreach and education, however, should not just be limited to an awareness of the website and its contents. A recent literature review that examined clinician search behavior concluded that providers may not be equipped with the skills for effective use of information resources that are available to them (39). Outreach librarians, as well as others invested in seeing clinicians utilize information resources to improve patient outcomes, need to do more to educate health professionals on locating, accessing, and utilizing health information resources.

Established through the Medical Library Assistance Act of 1965, the National Network of Libraries of Medicine member libraries and information centers provide health professionals and the general public with health information resources and services (54). Services include education on effective search techniques and applying health information to decision-making processes. Some of these medical libraries have outreach librarians who not only work directly with health professionals and consumers but also librarians at non-medical libraries to enhance resources available to the general public. Studies have shown that professionally led library services have an impact on health outcomes for patients and may lead to time savings for health-care professionals (55). Yet it remains unclear how often these services are utilized. Furthermore, health professionals and consumers cannot be forced to use library resources or services. Additional interventions within libraries, as well as interventions from other groups such as professional societies, EHR implementation teams, consumer advocacy groups, and informatics programs, might improve providers and consumers’ access and comprehension of health information resources available to them.

In addition to outreach and education about SHINE and how to use the site’s contents to make informed health care decisions, the SHINE Team should explore ways to integrate consumer-focused resources into the EHR systems in use by clinicians throughout central Indiana. Currently there is little knowledge about EHR systems’ support for such “Patient Education Materials.” However, requirements of the U.S. Centers for Medicare and Medicaid Services (CMS) EHR Incentive Program for hospitals and physician practices include making consumer-focused materials available to patients during a clinical encounter (56). In support of this requirement, the U.S. National Library of Medicine (NLM) launched an initiative to integrate MedlinePlus resources into EHR systems and patient portals (57). Additional research and development work will likely be necessary to fully integrate patient materials available from public health information resources like SHINE into EHR systems and patient portals, but doing so will support greater clinician access to (and possibly prescription for) materials that can meet the information needs of PLWHA.


Participants were selected using convenience and non-randomized methods. Therefore the sample populations represented in the usability testing may not reflect the general population of clinicians and PLWHA. The sample sizes were small, which is appropriate for usability testing but makes generalization of the findings to larger populations difficult. Preferences for a website that disseminates information resources about HIV/AIDS in Indiana may not reflect the desires of clinicians and patients in other U.S. states or other nations.


A number of factors are moving health care delivery processes to be more patient-centric and consumer-driven in which the patient plays a stronger decision-making role in his or her care. In such a world, higher quality care and outcomes requires clinicians to understand disease and medicine as well as strategies for engaging consumers in self-management. Sites like SHINE aim to provide information resources to consumers and clinicians to support and enhance communication and collaboration between patient and providers. However, access to information is just one part of the equation. SHINE must also meet the needs of PLWHA, over time, as they age with HIV/AIDS and their health status changes. SHINE must further consider the health literacy needs of PLWHA and others who access the site in support of caring for PLWHA. This includes clinicians who require access to patient-focused educational materials as well as information on social, nutritional, and other community resources that support care for the whole person. This is not easy, and the usability testing reveals that SHINE can do more to meet its aims. The testing does show, however, that progress is being made and reveals insights for other public health, clinical, and informatics professionals also striving to make patient-centric and consumer-driven health care a reality. We hope the lessons from SHINE support the development of a wide range of informatics applications that better support PLWHA, caregivers, clinicians, and others living with a chronic illness.


Conflicts of Interest

The authors declare that they have no conflicts of interests.


The authors would like to thank Michael Wilkinson, MLS, and John D. Patton for their dedication to the SHINE Project and its mission. They further thank Mr. Patton for his role in assisting in the collection of data for the study described here. The authors also thank Julie J. McGowan, PhD, for her feedback on early drafts of this manuscript.

The SHINE Project is a project of the Indiana University School of Medicine Library. This work was funded in part by the National Library of Medicine, National Institutes of Health, U.S. Department of Health and Human Services under Purchase Order No. HHSN276200800492P.

The views expressed in this article are those of the authors and do not necessarily represent the views of the National Library of Medicine, National Institutes of Health, Department of Health and Human Services, or Department of Veterans Affairs.

[1]. Joint United Nations Programme on HIV/AIDS (UNAIDS). 2008 Report on the global AIDS epidemic. Mexico City: 2008 JC1510E
[2]. Centers for Disease Control and Prevention. HIV/AIDS Surveillance Report, 2007; 2009 [cited 2010 May 8]; 19. Available from: http://www.cdc.gov/hiv/topics/surveillance/resources/reports/
[3]. Kates J, Levi J. Insurance coverage and access to HIV testing and treatment: considerations for individuals at risk for infection and for those with undiagnosed infectionClin Infect Dis 2007;45(Suppl 4):S255–60. Epub 2008/02/08
[4]. Kates J, Penner M, Kern D, Carbaugh A, Ginsburg B, Jorstad C. THE NATIONAL HIV PREVENTION INVENTORY THE STATE OF HIV PREVENTION ACROSS THE US. Menlo Park, CA: NASTAD and the Kaiser Family Foundation; 2009
[5]. Health Resources and Services Administration. Protecting Health Information Privacy and Complying with Federal Regulations2004 June 28, 2010. Available from: ftp://ftp.hrsa.gov/hab/hipaa04.pdf
[6]. Silk BJ, Berkelman RL. A review of strategies for enhancing the completeness of notifiable disease reportingJ Public Health Manag Pract 2005;11(3):191–200. Epub 2005/04/15
[7]. Overhage JM, Grannis S, McDonald CJ. A comparison of the completeness and timeliness of automated electronic laboratory reporting and spontaneous reporting of notifiable conditionsAm J Public Health 2008;98(2):344–50. Epub 2008/01/04
[8]. Glynn MK, Rhodes P, Kajese TChanges in the estimated number of persons diagnosed and living with HIV from 2000 to 2003 in the United StatesProgram and abstracts of the 12th Conference on Retroviruses and Opportunistic Infections2005Boston, MA
[9]. Kitahata MM, Koepsell TD, Deyo RA, Maxwell CL, Dodge WT, Wagner EH. Physicians’ experience with the acquired immunodeficiency syndrome as a factor in patients’ survivalN Engl J Med 1996;334(11):701–6. Epub 1996/03/14
[10]. Kitahata MM, Van Rompaey SE, Shields AW. Physician experience in the care of HIV-infected persons is associated with earlier adoption of new antiretroviral therapyJ Acquir Immune Defic Syndr 2000;24(2):106–14. Epub 2000/08/10
[11]. Kitahata MM, Van Rompaey SE, Dillingham PW, Koepsell TD, Deyo RA, Dodge W, et al. Primary care delivery is associated with greater physician experience and improved survival among persons with AIDSJ Gen Intern Med 2003;18(2):95–103. Epub 2003/01/25
[12]. Landon BE, Wilson IB, McInnes K, Landrum MB, Hirschhorn LR, Marsden PV, et al. Physician specialization and the quality of care for human immunodeficiency virus infectionArch Intern Med 2005;165(10):1133–9. Epub 2005/05/25
[13]. Saag MS. Opt-out testing: who can afford to take care of patients with newly diagnosed HIV infection?Clin Infect Dis 2007;45(Suppl 4):S261–5. Epub 2008/02/08
[14]. Penner M, Leone PA. Integration of testing for, prevention of, and access to treatment for HIV infection: state and local perspectivesClin Infect Dis 2007;45(Suppl 4):S281–6. Epub 2008/02/08
[15]. Gallant JE. What does the generalist need to know about HIV infection?Adv Chronic Kidney Dis 2010;17(1):5–18. Epub 2009/12/17
[16]. Liljestrand P. HIV care: continuing medical education and consultation needs of nurses, physicians, and pharmacistsJ Assoc Nurses AIDS Care 2004;15(2):38–50. Epub 2004/04/20
[17]. Benotsch EG, Kalichman S, Weinhardt LS. HIV-AIDS patients’ evaluation of health information on the internet: the digital divide and vulnerability to fraudulent claimsJ Consult Clin Psychol 2004;72(6):1004–11. Epub 2004/12/23
[18]. Hogan TP, Palmer CL. Information preferences and practices among people living with HIV/AIDS: results from a nationwide surveyJournal of the Medical Library Association 2005;93(4):431–9.
[19]. Veinot TC. Interactive acquisition and sharing: Understanding the dynamics of HIV/AIDS information networksJournal of the American Society for Information Science and Technology 2009;60(11):2313–32.
[20]. Horvath KJ, Courtenay-Quirk C, Harwood E, Fisher H, Kachur R, McFarlane M, et al. Using the Internet to provide care for persons living with HIVAIDS Patient Care STDS 2009;23(12):1033–41. Epub 2009/12/23
[21]. Minion J, Bath P, Albright K. From Sponge to Source: Health Information in the Lives of Gay Men Living with HIV8th Global Conference Making Sense of Health, Illness and DiseaseMansfield College, Oxford United Kingdom2009, July 3–5
[22]. Wood FB, Siegel ER, LaCroix E-M, Lyon BJ, Benson DA, Cid V, et al. A practical approach to e-government Web evaluationIT Professional 2003;5(3):22–8.
[23]. Dixon BE. Enhancing the informatics evaluation toolkit with remote usability testingAMIA Annu Symp Proc 2009;2009:147–51. Epub 2009/01/01
[24]. Brooke J. SUS - A quick and dirty usability scaleJordan PW, Thomas B, McClelland IL, Weerdmeester BUsability Evaluation in Industry Bristol, PA: Taylor and Francis Ltd; 2003:189–92.
[25]. From the Centers for Disease Control and Prevention. Update: provisional public health service recommendations for chemoprophylaxis after occupational exposure to HIVJAMA 1996;276(2):90–2. Epub 1996/07/10
[26]. Update: provisional Public Health Service recommendations for chemoprophylaxis after occupational exposure to HIVMMWR Morb Mortal Wkly Rep 1996;45(22):468–80. Epub 1996/06/07
[27]. Panlilio AL, Cardo DM, Grohskopf LA, Heneine W, Ross CS. Updated U.S. Public Health Service guidelines for the management of occupational exposures to HIV and recommendations for postexposure prophylaxisMMWR Recomm Rep 2005;54(RR-9):1–17. Epub 2005/10/01
[28]. Smith DK, Grohskopf LA, Black RJ, Auerbach JD, Veronese F, Struble KA, et al. Antiretroviral postexposure prophylaxis after sexual, injection-drug use, or other nonoccupational exposure to HIV in the United States: recommendations from the U.SDepartment of Health and Human Services MMWR Recomm Rep 2005;54(RR-2):1–20. Epub 2005/01/22
[29]. Denning P, DiNenno E. Communities in Crisis: Is There a Generalized HIV Epidemic in Impoverished Urban Areas of the United States?. Atlanta, GA: Centers for Disease Control and Prevention; 2010 [updated August 6; cited 2010 September 2]; Available from: http://www.cdc.gov/hiv/topics/surveillance/resources/other/poverty.htm
[30]. Kalichman SC, Eaton L, Cain D, Cherry C, Pope H, Kalichman MO. Community-Based Internet Access for People Living with HIV/AIDS -- Bridging the Digital Divide in AIDS CareJournal of HIV/AIDS & Social Services 2006;5(1):21–38.
[31]. Dee C, Blazek R. Information needs of the rural physician: a descriptive studyBull Med Libr Assoc 1993;81(3):259–64. Epub 1993/07/01
[32]. Winters CA, Lee HJ, Besel J, Strand A, Echeverri R, Jorgensen KP, et al. Access to and use of research by rural nursesRural Remote Health 2007;7(3):758. Epub 2007/09/26
[33]. Andrews JE, Pearce KA, Ireson C, Love MM. Information-seeking behaviors of practitioners in a primary care practice-based research network (PBRN)J Med Libr Assoc 2005;93(2):206–12. Epub 2005/04/29
[34]. Shaw LJ, de Berker DA. Strengths and weaknesses of electronic referral: comparison of data content and clinical value of electronic and paper referrals in dermatologyBr J Gen Pract 2007;57(536):223–4. Epub 2007/03/16
[35]. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic reviewJAMA 1998;280(15):1339–46. Epub 1998/10/30
[36]. Medow MA, Wilt TJ, Dysken S, Hillson SD, Woods S, Borowsky SJ. Effect of written and computerized decision support aids for the U.S. Agency for Health Care Policy and Research depression guidelines on the evaluation of hypothetical clinical scenariosMed Decis Making 2001;21(5):344–56. Epub 2001/09/29
[37]. Kim Y, Chen AH, Keith E, Yee HF Jr, Kushel MB. Not perfect, but better: primary care providers’ experiences with electronic referrals in a safety net health systemJ Gen Intern Med 2009;24(5):614–9. Epub 2009/03/25
[38]. Dawes M, Sampson U. Knowledge management in clinical practice: a systematic review of information seeking behavior in physiciansInt J Med Inform 2003;71(1):9–15. Epub 2003/08/12
[39]. Younger P. Internet-based information-seeking behaviour amongst doctors and nurses: a short review of the literatureHealth Info Libr J 2010;27(1):2–10. Epub 2010/04/21
[40]. Gonzalez-Gonzalez AI, Dawes M, Sanchez-Mateos J, Riesgo-Fuertes R, Escortell-Mayor E, Sanz-Cuesta T, et al. Information needs and information-seeking behavior of primary care physiciansAnnals of family medicine 2007;5(4):345–52. Epub 2007/08/01
[41]. Wilkerson JM, Smolenski DJ, Horvath KJ, Danilenko GP, Simon Rosser BR. Online and offline sexual health-seeking patterns of HIV-negative men who have sex with menAIDS and behavior 2010;14(6):1362–70. Epub 2010/08/28
[42]. Kalichman SC, Cain D, Cherry C, Pope H, Eaton L, Kalichman MO. Internet use among people living with HIV/AIDS: coping and health-related correlatesAIDS patient care and STDs 2005;19(7):439–48. Epub 2005/08/02
[43]. Bankole A, Biddlecom A, Guiella G, Singh S, Zulu E. Sexual behavior, knowledge and information sources of very young adolescents in four sub-Saharan African countriesAfrican journal of reproductive health 2007;11(3):28–43. Epub 2008/05/07
[44]. Feldman BS, Kark JD, Zarka S, Ankol O, Letyagina V, Shtarkshall RA. Behavioral surveillance of knowledge about HIV/AIDS transmission and perceived need for additional knowledge in a national sample of young israeli men and women between 1993 and 2005AIDS and behavior 2011;15(1):193–203. Epub 2010/01/09
[45]. O’Grady L. Meeting health information needs of people with HIV/AIDS: sources and means of collaborationHealth Info Libr J 2008;25(4):261–9. Epub 2008/12/17
[46]. Gage EA, Panagakis C. The devil you know: parents seeking information online for paediatric cancerSociology of health & illness. 2011 Epub 2011/08/23
[47]. Gazmararian JA, Baker DW, Williams MV, Parker RM, Scott TL, Green DC, et al. Health literacy among Medicare enrollees in a managed care organizationJAMA 1999;281(6):545–51. Epub 1999/02/18
[48]. Wilson M. Readability and patient education materials used for low-income populationsClin Nurse Spec 2009;23(1):33–40. quiz 1–2
[49]. Gal I, Prigat A. Why organizations continue to create patient information leaflets with readability and usability problems: an exploratory studyHealth Educ Res 2005;20(4):485–93.
[50]. Vallance JK, Taylor LM, Lavallee C. Suitability and readability assessment of educational print resources related to physical activity: implications and recommendations for practicePatient Educ Couns 2008;72(2):342–9.
[51]. Friedman DB, Hoffman-Goetz L, Arocha JF. Health literacy and the World Wide Web: comparing the readability of leading incident cancers on the InternetMed Inform Internet Med 2006;31(1):67–87. Epub 2006/06/07
[52]. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, et al. Health information on the Internet: accessibility, quality, and readability in English and SpanishJAMA 2001;285(20):2612–21. Epub 2001/05/23
[53]. Eichner J, Dullabh P. Accessible Health Information Technology (IT) for Populations with Limited Literacy: A Guide for Developers and Purchasers of Health IT. Rockville, MD: Agency for Healthcare Research and Quality; 2007
[54]. National Network of Libraries of Medicine. About the National Network of Libraries of Medicine (NN/LM). 2006 [updated 27 July; cited 2010 August 6]; Available from: http://nnlm.gov/about/
[55]. Weightman AL, Williamson J. The value and impact of information provided through library services for patient care: a systematic reviewHealth Info Libr J 2005;22(1):4–25. Epub 2005/04/07
[56]. CMS. Medicare and Medicaid Programs; Electronic Health Record Incentive Program; Proposed Rule. Federal Register [Internet]. 2010 January 25, 2010. [cited 2010 January 25]; 75(8):[1844–92. pp.]. Available from: http://frwebgate5.access.gpo.gov/cgi-bin/PDFgate.cgi?WAISdocID=4736101841+0+2+0&WAISaction=retrieve
[57]. National Library of Medicine. MedlinePlus Connect. 2011 [updated 16 June 2011; cited 2011 October 20]; Available from: http://www.nlm.nih.gov/medlineplus/connect/overview.html

AIDS: acquired immunodeficiency syndrome
ART: antiretroviral therapy
CDC: Centers for Disease Control and Prevention
CNN: Cable News Network
EHR: electronic health record
HIV: Human immunodeficiency virus
IRHA: Indiana Rural Health Association
IUPUI: Indiana University-Purdue University Indianapolis
MATEC: Midwest AIDS Training and Education Center
MSN: Microsoft Network
PEP: post-exposure prophylaxis
PLWHA: people living with HIV and AIDS
SHINE: Statewide HIV/AIDS Information Network
STI: sexually transmitted infection
SUS: system usability scale

Article Categories:
  • Articles

Keywords: HIV, Acquired Immunodeficiency Syndrome, Information Seeking Behavior, Internet, Public Health Informatics.

Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org