Heuristic evaluations of the accessibility of learning management systems (LMSs) as authoring tools for teachers
First Monday

Heuristic evaluations of the accessibility of learning management systems (LMSs) as authoring tools for teachers by Weiqin Chen, Norun C. Sanderson, Siri Kessel, and Aleksandra Krolak

With the development of e-education and e-society it is important for information and communication systems and services to be equally accessible to all users, including people with disabilities. Learning management systems (LMSs) are becoming a necessity in higher education and making them accessible to all users, both students and teachers, is crucial. Existing studies on accessibility of LMSs mostly focus on students and their access to learning materials. Very few studies have focused on the accessibility of LMSs for teachers who are responsible for using LMSs to create accessible contents for students. This paper presents the heuristic evaluations of two learning management systems (Fronter and Sakai) focusing on their accessibility from teachers’ perspective. Based on universal design principles and guidelines we aim to identify accessibility issues and to propose possible improvements. The Authoring Tool Accessibility Guidelines (ATAG) 2.0 were adopted in the heuristic evaluations and qualitative data have been collected. We have also compared the findings from the evaluations of Fronter and Sakai, as well as Moodle, an open source LMS which was evaluated in our previous heuristic evaluation. The analysis of the data shows that the level of conformance of the systems to the ATAG2.0 guidelines is very low and many issues needs to be solved for them to be fully accessible for teachers with disabilities.


1. Introduction
2. Related work
3. Task analysis
4. Methodology
5. Findings
6. Recommendations
7. Discussion and conclusions



1. Introduction

The transformations in higher education in developed countries over the past few years impose the need to introduce information and communication technologies (ICT). ICT systems are expected to strengthen the learning environments and therefore should be accessible to all users, including people with disabilities. Promoting the accessibility to ICT systems and services is required by the U.N. Convention on the Rights of Persons with Disabilities in countries that ratified this convention. According to the United Nations’ fact sheet, around 10 percent of the world population lives with a disability. The International Labor Organization (ILO) estimates that 386 million of the world’s working-age people have some kind of disability. The Norwegian Labor Market Supplementary Survey on persons with disabilities shows that the percentage of persons with disabilities in teaching positions amongst persons with disabilities in the work force is higher than the percentage of persons in teaching positions amongst the total work force. In 2013, 11.1 percent of persons with disabilities in the work force were employed in teaching positions, which is 2.5 percent higher than the percentage of persons in teaching positions amongst the total work force. The survey also shows that this difference increased in the two previous years, from 1.6 in 2011 to 1.9 in 2012 (Bø and Håland, 2013). Therefore, it is crucial that ICT systems used by teachers should be accessible.

Learning management systems (LMSs) are becoming a necessity in higher education and making them accessible to all users, both students and teachers, is crucial. A number of studies have been carried out on this topic. However, most of these studies focus on one particular user group such as students with dyslexia or visual impairments. Very few studies have considered the accessibility of LMSs for teachers. LMSs are authoring tools for teachers, who typically carry out many tasks with these systems in their daily work. According to Treviranus (2008), it is as important that people with disabilities be able to use authoring tools to produce Web content as it is that content be accessible to people with disabilities. This requires that the interfaces for LMSs follow standard user interface accessibility guidelines. In our previous research we have conducted heuristic evaluations of Moodle, an open source LMS, to study its accessibility from teachers’ perspective (Chen, et al., 2013). The research reported in this paper is a continuation of the previous research. Two more LMSs have been evaluated in order to assess their accessibility for teachers with different disabilities. The research has two main goals: evaluating the compliance of the LMSs with existing guidelines defined by ATAG (Authoring Tool Accessibility Guidelines), and identifying and addressing accessibility issues in the LMSs.



2. Related work

Prior studies on LMSs show that as the complexity of these systems increases, the use of LMSs becomes more and more difficult, especially for users with disabilities. LMSs can make teaching more efficient (Lonn and Teasley, 2009; Swan, 2001). However, it requires that teachers can plan, design, and teach the courses, but also be able to handle technological and managerial domains (Alvarez, et al., 2009). Therefore, it is important to improve the usability and accessibility of LMSs. The existing research studies focusing on usability of LMSs (such as Granić and Adams, 2011) have very limited findings on the accessibility issues, although accessibility can be seen as an essential part of usability.

Studies have also been conducted on accessibility of LMSs (Habib, et al., 2012; Woodfine, et al., 2008; Ulbricht, et al., 2012). Most of these studies focus on accessibility of one particular LMS in relation to users with specific disabilities, such as dyslexia (Habib, et al., 2012), hearing or visual impairments (Kyun, et al., 2007; Debevc, et al., 2012; Ulbricht, et al., 2012). For example, Habib, et al., (2012) have targeted students with dyslexia in higher education and identified several challenges associated with the use of Fronter based on an in-depth interview of 11 informants. The challenges include information overload, imperfect word processing tools, inadequate search functions, and having to relate to more than one system at a time. Another example is the study conducted by Drigas, et al. (2005) who evaluated a Moodle course adapted to the needs of users with hearing impairments. The system satisfied the requirements by providing text and information in sign language, as well as high level of visualization and learning with peers through the video conferencing.

In order to ensure the accessibility of ICT systems a number of guidelines and standards have been developed. These include the Web Content Accessibility Guidelines (WCAG), Authoring Tool Accessibility Guidelines (ATAG) and Accessible Rich Internet Applications (ARIA) from the World Wide Web Consortium (W3C). Available standards include ISO 9241-171 Ergonomics of human-system interaction Part 171: Guidance on software accessibility, ISO/IEC 24751-1 Individualized adaptability and accessibility in e-learning, education and training, ISO/IEC 24751-2 Part 2 “Access for all” personal needs and preferences for digital delivery, and ISO/IEC 24751-3 Part 3 “Access for all” digital resource description and IMS Guidelines for Developing Accessible Learning Applications (IMS DALA). A list of guidelines was presented in the research of Periša, et al. (2011) for adapting LMSs to blind and visually impaired users. The authors evaluated the Merlin LMS used at the University of Zagreb, and defined the guidelines for designing accessible e-learning systems covering aspects such as Web site architecture, navigation and design, descriptive tags, keyboard shortcuts and auditory feedback. An approach for evaluating the accessibility, usability and utility of LMSs for blind and visually impaired students was proposed by Babu and Singh (2013) based on task-oriented user-centered multi-method evaluation (TUME) technique. Accessibility evaluation tools have also been developed. For example, WAVE, AChecker, and Cynthia are tools to assess Web accessibility.

A number of tools have been proposed and developed to facilitate creating accessible e-learning courses and contents. Seale and Cooper (2010) proposed to combine generic pedagogical tools with specific accessibility tools to help teachers developing accessible e-learning material and activities for learners with disabilities. Ulbricht, et al. (2012) presented a tool designed for Moodle allowing for wider use of the course for users with visual impairments. Laabidi, et al. (2014) proposed another modification of Moodle, namely MoodleAcc+, which offers several additional services: Learners Assistance Tool, Author Assistance Tool, Accessible Course Generation Tool and Platform Accessibility Evaluation Tool. The teachers have the possibility to index the learning resources with accessibility metadata. The students must first specify their types of disability before they can use the personalized content and interface provided by the modified Moodle.

A comparison of accessibility among four LMSs (Blackboard 9.1 Service Pack 3, Desire2Learn 9.2, Moodle 1.9, Sakai 2.8.0) was conducted by Rangin and colleagues (2011). Their study can be considered as a heuristic evaluation of the four LMSs. The evaluation was based on eight categories of interaction: six common for students and teachers (login and configuration/compatibility testing, personalization, navigation, forms, help and documentation, features affecting accessibility that are unique to each LMS), one student-oriented category (common modules/tools), and one teacher-oriented category (authoring tools/content creation). The set of five assessment criteria for the above categories was influenced by the WCAG guidelines. The study found that many accessibility issues were in fact general usability issues, but had an overwhelmingly negative impact on the accessibility of the LMSs.



3. Task analysis

3.1. Teachers’ use of LMSs — A scenario

Before the start of a course, school administration staff will have created necessary course areas, and given teachers access to the areas so that the teachers can organize and add course content and teaching material. Student enrolment is usually handled automatically or by school administration staff, but the teacher will have options to make additional enrolments and grant (or deny) access to the course area as appropriate. The course area itself will be furnished with a set of default elements at creation, for instance forum search, calendar, messages, etc., but the teachers can make changes as preferred. The teacher will normally organize the course area only the first time a course is arranged, and then reuse the same overall organization the next time the course is arranged, only updating the content.

At the onset of a course, the teacher has to add, update, and organize course contents and how it should be presented to the students. The first thing the teacher has to do is to create (or upload a file containing) the syllabus and a course plan with weekly activities. This weekly schedule is often used as the basis for the organization of the whole course — with information about topics, lectures, reading material, lessons and assignments. The relevant teaching material for each week or topic may be added and published for the whole course at once, or added throughout the course, but in any case updates are likely to occur relatively frequently.

There are usually activities of some sort every week that the students have to take part in: a lesson, an assignment or a quiz, and the teacher has to create and administer these as well as give feedback about results to students. In addition, the teacher will use the LMS as the main channel for communicating with the students — announcing changes to the schedule (news), answering questions through forums and messages, and sometimes giving instructions for assignments and quizzes. The teacher usually arranges some form of evaluation of the course as well, often at the end of the course, and sometimes also during the course. LMSs are commonly used to manage this kind of course evaluation.

3.2. Main tasks for teachers

Based on the scenario, we have identified the following main tasks that a teacher carries out and the frequencies of these tasks during a semester.


Table 1: Main tasks for teachers in a learning management system.
Main tasksDetails/subtasksFrequency
Course access managementEnroll students.
Note: The main part of this task is commonly handled by administrative staff. The teacher usually enrolls single students and adds other people (teachers, examiners) that need to access the course area and teaching material.
Remove people (students, others) that no longer need access.Seldom
Creating and organizing course contentCourse plan (planned weekly activities in course).Create once, update can be often
Syllabus list.Create once, update can be often
Resources (references, papers, links, etc.).Create once, update can be often
Lectures (slides) & other teaching material.Often/weekly
Assignments and announcing of deadlines.Often/weekly
Hand-in folders for assignments.Once per semester or for each assignment
Quiz/test (with appropriate alternatives for answering).Depending on teacher, can be weekly, monthly or irregularly
Grading and feedbackGrade assignments and give feedback to students.For each assignment




4. Methodology

The heuristic evaluations were conducted with four accessibility experts, all of whom are teachers in higher education and use LMSs daily in their teaching. Three of the experts have a background in computer science and one has a social science background. One of the experts is partially sighted and uses a screen reader and magnification software on a daily basis in her work.

The evaluation focused on content creation, thus we adopted the ATAG 2.0 principles and criteria. After examining ATAG 2.0 Part A, we decided on the principles and criteria that are relevant for evaluating the chosen LMSs (Table 2). We have also made a list describing the different types of impairments and their main challenges (Table 3).


Table 2: Principles and success criteria.
Note: Adapted from ATAG 2.0, W3C Candidate Recommendation 7 November 2013.
PrinciplesGuidelinesSuccess criteria
A.2 Editing — views are perceivableA.2.1 Make alternative content available to authorsA.2.1.1 Text Alternatives for Rendered Non-Text Content (Level A)
A.2.1.2 Alternatives for Rendered Time-Based Media (Level A)
A.2.2 Ensure that editing-view presentations can be programmatically determinedA.2.2.1 Editing-View Status Indicators (Level A)
A.2.2.2 Access to Rendered Text Properties ((Level AA)
A.3: Editing — views are operableA.3.1 Provide keyboard access to authoring featuresA.3.1.1 Keyboard Access (Minimum) (Level A)
A.3.1.2 No Keyboard Traps (Level A)
A.3.1.3 Efficient Keyboard Access (Level AA)
A.3.1.4 Keyboard Access (Enhanced) (Level AAA)
A.3.1.5 Customize Keyboard Access (Level AAA)
A.3.1.6 Present Keyboard Commands (Level AAA)
A.3.2 Provide authors with enough timeA.3.2.1 Auto-Save (Minimum) (Level A)
A.3.2.2 Timing Adjustable (Level A)
A.3.2.3 Static Input Components (Level A)
A.3.2.4 Content Edits Saved (Extended) (Level AAA)
A.3.3 Help authors avoid flashing that could cause seizuresA.3.3.1 Static View Option (Level A)
A.3.4 Enhance navigation and editing via content structureA.3.4.1 Navigate By Structure (Level AA)
A.3.4.2 Navigate by Programmatic Relationships ((Level AAA)
A.3.5 Provide text search of the contentA.3.5.1 Text Search (Level AA)
A.3.6 Manage preference settingsA.3.6.1 Independence of Display (Level A)
A.3.6.2 Save Settings (Level AA)
A.3.6.3 Apply Platform Settings (Level AA)
A.3.7 Ensure that previews are at least as accessible as in-market user agentsA.3.7.1 Preview (Minimum) (Level A)
A.3.7.2 Preview (Enhanced) (Level AAA)
A.4: Editing — views are understandableA.4.1 Help authors avoid and correct mistakesA.4.1.1 Content Changes Reversible (Minimum) (Level A)
A.4.1.2 Settings Change Confirmation (Level A)
A.4.1.3 Content Changes Reversible (Enhanced) (Level AAA)
A.4.2 Document the user interface, including all accessibility featuresA.4.2.1 Describe Accessibility Features (Level A)
A.4.2.2 Document All Features (Level AA)



Table 3: Impairments description.
MobilityM1Decreased motoric precision of hand/arm (Parkinson’s disease, Rheumatoid arthritis, MS/Multiple Sclerosis)Difficulties with precise movements of fingers/hands/arms; due to e.g., tremor, stiffness, deformities, pain and slowness.
M2No hand/arm function (paralysis, amputation, deformities, spasticity)In need of alternative input systems (e.g., voice recognition, head stylus, eye blink, gesture recognition)
HearingH1Hard of hearingDifficult to understand sound information. Background sounds are likely to be disturbing noise for users of hearing aids (hearing instruments, Cochlea implant).
H2DeafIn need of alternative output for sound information (text or sign interpretation)
VisionV1Color blindness, and low vision due to e.g., blurred vision (cataracts), light-sensitiveness (albinism)In need of contrasting colors, high-level contrast and minor font enlargement. Orientation and reading difficulties. (Enhanced contrast and magnification)
V2Decreased visual field (retinitis pigmentosa/tunnel visionOrientation difficulties — cannot see the whole screen, but can often read when text is in the visual field. (Keyboard navigation/screen reader navigation, audio descriptions of visual information)
V3Decreased sharp vision, but good peripheral vision (macular degeneration)In need of high magnification to be able to read, and face screen orientation difficulties since only a small part of the screen is visible. (Magnification, screen reader for synthetic speech, audio descriptions of visual information)
V4BlindIn need of alternative output for orientation and reading on Web (screen readers for braille and synthetic speech, audio descriptions of visual information)
CognitionC1Decreased concentration and/or memory (ADHD, dementia)Difficulties with complexity.
C2Learning difficulties (dyslexia)Difficulties with complexity, low reading level, narrow line space and small font size and fonts with serifs and hairlines.
C3Epileptic seizuresDifficulties with rapid movement of objects.
Psychology/affectP1Worry, anxietyPerformance difficulties.


Through analyzing the teachers’ tasks, we have identified the most frequent tasks in LMSs: creating and organizing course content, including resources, and grading and giving feedback. The heuristic evaluations were based on these tasks. More specifically, the activities the experts carried out when conducting the heuristic evaluations were uploading resources of different types, such as text files, HTML files, documents, images, videos, audios and links, creating assignments with deadlines and quizzes that include multiple choice questions, true/false questions and text-based questions and answers. Since text editing tools are widely used in activities in LMSs, such as creating and editing news, course descriptions, and assignment descriptions, we have also evaluated the text editors in the LMSs.

During the evaluation, each expert individually carried out the tasks and inspected the elements in each LMS based on the ATAG 2.0 criteria while taking the different impairments (Table 3) into consideration. The experts took notes about the conformance to the criteria and made screenshots to illustrate the accessibility issues. After the experts had conducted their individual evaluation on each LMS, an analysis session was organized to compare notes, discuss and summarize the results.

We evaluated Fronter Y12 and SAKAI 2.9.3. The platforms included Firefox 27.0.1, Internet Explorer 9, and Google Chrome: 32.0.1700.107, on Windows 7 desktops and laptops. Screen readers included open sourced NVDA 2014.2 and subscription-based ZoomText 10 with InfoVox (v. 4 for laptop and v. 2.2 for desktop).



5. Findings

5.1. Compliance to ATAG 2.0 and accessibility issues in Fronter

Among the 28 success criteria, Fronter was found to fully comply with three, partially comply with 15, and fail to comply with seven criteria. Three criteria (A.3.6.1, A.3.6.2 and A.4.1.2) were found not applicable. This is because the Fronter system we evaluated was configured by the administrator in our organization so that the preference settings were disabled. We were therefore not able to inspect the related criteria. Table 4 shows the main accessibility issues categorized by the guidelines.


Table 4: Accessibility issues found in Fronter.
GuidelinesAccessibility issues
A.2.1— There are no possibilities to add text description when uploading video or audio files, or multiple images.
— Adding alt text when uploading images is not enforced.
— Pasting an image into the text editor requires that the author enters the image properties to add alt text.
— There are no options for rendering Flash-based video. (Rendered by user agent.)
A.2.2— Status indicators (primarily spelling errors) are only indicated visually (e.g., not identified by a screen reader).
— There is no possibility for programmatically determining properties (e.g., CSS).
A.3.1— Keyboard cannot access all functionality available in tool.
— There are keyboard traps when playing videos, and in the tool itself.
— Only very few shortcuts (7) and WAI-ARIA landmarks are included for more efficient keyboard navigation.
— No customization of keyboard commands is possible (unless with programming).
A.3.2— Session time is set to six hours by default.
— There is no auto-save option available for authors.
A.3.4— Some structure for navigation and elements exists, but access depends on use of assistive technology. Search for elements is not possible. Not sufficient heading structure.
— Editing programmatic relationships is not possible.
A.3.5— Text search is not possible in source view in text editor (This is an issue with the editor, not Fronter).
— Not all information is searchable in the edit view in text editor, only text (not alternative content, not tags). Screen readers do not give information about highlighted “hits” when choosing to highlight all occurrences (in search). There is no two-way search available in the text editor. (These are issues with the editor, not Fronter itself)
— In Fronter documents, it is not possible to search the different elements that comprise the document.
A.3.6— System/platform settings are respected depending on browser settings, thus it will vary with the browser used. For instance, in Firefox, the user needs to uncheck an option allowing Web pages override system settings to display high contrast (platform/system settings). (This is a browser issue, not a Fronter issue.)
A.3.7— There is no preview available for resources.
— There is no possibility for the user to specify a user agent for preview.
A.4.1— There is no confirmation for “Cancel”.
— Content changes are not reversible in forms.
A.4.2— Accessibility information is only available in the “Help” function, not in the interface.
— Not all available features are documented, e.g., “alternative text for images”.
— Fronter informs in their help/manuals: “You may not find the manual you are looking for here. As some of Fronter’s flexible features are used according to each country’s pedagogical practice, we recommend training and consultation rather than written manuals.”


Access and navigation with keyboard in Fronter was found to be especially inefficient and difficult. As can be seen in Table 4, several issues concerning keyboard access to authoring features (A.3.1) have been identified, including keyboard traps (A.3.1.2 — Level A) and inaccessible functionalities by keyboard (A.3.1.1 — Level A, A.3.1.3 — Level AA, A.3.1.4 — Level AAA).

The text editor is a heavily used tool and is essential to carry out several important tasks such as editing documents, news, assignments, and tests. The default text editor in Fronter is CKEditorTM. Although this open source editor provides an accessibility guide with an extensive list of keyboard shortcuts for navigation and editing, it still poses some challenges to keyboard access (Figure 1). In the evaluated Fronter system, the toolbar in the editor is hidden by default in most of the cases where it is used. A very small button exists for making the toolbar visible. This button is not accessible by keyboard, and no shortcut key is available for opening the toolbar. In addition, there is no option in the editor or in Fronter for setting the toolbar to be visible by default. Fronter provides the possibility for individual users to assign a different text editor in their profile. However, this requires technical competence and knowledge about alternative editors. In our organization, the administrator responsible for managing the Fronter system has removed this function from our profile and all users are assigned with the CKEditor built into Fronter.


Text editor and its accessibility issue
Text editor and its accessibility issue
Figure 1: Text editor and its accessibility issue (the small button for toggling hidden/visible is marked with red circle).


Keyboard traps are found in several tasks. For example, when using Tab forward for browsing the questions in a test quiz, you will eventually reach the organization banner at the top of the Web page, only to find that the navigation is trapped and it is not possible to Tab either backwards or forwards. When adding resources, the cursor is “lost” when you create a Fronter document resource, as well as after adding elements and/or when you have finished adding elements and want to save. The ESC key is only helpful in some situations.

The difficulties with keyboard access and navigation may cause confusion and frustration for teachers who have temporary or permanent motoric and visual impairments and rely on keyboard to carry out tasks in Fronter.

5.2. Compliance to ATAG 2.0 and accessibility issues in Sakai

Among the 28 success criteria Sakai was found to fully comply with three, partially comply with 18 and fail to comply with six criteria. One criterion (A.4.1.2) was found not applicable. This is because Sakai does not provide mechanisms for changing user interface settings. We therefore consider this criterion not applicable. Table 5 shows the main accessibility issues categorized by the guidelines.


Table 5: Accessibility issues found in Sakai.
GuidelinesAccessibility issues
A.2.1— To insert an alternative text to images is not enforced and can therefor be ignored. No option for giving videos, audio files or tests a text description.
— No options for simultaneously uploading an alternative format for time based media.
A.2.2— Only one of the screen readers (NVDA) could identify spelling errors in edit and markup view.
— No style sheet for automatic rendering available in text editor (This is an issue with the editor, not Sakai).
A.3.1— Not all common “short cut keys” and browser specific access keys are available.
— Not possible with keyboard only to add images, links and videos to assignments.
— Not possible with keyboard only to get into the text editing area after selecting “Source view”, or start an iPhone videos.
— Keyboard access is not efficient due to, e.g., missing labels on text fields and different access to “Browse ...” buttons (Space and Enter). In addition, the tab functionality is inconsistent.
— Alerts are not “caught” by ZoomText; they do not get in focus and are not read out aloud.
— Not possible to customize keyboard commands.
— Access to accessibility help may depend on the screen reader used.
A.3.2— A time limit (30 minutes) exists.
— To auto-save in text editor requires downloading and installing a plug-in. No information about this option.
— The system logs off the user automatically without auto-save after 30 minutes of inactivity, but a warning dialog pops up about 10 minutes in advance.
A.3.4— Only some WAI-ARIA landmarks available. Inconsistent use of headings.
— Search for mark-up elements is not possible in the search feature. Access depends on use of assistive technology.
— No option for editing programmatic relationships within Web content.
A.3.5— Not possible to search using wild cards. No highlighting of all occurrences. There is no two-way search available in the text editor (This is an issue with the editor, not Sakai).
A.3.6— Sakai has no display or control settings. Hence, keyboard and display preference cannot be adjusted and saved.
— Personal color settings in the browser for background, links and opened links are not respected.
A.3.7— No option to specify user agent for preview.
A.4.1— No warning or confirmation when activating the “Cancel” button.
— “Undo” stops when reversed to an image.
A.4.2— Easy access to documentation can depend on screen readers.


Although there are no keyboard traps and Sakai provides common shortcut keys, some features remain inaccessible with keyboards, such as adding images, links and videos to assignments. Keyboard navigation is particularly inefficient and tedious Figure 2).


Using keyboard to navigate must sequentially go through all the items in Sakai
Figure 2: Using keyboard to navigate must sequentially go through all the items in Sakai.


Sakai seems to respect the platform settings with browser settings having a higher priority than system settings in FireFox and Chrome. However, the user must know the relationship between the system settings and the browser settings as well as how to configure the browser settings. There is very little information in Sakai documentation about settings such as high contrast and colors for background, text, and links, and how to allow system settings to override browser settings, which makes it very difficult for novice users and people with cognitive disabilities.

5.3. Comparison of compliance of Fronter, Sakai and Moodle

The findings from the evaluation of Moodle were reported in Chen, et al. (2013). Here we present the summary and comparison of the results from the evaluations of the three LMSs. Table 6 shows the summary of the compliance of each of the LMSs to the criteria in ATAG 2.0.

In Table 6, we can observe that the LMSs comply only partially with most of the criteria. In many cases, we found that the LMSs comply with a certain criterion in one task (e.g., resource), but not in another task (e.g., assignment). Each LMS complies fully with only very few criteria (Fronter 3, Sakai 4, and Moodle 3). Among those, Fronter complies fully with one Level 1 criterion, Moodle complies fully with three Level 1 criteria and Sakai complies fully with four Level 1 criteria.


Table 6: Summary of compliance of three LMSs.
A.2.1Level AA2.1.1partialpartialpartial
A.2.2Level AA2.2.1partialpartialpartial
Level AA2.2.2nonono
A.3.1Level AA3.1.1partialpartialpartial
Level AAA3.1.3partialpartialpartial
Level AAAA3.1.4partialpartialpartial
A.3.2Level AA3.2.1partialnono
Level AAAA3.2.4nonono
A.3.3Level AA3.3.1partialyesyes
A.3.4Level AAA3.4.1partialpartialno
Level AAAA3.4.2nonono
A.3.5Level AAA3.5.1partialpartialpartial
A.3.6Level AA3.6.1n/an/an/a
Level AAA3.6.2n/an/an/a
A.3.7Level AA3.7.1partialyesyes
Level AAAA3.7.2nonono
A.4.1Level AA4.1.1partialpartialpartial
Level AAAA4.1.3partialpartialpartial
A.4.2Level AA4.2.1partialpartialpartial
Level AAAA4.2.2partialpartialpartial


Many accessibility issues that cause difficulties for teachers with disabilities were found in the three LMSs. The most common problems include missing possibility to add text descriptions for audio and video files, lack of full keyboard access and efficient keyboard navigation, missing immediate confirmation and feedback for authors’ actions, insufficient search functions, missing auto-save function, and incomplete context-aware help and documentation of both accessibility features and general functionalities.

In addition to the many accessibility issues, we have also found a number of general usability issues. For example, Feedback is a usability design principle that suggests the system should give users confirmation that an action has been performed successfully or unsuccessfully. In Fronter there is no confirmation after the user has clicked the “Cancel” button. The violation of usability principles can also cause confusion and difficulties, and create additional barriers for users, especially those with temporary or permanent impairments.



6. Recommendations

Based on the evaluations and comparison of the three LMSs, we identified the following main recommendations for improving the accessibility of LMSs for all users, in particular for teachers with disabilities.

Providing possibilities for authors to add text descriptions for audio and video files. Currently, the evaluated LMSs only allow authors to provide alternative text for image files. With a meaningful text describing the content of the files, visually impaired users can use screen readers to read the text first before they decide whether to download or open the files. LMSs should not only allow teachers to add descriptive texts to multimedia elements, but also provide reminders when the teachers forget to do so.

Providing efficient keyboard navigation support. Currently, using Tab and Shift-Tab is still the main navigation method in most of the pages in the evaluated LMSs. Although some limited WAI-ARIA landmarks are included for more efficient keyboard navigation (Fronter and Sakai), and screen readers allow users to skip groups of elements, the navigation with keyboard is still mostly inefficient and tedious. Providing more and consistent WAI-ARIA landmarks, grouping items and allowing users to jump from group to group with keyboard will make the navigation more efficient.

Providing efficient search function. Search is an essential component in documents and Web sites. An efficient search function can also make navigation easier. Currently in the three LMSs, the search functions are rather limited. A more efficient search function should be provided in the LMSs to search for different elements in addition to the search function provided by the browsers.

Providing immediate feedback in different modalities. When users use screen readers to interact with the system, they need to listen to the information repeatedly and to receive confirmations continuously that they are on the right track. Currently in the evaluated LMSs, if an obligatory field is not filled in, there is no immediate audio feedback. So the user does not know that a mistake has been made. When the user chooses Cancel in Fronter and Sakai, there is no confirmation message from any of the systems. Immediate confirmation and warning in different modalities will increase the overall accessibility of the LMSs.

Providing auto-save function. When the session reaches its time limit, the user is automatically logged off and unsaved work is often lost, although Sakai shows a popup window 10 minutes before the time limit to warn the user. An auto-save function would be able to prevent users from losing unsaved work. In addition, the warning message should preferably be presented in different modalities to ensure that it reaches to all users.

Providing easily accessible customization. Currently in the evaluated LMSs, most of the customizations such as shortcut keys, session time limit, and display settings are done on the code level or through plugins. Users must be able to either read code to configure the system or find the right plugins. An accessible interface for easy configuration will provide users with freedom and personalized interactions.

Providing more informative and complete accessibility information through both Help function and documentation. The context-aware Help function is currently rather limited in the evaluated LMSs. Not all accessibility information is documented. For example, the following statement is found in Fronter’s user manual: “You may not find the manual you are looking for here. As some of Fronter’s flexible features are used according to each country’s pedagogical practice, we recommend training and consultation rather than written manuals.”



7. Discussion and conclusions

In this paper we have presented the results from heuristic evaluations of two learning management systems, Fronter and Sakai, based on ATAG 2.0 and focusing on their accessibility as authoring tools for teachers. We have also compared the results from the evaluation of Moodle, an open source LMS. We have identified individual and common accessibility issues and provided recommendations for improvements based on the analysis of the findings. We believe that identifying and removing ICT barriers in LMSs will contribute to equal participation in higher education for teachers and students with and without disabilities and that the experiences from this process will benefit other areas of the society in providing equal opportunities for all citizens.

We acknowledge that all three LMSs have devoted considerable efforts to make their systems accessible, through raising awareness and providing accessible guidelines to developers. Moodle [1] has an active community addressing accessibility issues and making new development accessible. In its documentation, there is an accessibility page showing the current state of accessibility and providing coding guidelines for developers. The Moodle documentation also contains a list of Web standards, guidelines and legislation as well as accessibility validation tools and other resources. Sakai [2] has established Accessibility Working Group responsible for ensuring the Sakai framework and its tools are accessible. Sakai provides updates on its accessibility states and encourages users to report accessibility issues. Fronter [3] has also a dedicated web page for accessibility where it states that “all new functionality released as of January 2013 will be accessible up to WCAG AA standard as a minimum”. Our research aims to identify further accessibility barriers that are not yet detected or addressed by these LMSs and may contribute to the improvement of these systems. In fact, after learning about our research, Fronter invited us to present our findings to their developers and they were very interested in our feedback and suggestions. Since then, Fronter has made some changes in their code to address the issues we have identified. For example, in our evaluation we found that it was not possible to use keyboard to open the toolbar in Fronter’s text editor. Now Fronter has added a keyboard accessible link “turn toolbars on/off” in the area over the editor.

One of the challenges with using the ATAG 2.0 guidelines as heuristics has been that the success criteria are very general and difficult to interpret and apply. Although ATAG 2.0 provides examples, we experienced difficulties in understanding and applying the criteria and different evaluators interpreted some of the criteria differently. Consequently, discussion among the evaluators, together with summarization of the individual notes has been a crucial step in the evaluation process. Based on our experience with applying ATAG 2.0 to three LMSs, we argue that the criteria should be formulated more clearly and more specific. In our research, only four accessibility experts and teachers in higher education participated in the heuristic evaluations, one of whom is visually impaired. Moreover, when conducting the evaluations we have paid little attention to users with cognitive disabilities. Universal design requires that all users, regardless of their abilities, ethnicity, digital competence, situations and context, should have access to ICT systems. We are currently planning an extensive user testing with diverse user groups including users with physical and cognitive disabilities, elderly users as well as users with different digital competence.

Rømen and Svanæs (2012) found in their research that only 27 percent of the accessibility problems identified by users with disabilities through user testing could have been identified through using WCAG 1.0 guideline alone. The limited number of participants and the lack of user testing in our study could have limited the results of the evaluation. In future evaluations, we plan to involve more accessibility experts, as well as conduct user testing to discover accessibility and usability issues that may not be covered in the ATAG guidelines. End of article


About the authors

Weiqin Chen is a professor in human-computer interaction in the Oslo and Akershus University College of Applied Sciences (HiOA) in Oslo, Norway. She is also an adjunct professor in the University of Bergen, Norway. She received her Ph.D. in computer science in the Chinese Academy of Sciences in 1997. Before moving to Norway, she worked as a research associate in Osaka University, Japan. Her current research interests focus on accessibility of interactive systems and technology-enhanced learning.
E-mail: weiqin [dot] chen [at] hioa [dot] no

Norun Christine Sanderson works as an associate professor in the Department of Computer Science at Oslo and Akershus University College of Applied Sciences (HiOA) in Oslo, Norway. She received her Ph.D. in informatics from the University of Oslo in 2008. Her current research interests focus on universal access of ICT systems and adaptive e-learning systems.
E-mail: Norun-Christine [dot] Sanderson [at] hioa [dot] no

Siri Kessel is an assistant professor at Oslo and Akershus University College of Applied Sciences (HiOA) in Oslo, Norway. She has a Master’s degree for teachers in health care educations from HiOA and University of Oslo in 1998. After this, she has worked with accessibility for students at HiOA and as a senior advisor at the Equality and Discrimination Ombud in Oslo, Norway, before joining the Department of Computer Science at HiOA in 2011. Her research interests focus on universal design of ICT systems and dismantling of ICT barriers.
E-mail: Siri [dot] Kessel [at] hioa [dot] no

Aleksandra Królak is an associate professor in the Medical Electronics Division, Lodz University of Technology (Politechnika Łódzka, TUL). She received her Ph.D. degree in computer science in 2009 at the Faculty of Electrical, Electronic, Computer and Control Engineering, TUL. She specializes in the fields of human-computer interaction and processing of biomedical signals and images.
E-mail: aleksandra [dot] krolak [at] p [dot] lodz [dot] pl



The authors would like to thank the anonymous reviewers for their constructive and helpful comments that greatly contributed to improving the final version of the paper.



1. Moodle Accessibility: https://docs.moodle.org/28/en/Accessibility.

2. Sakai Accessibility Working Group: https://confluence.sakaiproject.org/display/2ACC/Accessibility+Working+Group.

3. Fronter Accessibility: https://eng.fronter.com/support/technical-information/accessibility/.



Tor Petter Bø and Inger Håland, 2013: “Funksjonshemma på arbeidsmarknaden i 2013,” Rapport 51/2013, Statistisk Sentralbyrå (Tabell A5), at https://www.ssb.no/arbeid-og-lonn/artikler-og-publikasjoner/_attachment/148088?_ts=14247681200, accessed 10 May 2015.

Ibis Alvarez, Pascual Teresa Guasch, and Roca Anna Espasa, 2009. “University teacher roles and competencies in online learning environments: A theoretical analysis of teaching and learning practices,” European Journal of Teacher Education, volume 32, number 3, pp. 321–336.
doi: http://dx.doi.org/10.1080/02619760802624104, accessed 23 August 2015.

Rakesh Babu and Rahul Singh, 2013. “Enhancing learning management systems utility for blind students: A task-oriented, user-centered, multi-method evaluation technique,” Journal of Information Technology Education, volume 12, pp. 1–32, at http://www.jite.org/documents/Vol12/JITEv12ResearchP001-032Babu1185.pdf, accessed 23 August 2015.

Weiqin Chen, Norun Christine Sanderson, and Siri Kessel, 2013, “The accessibility of learning management systems from teachers’ perspective,” Proceedings of the 21st International Conference on Computers in Education, pp. 437–442.

Matjaž Debevc, Zoran Stjepanovič, and Andreas Holzinger, 2012. “Development and evaluation of an e-learning course for deaf and hard of hearing based on the advanced Adapted Pedagogical Index method,” Interactive Learning Environments, volume 22, number 1, pp. 35–50.
doi: http://dx.doi.org/10.1080/10494820.2011.641673, accessed 23 August 2015.

Athanasios S. Drigas, John Vrettaros, and Dimitris Kouremenos, 2005. “An e-learning management system for the deaf people,” AIKED ’05: Proceedings of the Fourth WSEAS International Conference on Artificial Intelligence, Knowledge Engineering Data Bases, article number 28.

Andrina Granić and Ray Adams, 2011. “User sensitive research in e-learning: Exploring the role of individual user characteristics,” Universal Access in the Information Society, volume 10, number 3, pp. 307–318.
doi: http://dx.doi.org/10.1007/s10209-010-0207-7, accessed 23 August 2015.

Laurence Habib, Gerd Berget, Frode Eika Sandnes, Norun Sanderson, Pia Kahn, Siri Fagernes, and Anita Olcay, 2012. “Dyslexic students in higher education and virtual learning environments: An exploratory study,” Journal of Computer Assisted Learning, volume 28, number 6, pp. 574–584.
doi: http://dx.doi.org/10.1111/j.1365-2729.2012.00486.x, accessed 23 August 2015.

Ng Chee Kyun, Liew Yeong Tat, M. Iqbal Saripan and Ahmad Fauzi Abas, 2007. “Education for all: Disabled friendly Flexi e-learning system,” Proceedings of AEESEAP Regional Symposium on Engineering Education, pp. 120–124.

Mohsen Laabidi, Mohamed Jemni, Leila Jemni Ben Ayed, Hejer Ben Brahim, and Amal Ben Jemaa, 2014. “Learning technologies for people with disabilities,” Journal of King Saud University — Computer and Information Sciences, volume 26, number 1, supplement, pp. 29–45.
doi: http://dx.doi.org/10.1016/j.jksuci.2013.10.005, accessed 23 August 2015.

Steven Lonn and Stephanie D. Teasley, 2009. “Saving time or innovating practice: Investigating perceptions and uses of learning management systems,” Computers & Education, volume 53, number 3, pp. 686–694.
doi: http://dx.doi.org/10.1016/j.compedu.2009.04.008, accessed 23 August 2015.

Marko Periša, Dragan Peraković, and Vladimir Remenar, 2011. “Guidelines for developing e-learning system for visually impaired,” Proceedings of International Conference on Universal Learning Design, pp. 115–116.

Hadi Rangin, Ken Petri, Brian Richwine, and Marc Thompson, 2011. “A comparison of learning management system accessibility,” at http://presentations.cita.illinois.edu/2011-03-csun-lms/, accessed 10 May 2015.

Dagfinn Rømen and Dag Svanæs, 2012. “Validating WCAG Versions 1.0 and 2.0 through usability testing with disabled users,” Universal Access in the Information Society, volume 11, number 4, pp. 375–385.
doi: http://dx.doi.org/10.1007/s10209-011-0259-3, accessed 23 August 2015.

Jane Seale and Martyn Cooper, 2010. “E-learning and accessibility: An exploration of the potential role of generic pedagogical tools,” Computers & Education, volume 54, number 4, pp. 1,107–1,116.
doi: http://dx.doi.org/10.1016/j.compedu.2009.10.017, accessed 23 August 2015.

Jutta Treviranus, 2008. “Authoring tools,” In: Simon Harper and Yeliz Yesilada (editors). Web accessibility: A foundation for research. Berlin: Springer, pp. 127–138.
doi: http://dx.doi.org/10.1007/978-1-84800-050-6_9, accessed 23 August 2015.

Karen Swan, 2001. “Virtual interaction: Design factors affecting student satisfaction and perceived learning in asynchronous online courses,” Distance Education, volume 22, number 2, pp. 306–331.
doi: http://dx.doi.org/10.1080/0158791010220208, accessed 23 August 2015.

Vania R. Ulbricht, Tarcísio Vanzin, Marília Amaral, Vilma Vilarouco, Silvia Regina P. de Quevedo, Luís Augusto Machado Moretto, and Angela R.B. Flores, 2012. “A tool to facilitate including accessible content in Moodle to the person with visual impairment,” Procedia Computer Science, volume 14, pp. 138–147.
doi: http://dx.doi.org/10.1016/j.procs.2012.10.016, accessed 23 August 2015.

Brain Woodfine, José Miguel Baptista Nunes, and David J. Wright, 2008. “Text-based synchronous e-learning and dyslexia: Not necessarily the perfect match!” Computers & Education, volume 50, number 3, pp. 703–717.
doi: http://dx.doi.org/10.1016/j.compedu.2006.08.010, accessed 23 August 2015.


Editorial history

Received 21 August 2015; accepted 22 August 2015.

Creative Commons License
This paper is in the Public Domain.

Heuristic evaluations of the accessibility of learning management systems (LMSs) as authoring tools for teachers
by Weiqin Chen, Norun C. Sanderson, Siri Kessel, and Aleksandra Królak.
First Monday, Volume 20, Number 9 - 7 September 2015
doi: http://dx.doi.org/10.5210/fm.v20i9.5430

A Great Cities Initiative of the University of Illinois at Chicago University Library.

© First Monday, 1995-2019. ISSN 1396-0466.