The BITV-Test is a tool to evaluate the accessibility of information-oriented websites which captures the requirements and success criteria of the BITV - Barrierefreie Informationstechnik-Verordnung (Accessible Information Technology Regulation) in a set of 52 concise and practical checkpoints. The BITV Test has been developed as a part of the BIK project in close co-operation with associations of and for disabled people, web designers and accessibility experts. The test was released in 2003 and since then it has been continually updated. The test covers both priority levels of the BITV regulation.
The BITV-Test is currently being updated to bring it in line with the soon to be released revision of BITV, which in turn closely follows the latest W3C recommendation, the Web Content Accessibility Guidelines 2.0.
Structure of the BITV-Test
BITV-Test encompasses altogether 52 checkpoints; for each checkpoint, in-depth explanations and instructions inform testers on what they should check, why the aspect to be checked is important, and how the check should be carried out. For each web site, at least 3 representative pages are tested.
The checkpoints are evaluated according to their impact on accessibility, each of them contributing between 1 and 3 points to the overall maximum score of 100. Web sites that reach more than 90 points are classed as 'accessible', sites with a score of more than 95 points are classed as 'very accessible'. The entire test procedure is publicly documented. For those tests that have been published, the individual checkpoint assessments are also publicly available.
The reliability of the BITV-Test
The result of a BITV-Test is based both on objective measurements and human assessment. This should not be seen as a deficiency of method. Any measurement has to be qualified by human judgement. However, the reliability of testing must be safeguarded, meaning that test results should be replicable within a margin of error. Different testers should arrive at comparable results. This is the rationale behind the tandem test approach.
Test conditions and procedures:
- To ensure the reliability of test results, testers must meet certain criteria regarding qualifications, knowledge and experience and the test environment is clearly defined. It prescribes the choice of operating system, browsers, or special measurement tools. In addition, the scope of testing is clearly defined at the outset.
- Coordination of decisions: testers are part of a team; here, all cases that cannot be readily decided based on the documented test procedure are discussed and such discussions also serve as the basis for the continued refinement and adaptation of the test procedure.
- Comparability of results: the test suite enables access to all prior assessments made for each checkpoint; these can be used as a guidance for testers.
The team-based approach enables testers to learn to apply objectiveness in their assessment as the web sites tested should neither be assessed benevolently nor severely. Conclusive tests are therefore always carried out by two testers who first assess the site independently and then co-ordinate and harmonise their results in a second step.
Policy Context
In 2002, the German Federal regulation Barrierefreie Informationstechnik-Verordnung (BITV) came into effect for German federal web sites. The BITV was modelled on the WCAG 1.0 - Web Content Accessibility Guidelines 1.0, a set of guide-lines explaining how to make web content accessible to persons with disabilities. BITV mandates that the web sites of institutions providing public tasks or services on behalf of the German Federal Government should be accessible also for people with disabilities; examples of such institutions are Federal Ministries and Agencies, supra-regional health insurance companies, or charitable trusts.
The BITV-Test was developed by the BIK project with funding from the German Ministry for Employment and Social Affairs (BMAS).
Other BIK activities involved the creation of local accessibility advice offices that provided accessibility consulting and familiarisation and also carried out the BITV-Test (design support test and final test).
Description of target users and groups
Since the first release of the web-based testing application in autumn 2003, the BITV-Test has been carried out over 900 times.
The target group of indirect users is anyone benefiting from accessible web sites, but more specifically, people with disabilities. Direct users are the Federal Ministries and Agencies and web-based information sites run or initiated by these; there are other organisations such as health insurance groups or the judiciary. A second group is constituted by administrations and public bodies in the German Federal states that are subject to individual States’ implementations of the Federal BITV.
Sites of organisations representing people with disabilities have also used the test to improve the accessibility of their sites.
The BITV-Test also serves as a basis for the inclusion of web design companies in a list of service providers with a proven ability to develop accessible web sites (Agenturliste 95plus). Since 2005, Web designers have been using a tool for the concurrent evaluation of web design and development: the BITV self-test (BITV-Selbstbewertung (in German only)).
The last group of target organisations did not choose to be tested, but was selected for the BIK project's "Test of the week" (later, Test of the Month). BIK often focused on sites that provide important public or private services to society, especially those of particular importance to people with disabilities including the web outlets of media companies (newspapers, radio and television), public transport sites, and often, sites of organisations related to health, employment and occupational services. In addition to the "Tests of the Month", a number of comparative tests have been carried out, for example, of political parties, online news sites, airlines, and federal Ministries.
Description of the way to implement the initiative
BIK is a joint project of DIAS GmbH and several German organisations of blind and visually impaired people.
The BITV-Test Development is one of two modules of the BIK project; the other module, BIK@work, promotes the improvement of internal accessibility of IT systems in large organisations and administrations. The services of BIK@work span from familiarisation workshops, status quo assessments, to in-depth consulting regarding the document workflow and the integration of staff with disabilities in overall work processes. The BITV-Test is also carried out for example to provide a measure of success of intranet applications and to produce input for design improvements.
Technology solution
The BITV-Test is an accessible web-based testing application. The quality assurance residing with the BIK test development module sets up any new test case after checking its suitability; it oversees the selection of testing scope and has to agree with any decision to exclude legacy parts of the web sites tested. The precondition is usually a clear demarcation of non-accessible content, and a demonstrated policy with a commitment to time frames and milestones on the part of the site owner to implement an accessibility-oriented workflow in the future. The BIK quality assurance is always involved in tandem tests to ensure reliable and valid results. It also examines all design support tests carried out by one tester.
Completed BITV-Tests can be published if the client agrees on that; the detailed results of published tests are then publicly available on the bitvtest.de website and must be linked to from any BIK 90plus and 95plus quality seal placed on clients’ web sites. A free version of the BITV-Test, the BITV self-test (BITV-Selbstbewertung - in German), is frequently used by web design companies to check the degree of accessibility of sites under development.
Technology choice: Proprietary technology, Accessibility-compliant (minimum WAI AA)Main results, benefits and impacts
Since its inception in 2002, the BITV-Test has become an important benchmark test procedure for German-language web sites. It is also used in other German-speaking countries such as Austria. Overall, more than 900 tests (partly design support tests, partly final tandem tests) have been carried out.Â
In many cases, the accessibility problems found have been corrected by site owners and the BIK website has reported such improvements made in response to test results. Partly as a result of the wide application of the BITV-Test, the accessibility, especially of public sector sites, has noticeably improved over the years.
The BITV-Test has also spilled over to the private sector, with more and more clients being aware that inaccessible sites exclude customers and can cause negative press.
In March 2010, a group of German web design and development companies (and one Austrian company) with proven competence in accessible web design has begun to co-operate with the BIK Test development Module and started the joint use of the BITV-Test during a one-year trial period. BIK’s main aim is to ensure the relevance of test steps in the light of ever more dynamic web technologies; a second aim is to carry the use of the BITV-Test beyond BIK project partners, by supporting or initiating the creation of competent external test departments or outlets in organisations of disabled people as well as in web design companies.
BIK will retain its role in providing quality assurance to external partners applying the test. It will train testers implicitly during the harmonisation phase of joint tests, and explicitly in workshops discussing testing issues, such as the right approach for checking dynamic content made accessible for assistive technology with WAI-ARIA roles and properties.
Return on investment
Return on investment: Not applicable / Not availableTrack record of sharing
BITV-Test has been an important source of information for other initiatives that have benefited from its development. For example, the German Bundesverwaltungsamt (the German Federal Administration Office) is currently developing a tool box for ministries to aid the assessment of web and intranet sites and applications produced in-house. This tool box makes use of the checks of BITV-Test.
BIK has also contributed to the discussion phase of WCAG 2.0. The project has made efforts to become involved in European projects such as the 2009 call for the eAccessibility thematic network. We actively seek co-operation with other entities involved in developing workable test procedures in Europe and beyond. Recently, BIK has carried out a review of other testing approaches of European organisations involved in the development and application of web accessibility tests.
Lessons learnt
- The tandem test approach implying two human testers testing independently and consolidating results during a final arbitration phase has made test results quite reliable and provided the best atmosphere for improving testing skills among peers.
- The co-operation with web design professionals has provided a "reality check" and helped to retain the applicability and relevance of the test procedure by updating it continuously.
- The co-operation with users of assistive technologies and with the INCOBS project that tests such technologies has helped to ensure that the BITV-Test reflects the actual impact of accessibility problems in its assessment scheme - independent of the mere technical compliance with (sometimes outdated) specifications.