Accessibility: Test Plan

Scope

A test plan for the Accessibility team's efforts in bringing accessibility to mono by implementing the managed UI Automation framework.

References

Product Test Case Plan (current page)
Product Test Case Specification
Product Test Log
Product Test Summary
Product Bug Specification
Product Test Coding Standard
Product Home

The Product roadmap (http://www.mono-project.com/Accessibility:_Roadmap)
The Architecture overview (http://www.mono-project.com/Accessibility#Architecture)
The Novell & Microsoft announcement (http://www.novell.com/news/press/microsoft-and-novell-celebrate-year-of-interoperability-expand-collaboration-agreement) to create cross-platform accessibility framework

QA Meetings

June 5 2008 August 13 2008

No meetings scheduled

Definitions

UIA (http://msdn2.microsoft.com/en-us/accessibility/bb892133.aspx)---Microsoft UI Automation. A managed code application programming interface (API), exposing user interface controls for test automation and assistive technology. Part of the .NET framework starting at 3.0. Successor of MSAA (Microsoft Active Accessibility)

UIA Clients---Applications such as screen readers and testing frameworks written in managed code (e.g., C#/VB).

UIA Providers---UI implementations or application controls such as checkboxes. Written in managed code or C/C++.

AT---Assistive technology. A generic term that includes assistive, adaptive, and rehabilitative devices and the process used in selecting, locating, and using them.

AT-SPI---A toolkit neutral way of providing accessibility facilities in applications. AT-SPI can also be used to automate testing of user interfaces. AT-SPI is currently supported by GTK+2, JAVA/Swing, Mozilla, and StarOffice/OpenOffice. For our product, AT-SPI will act as the equivalent of the UIA core.

ATK---Accessibility toolkit. A developer toolkit that allows programmers to use common GNOME accessibility features in their applications.

ATK/UIA Bridge---Mapping of ATK to the UIA provider APIs.

UIA/at-spi Bridge---Mapping of AT-SPI to the UIA provider APIs.

WinForms---One of the many GUI Toolkits for use with Mono, working towards compatibility with Microsoft's System.Windows.Forms.

Moonlight---The Mono-based implementation of Silverlight.

Accerciser (http://live.gnome.org/Accerciser)---An interactive Python accessibility explorer for the GNOME desktop. It uses AT-SPI to inspect and control widgets, allowing you to check if an application is providing correct information to assistive technologies and automated test frameworks.

Orca (http://live.gnome.org/Orca)---Open source scriptable screen reader. Using various combinations of speech, braille, and magnification, Orca helps provide access to applications and toolkits that support the AT-SPI (e.g., the GNOME desktop).

IronPython (http://www.codeplex.com/Wiki/View.aspx?ProjectName=IronPython)---Implementation of the Python programming language, targeting .NET and Mono. It makes .NET libraries easily available to Python programmers, while maintaining full compatibility with the Python language.

CPython (http://www.python.org)---The default, most-widely used implementation of the Python programming language. It is written in C, hence the name CPython.

Strongwind (http://medsphere.org/projects/strongwind)---A GUI test automation framework inspired by dogtail. Strongwind is object-oriented and extensible. Strongwind is written in Python and uses the pyatspi library to manipulate and query the state of applications.

Test plan

Purpose

This document is a comprehensive test strategy to be used by testers to ensure proper and exhaustive testing. It also acts as a reference for all stakeholders wishing to obtain information on current and past testing efforts.

Outline

Test plan identifier

The purpose of this section is to identify the current and previous versions of the test plan, its level, and the version of software it pertains to.

ID Level Software Version Modify Time Author
Accessibility-TP-V0.1 Draft N/A 03-25-2008 Brian & Calen
Accessibility-TP-V1.0 Initial Release N/A 03-25-2008 Brian & Calen

Introduction

Testing efforts will be related to the project goals, which are:

  • Make mono WinForms accessible. All WinForms currently supported by mono.
  • Make Moonlight accessible
  • Allow UI Automation based Accessibility Technologies to run on Linux

This plan includes integration, system, and acceptance testing. Unit testing is excluded, as it is being managed by individual developers.

Integration testing to test WinForms accessibility via UIA provider and UIA/ATK bridge to AT-SPI is scheduled to be completed in 2008.

System testing will be performed in 2009. Testing will be performed using an AT management tool to test the client and provider.

Acceptance testing will be performed later in 2009 before the product is released. During system testing, the product should be tested in its entirety from the end user's point of view.

Test items

According to the project plan, test items is below:

Test Components:

Test Items Priority(S1/S2/S3) Schedule
Winforms Providers S1
Moonlight Providers S1
Core-Provider S1
Core-Client S1
ATSPIBridge S1

Software risk issues

This section describes any risks resulting from lack of time and/or resources.

  • frequent changes to requirements and/or design.
  • loss of support from Novell/Microsoft upper management

Developers are also tracking problems they encounter:

Features to be tested

Abstraction of test items. What will be tested from the user or customer point of view.

WinForms

According to Q2 2008 of the Accessibility#Roadmap roadmap, testing contents need relate with below info:

  • The WinForms Controls list (http://www.mono-project.com/Accessibility:_Test_Plan_WinForms_Controls) defines which WinForms controls will be implemented and therefore need to be tested.
  • Create or use existing WinForms application samples to test against. (Some samples can be checked out via svn from http://anonsvn.mono-project.com/viewvc/trunk/winforms.)
  • Test WinForms applications samples using Accerciser to ensure that the samples are accessible.
  • Test WinForms applications samples using Orca to ensure that the samples are accessible.
  • Write automated scripts using Strongwind to verify accessibility of all WinForms controls.


Schedule for WinForms test:

  • Define what WinForms controls will be tested
  • Define what WinForms application samples will be used
  • Create custom WinForms applications if necessary
  • Test WinForms application samples with Accerciser to ensure accessibility
  • Test WinForms application samples with Orca to ensure accessibility
  • Create automated test suite to ensure accessibility of all WinForms controls
UIA Provider
Moonlight
ATK Bridge
AT-SPI Bridge
UIA Client


Features not to be tested

Any features that will not be tested and why. Decisions not to test a feature should be based on priority and risk. If a feature will not be tested, define what features (class, methods, properties) will not be tested.

At this time we plan to test all features exhaustively.

Approach

A description of how and where testing will be performed and explain any issues that have a major impact on the success of testing and ultimately on the project.

WinForms Testing

The accessibility of WinForms applications with be tested using Strongwind tests and WinForms sample applications. A test harness has also been developed to facilitate the execution and logging of a suite of tests. For more information see the WinForms portion of the Testing Howto.

General Guidelines
  • All testers shall be on the team IRC channel (#mono-a11y on irc.gimp.org) during work hours.

  • Tests shall be automated whenever possible. Time constraint is not a good excuse not to automate.

  • Test plan will be developed using the IEEE Std. 829-1998 Standard for Software Test Documentation.

  • All bugs shall be logged in Bugzilla (https://bugzilla.novell.com) at the time they are found.

  • Bugzilla can be accessed from http://bugzilla.novell.com with Novell account. The product category is "UI Automation," which can be found under the Mono classification.

  • When a bug's status is changed to VERIFIED in Bugzilla, the reporter of the bug should change the bug's status to CLOSED or reopen the bug as soon as possible. If the person who reported the bug verifies the bug, the bug can be closed without having its status changed to VERIFIED. A bug should not be closed by someone who did not report the bug unless the reporter is unavailable.

  • Types of system testing include function, performance, security, load, reliability, usability, documentation testing.

  • Acceptance criteria for patch acceptance: Before a patch is accepted, a QA engineer must ensure that the patch submitted from developer passes QA testing. A build engineer must ensure the patch builds properly and meets packaging standards. QA and build engineers will then create a patch acceptance report, and the patch can be included in the product.

  • Testers may perform system testing on the product only after development has verified that they have completed a development milestone and the build team has created a stable release.

  • WinForms samples will be created in C#, Boo, or IronPython. Automation scripts, that test the accessibility of the WinForms apps will be created in CPython. Strongwind and Orca Regression Tests will be used for the automation scripts.

  • No regularly scheduled meetings at this time

  • Minor editing (grammar and spelling corrections) of this test plan can be done at any time. Any change to the test plan that changes how the product will be tested shall be approved by the QA team who will determine if the changes are large enough to require a change to the test plan identifier.

  • Black box and white box testing methods are both acceptable. However, it is anticipated that black box testing will be the norm.

  • Integration testing will involve the iterative testing of new developer code as it becomes available and it is integrated into the product. System testing will begin after provider and client pieces has been implemented and released.

  • Smoke tests will be performed and will consist of (1) ensuring that the newest packages provided by the build team install successfully and (2) ensuring basic functionality and sanity of the newest packages.
  • Exit criteria has yet to be established. This should be discussed with project management.

  • At this time, Orca and Accerciser are the standard tools to be used to test the accessibility of an application. The Orca test harness and Strongwind will be used to automate accessibility tests.


Item pass/fail criteria

The pass/fail criteria for each of the items described in Test Items section.

Criteria for Test Components:

Test Items
 Pass Fail
Winforms Providers  ?
Moonlight Providers  ?
Core-Provider  ?
Core-Client  ?
ATKBridge  ?
ATSPIBridge  ?

Individual test case pass/fail criterion is defined by the automated script which performs the testing. Upon failure of a test case, the script should will log the failure. For exit criteria, see Approach.

Suspension criteria and resumption requirements

Suspension criteria:

  • Unavailability of external dependency during execution.
  • When a defect is introduced that cannot allow any further testing (i.e., a blocker bug).
  • Critical path deadline is missed so that the client will not accept delivery even if all testing is completed.
  • A specific holiday shuts down both development and testing.
  • Lack of testing resources (e.g., testers, hardware, etc).

Resumption requirements:

  • When the external dependent systems become available again.
  • When a fix is successfully implemented and the testing team is notified to continue testing.
  • The contract is renegotiated with the client to extend delivery.
  • The holiday period ends.
  • Testing resources become available

A failed build would not constitute suspension as we could generally continue to use the previous build. Major or critical defects would also not constitute suspension as other areas of the system could continue to be tested.

Test deliverables

All documents, tools, and other components that are to be developed and maintained in support of the testing effort

  • Test plan (this document)
  • Test cases - none at this time
  • Custom tools - test harness, test scripts, sample applications
  • Defect reports - none at this time
  • Test summary reports - none at this time
  • Other - none at this time

Testing preparation and setup

set of tasks necessary to prepare for and perform testing

Team Setup:

Task
 Finished
Build UIAutomation project on Bugzilla and Testopia. X
Prepare virtual machines for most recent releases of supported platforms (openSUSE, Ubuntu, Fedora) X
Setup test environment on VMs P
Obtain testable build P

X = Done P = In Progress

Individual Preparation:

  • Enable assistive technologies on the GNOME desktop. This is done from "Assistive Technology Preferences" from the GNOME Control Center.
  • Create Novell Bugzilla account
  • Install Accerciser on OS
  • Install most recent Mono
  • Install most recent Orca
  • Install most recent Strongwind
  • Install Python >=2.5
  • Install Iron Python (IPCE) >=1.1

Environmental needs

Hardware, software, data, interfaces, facilities, publications, other requirements that pertain to the testing effort

Testing may be done on physical and virtual machines.

All tests must be performed on the most recent official release of the following platforms:

x86 x86_64
openSUSE X X
Fedora X X
Ubuntu X X

Hardware:

  • No specific hardware requirements at this time

Software:

  • Mono - most recent release
  • Accerciser - most recent package for your platform
  • Orca - most recent package for your platform
  • Python >=2.5
  • IronPython (IPCE) - most recent package for your platform
  • Strongwind - most recent release

Responsibilities

All testers can work wherever they are needed, however, below is where our team focuses their efforts at this time:

Strongwind Tests: Calen Orca Tests: Brian Test Harness: Brian Smoke Tests: Ray Sample Applications: Ray

Staffing and training needs

  • QA Automation Engineer (3)
    • Solid programming experience C#, Python, and/or Boo
    • QA Engineer experience
    • Bugzilla experience
    • Iterative testing experience

Everyone in the open source community is encouraged to join our QA team!

Schedule

Built around the roadmap but specific to testing with testing milestones


Based on Q2 in roadmap, our initial testing schedule is below:

Task Start Time End Time Owner
Design test plan Mar. Apr. Brian & Calen
Prepare test environmental stuff(OS, VM, App samples) Mar. Brian & Calen
Set up test environment Apr. May. Brian & Calen
Design test case and test samples Apr. Jun. Brian & Calen
Automate smoke tests Sep. 10 2008 Sep. 12 2008 Brian
Design Strongwind test, Run testcase and submit defect (release 0.9) July Nov Calen & Brian
Design Strongwind test, Run testcase and submit defect (release 1.0 BETA) Dec Jan Calen & Brian & Ray
Design Strongwind test, Run testcase and submit defect (release 1.0 FINAL) Jan Feb Calen & Brian & Ray
Finish test report Brian & Calen

Risks and contingencies

Any activity that jeopardizes the testing schedule is a planning risk

  • Program release schedule delay will jeopardizes testing schedule
  • Program quality of release version couldn't satisfy with acceptance criteria
  • Test environmental stuff unobtainable easily
  • Delays in necessary QA training (e.g., understanding what needs to be tested, writing sample applications, test tools, and automation scripts)
  • Tester staff change or lack of tester resources
  • Changes to the original requirements or Designs
  • Frequent program design changes
  • Loss of support from Novell/Microsoft upper management

Approvals

Persons who declare that the software is ready to move to the next stage

Brian Merrell
Calen Chen
Brad Taylor