Quality Assurance

As a CMMI Level 2 Agile performing company, Quality Assurance is interwoven throughout our Key Process Areas. Here is an example from one of our Programs of a primary Quality Assurance section included in our work practices. The Links to further detail are disabled.

Our Quality Assurance approach pursues excellence in a number of project dimensions including documentation artifacts, executing code and development processes or practices.

Documentation Artifacts

The wiki is used to document everything project related. This documentation may include the following:

• Requirements in the form of Stories
• Analysis including log of questions and answers from client
• Design which can be a screen markup, state transition diagrams, sequence diagrams or whatever deemed usable/essential to communicate a design decision to technical staff
• Test Case information
• Business Service description
• Defect List
• Meeting Minutes
• Monthly Report
• Technical Environment information
• etc

In all of these cases, the information is contained in specific topics and a topic template is used where there is a discussion/feedback section to enable peer review of the information. All project team members have access to these topics and are expected to read and provide feedback constantly. Additionally, the client is asked to read topics to ensure that there is agreement on content. The topic templates and conventions for each topic can be described here, BlueJeanScrumTwikiAndProcessConventions.

Quality Assurance in the context of the Development workflow

The quality of the Java Screen replacement for the existing legacy LMCS Screen functionality is tested in a number of environments and multiple types of testing before being deployed to the production environment. This approach includes the following:

Unit Testing on developer workstation

• • within the Integrated Development Environment (IDE), which is Eclipse for the project, unit testing is achieved by the developer writing JUnit (an open source framework for implementing unit tests and suites) tests and executing them whenever changes are made to the software.
  • outside of the IDE a WAR (web archive) is deployed into the JBoss or Tomcat application servers and tested from a users point of view in a browser (Internet Explorer and/or Firefox. Sometimes Selenium Testing using the Test Tool developed for the project can be used. Upon successful Unit Testing the code is checked into the Repository and the developer issues a Build Request onto the IntegrationForDailyBuildDescriptions topic which causes the Build Master role to execute a build according to either a normal build schedule or, as an adhoc emergency process. If the request is routine, then it will be satisfied as part of the normal daily build process. A normal part of this process is for the build to be subject to Cross Testing by a crowd all of whom hit the application on a known URL. It is imperative that after the build is done, that all developers whose changes are contained in the deployed build, test their code on the Cross Testing Server.

Cross Testing

On the Blue Collar Screen Server which runs the Linux operating system and the Tomcat application server. Testing on this server currently is manual cross testing (testing by developers and client at times, though not usual). The results from this testing is documented in CrossTestingIntegrationTestingResults. Sometimes based upon the analysis of a result, a defect can be added to the Defect System. Testers render an opinion as to whether or not the code should be promoted to the next level of testing which occurs on a Open VMS environment which has Fortran and DCL code and represents a system very close to the Production environment.

Integration and System Testing

On the Simulation/Test Environment Server (Itanium) which runs Open VMS 8.2 operating system and Tomcat application server. Testing on this server is automated and uses the Selenium/Test Tool suite to simulate user inputs to a browser for selected business transactions. Defects found during this testing is reported on wiki based Defect List.

User Testing on Client Cluster Box (Test 9)

Runs Open VMS 8.2 and Tomcat application server. Testing on this box is manual and done by client and a developer/Tester only. Defects found during this testing are reported on the Defect Tracking System.

Parallel Testing on Client Cluster Box (Prod 9)

Runs Open VMS 8.2 and Tomcat application server. This box is used by Users to mimic transactions done on the production system to make sure that the behavior of the test and production systems are identical for existing functions. At some point, the users determine that the Test System is good enough and then the Test System is made to be the production system.

Defects not fixed within a sprint get prioritized for inclusion in future sprint unless it is an emergency in which case the Emergency Fix process is executed. The team works with Product Owner to determine severity of defect.

Additionally, the team follows an engineering practice called Test Driven Development where JUnit tests are written for the business services delivered vs. waiting for a separate test team to test the delivered code according to a test script. This approach not only improves the quality of the delivered code but, additionally serves to provide automated regression tests for future refactoring or design changes. It was this Test Driven Development philosophy which drove the development of the test tool as the initial activity in the overall project scope.

Training

Training will occur as needed in one of the following forms:

• Via self-study using documentation provided on the Wiki
• Combination of demonstration/training sessions either on-site or via phone conference and online help available via wiki
• Mentoring from experienced project members

Due to the highly interactive nature of the development approach, alot of training will occur through mentoring and Q&A via the wiki.

Schedule and Milestones

As mentioned in the Software LCS section, the Rational Unified Process method divides a project into Inception, Elaboration, Construction and Transition. On this project, Phase I includes Inception and Elaboration. Phase II and III include Construction. Some aspect of the Transition phase may occur in Phase II/III if integration challenges with Client can be addressed early on during the project.

Development Processes

Reviews and Audits

An individual has been identified to ensure that the established Engineering Practices, Coding Standards and Architectural Guidelines are being followed. A topic which defines her responsibilities follows:

• BlueTechnicalBigHead The Technical Big Head role requires that the architect develop side by side with developers on various teams, this way, being able to truly observe the behaviors of each developer, read their code and see quality both from a high level architect view and a developer perspective. The result of this approach has been a backlog of architectural changes to improve developer quality, updates to our build process to increase automation and thus remove opportunities for human error and a number of meetings to make sure that all of the Team members are following our Engineering standards.

Informal Design and Code reviews occur constantly throughout the project as project members read each others code. However, Formal Design/Code Reviews occur at important project transitions: from Elaboration to Construction, when developing a new type of system component as part of the Application Architecture.

The overall Application Architecture being used for this project has undergone numerous reviews as it has been used on numerous projects in different industry domains. However, the implementation must always be reviewed as the code may not always adhere to the architectural and coding standards.

Risks and Risk Management

The project team has created a RiskList which is reviewed at Iteration Planning meetings. The RiskList topic provides more detail on how they are managed. This list, being on the wiki, is available to all stakeholders at all times and there is a feedback section for team members to criticize risks and improve the quality of the content.

Security Considerations

• Development artifacts are stored on Linux servers locked in a server room. Client ensures that they don't house any proprietary information.
• The project team must avoid connection to the Client network.
• The project team will actively engage Client to learn security policies relevant to production deployment and comply with guidelines

Software Engineering Environment

The Software Engineering environment consists of the development, Cross Test, Integration Test, System Test and Parallel Test environments. Details on the configurations for these environments can be found in FfbLmcsComputingEnvironment and FfbTestToolPlatformInformation. Additionally, Configuration Server/Repository Environment (CMSE) is as follows:

• CVS is used for software source code (will be transitioning to Subversion during Transition Phase -- SolomonThompson - 31 Mar 2009 - 05:57)
• Wiki is used for development artifacts and general documentation

Software Standards, Processes, and Procedures

CCB Processes

During Construction Phase (II/III) a configuration control board will be put in place to determine priorities of story implementation and defect/enhancement work. There will be 5 members (should try to keep group less than 5) from Client and Blue Collar.

This group will remain in place through Transition (Phase IV) into Production (Operation/Maintenance). When the system is in production, as mentioned in the Software Configuration Management section, stories where the corresponding left bars have been enabled (by Security/Application Administrator) and are visible to Production Users will be considered baselined. Changes to these baselined stories shall be approved by the Product Owner (proxy to the stakeholders which are members of the CCB and attend the Big Head Meetings). These changes may be defects or enhancements. Additionally, development activities for stories not yet in production will also occur based upon approval given by the Product Owner.

Meetings will be Monthly as part of the Redesign (Big Head) meetings which occur at the Blue Collar Office or Client Office on the Third Thursday of each month (unless rescheduled on a particular month). A quorum is achieved when at least 2 humans from Client and 1 human from Blue Collar are present (present can mean over the phone).

Defect and Enhancement Request Processing

ProblemList (now using an Automated Defect Tracking System -- SolomonThompson - 31 Mar 2009 - 05:57) and ProductBacklog will be reviewed by team or board and prioritized monthly. The maintaining of the ProductBacklog and defects has evolved to be prioritized no less than twice a month and in some cases once per week. The Defects have been moved from the Problem List topic to an automated Defect Tracking Tool which uses a relational database.

Software Estimation

Project work will be defined as a set of business stories (representing business units of work or test units of work and tasks) that will continuously be refined, re-factored, & priortized (by Product Owner). During Sprint Planning, which occurs once per week for one team and every two weeks for other teams, stories which have not been estimated will be estimated using a Delphi type of approach based upon planning poker established by eXtreme Programming teams(a detailed description can be found here, planning poker described) which has the following characteristics

• Stories are estimated relative to each other vs. in terms of time to complete (hours, days, weeks, months etc)
• Potential relative difficulty values are Fibonacci numbers between 1 and 21 (1, 2, 3, 5, 8, 13, 21)
• Each team member has a vote which is kept secret until Scrum Master requests vote from each team member
• If all votes cluster around 2 values the top value is chosen, if there are outliers, then the individuals are questioned on why they estimated so high or so low
  • Team will keep voting until the responses cluster around one or two values and a voting decision is reached

Emphasis will be on identifying correct priorities, continuous integration of additional working stories, and updating estimates for remaining stories.

Ballpark estimates will constantly be developed by categorizing not-started stories into complexity, effort size and using multipliers (based on actual development experience for other stories developed as part of this project.)

Software Release Identification

• IntegrationForDailyBuildDescriptions

Testing

Test cases developed by project team with potential contributions from business users. The Test Tool has been developed to provide automation to the test cases. Testing at the unit, integration and system level will occur throughout the project. Details on testing will be located in separate topics:

• LMCSTestPlan
• CrossTestingIntegrationTestingResults
  • CrossTestingIntegrationTestingArchive
• TestCaseBuildTopic
• TestCaseConcerns
• TestCaseSetupAndExpectedResultsInfoForTestTool
• TestCaseSCREENSResultsList
• UserReviewStatus