F05SENG411 L18

From Craig

Software Quality and Quality Assurance

There is very little agreement on the definition of quality within in the fields of Computer Science and Software Engineering. Some people regard software quality as a series of aesthetic and practical issues while others regard quality as a strict adherence to requirements and the absence of faults.

What is quality software?

What characteristics does quality software exhibit?

• Conformance to requirements
  • Correctness
  • Reliability
• Robustness
• Completeness
• Absense of faults
  • Behaves as it should
• Fault-tolerant
• Flexible
• Maintainable
  • Low maintenance costs over the long term
  • Understandable
  • Documentation
  • Clearly written code
• Conforms to a vision
  • Specified architecture
  • Conceptual integrity
• Simplicity
• Scalability

In-class discussion

For all of the above characteristics, what are the positive and negative contributors? Is it possible to measure the relative conformance/non-conformance to the above characteristics of a given piece of software?

Issues of Quality Control/Engineering

History of Quality Control

• Terms quality control and quality engineering are relatively new although the ideas have been around for many years
  • Guilds took on the responsibility of quality control
    • Craftsmen had to join a local guild
    • Guilds levies punishments against craftsmen who turned out low quality products.
  • Governments were interested in Quality control
    • King John of England appoined William Wrotham to supervise the construction and repair of ships. Several centuries later, the Secretary of the Admiralty appointed multiple such overseers.

• Prior to the division of labour, it was possible for an individual workman to control the quality of his own products
  • Working conditions were more suited to producing quality products
  • With mechanisation and the inevitable division of labour, an individual workman could no longer guarantee the quality of the finished product.

• The industrial revolution changed the very nature of work
  • Large groups of people performing the same tasks were typically grouped together
  • A foreman was then assigned to provide quality control for the group

• During WWI, the manufacturing processes became more complex as the tools of war became increasingly complex.
  • Tanks, Machinery, Munitions.
  • Saw the introduction of mass production and piece work.
    • Piecework caused a large drop in quality as the salary of the producer was typically tied to the number of items which could be completed.
    • The drop in quality lead to the introduction of full-time inspectors (beginnings of modern quality control processes).
    • This lead to the development of inspection organizations in the 1920s and 1930s

• By WWII, it became necessary to introduce more stringent forms of quality control
  • Introduction of statistical methods
  • SPC: Statistical Process Control. Walter A. Shewhart, Bell Labs.
• It was observed that quality could not be inspected into a product
  • THe product itself was either of high or low quality or somewhere in between.
• Statistical methods lead to sampling of inspection rather than 100% inspection.

• After WWII, the general tendency was to explore new forms of quality control based on collaboration. In the US, the focus remained on non-collaborative statistical methods and the new forms of quality control were largely ignored.

Quality Assurance

• In terms of physical products, quality assurance covers all aspects of product development, installation and servicing. From a software perspective, quality assurance covers all aspects of development from requirements elicitation through to deployment and maintenance.

• A major process of quality assurance involves various forms of testing. In physical systems, testing is typically based on sampling.

• By the 1980s, poor quality tended to be viewed as a problem with process.
  • If a product's quality was poor/low, that reflected a defect within the process which was used to create the product.
  • Quality issues became a matter not only of the manufacturing process, but all of the surrounding/supporting processes as well
    • Job Management, Processes
    • Competence, knowledge, skills, experience, qualifications
    • Personal Integrity, confidence, teamwork

• Must be "buy-in" from upper management.
  • Allocate resources to support quality
• Must be "buy-in" from front-line workers.
  • Workers sabotaging their own work

Quality assurance protocols and processes

ISO 9000

• Widely accepted standards for Quality Assurance programs.
  • Based on the British Standard Institute's BS 5750.
  • The Standard is now maintained by the ISO

ISO 9000 - History

• During the second world war, deadly incidents within munitions manufacturing facilities were commonplace.
• This caused England to place quality control inspectors within munitions factories
• Typically, a safe process for manufacturing munitions was formulated and inspectors were charged with the responsibility of ensuring that these processes were followed.
• To supply the military or government with products, a company had to document their processes and then have those processes inspected and approved by the military or government
  • Focus shifted from manufacturing to reports/documents.
• As technology increased, the bureaucracy introduced made it difficult (if not impossible) to comply.
  • Processes were not generic

ISO9000 is about the creation of a generic quality control process which could be adopted by many different organizations.

• Initial drafts 1987 were focused on conformance with manufacturing processes.
• 1994 saw the introduction of quality assurance through preventative actions.
  • Overly focused on documentation
• The latest standards (2000) focus on process management.
  • "A documented system" rather than a "system of documents".
  • Use of process performance metrics

ISO 9001 Certification

• The applying organization is assessed based on an intensive inspection of its site.
• Certification must be renewed on an ongoing basis. (Usually every 3 years).
• Includes internal and external auditing procedures
  • Auditing provides a framework whereby the processes are continually reviewed
  • The review allows for subsequent improvement.
  • Auditing is primarily focused on:
    • Tell me what you do
    • Show me where what you do is documented
    • Prove to me that it actually happens

ISO 9000 Criticisms

• Too much focus on the standard itself with little understanding of the underlying philosophy
• May not be applicable to areas involving creativity where the process cannot be documented
• Processes and business environments are continually evolving
• Too much focus on documentation and too little focus on what is really going on
• Many companies certify with ISO 9001 because it is a good marketing tool for their marketplace rather than whether it is a standard which is applicable to their business.
• Few objective metrics actually show the effectiveness of ISO 9001.

Total Quality Management

• Originally called Total Quality Control. (Translated from Japanese where the words control and management are the same.

TQM comprises for processes:

• Continuous process improvement. Processes must be visible, repeatable and measurable
• Endeavour to understand the intangible effects on processes and ways to optimize and reduce their effects
• Examine the mechanism by which the user applies the product and improve the product based on those observations
• Focus management perspective beyond the immediate product

TQM Mechanisms

• Sample random selections of the product
• Test samples for conformance to users' goals
• When failures are detected, isolate components within process which lead to the failure
• Continual measurement to prevent "drift". Thresholds are identified in an attempt to identify quality drift. This will inevitably lead to changes in the process.

Six Sigma

• Based on standard deviation in statistics
  • Defects should be beyond 6 standard deviations from the mean.
  • This equates to approximately 1 defect / 2 billion opportunities
  • In reality, the process is expected to drift by as much as 1.5 standard deviations so application of six sigma reflects 3.4 defects/million opporunities
• An opportunity is defined as a chance for non-conformance or not meeting the the required product specifications.

Basic Methods of Six Sigma

DMAIC

• Define: how much the process needs to improve
• Measure: obtain baseline measurements for future comparison
• Analyze: determine the relationships between potential root causes of defects and their effects
• Improve: improve the process
• Control: control the causes and monitor the effects

DMADV

• Define:how much the process needs to improve
• Measure: obtain baseline measurements for future comparison
• Analyze:determine the relationships between potential root causes of defects and their effects
• Design: design the process to match the customer's needs
• Verify: the performance of the design

Six Sigma is implemented by two types of trained personnel

• Black belts: On site implementation experts who develop, mentor and manage the six sigma process. Directly responsible for the execution of projects
• Green belts: deploy six sigma techniques and lead small scale aspects of the process. Do the grunt-work and gathering data required by the Black Belts.

Criticisms of Six Sigma

• Statisticians are very critical of the statistical aspects of Six Sigma. Claim the name is really just a marketting term because "4.5 Sigma" wouldn't sound as great.
• Concern that universal standards are being applied to all six sigma projects
• Not really a methodology. Just an unending series of process improvements applied to tactical decisions rather than to strategic decisions.