This work is licensed under a Creative Commons Attribution-Noncommercial 3.0 Unported License Written permission from Douglas Hoffman is required for exceptions ( Contact Doug)

The materials available below are in pdf format. You need an Adobe(tm) Reader 5.0 or later to view them (Download)

Alphabetical Order

Management Topics

Technical Topics

Reverse Chronological Order

Background:
In software testing, the mechanism used to generate the expected results is called an oracle. (In this paper, the first letter will be capitalized when referring to the Oracle for a specific test.) Many different approaches can be used to create, capture, and compare test results. The author, for example, has used the following methods for generating expected results:

• Manual verification of results (human oracle)
• Separate program implementing the same algorithm
• Simulator of the software system to produce parallel results
• Debugged hardware simulator to emulate hardware and software operations
• Earlier version of the software
• Same version of software on a different hardware platform
• Check of specific values for known responses
• Verification of consistency of generated values and end points
• Sampling of values against independently generated expected results

Software tests themselves can be classified in many different ways. Automated tests that include evaluation of results need some kind of oracle regardless of the type or purpose of the tests. Yet, the mechanism for evaluation of results ranges from none (the program or system didn't crash) to exact (all values, displays, files, etc., are verified). Various levels of effort and exactness are appropriate under different circumstances. The nature and complexity of an oracle is also dependent upon those circumstances.

Presented at 1998 Quality Week (QW), and October, 1998 Los Altos Workshop on Software Testing (LAWST 5)

Taxonomy Slides Taxonomy Paper

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Architecture and Design of Automated Software Tests

Spring 2000 PNSQC Tutorial (1/4 Day Tutorial)

Topics emphasized:

• Why Automate Software Tests?
• Automated Test Design
• Test Automation Strategies
• Automated Test Oracles
• Automation Architectures

NOTE: This tutorial was not about how to use any particular test tool or about code in any particular test tool's programming language.

Presented at Spring 2000 Pacific Northwest Software Quality Conference Tutorials

Workshop Abstract Automation Architecture Tutorial Slides

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Testing Automation Beyond Regression Testing

Fall 2008 STPCon

Presented at Fall 2008 Software Testing and Performance Conference

Presented at 2007 Pacific Northwest Software Quality Conference

Talk presented at ASQ Silicon Valley Section Meeting, September 2007

Graphical Display Slides Graphical Display Zip File

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Mutating Automated Tests

2000 STAR East Conference; SSQA

Key points attendees take away:

• Benefits and shortcomings of automated tests
• Types of automated tests that are easy and hard to vary
• Some methods to improve the value of some automated tests
• Examples of non-deterministic automated tests
• Design approaches for creating more powerful automated tests

Summary:
Most automated tests are used as regression tests - doing the same exercises each time the test is run. The paper and talk describe a powerful type of automated test - one that does something different each time it runs. The technique doesn't apply to all situations of automated tests, but the author presents the pros and cons for mutating automated tests based on his experience with them. The paper also provides several examples.

Presented at 2000 Software Testing, Analysis, and Review (STAR) East Conference, and May, 2000 meeting of Silicon Valley Software Quality Association (SSQA)

Mutating Tests Slides SSQA May 2000 Slides Mutating Tests Paper

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Overview of ASQ's Certification in Software Quality Engineering

2002 Quality Week Conference

• Certification Requirements
• Levels Of Cognition (from Bloom's Taxonomy, 1956)

The Subject Areas of the CSQE 2002 Body of Knowledge are:

• General Knowledge, Conduct, and Ethics
• Software Quality Management
• Software Engineering Processes
• Program and Project Management
• Software Metrics, Measurement, and Analytical Methods
• Software Verification and Validation (V&V)
• Software Configuration Management
• Examples Of Performance Skill Levels
• Mapping Of Performance Levels To Job Requirements
• Describing Individual Performance Levels

The emphasis of the tutorial is on mapping knowledge and skill areas into performance measures. By identifying relevant skill areas and performance measures, the quality engineering team can understand what levels of performance are expected and what levels of performance are being shown. Only a brief BOK topical overview is presented, since defining of all the topics in the CSQE BOK in detail would take more time than is available in a one day tutorial.

Presented at 2002 Software Quality Week (QW)

2002 CSQE BOK Slides (6 up) 2002 CSQE BOK Slides (2 up) 2002 CSQE BOK Paper

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Divide and Conquer

January 2005 Better Software Magazine

Published as a "Front Line" article in Better Software Magazine January 2005

Talk presented at ASQ Silicon Valley Section Meeting, November 2005

Divide Slides Divide and Conquer Paper

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Exhausting Your Testing Options

July/August 2003 STQE Magazine

• Failures can lurk in incredibly obscure places
• You could need 4 billion computations to find 2 errors caused by a bug
• Even covering all of the input and result values isn't enough to be certain

Published as a "Bug Report" within Software Testing and Quality Engineering (STQE) Magazine July/August 2003

Presented at 2004 Better Software Conference

Early Testing Without the Test and Test Again Syndrome (Abstract) Early Testing (Slides)

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Avoiding the "Test and Test Again" Syndrome

2007 Conference for AST

CAST 07/2007

Presented at 2007 CAST

Avoiding the Test and Test Again Syndrome (Abstract) Avoiding the Test and Test Again Syndrome (Slides) Avoiding the Test and Test Again Syndrome (Paper)

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Automation Architecture Approaches: Beyond Regression Testing

2007 Conference for AST; 2006 SSQA

There are vast possibilities beyond that open to us when automating tests. When we think of test automation we should first think about doing things that we can't do manually. Based on experience creating non-regression automated tests, these presentations address what and how we can create more powerful automated tests.

The SSQA talk is a one hour presentation about the limitations and how other kinds of test automation may be much more valuable.

The CAST tutorial is a one-day presentation of advanced automated test architectures.

The Tutorial covers:

• The relative strengths and weaknesses of manual and automated testing
• The trouble with automated regression tests
• Architectures for automated oracles to establish pass/fail verdicts
• 8 frameworks for non-regression automated tests
• 12 types of errors discoverable only with automated tests

Automation Architecture Approaches: Beyond Regression (Tutorial Abstract) Automation Architecture (CAST 2007 Slides) SSQA Beyond Regression Slides

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Heuristic Test Oracles

April 1999 STQE Magazine

It is often impractical to exactly reproduce or compare accurate results, but it isn't necessary for an oracle to be perfect to be useful. In this article, I describe some ideas associated with what I call heuristic oracles. A heuristic oracle uses simpler consistency checks (heuristics) for the results of a test.

Published as an article "Heuristic Test Oracles" in Software Testing and Quality Engineering (STQE) Magazine April 1999

• Important differences between manual and automated testing that must be factored into a test automation architecture
• The role test results capture and comparison plays in test automation
• A model for software testing useful for designing test automation architectures
• The use of oracles for generation of expected results
• An approach to designing a test automation architecture based on the factors.

Presented at EBSSQA, March 1998 and 1998 International Quality Week Conference

Planning Ahead Slides (EBSSQA) Automation Architectures Slides Automation Architectures

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Automated Testing of Embedded Software

2003 Spring/Fall Software Test Automation Conference

Presented at Spring 2003 Software Test Automation Conference

The presentation focuses on a metric that I've seen used in many organizations (readiness for release) and some of the disruptive results in those organizations. I've focused on three examples of different metrics that have been used and a few examples of the behaviors elicited by using the metrics. For obvious reasons, the examples have been subtly altered to protect the innocent (or guilty). The three metrics are:

1. Defect find/fix rate
2. Percent of tests running/Percent of tests passing
3. Complex model based metrics (e.g., COCOMO)

Some of the observed behaviors include:

• testers withholding defect reports
• punishment of test groups (and individual testers) for not finding defects sooner
• use of "Pocket lists" of defects by developers and testers blocks of unrelated defects being marked as duplicates of one new consolidated defect (to reduce the defect count)
• artificial shifting of defects to other projects or temporary "unassigning" of defects to reduce the defect count
• changing definitions of what a test or test case is to change the count of tests
• shipping of products with known missing features because 100% testing was achieved
• routine changing of expected results to known incorrect results so the test would pass
• lowered ranking of testers because they weren't finding defects as quickly as the model showed they should
• holding back on defect reporting and testing because the model showed they shouldn't be found yet

  Presented at 2000 Pacific Northwest Software Quality Conference

  Presented at CAST 2006

  Dark Side Abstract PNSQC Dark Side Slides PNSQC Dark Side Paper
  CAST Dark Side (Slides)
  Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

  **********************************************************************

  Cost Benefits for Test Automation

  1999 STAR

  There are many factors to consider when planning for test automation. Automation changes the complexion of testing from design through implementation and test execution. It is important to understand the potential costs and benefits before undertaking the kind of change automation implies. Automated tests can be incredibly effective, giving more coverage and new visibility into the software under test. However, it often provides us with opportunities for testing in ways impractical or impossible for manual testing; yet conventional metrics may not show any improvements. This presentation describes financial, organizational, and test effectiveness impacts observed when software test automation is installed. Equations, suggestions and examples are provided to help decide when automation is beneficial.

  Cost benefits from automation are viewed as trade-offs in comparison to manual testing (or the current situation). Financial impacts are computed in comparison to two alternatives: manually testing the same thing or not testing (accepting the risk of not knowing). Organizational impacts such as the skills needed to design and implement automated tests, automation tools, automation environments, development, and maintaining automation tools and environments are also discussed.

  Test automation is not always necessary or appropriate. Automating existing manual tests is a path frequently chosen by default, but usually is not cost beneficial and sometimes results in decreased test effectiveness. The costs and benefits of test automation can be identified and estimated, and good management decisions made about using automation to improve testing.

  Paper presented at 1999 Software Testing, Analysis, and Review (STAR) Conference.

  Cost Benefits Abstract Cost Benefits Paper Cost Benefits Slides
  Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

  **********************************************************************

  Using Test Oracles in Automation

  Spring 2003 Software Test Automation Conference; PNSQC 2001; and Quality Week 2000

  These presentations and slides show a progression of understanding and applying of automated software test oracles.

  Software test automation is often a difficult and complex process. The most familiar aspects of test automation are organizing and running of test cases and capturing and verifying test results. A set of expected results are needed for each test case in order to check the test results. Verification of these expected results is often done using a mechanism called a test oracle. The paper and talks describe the use of oracles in automated software verification and validation. Several relevant characteristics of oracles are included with the advantages, disadvantages, and implications for test automation.

  Real world oracles vary widely in their characteristics. Although the mechanics of various oracles may be vastly different, a few classes can be identified which correspond with automated test approaches. These types of oracles are categorized based upon the strategy for verification using the oracle. Thus, an oracle strategy using a lookup table to generate expected results can be the same as one that uses an alternate algorithm implementation to compute them. Four types of oracle strategies (and not using any oracle) are identified and defined. The strategies are labeled True, Heuristic, Consistency, and Self Referential.

  Slides presented at Spring 2003 Software Test Automation Conference 03/2003

  Oracles In Automation Slides

  Paper presented at PNSQC 2001

  Oracles In Automation Paper Oracles In Automation Slides

  Tutorial Slides presented at Quality Week 2000

  Oracles In Automation Tutorial Slides
  Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

  **********************************************************************

  Measuring the Quality of Consulting

  Quality Week 1994; PNSQC 1994; 4ICSQ

  These papers describe an innovative process for measuring and improving quality for quality assurance professionals software contracted through an agency. The process was implemented over a three-year period and used by their clients and software quality professionals of Systems Partners (a consulting agency). The papers provide brief histories of the program, the main elements and mechanisms, and some of the results obtained.

  Paper and presentation made at Quality Week, 1994 PNSQC 1994, and the 4th International Conference on Software Quality (4ICSQ))

  PNSQC Measuring Consulting Paper 4ICSQ Measuring the Quality of Software Consulting (paper)
  Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

  **********************************************************************

  Requirements for Test Automation

  2000

  Automating testing is like any software development project, and as such we need to articulate the requirements. Test automation should not be done piecemeal or in an ad hoc fashion because the resulting work products will become more and more expensive to use, eventually becoming obsolete or insupportable. The presentation describes many special considerations for automation requirements and a method for understanding and organizing them.

  Presentation at PNSQC 2000 and SSQA 12/2000

  Requirements for Test Automation
  Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

  **********************************************************************

  The Software Quality Group's Relationship to Development

  Quality Week 2003

  This paper presents the roles of the Software Quality Organization in software development as observed in dozens of commercial organizations. It describes the development process as the way quality is built into software. It also looks at the different ways the quality group's purpose and charters were viewed. The potential benefits and drawbacks for various charters are described, along with the organizational structure and typical activities for each. The idea that the charter for the quality group changes over time is also presented, along with observed progressions in organizations. The various possible organizations, charters, and roles are described and related briefly to quality systems described in both the SEI Maturity Model and ISO 9000 Standards (ISO 9001 and ISO 9000-3). In summary, it describes the impact on product quality of the different types of development process and possible roles for the software quality group.

  Paper and presentation made at 1993 Quality Week 10/1993

  Relationship to Development Paper Relationship to Development Slides
  Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

  **********************************************************************

  Review Comments on the Proposed IEEE Software Engineering Body of Knowledge V1.00 (2003)

  Personal Email to IEEE SWEBOK Committee, June 2003

  The IEEE Software Engineering Coordinating Committee put out the SWEBOK (Trial Version 1.00 - May 2001) for review and feedback. I felt compelled to provide 12 pages of comments because of my extremely strong [negative] reaction.

  I spent several hours going into the first two chapters before skipping to the chapters on Testing and then Software Quality. I was encouraged by the explicit recognition that different organizations, users, and products require different techniques in both chapters. But, I was discouraged by the many deficiencies in the Testing chapter and the gaping holes in the Software Quality chapter. I was shocked by the fact that no reference is made to ASQ’s CSQE, if only to criticize it. Either the drafters of the SQEBOK were really ignorant of the existence of a sister society’s related Software Quality Engineering Body of Knowledge, or they choose to ignore it because it was inconvenient or at odds with SWEBOK. In my opinion, if the first case is true, they were incompetent. In my opinion, if the second case was true, they committed professional malpractice.

  In any case, they choose not to respond to or even acknowledge any of my input in the 2004 publication of the "SWEBOK_Guide_2004". The 2004 SWEBOK still does not acknowledge the ASQ or the CSQE BOK. I also find it curious that the Preface states that the ACM was active in creating the joint committee, that they approved the code of ethics in 1998, and they were working on an alternate educational curriculum, but the Preface fails to mention that ACM rejected the SWEBOK itself.

  "SWEBOK Observations" for IEEE-SWEBOK V1.0 1 Page Summary of My "SWEBOK Observations"
  Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

  **********************************************************************

  Metrics for Metrics: Cost Analysis and Justifications

  Developing Strategic I/T Metrics, May 1988

  This talk was targeted at IT managers and CFOs who were developing metrics programs. The talk discusses some of the key elements in determining where to direct and evaluate a software metrics program:
  • Determining metrics expenditures' place within your I/T budget
  • Analyzing metrics results
  • Refining your metrics program
  • How do you measure the ROI of your metrics program?
  • Short term testing for metrics justification
  • Ensuring yours is a flexible metrics program for optimal long-term performance

  The speaker notes are available with the slides.

  Metrics for Metrics (Slides)" Metrics for Metrics (Notes)"
  Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

  **********************************************************************

  What Makes Software Quality Different?

  ASQ SV Section 2003

  Quality models for software are often based on hardware parallels or assembly line concepts. Although these models may be convenient for people coming from other industry segments, they ignore significant differences that can lead to counter productive processes, metrics, and controls. The talk discusses some of the reasons software quality assurance is different, for example:
  • Software is developed, not manufactured
  • Software development processes vary tremendously
  • Software is easily modified
  • Side effects of software changes are not well understood
  • Change control and version management frequently aren’t rigorous
  • Quality records are easily modified and rarely secure
  • Standard MTBF and MTTR types of measures don’t mean the same as with hardware
  "What Makes Software Quality Different (Slides)
  Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

  **********************************************************************

  The Role of the Quality Group in Software Development

  PNSQC 2003

  This paper describes the role of the quality organization in software development as observed in dozens of commercial organizations. It views the different charters and purposes the quality groups have. The potential benefits and drawbacks for various charters are presented, along with the organizational structure and typical activities in each. The charter for the quality group changes over time, and observations of progressions in organizations are made. The various possible organizations, charters, and roles are described and related briefly to quality systems described in both the SEI Capability Maturity Model and ISO 9000 Standards (ISO 9001 and ISO 9000-3). The impact on product quality of the different types of development process, and possible roles for the quality group are also covered.

  Paper and presentation made at 1993 PNSQC 10/1993

  Organization Roles Paper Organization Roles Slides
  Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

  **********************************************************************

  Failure Modes and Effects Analysis

  May, 2000 ASQ Presentation

  Failure Modes and Effects Analysis (FMEA) is a risk analysis and prioritization method developed for the aerospace industry in the 1960s. This talk describes how the FMEA techniques can be applied to software.

  Talk presented at May 19, 2000 ASQ Silicon Valley Section Dinner Meeting.

  FMEA Slides
  Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

  **********************************************************************

  21CFR Part 11: Electronic Signatures, Electronic Records

  May, 2000 ASQ Presentation

  The US Food and Drug Administration (FDA) has issued regulations that apply to the security of Electronic Signatures and Electronic Records. The Code of Federal Regulations, Section 21, Part 11 (21CFR11) is the model for information security being considered for all government agencies.

  This talk describes the major characteristics of the regulations and implications for software testing.

  Talk presented at May 19, 2000 ASQ Silicon Valley Section Dinner Meeting.

  21CFR11 Slides
  Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

  **********************************************************************

  
  

  Updated April 18, 2008

  Copyright © 1995-2007 Software Quality Methods, LLC. All Rights Reserved.