This work is licensed under a Creative Commons Attribution-Noncommercial 3.0 Unported License Written permission from Douglas Hoffman is required for exceptions (Contact Doug)

The materials available below are in pdf format. You need an Adobe(tm) Reader 5.0 or later to view them (Download)

Alphabetical Order

Management Topics

Technical Topics

Reverse Chronological Order

Background:
In software testing, the mechanism used to generate the expected results is called an oracle. (In this paper, the first letter will be capitalized when referring to the Oracle for a specific test.) Many different approaches can be used to create, capture, and compare test results. The author, for example, has used the following methods for generating expected results:

• Manual verification of results (human oracle)
• Separate program implementing the same algorithm
• Simulator of the software system to produce parallel results
• Debugged hardware simulator to emulate hardware and software operations
• Earlier version of the software
• Same version of software on a different hardware platform
• Check of specific values for known responses
• Verification of consistency of generated values and end points
• Sampling of values against independently generated expected results

Software tests themselves can be classified in many different ways. Automated tests that include evaluation of results need some kind of oracle regardless of the type or purpose of the tests. Yet, the mechanism for evaluation of results ranges from none (the program or system didn’t crash) to exact (all values, displays, files, etc., are verified). Various levels of effort and exactness are appropriate under different circumstances. The nature and complexity of an oracle is also dependent upon those circumstances.

Presented at 1998 Quality Week (QW), and October, 1998 Los Altos Workshop on Software Testing (LAWST 5)

Taxonomy Slides Taxonomy Paper

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Architecture and Design of Automated Software Tests

Spring 2000 PNSQC Tutorial (1/4 Day Tutorial)

Topics emphasized:

• Why Automate Software Tests?
• Automated Test Design
• Test Automation Strategies
• Automated Test Oracles
• Automation Architectures

NOTE: This tutorial was not about how to use any particular test tool or about code in any particular test tool’s programming language.

Presented at Spring 2000 Pacific Northwest Software Quality Conference Tutorials

Workshop Abstract Automation Architecture Tutorial Slides

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Design of Oracle Based Automated Tests

SQC Germany 2009 (1/2 Day Class)

Five factors about automating results comparison need to be factored into any test automation effort. The talk describes the five factors (listed below), explains why each is an issue, and goes into some of the implications and possible actions to deal with them..

The five factors are:

• Which [potential] results to compare
• What results to expect
• Not all expected differences indicate errors
• "Fuzzy" comparisons
• When to generate and compare results

Presented at 2000 Recife Summer School 2009 (Brazil)

• Accounting Tenents are based on Assumptions, Principles, and Guidelines
• All accounting is based on monetary units (dollars)
• Accounting is done based on Generally Accepted Accounting Principles (GAAP)
• The profession is self-regulated by the Financial Accounting Standards Board (FASB)
• GAAP may be modified to fit different industries

Presented at 2008 CAST

Lessons from Finance paper Lessons from Finance Slides

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Testing Automation Beyond Regression Testing

Fall 2008 STPCon

Presented at Fall 2008 Software Testing and Performance Conference

Presented at 2007 Pacific Northwest Software Quality Conference

Talk presented at ASQ Silicon Valley Section Meeting, September 2007

Graphical Display Slides Graphical Display Zip File

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Why Tests Don't Pass

CAST July, 2009; PNSQC October, 2009

Likewise, failing a test is no guarantee that a bug is present. There could be a bug in the test itself, a configuration problem, corrupted data, or a host of other explainable reasons that do not mean that there is anything wrong with the software being tested. Failing really only means that something that was noticed warrants further investigation.

The talk explains the ideas further, explores some of the implications, and suggests some ways to benefit from this new way of thinking about test outcomes. The talk concludes with examination of how to use this viewpoint to better prepare tests and report results.

Presented at Toronto Association of Systems and Software Quality (TASSQ) March 31, 2009

Presented at Kitchner Waterloo Software Quality Association (KWSQA) April 1, 2009

Paper and presentation at the Conference for the Association for Software Testing (CAST) July 14, 2009

TASSQ Slides CAST Slides Why Not Pass Paper (CAST)

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Non-Regression Test Automation

Pacific Northwest Software Quality Conference 2008

Presented at 2008 Pacific Northwest Software Quality Conference

Non-Regression Slides Non-Regression Paper

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Mutating Automated Tests

2000 STAR East Conference; SSQA

Key points attendees take away:

• Benefits and shortcomings of automated tests
• Types of automated tests that are easy and hard to vary
• Some methods to improve the value of some automated tests
• Examples of non-deterministic automated tests
• Design approaches for creating more powerful automated tests

Summary:
Most automated tests are used as regression tests - doing the same exercises each time the test is run. The paper and talk describe a powerful type of automated test - one that does something different each time it runs. The technique does not apply to all situations of automated tests, but the author presents the pros and cons for mutating automated tests based on his experience with them. The paper also provides several examples.

Presented at 2000 Software Testing, Analysis, and Review (STAR) East Conference, and May, 2000 meeting of Silicon Valley Software Quality Association (SSQA)

Mutating Tests Slides SSQA May 2000 Slides Mutating Tests Paper

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Overview of ASQ’s Certification in Software Quality Engineering

2002 Quality Week Conference

• Certification Requirements
• Levels Of Cognition (from Bloom’s Taxonomy, 1956)

The Subject Areas of the CSQE 2002 Body of Knowledge are:

• General Knowledge, Conduct, and Ethics
• Software Quality Management
• Software Engineering Processes
• Program and Project Management
• Software Metrics, Measurement, and Analytical Methods
• Software Verification and Validation (V&V)
• Software Configuration Management
• Examples Of Performance Skill Levels
• Mapping Of Performance Levels To Job Requirements
• Describing Individual Performance Levels

The emphasis of the tutorial is on mapping knowledge and skill areas into performance measures. By identifying relevant skill areas and performance measures, the quality engineering team can understand what levels of performance are expected and what levels of performance are being shown. Only a brief BOK topical overview is presented, since defining of all the topics in the CSQE BOK in detail would take more time than is available in a one day tutorial.

Presented at 2002 Software Quality Week (QW)

2002 CSQE BOK Slides (6 up) 2002 CSQE BOK Slides (2 up) 2002 CSQE BOK Paper

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Divide and Conquer

January 2005 Better Software Magazine

Published as a "Front Line" article in Better Software Magazine January 2005

Talk presented at ASQ Silicon Valley Section Meeting, November 2005

Divide Slides Divide and Conquer Paper

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Exhausting Your Testing Options

July/August 2003 STQE Magazine

• Failures can lurk in incredibly obscure places
• You could need 4 billion computations to find 2 errors caused by a bug
• Even covering all of the input and result values isn’t enough to be certain

Published as a "Bug Report" within Software Testing and Quality Engineering (STQE) Magazine July/August 2003

The presentation covers the fallacies, how to find more bugs with automated tests, what makes automated tests different from manual tests, typical errors in test automation, the difficulties with most automated results comparisons, where automated tests are valuable, and actions that can be taken to avoid trouble over these problems.

• Automated tests find many bugs.
• Manual tests make good automated tests.
• What to check is clear and simple.
• We know what to expect.
• More automated regression tests are always better.

Presented at Better Software Conference, June, 2009

Presented at 2004 Better Software Conference

Early Testing Without the Test and Test Again Syndrome (Abstract) Early Testing (Slides)

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Avoiding the "Test and Test Again" Syndrome

2007 Conference for AST

CAST 07/2007

Presented at 2007 CAST

Avoiding the Test and Test Again Syndrome (Abstract) Avoiding the Test and Test Again Syndrome (Slides) Avoiding the Test and Test Again Syndrome (Paper)

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Automation Architecture Approaches: Beyond Regression Testing

2007 Conference for AST; 2006 SSQA

There are vast possibilities beyond that open to us when automating tests. When we think of test automation we should first think about doing things that we can not do manually. Based on experience creating non-regression automated tests, these presentations address what and how we can create more powerful automated tests.

The SSQA talk is a one hour presentation about the limitations and how other kinds of test automation may be much more valuable.

The CAST tutorial is a one-day presentation of advanced automated test architectures.

The Tutorial covers:

• The relative strengths and weaknesses of manual and automated testing
• The trouble with automated regression tests
• Architectures for automated oracles to establish pass/fail verdicts
• 8 frameworks for non-regression automated tests
• 12 types of errors discoverable only with automated tests

Automation Architecture Approaches: Beyond Regression (Tutorial Abstract) Automation Architecture (CAST 2007 Slides) SSQA Beyond Regression Slides

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Heuristic Test Oracles

April 1999 STQE Magazine

It is often impractical to exactly reproduce or compare accurate results, but it is not necessary for an oracle to be perfect to be useful. In this article, I describe some ideas associated with what I call heuristic oracles. A heuristic oracle uses simpler consistency checks (heuristics) for the results of a test.

Published as an article "Heuristic Test Oracles" in Software Testing and Quality Engineering (STQE) Magazine April 1999

• Important differences between manual and automated testing that must be factored into a test automation architecture
• The role test results capture and comparison plays in test automation
• A model for software testing useful for designing test automation architectures
• The use of oracles for generation of expected results
• An approach to designing a test automation architecture based on the factors.

Presented at EBSSQA, March 1998 and 1998 International Quality Week Conference

Planning Ahead Slides (EBSSQA) Automation Architectures Slides Automation Architectures

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Automated Testing of Embedded Software

2003 Spring/Fall Software Test Automation Conference

Presented at Spring 2003 Software Test Automation Conference

The presentation focuses on a metric that I’ve seen used in many organizations (readiness for release) and some of the disruptive results in those organizations. I’ve focused on three examples of different metrics that have been used and a few examples of the behaviors elicited by using the metrics. For obvious reasons, the examples have been subtly altered to protect the innocent (or guilty). The three metrics are:

1. Defect find/fix rate
2. Percent of tests running/Percent of tests passing
3. Complex model based metrics (e.g., COCOMO)

Some of the observed behaviors include:

• testers withholding defect reports
• punishment of test groups (and individual testers) for not finding defects sooner
• use of "Pocket lists" of defects by developers and testers blocks of unrelated defects being marked as duplicates of one new consolidated defect (to reduce the defect count)
• artificial shifting of defects to other projects or temporary "unassigning" of defects to reduce the defect count
• changing definitions of what a test or test case is to change the count of tests
• shipping of products with known missing features because 100% testing was achieved
• routine changing of expected results to known incorrect results so the test would pass
• lowered ranking of testers because they weren’t finding defects as quickly as the model showed they should
• holding back on defect reporting and testing because the model showed they shouldn’t be found yet

Presented at 2000 Pacific Northwest Software Quality Conference

Presented at CAST 2006

Dark Side Abstract PNSQC Dark Side Slides PNSQC Dark Side Paper
CAST Dark Side (Slides)

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Cost Benefits for Test Automation

1999 STAR

Cost benefits from automation are viewed as trade-offs in comparison to manual testing (or the current situation). Financial impacts are computed in comparison to two alternatives: manually testing the same thing or not testing (accepting the risk of not knowing). Organizational impacts such as the skills needed to design and implement automated tests, automation tools, automation environments, development, and maintaining automation tools and environments are also discussed.

Test automation is not always necessary or appropriate. Automating existing manual tests is a path frequently chosen by default, but usually is not cost beneficial and sometimes results in decreased test effectiveness. The costs and benefits of test automation can be identified and estimated, and good management decisions made about using automation to improve testing.

Paper presented at 1999 Software Testing, Analysis, and Review (STAR) Conference.

Cost Benefits Abstract Cost Benefits Paper Cost Benefits Slides

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Using Test Oracles in Automation

Spring 2003 Software Test Automation Conference; PNSQC 2001; and Quality Week 2000

Software test automation is often a difficult and complex process. The most familiar aspects of test automation are organizing and running of test cases and capturing and verifying test results. A set of expected results are needed for each test case in order to check the test results. Verification of these expected results is often done using a mechanism called a test oracle. The paper and talks describe the use of oracles in automated software verification and validation. Several relevant characteristics of oracles are included with the advantages, disadvantages, and implications for test automation.

Real world oracles vary widely in their characteristics. Although the mechanics of various oracles may be vastly different, a few classes can be identified which correspond with automated test approaches. These types of oracles are categorized based upon the strategy for verification using the oracle. Thus, an oracle strategy using a lookup table to generate expected results can be the same as one that uses an alternate algorithm implementation to compute them. Four types of oracle strategies (and not using any oracle) are identified and defined. The strategies are labeled True, Heuristic, Consistency, and Self Referential.

Slides presented at Spring 2003 Software Test Automation Conference 03/2003

Paper presented at PNSQC 2001

Oracles In Automation Paper Oracles In Automation Slides

Tutorial Slides presented at Quality Week 2000

Paper and presentation made at Quality Week, 1994 PNSQC 1994, and the 4th International Conference on Software Quality (4ICSQ))

PNSQC Measuring Consulting Paper 4ICSQ Measuring the Quality of Software Consulting (paper)

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Requirements for Test Automation

2000

Presentation at PNSQC 2000 and SSQA 12/2000

Paper and presentation made at 1993 Quality Week 10/1993

Relationship to Development Paper Relationship to Development Slides

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Review Comments on the Proposed IEEE Software Engineering Body of Knowledge V1.00 (2003)

Personal Email to IEEE SWEBOK Committee, June 2003

I spent several hours going into the first two chapters before skipping to the chapters on Testing and then Software Quality. I was encouraged by the explicit recognition that different organizations, users, and products require different techniques in both chapters. But, I was discouraged by the many deficiencies in the Testing chapter and the gaping holes in the Software Quality chapter. I was shocked by the fact that no reference is made to ASQ’s CSQE, if only to criticize it. Either the drafters of the SQEBOK were really ignorant of the existence of a sister society’s related Software Quality Engineering Body of Knowledge, or they choose to ignore it because it was inconvenient or at odds with SWEBOK. In my opinion, if the first case is true, they were incompetent. In my opinion, if the second case was true, they committed professional malpractice.

In any case, they choose not to respond to or even acknowledge any of my input in the 2004 publication of the "SWEBOK_Guide_2004". The 2004 SWEBOK still does not acknowledge the ASQ or the CSQE BOK. I also find it curious that the Preface states that the ACM was active in creating the joint committee, that they approved the code of ethics in 1998, and they were working on an alternate educational curriculum, but the Preface fails to mention that ACM rejected the SWEBOK itself.

• Determining metrics expenditures’ place within your I/T budget
• Analyzing metrics results
• Refining your metrics program
• How do you measure the ROI of your metrics program?
• Short term testing for metrics justification
• Ensuring yours is a flexible metrics program for optimal long-term performance

The speaker notes are available with the slides.

Metrics for Metrics (Slides)" Metrics for Metrics (Notes)"

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

What Makes Software Quality Different?

ASQ SV Section 2003

• Software is developed, not manufactured
• Software development processes vary tremendously
• Software is easily modified
• Side effects of software changes are not well understood
• Change control and version management frequently aren’t rigorous
• Quality records are easily modified and rarely secure
• Standard MTBF and MTTR types of measures don’t mean the same as with hardware

Paper and presentation made at 1993 PNSQC 10/1993

Organization Roles Paper Organization Roles Slides

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Failure Modes and Effects Analysis

May, 2000 ASQ Presentation

Talk presented at May 19, 2000 ASQ Silicon Valley Section Dinner Meeting.

This talk describes the major characteristics of the regulations and implications for software testing.

Talk presented at May 19, 2000 ASQ Silicon Valley Section Dinner Meeting.

Likewise, failing a test is no guarantee that a bug is present. There could be a bug in the test itself, a configuration problem, corrupted data, or a host of other explainable reasons that do not mean that there is anything wrong with the software being tested. Failing really only means that something that was noticed warrants further investigation.

The paper explains the ideas further, explores some of the implications, and suggests some ways to benefit from this new way of thinking about test outcomes. The paper concludes with examination of how to use this viewpoint to better prepare tests and report results.

Talk presented at October, 2009 Pacific Northwest Software Quality Conference (PNSQC).

Why Tests Do Not Pass (Slides) Why Tests Do Not Pass (Paper)

Return to Top | Alphabetical Order | Management Topics | Technical Topics | Reverse Chronological Order

**********************************************************************

Bugs For Sale

April, 2010 IV EBTS

The talk explains how to sell bugs and how the selling of bugs helps get more bugs fixed and better decisions made. It describes the kind of information it takes to sell a bug.

The main points made include:

1. Motivating a person to buy
2. Overcoming objections
3. Identifying the audience
4. Capturing the important information
5. Keeping it simple

Talk presented at IV Encontro Brasileiro de Testes de Software (IV EBTS).

The talk describes the nine types (listed below), explains what each is, and goes into some of their applications and possible mechanisms to use them.

The nine types are:

1. Complete
2. Heuristic
3. Statistical
4. Consistency
5. Self-verifying
6. Model-based
7. Inverse function
8. Hand crafted
9. None

Talk presented at IV Encontro Brasileiro de Testes de Software (IV EBTS).

Updated April 20, 2010

Copyright (C) 1995-2010 Software Quality Methods, LLC. All Rights Reserved.

If buttons at top or on the side of the page are not showing, click here to download free java software.