This work is licensed under a
Creative Commons Attribution-Noncommercial
3.0 Unported License
Written permission from Douglas Hoffman is required for exceptions (
Contact Doug)
The materials available below are in pdf format. You need an Adobe(tm) Reader 5.0 or later to view them (Download)
| Title | Date | Purpose |
|---|---|---|
| Advanced Automation Architectures (Tutorial) | 07/2007 | Conference for the Association for Software Testing (CAST) 2007 |
| Analysis of The Taxonomy of Test Oracles | 10/1998 | Fifth Los Altos Workshop on Software Testing |
| Architecture and Design of Automated Software Tests | 05/2000 | PNSQC Spring Workshop 2000 |
| Automated Testing of Embedded Software | 03/2003 | Spring 2003 Software Test Automation Conference |
| Avoiding the "Test and Test Again" Syndrome | 07/2007 | Conference for the Association for Software Testing (CAST) 2007 |
| Cost Benefits for Test Automation | 10/1999 | STAR West 1999 |
| The Darker Side of Metrics | 10/2000 | PNSQC 2000 |
| Divide and Conquer | 1/2005 | Better Software, "Front Line" January 2005 |
| Early Testing Without the Test and Test Again Syndrome | 11/2006 | SSQA |
| Exhausting Your Test Options | 07/2003 | Software Testing and Quality Engineering |
| Failure Mode and Effects Analysis | 05/2000 | ASQ Section 0613 |
| Foundations of Software Quality | 1994 | ASQ Section 0613 Tutorial |
| Fundamentals of Software Testing | 1995 | ASQ Section 0613 Tutorial |
| Fundamentals of Software Quality Assurance | 1992-1995 | ASQ Section 0613 Tutorial |
| A Graphical Display of Testing Status for Complex Configurations | 10/2008 | PNSQC |
| Heuristic Test Oracles | 04/1999 | Software Testing and Quality Engineering |
| Measuring the Quality of Software Consulting | 10/1994 | Fourth International Conference on Software Quality |
| A Method for Measuring Quality of Software Consulting | 1994 | ASQ Section 0613 |
| Metrics for Metrics: Cost Analysis and Justification | 05/1998 | Developing Strategic I/T Metrics Conference 1998 |
| Mutating Automated Tests | 05/2000 | SSQA |
| Mutating Automated Tests | 04/2000 | Software Testing Analysis & Review (STAR) East 2000 |
| Overview of ASQ's Certified Software Quality Engineer (CSQE) | 09/2002 | Quality Week 2002 |
| Practical Software Metrics | 1993 | |
| A Process for Measuring the Quality of Software Consulting | 05/1994 | PNSQC |
| Requirements for Test Automation | 12/2000 | SSQA |
| Requirements for Test Automation | 10/1999 | PNSQC 1999 |
| The Software Quality Group's Relationship to Development | 05/1993 | Quality Week 1993 |
| Some Measures of Quality of Software Consulting 1994 (ASQ) | 1994 | SSQA |
| SWEBOK, Feedback to IEEE on | 06/2003 | Review feedback to IEEE |
| Taxonomy of Test Oracles, Analysis of The | 05/1998 | Quality Week 1998 |
| The "Test and Test Again" Syndrome | 10/2004 | 2004 Better Software Conference |
| Test Automation Architectures: Planning for Test Automation | 05/1999 | Quality Week Conference 1999 |
| Test Oracles; Planning Ahead for Test Automation | 03/1998 | East Bay SSQA (EBSSQA) |
| Testing Automation Beyond Regression Testing | 4/2008 | Spring STPCon |
| 21 CFR Part 11: Electronic Signatures, Electronic Records | 05/2000 | ASQ Section 0613 |
| Using Test Oracles in Automation | 03/2003 | Spring 2003 Software Test Automation Conference |
| Using Test Oracles in Automation | 10/2001 | 2001 Pacific Northwest Software Quality Conference (PNSQC 2001) |
| Using Test Oracles in Automation | 05/2000 | Quality Week 2000 Tutorial |
|
What's Different About Software |
11/2001 | ASQ Golden Gate Section |
|
What's Different About Software |
04/2001 | ASQ Section 0613 |
| Title | Date | Purpose |
|---|---|---|
| 21 CFR Part 11: Electronic Signatures, Electronic Records | 05/2000 | ASQ Section 0613 |
| A Process for Measuring the Quality of Software Consulting | 05/1994 | PNSQC |
| Avoiding the "Test and Test Again" Syndrome | 07/2007 | Conference for the Association for Software Testing (CAST) 2007 |
| Cost Benefits for Test Automation | 10/1999 | STAR West 1999 |
| Divide and Conquer | 1/2005 | Better Software, "Front Line" January 2005 |
| Failure Mode and Effects Analysis | 05/2000 | ASQ Section 0613 |
| Fundamentals of Software Quality Assurance | 1992-1995 | ASQ Section 0613 Tutorial |
| Measuring the Quality of Software Consulting | 10/1994 | Fourth International Conference on Software Quality |
| Metrics for Metrics: Cost Analysis and Justification | 05/1998 | Developing Strategic I/T Metrics Conference 1998 |
| Overview of ASQ's Certified Software Quality Engineer (CSQE) | 09/2002 | Quality Week 2002 |
| Requirements for Test Automation | 10/1999 | PNSQC 1999 |
| The Darker Side of Metrics | 10/2000 | PNSQC 2000 |
| The Software Quality Group's Relationship to Development | 05/1993 | Quality Week 1993 |
| The "Test and Test Again" Syndrome | 10/2004 | 2004 Better Software Conference |
| Early Testing Without the Test and Test Again Syndrome | 07/2007 | Conference for the Association for Software Testing (CAST) 2007 |
|
What's Different About Software |
04/2001 | ASQ Section 0613 |
| Title | Date | Purpose |
|---|---|---|
| Advanced Automation Architectures (Tutorial) | 07/2007 | Conference for the Association for Software Testing (CAST) 2007 |
| Analysis of The Taxonomy of Test Oracles | 05/1998 | Quality Week 1998 |
| Architecture and Design of Automated Software Tests | 05/2000 | PNSQC Spring Workshop 2000 |
| Automated Testing of Embedded Software | 03/2003 | Spring 2003 Software Test Automation Conference |
| Exhausting Your Test Options | 07/2003 | Software Testing and Quality Engineering |
| Failure Mode and Effects Analysis | 05/2000 | ASQ Section 0613 |
| Fundamentals of Software Quality Assurance | 1992-1995 | ASQ Section 0613 Tutorial |
| A Graphical Display of Testing Status for Complex Configurations | 10/2008 | PNSQC |
| Heuristic Test Oracles | 04/1999 | Software Testing and Quality Engineering |
| Mutating Automated Tests | 04/2000 | Software Testing Analysis & Review (STAR) East 2000 |
| Overview of ASQ's Certified Software Quality Engineer (CSQE) | 09/2002 | Quality Week 2002 |
| Requirements for Test Automation | 10/1999 | PNSQC 1999 |
| Feedback to IEEE on SWEBOK | 06/2003 | Review feedback to IEEE |
| Test Automation Architectures: Planning for Test Automation | 05/1999 | Quality Week Conference 1999 |
| Test Oracles; Planning Ahead for Test Automation | 03/1998 | East Bay SSQA (EBSSQA) |
| Testing Automation Beyond Regression Testing | 4/2008 | Spring STPCon |
| Using Test Oracles in Automation | 05/2000 | Quality Week 2000 Tutorial |
| Using Test Oracles in Automation | 10/2001 | 2001 Pacific Northwest Software Quality Conference (PNSQC 2001) |
| Using Test Oracles in Automation | 03/2003 | Spring 2003 Software Test Automation Conference |
| Title | Date | Purpose |
|---|---|---|
| Testing Automation Beyond Regression Testing | 4/2008 | Spring STPCon |
| A Graphical Display of Testing Status for Complex Configurations | 10/2007 | PNSQC |
| Advanced Automation Architectures (Tutorial) | 07/2007 | Conference for the Association for Software Testing (CAST) 2007 |
| Avoiding the "Test and Test Again" Syndrome | 07/2007 | Conference for the Association for Software Testing (CAST) 2007 |
| Early Testing Without the Test and Test Again Syndrome | 11/2006 | SSQA |
| Divide and Conquer | 1/2005 | Better Software, "Front Line" |
| The "Test and Test Again" Syndrome | 10/2004 | 2004 Better Software Conference |
| Exhausting Your Test Options | 07/2003 | Software Testing and Quality Engineering, "Bug Report" |
| Feedback to IEEE on SWEBOK | 06/2003 | Review feedback to IEEE |
| Using Test Oracles in Automation | 03/2003 | Spring 2003 Software Test Automation Conference |
| Automated Testing of Embedded Software | 03/2003 | Spring 2003 Software Test Automation Conference |
| Overview of ASQ's Certified Software Quality Engineer (CSQE) | 09/2002 | Quality Week 2002 |
|
What's Different About Software |
11/2001 | ASQ Golden Gate Section |
| Using Test Oracles in Automation | 10/2001 | 2001 Pacific Northwest Software Quality Conference (PNSQC 2001) |
|
What's Different About Software |
04/2001 | ASQ Section 0613 |
| Requirements for Test Automation | 12/2000 | SSQA |
| The Darker Side of Metrics | 10/2000 | PNSQC 2000 |
| Mutating Automated Tests | 05/2000 | SSQA |
| Using Test Oracles in Automation | 05/2000 | Quality Week 2000 Tutorial |
| Architecture and Design of Automated Software Tests | 05/2000 | PNSQC Spring Workshop 2000 |
| 21 CFR Part 11: Electronic Signatures, Electronic Records | 05/2000 | ASQ Section 0613 |
| Failure Mode and Effects Analysis | 05/2000 | ASQ Section 0613 |
| Mutating Automated Tests | 04/2000 | Software Testing Analysis & Review (STAR) East 2000 |
| Cost Benefits for Test Automation | 10/1999 | STAR West 1999 |
| Requirements for Test Automation | 10/1999 | PNSQC 1999 |
| Test Automation Architectures: Planning for Test Automation | 05/1999 | Quality Week Conference 1999 |
| Heuristic Test Oracles | 04/1999 | Software Testing and Quality Engineering |
| Analysis of The Taxonomy of Test Oracles | 10/1998 | Fifth Los Altos Workshop on Software Testing |
| Metrics for Metrics: Cost Analysis and Justification | 05/1998 | Developing Strategic I/T Metrics Conference 1998 |
| Analysis of The Taxonomy of Test Oracles | 05/1998 | Quality Week 1998 |
| Test Oracles; Planning Ahead for Test Automation | 03/1998 | East Bay SSQA (EBSSQA) |
| Fundamentals of Software Testing | 1995 | ASQ Section 0613 Tutorial |
| Foundations of Software Quality | 1994 | ASQ Section 0613 Tutorial |
| Measuring the Quality of Software Consulting | 10/1994 | Fourth International Conference on Software Quality |
| A Process for Measuring the Quality of Software Consulting | 05/1994 | PNSQC |
| Some Measures of Quality of Software Consulting 1994 (ASQ) | 1994 | SSQA |
| A Method for Measuring Quality of Software Consulting | 1994 | ASQ Section 0613 |
| Practical Software Metrics | 1993 | |
| The Software Quality Group's Relationship to Development | 05/1993 | Quality Week 1993 |
**********************************************************************
Background:
In software testing, the mechanism used to generate the expected results is called an oracle.
(In this paper, the first letter will be capitalized when referring to the Oracle for a specific test.)
Many different approaches can be used to create, capture, and compare test results.
The author, for example, has used the following methods for generating expected results:
Software tests themselves can be classified in many different ways. Automated tests that include evaluation of results need some kind of oracle regardless of the type or purpose of the tests. Yet, the mechanism for evaluation of results ranges from none (the program or system didn't crash) to exact (all values, displays, files, etc., are verified). Various levels of effort and exactness are appropriate under different circumstances. The nature and complexity of an oracle is also dependent upon those circumstances.
Presented at 1998 Quality Week (QW), and October, 1998 Los Altos Workshop on Software Testing (LAWST 5)
**********************************************************************
Topics emphasized:
NOTE: This tutorial was not about how to use any particular test tool or about code in any particular test tool's programming language.
Presented at Spring 2000 Pacific Northwest Software Quality Conference Tutorials
**********************************************************************
Presented at Fall 2008 Software Testing and Performance Conference
**********************************************************************
Presented at 2007 Pacific Northwest Software Quality Conference
Talk presented at ASQ Silicon Valley Section Meeting, September 2007
**********************************************************************
Key points attendees take away:
Summary:
Most automated tests are used as regression tests - doing the same exercises each time the test is
run. The paper and talk describe a powerful type of automated test - one that does something
different each time it runs. The technique doesn't apply to all situations of automated tests, but
the author presents the pros and cons for mutating automated tests based on his experience with
them. The paper also provides several examples.
Presented at 2000 Software Testing, Analysis, and Review (STAR) East Conference, and May, 2000 meeting of Silicon Valley Software Quality Association (SSQA)
**********************************************************************
The Subject Areas of the CSQE 2002 Body of Knowledge are:
The emphasis of the tutorial is on mapping knowledge and skill areas into performance measures. By identifying relevant skill areas and performance measures, the quality engineering team can understand what levels of performance are expected and what levels of performance are being shown. Only a brief BOK topical overview is presented, since defining of all the topics in the CSQE BOK in detail would take more time than is available in a one day tutorial.
Presented at 2002 Software Quality Week (QW)
**********************************************************************
Published as a "Front Line" article in Better Software Magazine January 2005
Talk presented at ASQ Silicon Valley Section Meeting, November 2005
**********************************************************************
Published as a "Bug Report" within Software Testing and Quality Engineering (STQE) Magazine July/August 2003
**********************************************************************
SSQA 11/2006
Presented at 2004 Better Software Conference
**********************************************************************
CAST 07/2007
Presented at 2007 CAST
**********************************************************************
There are vast possibilities beyond that open to us when automating tests. When we think of test automation we should first think about doing things that we can't do manually. Based on experience creating non-regression automated tests, these presentations address what and how we can create more powerful automated tests.
The SSQA talk is a one hour presentation about the limitations and how other kinds of test automation may be much more valuable.
The CAST tutorial is a one-day presentation of advanced automated test architectures.
The Tutorial covers:
**********************************************************************
It is often impractical to exactly reproduce or compare accurate results, but it isn't necessary for an oracle to be perfect to be useful. In this article, I describe some ideas associated with what I call heuristic oracles. A heuristic oracle uses simpler consistency checks (heuristics) for the results of a test.
Published as an article "Heuristic Test Oracles" in Software Testing and Quality Engineering (STQE) Magazine April 1999
**********************************************************************
1999 International Quality Week Conference; East Bay SSQA (EBSSQA)
Presented at EBSSQA, March 1998 and 1998 International Quality Week Conference
**********************************************************************
2003 Spring/Fall Software Test Automation Conference
More and more software is being embedded in everyday devices ranging from computer peripherals to toys. Development and testing of embedded software offers new challenges because of the combination of the hardware devices and environments. This presentation describes some of the common issues and some ways they have been addressed.
Presented at Spring 2003 Software Test Automation Conference
**********************************************************************
2000 PNSQC
The presentation focuses on a metric that I've seen used in many organizations (readiness for release) and some of the disruptive results in those organizations. I've focused on three examples of different metrics that have been used and a few examples of the behaviors elicited by using the metrics. For obvious reasons, the examples have been subtly altered to protect the innocent (or guilty). The three metrics are:
1. Defect find/fix rate
2. Percent of tests running/Percent of tests passing
3. Complex model based metrics (e.g., COCOMO)
Some of the observed behaviors include:
Presented at 2000 Pacific Northwest Software Quality Conference
Presented at CAST 2006
**********************************************************************
1999 STAR
Cost benefits from automation are viewed as trade-offs in comparison to manual testing (or the current situation). Financial impacts are computed in comparison to two alternatives: manually testing the same thing or not testing (accepting the risk of not knowing). Organizational impacts such as the skills needed to design and implement automated tests, automation tools, automation environments, development, and maintaining automation tools and environments are also discussed.
Test automation is not always necessary or appropriate. Automating existing manual tests is a path frequently chosen by default, but usually is not cost beneficial and sometimes results in decreased test effectiveness. The costs and benefits of test automation can be identified and estimated, and good management decisions made about using automation to improve testing.
Paper presented at 1999 Software Testing, Analysis, and Review (STAR) Conference.
**********************************************************************
Spring 2003 Software Test Automation Conference; PNSQC 2001; and Quality Week 2000
Software test automation is often a difficult and complex process. The most familiar aspects of test automation are organizing and running of test cases and capturing and verifying test results. A set of expected results are needed for each test case in order to check the test results. Verification of these expected results is often done using a mechanism called a test oracle. The paper and talks describe the use of oracles in automated software verification and validation. Several relevant characteristics of oracles are included with the advantages, disadvantages, and implications for test automation.
Real world oracles vary widely in their characteristics. Although the mechanics of various oracles may be vastly different, a few classes can be identified which correspond with automated test approaches. These types of oracles are categorized based upon the strategy for verification using the oracle. Thus, an oracle strategy using a lookup table to generate expected results can be the same as one that uses an alternate algorithm implementation to compute them. Four types of oracle strategies (and not using any oracle) are identified and defined. The strategies are labeled True, Heuristic, Consistency, and Self Referential.
Slides presented at Spring 2003 Software Test Automation Conference 03/2003
Paper presented at PNSQC 2001
Tutorial Slides presented at Quality Week 2000
**********************************************************************
Quality Week 1994; PNSQC 1994; 4ICSQ
Paper and presentation made at Quality Week, 1994 PNSQC 1994, and the 4th International Conference on Software Quality (4ICSQ))
**********************************************************************
2000
Automating testing is like any software development project, and as such we need to articulate the requirements. Test automation should not be done piecemeal or in an ad hoc fashion because the resulting work products will become more and more expensive to use, eventually becoming obsolete or insupportable. The presentation describes many special considerations for automation requirements and a method for understanding and organizing them.
Presentation at PNSQC 2000 and SSQA 12/2000
**********************************************************************
Quality Week 2003
This paper presents the roles of the Software Quality Organization in software development as observed in dozens of commercial organizations. It describes the development process as the way quality is built into software. It also looks at the different ways the quality group's purpose and charters were viewed. The potential benefits and drawbacks for various charters are described, along with the organizational structure and typical activities for each. The idea that the charter for the quality group changes over time is also presented, along with observed progressions in organizations. The various possible organizations, charters, and roles are described and related briefly to quality systems described in both the SEI Maturity Model and ISO 9000 Standards (ISO 9001 and ISO 9000-3). In summary, it describes the impact on product quality of the different types of development process and possible roles for the software quality group.
Paper and presentation made at 1993 Quality Week 10/1993
**********************************************************************
Personal Email to IEEE SWEBOK Committee, June 2003
The IEEE Software Engineering Coordinating Committee put out the SWEBOK (Trial Version 1.00 - May 2001) for review and feedback. I felt compelled to provide 12 pages of comments because of my extremely strong [negative] reaction.
I spent several hours going into the first two chapters before skipping to the chapters on Testing and then Software Quality. I was encouraged by the explicit recognition that different organizations, users, and products require different techniques in both chapters. But, I was discouraged by the many deficiencies in the Testing chapter and the gaping holes in the Software Quality chapter. I was shocked by the fact that no reference is made to ASQ’s CSQE, if only to criticize it. Either the drafters of the SQEBOK were really ignorant of the existence of a sister society’s related Software Quality Engineering Body of Knowledge, or they choose to ignore it because it was inconvenient or at odds with SWEBOK. In my opinion, if the first case is true, they were incompetent. In my opinion, if the second case was true, they committed professional malpractice.
In any case, they choose not to respond to or even acknowledge any of my input in the 2004 publication of the "SWEBOK_Guide_2004". The 2004 SWEBOK still does not acknowledge the ASQ or the CSQE BOK. I also find it curious that the Preface states that the ACM was active in creating the joint committee, that they approved the code of ethics in 1998, and they were working on an alternate educational curriculum, but the Preface fails to mention that ACM rejected the SWEBOK itself.
**********************************************************************
Developing Strategic I/T Metrics, May 1988
The speaker notes are available with the slides.
**********************************************************************
ASQ SV Section 2003
**********************************************************************
PNSQC 2003
Paper and presentation made at 1993 PNSQC 10/1993
**********************************************************************
May, 2000 ASQ Presentation
Failure Modes and Effects Analysis (FMEA) is a risk analysis and prioritization method developed for the aerospace industry in the 1960s. This talk describes how the FMEA techniques can be applied to software.
Talk presented at May 19, 2000 ASQ Silicon Valley Section Dinner Meeting.
**********************************************************************
May, 2000 ASQ Presentation
This talk describes the major characteristics of the regulations and implications for software testing.
Talk presented at May 19, 2000 ASQ Silicon Valley Section Dinner Meeting.
**********************************************************************
![]()
Updated April 18, 2008
Copyright © 1995-2007 Software Quality Methods, LLC. All Rights Reserved.