Years ago I worked in a shop that was one of the pioneers of Extreme Programming. One of the processes we used on some projects was to treat our business facing automated acceptance tests as having two states in the iteration lifecycle:
- Acceptance -- the specification is a declaration of what is supposed to happen, but may be a work in progress.
- Regression -- the work described in the specification is supposed to be complete and should never fail.
Today, Storyteller formalizes this lifecyle in its continuous integration support. When specifications are first created in the editing tool, a specification will have its lifecycle set to Acceptance to mark it as a work in progress that will not cause build failures when the Storyteller specifications are executed in the batch mode. When the development team is satisfied that the functionality of a specification is "done," the team should mark that specification as Regression so that Storyteller treats that specification as a mandatory check in the system.