Storyteller 5
In Storyteller 5, you can run all the specifications with the project console application itself, so from the root directory of the project at the command line, it's just:
dotnet run
To only run one suite at a time, try:
dotnet run -- run "suite path"
Storyteller 4
Storyteller specifications can be executed from the command line tool with the dotnet storyteller run command. You may either run all the specifications (for smaller projects or faster systems), or run one suite of specifications at a time.
Storyteller 3
Storyteller specifications can be executed from the command line tool with the st run command. You may either run all the specifications (for smaller projects or faster systems), or run one suite of specifications at a time.
Fast Build, Slow Build
In usage over the years, I will run the Storyteller specifications as part of the normal automated build for a codebase if they feel fast enough. Since Storyteller is going to be more frequently used for integration testing, the Storyteller specifications can easily become slower or intolerably long running for the kind of quick feedback you want in your normal automated build. In this common case, the Storyteller team recommends a cascading build where your normal build runs first with your "fast" tests, then on success triggers a secondary build to run the Storyteller specifications.
The reasoning is to create more effective and quicker feedback cycles. A team using a cascading build may follow a policy of waiting on the fast build before continuing any work after a push to source control, but only deal with the slower, cascading build when and if it fails.
Critical and Catastropic Exceptions
Storyteller 3 introduces two new exceptions that you can choose to throw yourself:
StorytellerCriticalException-- tells Storyteller that the current specification cannot possibly succeed and causes Storyteller to stop executing the specification wherever it's at. Internally, Storyteller treats any failure inFixture.SetUp()orFixture.TearDown()as a critical exception. A classic example is using Storyteller to test a web application where a section of a specification starts by trying to open the application to a certain Url. If the web server returns an error on the request, Storyteller should not bother to try to find the expected elements on the screen. It should fail quickly with a clear indication that it was unable to load the web page.StorytellerCatastropicException-- tells Storyteller that the current system under test is in an invalid state and to stop trying to execute anything else. You can throw this exception yourself if you detect something like an unavailable database connection that makes the system unusable. Internally, Storyteller throws this exception if theISystemfails to start or if it cannot create anIExecutionContext.
See Connecting Storyteller to your System for more information.
Retries
I hate to tell you this, but it is somewhat likely that your Storyteller specifications are going to fail sometimes in your continuous integration runs. It's also somewhat likely that you may hit specifications that are consistently reliable -- especially when you are testing against any kind of system that involves a lot of asynchronous operations like just about any modern web application. To this end, the original Storyteller team adopted the idea of "retries," where a specification that fails can be retried a configured number of times (we got this idea from jQuery as I recall).
This "retry" capability has produced very mixed results in real usage. In the early days of Selenium when the browser automation was so unreliable, the retry capability made our automated testing much more useful. Today, the retry capability is still valuable when we use Storyteller against asynchronous messaging systems. That being said, the retry capability in older versions of Storyteller has also been detrimental when it too aggressively tries to retry specifications that have no chance of ever succeeding or when the system under test itself is in an invalid state -- often hitting the timeout for each specification on each retry.
To that end, Storyteller 3 changes the retry policy a bit to fail faster:
- The retry count as of now is specified per specification with the default being no retries unless this is overridden by the
st run --retries [#]orst open --retries [#]flags. This may change later based on our shop's experience with Storyteller 3. For now, however, the attitude we are taking is that retries are a potentially harmful property that must be explicitly enabled. - Specifications marked as Acceptance will never be retried.
- If in the course of executing a Specification the Storyteller engine hits a "critical" or "catastropic" exception, the specification will no longer be retried even if it has not exceeded its maxiumum number of retries. A "critical" exception is any failure from a
Fixture.SetUp(),Fixture.TearDown(), or theStorytellerCriticalExceptionbeing thrown by grammars. - If the system under test itself fails to start or cannot create an execution context, the entire batch of specifications is aborted, but the results are still shown.
See Instrumenting and Performance Logging for more information on adding and using instrumentation against your Storyteller specifications.
Lifecycle
Storyteller has always treated Acceptance and Regression specifications somewhat differently in the command line execution. Acceptance specifications are executed strictly for informational purposes, are never retried in the case of failures, and do not count toward the success or failure of the execution run. Regression specifications on the other hand can be retried and any number of Regression failures will cause the st run command to return a non-zero return code denoting a failure.
See The Specification Lifecycle for more information.
Results
The specification execution results are written to a single html file with no other file dependencies. In continuous integration scenarios, you would want to have this file written to a place where your CI server will save this file as an artifact tagged to the build. For example, my teams write the Storyteller results to an /artifacts folder and configure our TeamCity build server to automatically store any file written to that directory as build artifacts. Most CI servers today will allow you to expose this html file from the build server website to browse the results.
The results page is a single page application with a tabular summary view of all the specifications executed, how long each took to execute, the high level result counts, and the number of retries. From the summary, you can drill into individual specifications by clicking on the specification name.
AppVeyor Integration
For better AppVeyor results, use the dotnet storyteller run --appveyor flag to direct Storyteller to output the Console logging
that AppVeyor expects.
Command Usage
st runRun a suite of StoryTeller tests
| Usages | |
|---|---|
| Execute from the current project path | dotnet storyteller run [-r, --results-path <resultspath>] [-w, --workspace <workspace>] [-e, --exclude-tags <excludetags>] [-o, --open] [-c, --csv <csv>] [-j, --json <json>] [-d, --dump <dump>] [-v, --validate] [-v, --verbose] [-r, --retries <retries>] [-z, --teamcity] [-l, --lifecycle Acceptance|Regression|Any] [-f, --framework <framework>] [-b, --build <build>] [-p, --profile <profile>] [-t, --timeout <timeout>] [-c, --config <confi>] [-s, --specs <specs>] [-f, --fixtures <fixtures>] [-c, --culture <culture>] [-s, --system-name <systemname>] |
| Execute | dotnet storyteller run <path> [-r, --results-path <resultspath>] [-w, --workspace <workspace>] [-e, --exclude-tags <excludetags>] [-o, --open] [-c, --csv <csv>] [-j, --json <json>] [-d, --dump <dump>] [-v, --validate] [-v, --verbose] [-r, --retries <retries>] [-z, --teamcity] [-l, --lifecycle Acceptance|Regression|Any] [-f, --framework <framework>] [-b, --build <build>] [-p, --profile <profile>] [-t, --timeout <timeout>] [-c, --config <confi>] [-s, --specs <specs>] [-f, --fixtures <fixtures>] [-c, --culture <culture>] [-s, --system-name <systemname>] |
| Arguments | |
|---|---|
| path | Path to the StoryTeller project file or the project directory |
| Flags | |
|---|---|
| [-r, --results-path <resultspath>] | Path to write out the results. Default is stresults.htm |
| [-w, --workspace <workspace>] | Optional. Runs only one workspace |
| [-e, --exclude-tags <excludetags>] | Optional. Excludes specs with any of the specified tags (comma-delimited) |
| [-o, --open] | Open the results in a browser after the run. DO NOT DO THIS IN CI! |
| [-c, --csv <csv>] | WriteToText the performance data in CSV format to the specified path |
| [-j, --json <json>] | WriteToText the raw result information to JSON format at the specified path |
| [-d, --dump <dump>] | Dump the raw JSON history of the batch run to the specified path |
| [-v, --validate] | Only validates the specifications and writes out a list of the encountered errors. Will fail if there are any errors. |
| [-v, --verbose] | Writes extra console logging for individual specs |
| [-r, --retries <retries>] | Sets a minimum number of retry attempts for this execution |
| [-z, --teamcity] | Optional. Applies extra logging to see progress within TeamCity during CI runs |
| [-l, --lifecycle Acceptance|Regression|Any] | Optional. Only runs tests with desired lifecyle |
| [-f, --framework <framework>] | Override the dotnet framework if multi-targeting |
| [-b, --build <build>] | Specify a build target to force Storyteller to choose that profile. By default, ST will use 'Debug' |
| [-p, --profile <profile>] | Storyteller test mode profile for systems like Serenity that use this |
| [-t, --timeout <timeout>] | Optional. Default project timeout in seconds. |
| [-c, --config <confi>] | Optional. Override the config file selection of the Storyteller test running AppDomain |
| [-s, --specs <specs>] | Optional. Override the spec directory |
| [-f, --fixtures <fixtures>] | Optional. Override the fixtures directory |
| [-c, --culture <culture>] | Force Storyteller to use this culture in all value conversions |
| [-s, --system-name <systemname>] | Optional. Tell Storyteller which which ISystem to use |