Performance Thresholds Edit on GitHub

Storyteller 4.0 introduced a new feature to designate performance thresholds that are enforced when specifications execute. In effect, this means that you can have Storyteller fail a specification that while passing functionally, ran slower than your desired threshold.

The easiest way to use this feature is with the [PerfLimit] attribute on grammar methods within Fixture classes like this:

[PerfLimit(100), FormatAs("Sentence w/ 100 ms threshold")]
public void Sentence()
{
    Thread.Sleep(WaitTime);
}

[PerfLimit(100), FormatAs("Fact w/ 100 ms threshold")]
public bool Fact()
{
    Thread.Sleep(WaitTime);
    return true;
}

[PerfLimit(100)]
public IGrammar SetVerification()
{
    return VerifyStringList(names).Titled("Check the names within 100ms");
}

Today, the [PerfLimit] attribute can be used on Sentence, Fact, and SetVerification grammars. The [PerfLimit] attribute can also be used against the individual rows within a Table grammar. Storyteller doesn't yet support a performance threshold against Paragraph's or Embedded Section grammars.

Performance threshold violations are shown in the specification results both inline and with an aggregated summary at the top of the page like this:

Threshold Policies

You can also use the static PerformancePolicies class to express additional performance thresholds that are applied conventionally at the end of the specification run:

// All grammars of any kind should run within 5 seconds
PerformancePolicies.PerfLimit(5000, r => r.Type == "Grammar");

// No specification should exceed 15 seconds
PerformancePolicies.PerfLimitBySubject(15000, "Specification");

// All grammars with "Open" in their name should run in under a second
PerformancePolicies.PerfLimit(1000, r => r.Subject.Contains("Open"));