Extending NUnit 3 with command wrappers

27.10.2017

Ryszard Tarajkowski
Team Leader
Ryszard Tarajkowski

If we want to decorate a test method in NUnit, in most cases we can use NUnit’s Action Attributes. However, they’re not enough in every case.

There is another, undocumented method of decorating test methods: command wrappers. It is much more powerful than action attributes, as we can actually decorate test method’s execution, in contrary to just implementing before and after actions. We also get access to more of NUnit’s infrastructure through TestExecutionContext object.

Creating Retrying attribute

To explain how command wrappers work, let’s create an attribute that will retry a test few times when it’s failing randomly (I’m looking at you, Selenium). There is similar Retry attribute already built in NUnit, but it retries only on failed assertions, which is not very convenient in some cases.

To create a command wrapper, you have to create an attribute class implementing either IWrapTestMethod or IWrapSetUpTearDown. Both of these interfaces extend ICommandWrapper and add no methods of their own, but NUnit will use them in a different way:

IWrapTestMethod will only wrap test method execution, without setup and teardown and
IWrapSetUpTearDown will wrap test along with setup and teardown.

I believe retrying only a test method may cause some hard to debug state inconsistencies, so we’ll go with IWrapSetUpTearDown:

[AttributeUsage(AttributeTargets.Method)]
public class RetryingAttribute : Attribute, IWrapSetUpTearDown
{
    public TestCommand Wrap(TestCommand command)
    {
        throw new NotImplementedException();
    }
}

Mind, that we only allow to use attribute on methods, since I believe command wrappers don’t support decorating test fixtures.

Moving on, we can see, that Wrap method gives us a TestCommand and expects a TestCommand in return. Let’s see how TestCommand class looks like:

public abstract class TestCommand
{
    public TestCommand(Test test)
    {
        this.Test = test;
    }

    public Test Test { get; private set; }

    public abstract TestResult Execute(TestExecutionContext context);
}

Great, looks like we can easily decorate TestCommand`s by implementing Execute method. There is even a helper abstract class for it – DelegatingTestCommand, which we can use to implement retrying. Most basic command wrapper could look like this:

public class RetryingCommand : DelegatingTestCommand
{
    public RetryingCommand(TestCommand innerCommand)
        : base(innerCommand)
    {
    }

    public override TestResult Execute(TestExecutionContext context)
    {
        context.CurrentResult = innerCommand.Execute(context);
        
        return context.CurrentResult;
    }
}

You might be surprised, how we both return TestResult from inner command and assign it to context.CurrentResult. This is indeed peculiar, but it seems NUnit does it this way.

I suspect that one of the reasons for this duplication is because inner commands can modify test results (instead of just returning new ones). For example, they accumulate assertion failures. This is bad news for us, since we want to run inner command multiple times. If we don’t clear CurrentResult of past failed assertions, our test will fail no matter if retrying will succeed or not. However, there is a fix for that, that creates a fresh TestResult for a given test:

context.CurrentResult = context.CurrentTest.MakeTestResult();

Knowing all this, we can proceed to implement retrying:

class RetryingCommand : DelegatingTestCommand
{
    private readonly int _times;

    public RetryingCommand(TestCommand innerCommand, int times)
        : base(innerCommand)
    {
        _times = times;
    }

    public override TestResult Execute(TestExecutionContext context)
    {
        var retriesLeft = _times;

        RunTest(context);

        while (TestFailed(context) && retriesLeft > 0)
        {
            ClearTestResult(context);
            RunTest(context);

            retriesLeft--;
        }

        var performedRetries = _times - retriesLeft;
                
        if (performedRetries > 0)
        {
            context.OutWriter.WriteLine();
            context.OutWriter.WriteLine($"Test retried {performedRetries} time/s.");
        }

        return context.CurrentResult;
    }

    private void RunTest(TestExecutionContext context)
    {
        context.CurrentResult = innerCommand.Execute(context);
    }

    private static void ClearTestResult(TestExecutionContext context)
    {
        context.CurrentResult = context.CurrentTest.MakeTestResult();
    }

    private static bool TestFailed(TestExecutionContext context)
    {
        return UnsuccessfulResultStates.Contains(context.CurrentResult.ResultState);
    }

    private static ResultState[] UnsuccessfulResultStates => new[]
    {
        ResultState.Failure,
        ResultState.Error
    };
}

Note how we’re checking whether test failed or not. Take a look at TestFailed method:

private static bool TestFailed(TestExecutionContext context)
{
    return UnsuccessfulResultStates.Contains(context.CurrentResult.ResultState);
}

private static ResultState[] UnsuccessfulResultStates => new[]
{
    ResultState.Failure,
    ResultState.Error
};

When an assertion is not met in a test, it’s state is set to ResultState.Failure; when any other exception is thrown, it’s ResultState.Error. You might want to check a list of these statuses and adjust verification to your needs.

Lastly, it’s probably a good idea to take note of tests which cause retrying, so maybe we can actually fix them one day:

var performedRetries = _timesToRetry - retriesLeft;

if (performedRetries > 0)
{
    context.OutWriter.WriteLine();
    context.OutWriter.WriteLine($"Test retried {performedRetries} time/s.");
}

Now we can finish our attribute:

[AttributeUsage(AttributeTargets.Method)]
public class RetryingAttribute : Attribute, IWrapSetUpTearDown
{
    public int Times { get; set; } = 1;

    public TestCommand Wrap(TestCommand command)
    {
        return new RetryingCommand(command, Times);
    }
}

…and use it:

public class UnstableTests
{
    private int _failsLeft = 2;

    [Test]
    [Retrying(Times = 2)]
    public void will_succeed_on_a_third_run()
    {
        if (_failsLeft > 0)
        {
            _failsLeft--;

            throw new Exception("oh no");
        }
    }
}

Testing command wrappers

Let’s face it: you wouldn’t be extending NUnit if you tested your code manually. So how do we test NUnit extensions, e. g. command wrappers? The obvious answer is to decouple the command wrapper’s logic and unit test it. Maybe also mock all the NUnit’s abstract classes and interfaces.

This all sounds great. However, I think that integration with NUnit is the biggest problem here, not retrying logic itself. Thus, in order to be sure, that everything _Really Works_, I’d like some basic integration tests. That is tests, that will run other tests with our retrying attribute attached and check if they really retry.

I’m looking for a following use cases:

1. Given test fails assertion two times in a row…
2. …when it has [Retrying(Times = 2)] attribute, it will pass.
3. …when it has [Retrying(Times = 1)] attribute, it will fail with ResultState.Failure status.
4. Given test throws unhandled exception two times in a row…
5. …when it has [Retrying(Times = 2)] attribute, it will pass.
6. …when it has [Retrying(Times = 1)] attribute, it will fail with ResultState.Error status.

Implementation

Testing the tests doesn’t sound easy and it’s not. Fortunately, NUnit provides us with an API to programmatically run NUnit tests. I haven’t seen it documented anywhere, but don’t worry – I’m here for you.

I’ve created a class that allows us to run single test of our choice: TestRunner. Here’s how to use it:

ITestResult result = TestRunner<SomeTestFixture>.Run(fixture => fixture.some_test());
result.ResultState.Should().Be(ResultState.Success);

Explaining TestRunner implementation is beyond the scope of this post. If you’re interested, you can check out the source code.

System under tests could look like this:

[Explicit("These tests should only run programmatically by other tests or for debugging purposes.")]
public class RetryingSystemUnderTests
{
    private int _timesToFail = 2;

    [Test]
    [Retrying(Times = 2)]
    public void fails_assertion_two_times_and_retries_two_times()
    {
        MaybeFailAssertion();
    }

    [Test]
    [Retrying(Times = 1)]
    public void fails_assertion_two_times_and_retries_one_time()
    {
        MaybeFailAssertion();
    }

    private void MaybeFailAssertion()
    {
        if (_timesToFail > 0)
        {
            _timesToFail--;

            Assert.Fail("welp!");
        }
    }

    [Test]
    [Retrying(Times = 2)]
    public void throws_exception_two_times_and_retries_two_times()
    {
        MaybeThrowException();
    }

    [Test]
    [Retrying(Times = 1)]
    public void throws_exception_two_times_and_retries_one_time()
    {
        MaybeThrowException();
    }

    private void MaybeThrowException()
    {
        if (_timesToFail > 0)
        {
            _timesToFail--;

            throw new Exception("oops!");
        }
    }
}

The actual tests:

public class RetryingAttributeTests
{
    [Test]
    public void test_succeeds_when_single_retry_has_no_assertion_failures()
    {
        var result = TestRunner<RetryingSystemUnderTests>.Run(fixture =>
            fixture.fails_assertion_two_times_and_retries_two_times());

        result.ResultState.Should().Be(ResultState.Success);
        result.Output.Should().Contain("Test retried 2 time/s.");
    }

    [Test]
    public void test_result_is_failure_when_last_retry_fails_assertion()
    {
        var result = TestRunner<RetryingSystemUnderTests>.Run(fixture =>
            fixture.fails_assertion_two_times_and_retries_one_time());

        result.ResultState.Should().Be(ResultState.Failure);
        result.Output.Should().Contain("Test retried 1 time/s.");
    }

    [Test]
    public void test_succeeds_when_single_retry_throws_no_unhandled_exceptions()
    {
        var result = TestRunner<RetryingSystemUnderTests>.Run(fixture =>
            fixture.throws_exception_two_times_and_retries_two_times());

        result.ResultState.Should().Be(ResultState.Success);
        result.Output.Should().Contain("Test retried 2 time/s.");
    }

    [Test]
    public void test_result_is_error_when_last_retry_throws_exception()
    {
        var result = TestRunner<RetryingSystemUnderTests>.Run(fixture =>
            fixture.throws_exception_two_times_and_retries_one_time());

        result.ResultState.Should().Be(ResultState.Error);
        result.Output.Should().Contain("Test retried 1 time/s.");
    }
}

Final notes

Source code for this example can be found here.

One final warning: the techniques I described are not documented and I mostly found them in NUnit’s source code. It means, that there is a chance that they will stop working at some point. Actually, while I was writing this post (which took way too long) NUnit’s API did change and I had to adjust my implementation.

That’s it. I hope this information will help you write some awesome NUnit plugins or whatever.