Running Acceptance Tests in Isolation

In my last blog post, I discussed the advantages of running acceptance tests at a level where it is still possible to isolate the code under test from third-party dependencies. Here I'm going to detail (using real-life examples) how Varealis has designed its .Net Core applications to run acceptance tests against one of the microservices that helps power RadSpider.

ASP.Net Core Acceptance Tests

ASP.Net Core offers an in-memory TestServer, made available in the Microsoft.AspNetCore.Mvc.Testing nuget package, to allow for integration/acceptance tests to be run without deploying the application to a test environment. A WebApplicationFactory is also provided to streamline the bootstrapping the TestServer and generate a HttpClient that can be used to interact with the web application. For more information about this Microsoft has provided detailed documentation which can be found here.

Generating a Client

By initialising a WebApplicationFactory using the same Startup class that is used to bootstrap the web application, we can generate a basic HttpClient that can interact with the web application.

var pipelineId = Guid.NewGuid();

var httpClient = new WebApplicationFactory<Startup>().CreateClient();
HttpOperationResponse httpResponse = 
    await httpClient.GetAsync($"/Pipeline/{pipelineId}");

While this client gives us everything we need to be able to integrate with a web application, since my web applications also automatically generate OpenAPI swagger files using Swashbuckle.AspNetCore I like to use this file to help generate a HttpClient. Not only does this greatly simplify the interactions with the service under test, but it also makes it possible to verify that our documentation effectively describes how to interact with that service. We are using the tools that we will later provide our customers in order to test our application, even at an acceptance test level.

To do this, I used 'autorest' to automatically generate a client that can be used to interact with the web application. The HttpClient generated by the WebApplicationFactory can then be used to initialise the 'autorest' client which provides a friendly interface which can be used to integrate with the service.

var httpClient = new WebApplicationFactory<Startup>().CreateClient();

// This PipelineClient was automatically generated by autorest
var pipelineClient = new PipelineClient(
    new ClientCredentials(), httpClient, true);

HttpOperationResponse<PipelineDetails> pipeline = await pipelineClient
    .GetPipelineDetailsWithHttpMessagesAsync(Guid.NewGuid());

* Note that automatically generated extension methods can also be used when we don't care to know about the HttpOperationResponse and only the response body. These look something like this...

PipelineDetails pipeline = await pipelineClient
    .GetPipelineDetails(Guid.NewGuid());

Mocking Dependencies

So far, because we are using the application's original Startup class, we are also relying on all of the original application's dependencies. For these tests to work in instances where we don't have access to third-party dependencies like databases and/or service buses we need to replace these dependencies with in-memory alternatives or mocks. To achieve this we have to make some minor changes to the Startup class to allow selected dependencies to be overridden. There are many ways of doing this, but I did this by introducing a base class. The StartupBase class is then responsible for configuring all of the fixed internal dependencies while the now elevated Startup is now responsible for adding/configuring the dependencies that we want to override.

public class Startup : StartupBase
{
    protected override IServiceCollection
        ConfigurePipelineRepository(IServiceCollection services)
    {
        var connectionString = this.Configuration
            .GetSection("Persistence")
            .GetValue<string>("PipelineRepository");
        
        var mongoUrl = new MongoUrl(connectionString);
        var mongoSettings = MongoClientSettings.FromUrl(mongoUrl);

        services.AddSingleton<IMongoClient>(new MongoClient(mongoSettings));
        services.AddSingleton<IPipelineRepository, PipelineRepository>();

        return services;
    }
}

public abstract class StartupBase
{
    protected IConfiguration Configuration { get; }

    // A lot of this method has been removed to maintain brevity
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvc();
        services.AddSwaggerGen();

        this.ConfigurePipelineRepository(services);
    }

    protected abstract IServiceCollection
        ConfigurePipelineRepository(IServiceCollection services);
}

By doing this any class that extends the StartupBase is now able to configure mocked dependencies in place of third-party dependencies that we don't want to rely on at this stage. We do this by creating a special MockStartup class which injects mocks into the IServiceCollection in place of the real implementations.

using NSubstitute;

public class MockStartup : StartupBase
{
    protected override IServiceCollection
        ConfigurePipelineRepository(IServiceCollection services)
    {
        services.AddSingleton(_ => Substitute.For<IPipelineService>());
        return services;
    }
}

This approach is not perfect, and there are two important things to note here. Firstly, whenever dependencies are replaced with mocks you are limiting the scope that the tests can cover. Ideally, you should look to replace dependencies with in-memory implementations of dependencies rather than mocks. For example, if you are using Entity Framework Core then you can use the in-memory database provider in place of your usual provider. This allows you to cover much of the database interactions at this level than a mock would allow since you are still interacting with a 'real' database. In my case, I am forced to mock out my IPipelineRepository because it wraps a mongo database which doesn't have (at the time of writing) an equivalent to this. Unfortunately this means that a lot of database interactions that I would love to test at this level (like the QueryBuilders) can not be tested at this level of the testing triangle.

Secondly, we need to be careful to reduce the amount of work that is done in the override methods, so steer clear of broad methods such as protected abstract IServiceCollection ConfigureDependencies(IServiceCollection services) instead favouring many smaller methods for each dependency that you are trying to replace. Because the override methods implemented in the Startup class are not covered by the tests it can be easy to forget/misconfigure dependencies in the real methods and be lulled into a false sense of security that these tests can provide.

All of this goes to say that, while these tests are extremely valuable, they can not replace full end-to-end integration tests. Hopefully, you can drastically reduce the number of more expensive integration tests that you need to run, especially since these tests allow you to easily manufacture scenarios that may not otherwise we easy to replicate with real dependencies.

Other Applications

In cases where you can't host your application in an ASP.Net in-memory test server, you have to get a little more creative. In my case, I also have 'MassTransit' consumers which also use Startup style classes to initialise the IBusControl. However, since these consumers are hosted in Service Fabric ICommunicationListeners, I am not able to load them into a TestServer like I did before. Fortunately, all we really have to do here is make it possible to override MassTransit's transportation layer on top of the third-party dependencies.

public class Startup : StartupBase
{
    public Startup()
    {
        var connectionString = this.Configuration
            .GetSection("Persistence")
            .GetValue<string>("SagaDatabase");

        var mongoUrl = new MongoUrl(connectionString);
        var mongoSettings = MongoClientSettings.FromUrl(mongoUrl);

        this.sagaDatabase = new MongoClient(mongoSettings)
            .GetDatabase("Pipeline");
    }

    protected override ISagaRepository<PipelineSaga> PipelineSagaDatabase =>
            new MongoDbSagaRepository<PipelineSaga>(
                this.sagaDatabase,
                new MongoDbSagaConsumeContextFactory(),
                nameof(PipelineSaga));

    protected override IBusControl 
        ConfigureMassTransit(IServiceCollection services)
    {
        var messageBusSettings = this.Configuration.GetSection("ServiceBus");

        return Bus.Factory.CreateUsingAzureServiceBus(cfg =>
        {
            var host = cfg.Host(
                messageBusSettings.GetValue<string>("ConnectionString"),
                config => { });

            cfg.ReceiveEndpoint(
                host,
                this.RecieveEndpointQueue,
                ec => this.ConfigureRecieveEndpoint(
                    ec,
                    services.BuildServiceProvider()));
        });
    }

    protected override IServiceCollection
        ConfigurePipelineRepository(IServiceCollection services)
    {
        var connectionString = this.Configuration
            .GetSection("Persistence")
            .GetValue<string>("PipelineRepository");
        
        var mongoUrl = new MongoUrl(connectionString);
        var mongoSettings = MongoClientSettings.FromUrl(mongoUrl);

        services.AddSingleton<IMongoClient>(new MongoClient(mongoSettings));
        services.AddSingleton<IPipelineRepository, PipelineRepository>();

        return services;
    }
}

public abstract class StartupBase
{
    protected IConfiguration Configuration { get; }

    protected string RecieveEndpointQueue =>
        "Varealis.RadSpider.Services.Pipeline.Consumer";

    protected abstract ISagaRepository<PipelineSaga> PipelineSagaDatabase { get; }

    // A lot of this method has been removed to maintain brevity
    public void ConfigureServices(IServiceCollection services)
    {
        this.ConfigurePipelineRepository(services);

        services.AddSingleton<PipelineConsumer>();

        services.AddSingleton(_ => this.ConfigureMassTransit(services));
        services.AddSingleton<IPublishEndpoint>(provider =>
            provider.GetRequiredService<IBusControl>());
    }

    protected void ConfigureRecieveEndpoint(
        IReceiveEndpointConfigurator recieveConfig,
        IServiceProvider serviceProvider)
    {
        recieveConfig.Consumer(() => serviceProvider.GetRequiredService<PipelineConsumer>());
        recieveConfig.Saga(this.PipelineSagaDatabase);
    }    

    protected abstract IServiceCollection 
        ConfigurePipelineRepository(IServiceCollection services);

    protected abstract IBusControl 
        ConfigureMassTransit(IServiceCollection services);
}
    

From this point, all I have to do is create a special MockStartup class that extends BaseStartup and implements all of the abstract methods. However, in place of CreateUsingAzureServiceBus I'm going to use CreateUsingInMemory. This tells MassTransit to use MassTransit's In-Memory transportation layer, which means I'm able to run the consumer without having a dependency on a service bus like RabbitMQ and Azure Service Bus.

public class MockStartup : StartupBase
{
    private readonly InMemorySagaRepository<PipelineSaga> pipelineSagaDatabase;

    public MockStartup(InMemorySagaRepository<PipelineSaga> pipelineSagaDatabase)
    {
        this.pipelineSagaDatabase = pipelineSagaDatabase;
    }

    protected override ISagaRepository<PipelineSaga> PipelineSagaDatabase =>
        this.pipelineSagaDatabase;

    protected override IBusControl
        ConfigureMassTransit(IServiceCollection services)
    {
        return Bus.Factory.CreateUsingInMemory(cfg =>
        {
            cfg.ReceiveEndpoint(
                this.RecieveEndpointQueue, 
                ec => this.ConfigureRecieveEndpoint(
                    ec,
                    services.BuildServiceProvider()));
        });
    }

    protected override IServiceCollection
        ConfigurePipelineRepository(IServiceCollection services)
    {
        services.AddSingleton(_ => Substitute.For<IPipelineService>());
        return services;
    }
}

Happy Testing

I hope that these two examples can help you experiment with this approach to testing since I've found it invaluable when it comes to testing microservices. For one of my microservices, at the time of writing, I have 388 unit tests which can be run in 2.61 seconds and then a further 92 acceptance tests that can be run in 6.39 seconds. These acceptance tests have enabled me to catch a number of bugs with how the code was integrated, bugs that are very difficult to find with unit tests alone. Not bad considering all 480 tests can be run in 7 seconds when running in parallel, and that these tests can be run anywhere.