Last week I started a new ASP.NET Core project. It’s still very small and in active development. So, there is no fancy staging or production environment as of yet (there may never be). The project runs on my local machine and that’s just fine for its scope.

It contains a lot of unit tests so business logic and flow is validated, but I wanted to add end-to-end tests as well, so the web UI gets some testing, too.

I remembered Microsoft mentioned Playwright at last year’s build conference link. So I started to investigate this new testing library and got stuck pretty fast…

What is Playwright?

It’s a testing library for end-to-end tests where you write (or record) testing code that instructs a browser session to do certain things on a web page and validate the results with asserts on the elements and properties of that page. So it’s basically just like unit tests for a web application. You can find on GitHub if you like more info.

They also have a good example of a basic test case at https://playwright.dev/dotnet/docs/intro

So, what’s the catch?

Where I got stuck was: you need a running instance of your web app, so the tests can load the pages and execute the steps defined in the test code.

Makes sense, right? But what’s the recommendation here?

Digging through the docs and finally asking in the designated Discord channel I didn’t get a satisfying answer. You are supposed to run the end-to-end tests against a staging environment (or alternatively production).

So, no end-to-end tests without staging environment then?

One alternative pointed out was to start the app locally before running the tests. This might be a workaround. But it comes with two drawbacks:

1. The local port of the app may differ on each developer’s machine or environment. So the URL in the test classes needs to be configured accordingly.
2. When working with Visual Studio running tests is just a shortcut away (or a few clicks in the Test Explorer). If you need to debug the tests it’s not very developer friendly to start the web app first and stop it afterwards.

Another issue may be testing with CI. Having hosted this small project on GitHub I wanted to use GitHub Actions for automatic testing as well.

While not impossible, it takes more effort to

1. build
2. start the app
3. run the tests
4. shutdown the app

with GitHub Actions.

An alternative approach

I wasn’t satisfied with this situation and a bit shocked that nobody seemed to have this issue before (or cared to share a solution).
Anyway here is my solution to it.

Make starting the app part of the test initialization phase

I figured it would be best to start the app each time the tests need to run and end the instance once the tests are completed. There is a lot to improve on (seeding test data for example), but this works for all apps that start fast.

You would have a staging enviroment for bigger apps anyway, right?!

The guide

As a prerequisite you should already have an ASP.NET Core project with an app to test. The example project created by dotnet cli or Visual Studio will suffice. I will skip this part. Let’s just assume it’s a project called MyApp.

Your Playwright tests will live inside another project. A testing project. You may use whatever testing framework you like. xUnit, nUnit and MS test are the three big choices here. I’ll use MS test for these examples. But equal solutions are possible in each one of them.

Setup a new testing project

To create a new testing project run dotnet new mstest or add a new MS test project via Visual Studio.

dotnet new mstest -n MyApp.E2ETests

You need to reference the web app project from the new testing project, so the test framework can run the web app before running the tests. Do this in Visual Studio by dragging the web app project onto the Dependencies of the testing project inside Solution Explorer. Or add an ProjectReference like the following to the MyApp.E2ETests.csproj file manually (check that the relative file path is correct!)

<!-- Project-Reference inside MyApp.E2ETests.csproj -->
<ItemGroup>
  <ProjectReference Include="..\MyApp\MyApp.csproj" />
</ItemGroup>

Prepare the web app to be started externally

You need to change the Main method of your web app project to enable the testing project to run the app.

This is what your Main looks like at the moment:

namespace MyApp;

public static class Program
{
    public static void Main(string[] args)
    {
        var builder = WebApplication.CreateBuilder(args);
        // You add some services to the builder:
        // builder.Services.AddXYZ()
        var app = builder.Build();
        // You define the middleware
        // app.UseXYZ();
        app.Run();
    }
}

If you use top level statements it might look a little different, but that’s no problem. Just adapt the following section accordingly.

You want to separate the app building from the app execution so the testing framework can start the app by itself. So, this is what needs to happen to your startup code:

1. Extract all code that defines your app (services + middleware) into a builder method
2. Call that builder method in Main
3. Run the app fromMain

[assembly:InternalsVisibleTo("MyApp.E2ETests")]

namespace MyApp;

public static class Program
{
    public static void Main(string[] args) =>
        BuildApp(args).Run();

    internal static WebApplication BuildApp(string[] args)
    {
        var builder = WebApplication.CreateBuilder(args);
        // You add some services to the builder:
        // builder.Services.AddXYZ()
        var app = builder.Build();
        // You define the middleware
        // app.UseXYZ();

        // ! return the app instead of running it here
        return app;
    }
}

Tipp: the InternalsVisibleTo attribute will allow your testing project to call the BuildApp method even though its internalvisibility modifier while other projects can’t. You may use this for unit tests, too ;)

Automatically start the app before tests are run

To setup your web app instance and finally running it I created this small util class. Add it to the testing project:

using Microsoft.AspNetCore.Builder;

namespace MyApp.E2ETests.Utils;

internal static class TestEnvironment
{
    public static string Url => Application?.Urls?.First() ?? string.Empty;

    private static WebApplication? Application { get; set; }

    public static void StartApp()
    {
        Application = Program.BuildApp([]);
        Application.RunAsync();
    }

    public static void StopApp()
    {
        if (Application == null) return;
        Application.StopAsync().Wait();
    }
}

Note that StartApp uses the BuildApp method from before. It uses the new C#12 notation for empty arrays []. You may provide test arguments for your app here, if needed.

Also note that RunAsync is called without await! This isn’t a bug. It’s exactly what we need: Start the web app without waiting on it’s exit (web apps normaly never exit by themselfs anyway).

With this in place all that’s left is a test class that instructs the testing framework to run intialization code when loading the test project assembly. Whit MS test the code looks like this:

namespace MyApp.E2ETests.Utils;

[TestClass]
public class TestEnvironmentInitialization
{
    [AssemblyInitialize]
    public static void AssemblyInitialize(TestContext _) => 
        TestEnvironment.StartApp();

    [AssemblyCleanup]
    public static void AssemblyCleanup() => 
        TestEnvironment.StopApp();
}

You need to adapt this class when using nUnit or xUnit.

Inside your actual test methods you’ll want to load your just launched web app via await Page.GotoAsync(TestEnvironment.Url);. This is where all comes together.

Conclution

Using the descripted method you’re able to run Playwright tests both via dotnet test cli and from Visual Studio’s Test Explorer without having to start your web app beforehand manually. This is usefull in smaller projects that havn’t got a staging environment (yet).

Bonus: GitHub Actions

If you use GitHub to host your project, chances are high your want to use GitHub Actions as well to build and test the code. Here is a litte goodie for you. Store it as test.yml inside .github/workflows:

name: Test
on:
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]
env:
    DOTNET_VERSION: '8.0'
    BUILD_CONFIGURATION: 'release'  # 'debug' or 'release' (lowercase!)
    E2ETEST_PROJECT: 'MyApp.E2ETests'
    SOURCE_DIR: './src'
jobs:
  test:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: setup dotnet
      uses: actions/setup-dotnet@v3
      with:
        dotnet-version: ${{ env.DOTNET_VERSION }}.x

    - name: cache NuGet packages
      id: nuget-packages
      uses: actions/cache@v3
      env:
        cache-name: nuget-package-cache
      with:
        path: ~/.nuget/packages
        key: ${{ runner.os }}-${{ env.cache-name }}

    - name: build solution
      working-directory: ${{ env.SOURCE_DIR }}
      run: dotnet build --configuration ${{ env.BUILD_CONFIGURATION }}

    - name: ensure browsers are installed
      run: pwsh ${{ env.SOURCE_DIR }}/${{ env.E2ETEST_PROJECT }}/bin/${{ env.BUILD_CONFIGURATION }}/net${{ env.DOTNET_VERSION }}/playwright.ps1 install --with-deps

    - name: Run tests
      working-directory: ${{ env.SOURCE_DIR }}
      run: dotnet test --configuration ${{ env.BUILD_CONFIGURATION }} --no-restore