User Interface (UI) Testing Azure Web App with Selenium on GitHub Actions

Before diving into the nitty gritty details of how we can do UI testing with GitHub Actions, let’s take a moment to understand some considerations on the technology stack and the background story.

I wanted to develop a personal web based password manager before I start losing my mind, trying to keep track of so many passwords. In my mind, we already have a perfect secrets store i.e. Azure Key Vault. We can certainly access Azure Key Vault on the Azure Portal and leverage it directly, but this is not ideal as I need to access the Azure Portal (and for other reasons, which you can read more about here). I just need to login (preferably with SSO) and access my passwords. I know that I can access Azure Key Vault via managed identity, which means there’s no need to keep secrets in my web application. This web application would most likely be an Azure App Service. My other technology consideration would be related to what platform I should be writing my web application in. I am a .NET developer, so my natural choices would be MVC, SPA with a Web API backend or maybe the new kid on the block, Blazor. From a development perspective, I like to learn Blazor, so Blazor it is.

I don’t really plan to dive into my development journey here, but the final product is a server based Blazor web application which is hosted as an Azure App Service, and is secured by Azure Active Directory. It is integrated with Azure Key Vault with any backend data stored on Azure Storage. Both Azure Key Vault and Azure Storage are accessed via managed identity which means I never have to manage any secrets within my Azure App Service.

Azure Active Directory and MFA

Now that we understand the technology stack and background story, let’s start talking about the requirements around UI testing. The first thing we need to consider is authentication. When a user attempts to login to the application, the user will be prompted to enter username and password. In addition, based on your Azure Active Directory (AAD) conditional access policy, you may also be challenged with MFA. In my case, that’s what is configured, and so, I started creating “automation” user accounts that I plan to use for my UI testing and configured them to be excluded via a user group called “automation bots” since these users are not real users.

The next consideration would be handling prompts from the sign in process to skip registering an MFA device. The following is an example of what you would encounter.

We will always need to skip setup and be able to access the application. Here, we should note that for a new user, the new user will not have consented to access the application. Hence, before using any automation accounts, you want to manually go through the process and consent. I don’t know if it is worth the effort to automate this part as well, but feel free to disagree.

Selenium

With Selenium, it is fairly easy to pick a UI element once you are familiar with XPATH. If you are not aware, a good way to construct and test if your XPATH is valid is via the debugger console. Use the $x function.

Depending on the browser, we will also want to make sure to download the correct Selenium driver which is Edge in my case. For more information with Edge, see this link.

What we really want to make sure is that when finding HTML element, there is most likely going to be a delay for when the element is in DOM. As such, we need to “poll” for the element. One technique is to configure an appropriate implicit wait for all elements with ImplicitWait property.

edge.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(5);

Another technique is to use WebDriverWait — which is part of the DotNetSeleniumExtras.WaitHelpers nuget package. The Until method exposes a Func so we can configure for the right element.

var wait = new WebDriverWait(edge, TimeSpan.FromSeconds(15));
wait.Until(...

The last technique involves waiting for page to be redirected (especially during sign in with Azure Active Directory), which can be confirmed by URL. Again, we can use the same wait.Until syntax for this.

You may notice that some of these techniques are not available when you are coding. Let’s make sure we are using the appropriate selenium nuget packages. With Edge, these are my nuget packages (other than the standard Selenium.WebDriver and Selenium.Support): DotNetSeleniumExtras.WaitHelpers and, Microsoft.Edge.SeleniumTools. That should conver the missing (extension) methods.

GitHub Actions

Once we have developed and tested UI tests locally with Selenium, we are ready to start pushing the code and running the same tests on GitHub with GitHub Actions.

The first consideration is to setup the Azure Environment with the necessary resources. With Bicep, it is straight forward to do. We can then run the az bicep build command to generate the ARM template for deployment. Next, we can configure the UI tests to point to the App Service resource URL during execution. Here, we should mention that the execution is similar to how we are testing locally with dotnet test. We will want to create a script that runs the tests.

The output of the tests will show in the logs. These are just raw console output and not properly formatted results for easy viewing. With Azure DevOps, I know the results will auto-magically surface. An example of that below.

However, this is not the case for GitHub Actions. Fortunately, someone already built an action which consumes the results in JUNIT format. Here’s a code snippet of my dotnet test with JUNIT output.

dotnet test --logger "junit;LogFilePath=$testFilePath"

My next line is to publish this as a step output, so that the GitHub Action can consume.

Write-Output "::set-output name=test_file_path::$testFilePath"

Now, we can use the following GitHub Action to publish the results in a format that anyone can follow.

- name: Publish functional test results
  uses: EnricoMi/publish-unit-test-result-action/composite@v1
  with:
    files: ${{ steps.testresults.outputs.test_file_path }}

The result of this will produce the following example.

Wrapping Up

As part of teardown, I have a step that runs the Remove-AzResource command to remove resources generated. The outcome here is that we now have an automated way to generate an Azure Environment on the fly, run UI testing and finally destroy the Azure Environment. This means we are effective in controlling the cloud cost which we can’t normally do on premise because it is typically static virtual machines running all the time in a corporate data center.

The full code discussed in this article can be found on my github project i.e. the Azure Key Vault Password Manager. Feel free to use that as a reference if you have something similar cooking. I do hope you have enjoyed this reading and learned something new today.