MS Test v2 RunSettings for Selenium Tests on Multiple Browsers

Our team has been using the DataSource attribute to run multiple permutations of browser sizes since 2015. We’ve used the .testsettings file and the DeploymentItems to get the TestMatrix.csv to the Vm, which uses that as the DataSource.

We want to move to .runSettings . The TestRun Parameters and TFS/VSTS build steps to provide the browser type and size (ex. Chrome, Large) at runtime.

The DataSource attribute is great for MSTest running the same test multiple times, just based on how many lines in the .csv file. That feature was useful to get large and small showing next to eachother in the build results (they’d look like MyTest (Data Row 0) and MyTest (Data Row 1)). This was a quick indication that it wasn’t a timing issue. It was also good to add 10 lines in the .csv to try and replicate a difficult to replicate failure.

However, .runSettings seems the way forward and the VSTS build definitions have made it much easier than the old XAML builds to add in new build steps.

Note: I’m on an OnPremise TFS installation Version 15.112.26307.0. VSTS will be different. I need to spend time there with the Deployment Groups and Release Management and then come back and update this article. Thanks for your understanding.

My Approach to moving to runsettings

I created a new local.runsettings files for running Selenium test from Visual Studio (2017 at this moment) with TestRunParameters.

  <TestRunParameters>
    <Parameter name="browserType" value="Chrome" />
    <Parameter name="browserSize" value="Large" />
  </TestRunParameters>

We can read the value from the TestContext.Properties dictionary. This StackOverflow answer pointed the way .

    var browserType = TestContext.Properties["browserType"].ToString();

Now we can use the browser type and size to tell Selenium how to run the test.

Using the parameters in the Release

First we need to use Windows Machine File Copy to get all the dlls and needed Selenium drivers to the VM. I’m using the $(UITestFiles) variable to send them to “C:\UiTestFiles”.

Then we can use the Deploy Test Agent and Run Fuctional Tests steps in the release management.

The is version 2, but on VSTS it looks different and there is only version 1. Make sure to check Run UI tests.

The settings will be similar to this. The $(TestMachines) variable is a comma list of VMs we have in our VMWare setup. UILocal.runsettings has the <TestRunParameters> I have above.

The Override Test Run Parameters allows us to override the TestRunParameter.

Here’s a similar article on MSDN from 2015.

Bonus Link: run more with Docker

Using Docker with VSTS and Selenium we may be able to get more browsers running in parallel without more Vms. This isn’t directly related to the runsettings, but a link I came across and something I want to investigate.