Sunday 8 December 2013

Moving to NUnit from MsTest - experience report

Recently, I was a part of migration of some tests from MsTest to NUnit. I was in a situation where some of the tests were written in NUnit and some (legacy ones) in MsTest and it was too costly to maintain both tools, so the money were put on NUnit. The goal was to convert existing tests as fast as possible, leaving refactoring and cleanup of test and production code for later. I didn't know much about MsTest before, and here's what I learned about it and the transition:

Assertions

Much to my astonishment, the number of assertions MsTest supports is rather small, compared to NUnit. You cannot assert something is greater than something else, cannot assert a value is in a certain range etc. This way, a lot of assertions looked like this (remember this was ugly legacy test code :-)):

Assert.True(numberOfMessages <= 1000 
  && numberOfMessages >= 950, "numberOfMessages")

This, at failure, gives you the a message like: "Expected true, got false". Pretty helpful, isn't it (sarcasm)? Sure, you can make the message better, but you have to work hard on it yourself. The NUnit's (better) equivalent is:

Assert.That(numberOfMessages, Is.InRange(950, 1000));

Usually, when writing good unit tests, I don't really need this kind of assertions, but this was legacy code and it tested many kinds of weird situations over a large set of objects put together.

On the other hand, MsTest has a strongly typed assertions in form of Assert.AreEqual<T>() which are missing from NUnit. This, however, is not a big deal, because creating a custom wrapping assertions which will give you this benefit in NUnit is trivial. For example:

public class XAssert
{
  public static void AreEqual<T>(T expected, T actual)
  {
    NUnit.Framework.Assert.AreEqual(expected, actual);
  }
}

Lessons learned:

  1. If you ever use MsTest, do yourself a favor and pick an external assertion library.
  2. If you use NUnit, writing strongly typed wrappers over common assertions may be a good idea.

Deployment Items in .testsettings file

MsTest has this little feature of deployment items that can be configured in solution-wide .testsettings file to be copied to the output directory before a test is run (usually these are some extra files etc.).

But wait! Why on earth would someone be willing to use that kind of feature in unit tests at all?

Unfortunately, this was my case - the deployment items were used to load an XML configuration :-(. The problem with Deployment Items is that some 3rd party runners support it, and some don't (namely: NCrunch and Gallio, which makes running tools like Sonar over such tests pretty difficult). Also, some 3rd party unit testing tools do not run tests from the build output directory, but create their owne custom directories somewhere on the hard drive (e.g. NCrunch does), but the paths coded in Deployment Items are not adjusted and get gopied to the same directories as always, leading to failed runs.

Lessons learned:

  1. If you're using Deployment Items in MsTest for unit tests, you don't even give me a chance to think you're serious.
  2. When Deployment Items are used to copy some additional files to the build output, this mechanism can be replaced with including these config files in test project as links and setting their "Copy Local" property to copy them to the build output directory.
  3. Do not write or load files in unit tests!!!

PrivateType and PrivateObject

I was shocked when I discovered that MsTest includes helpers for accessing private instance/static methods: PrivateType and PrivateObject. Even for legacy code, there are better strategies to apply. Anyway, what shocks me more is that anyone can even think that using these may be a good idea (it is not). Thus, when converting to NUnit, I found such constructs in the code. They lead to brittle tests that are accessing members using strings and reflection. It seems like MsTest allows generating accessors that are strongly typed and use strings and reflection under the hood, but this is still coupling to implementation details, plus I saw PrivateType and PrivateObject being used directly many times.

Lessons Learned:

  1. When writing new tests, forget that PrivateType and PrivateObject exist. Look for the real problem.
  2. PrivateType and PrivateObject are just helpers over a reflection API, so they don't require being executed under MsTest to work properly. During transition to NUnit, you can leave references to MsTest assembly and change all code references to include namespace e.g. from PrivateObject to Microsoft.VisualStudio.TestTools.UnitTesting.PrivateObject. The code will still work.

Cost of Transition

In general, it's a lot simpler to migrate from MsTest to NUnit than the other way round. This is because NUnit requires only that an assembly references nunit.framework.dll and marks certain methods with [Test] attribute. On the other hand, MsTest requires a special type of project (a Test Project) to hold the tests. Also, as I said, MsTest's assertions are for the most part a subset of NUnit's. Anyway, to convert a project from MsTest to NUnit, the following steps are usually sufficient:

  1. Reference nunit.framework.dll from the test project.
  2. Convert Deployment Items into links in project copied to the build output folder.
  3. Search-Replace the following:
    1. [TestClass] -> [TestFixture]
    2. [TestMethod] -> [Test]
    3. [TestInitialize] -> [SetUp]
    4. [TestCleanup] -> [TearDown]
    5. [TestClassInitialize] -> [TestFixtureSetUp]
    6. [TestClassCleanup] -> [TestFixtureTearDown]
    7. using Microsoft.VisualStudio.TestTools.UnitTesting; -> using NUnit.Framework;
  4. Change Assert.IsInstanceOfType(x, typeof(X)) to Assert.IsInstanceOf<X>(x);
  5. Change Assert.AreEqual<X>(x, y) to Assert.AreEqual(x, y)
  6. Fix references to some MsTest-specific classes (like PrivateObject)

Summary

And that's it. These were the lessons learned and tips for dealing with the transition as fast as possible. I advise to do a real cleanup of the test and production code sooner than later, but the tips I outlined here let you defer this cost.

7 comments:

Aaron said...

Thanks, this is a really useful guide and saved me a lot of time. I'm going through the same migration at the moment. The generalisations are pretty accurate.

Grzegorz Gałęzowski said...

Aaron, I am very happy it helped!

If you like, please leave a comment on additional things you find out when you're finished migrating. Hopefully others looking at this post and comments will benefit!

Mark said...


StringAssert.EndsWith(actual, expected) <=> StringAssert.EndsWith(expected, actual)

Unknown said...

Hi,
[TestInitialize] -> [SetUp]
[TestCleanup] -> [TearDown]

The above setup are not working as it does with MSTEST on NUNIT. With MSTest using it in conjunction with Selenium, every time a test runs, it closes the window and moves on to the next test. What I've noticed with Nunit is that it runs the first test, but after that everything elses falls apart. It just fails to re use the SetUp and Teardown. Has anyone faced this issue before?

Grzegorz Gałęzowski said...

Hi, Kam fet,

I never used Selenium, the only big difference I know is that MsTest reinitializes all fieds for each new test, but NUnit doesn't unless you do so explicitly in [SetUp] method. Maybe you have some fields initialized in constructor or inline in a test class?

Unknown said...

Hi, Gregoz,

I was going to send you a direct msg but can't see a way to do it. Below is a sample of my tests.

using System.Threading;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;

namespace SmokeTests.SIT
{
[TestClass]
public class SIT2ContactUs
{
IWebDriver driver = new FirefoxDriver();

[TestMethod]
public void VerifyEmailValMsgContactUsSIT2()
{
var emailMessage =
driver.FindElement(By.XPath("/html/body/form/div[4]/div/div/div[2]/div[1]/section/div[1]/div/ul/li[1]"));
Assert.AreEqual("Email is required", emailMessage.Text);
}

[TestMethod]
public void VerifyTitleValMsgContactUsSIT2()
{
var titleMessage =
driver.FindElement(By.XPath("/html/body/form/div[4]/div/div/div[2]/div[1]/section/div[1]/div/ul/li[2]"));
Assert.AreEqual("Title is required", titleMessage.Text);
}

[TestMethod]
public void VerifyFirstNameValMsgContactUsSIT2()
{
var firstNameMessage =
driver.FindElement(By.XPath("/html/body/form/div[4]/div/div/div[2]/div[1]/section/div[1]/div/ul/li[3]"));
Assert.AreEqual("First name is required", firstNameMessage.Text);
}

[TestMethod]
public void VerifyLastNameValMsgContactUsSIT2()
{
var lastNameMessage =
driver.FindElement(By.XPath("/html/body/form/div[4]/div/div/div[2]/div[1]/section/div[1]/div/ul/li[4]"));
Assert.AreEqual("Last name is required", lastNameMessage.Text);
}

[TestMethod]
public void VerifyAppQuestionContactUsSIT2()
{
var appropriateQuestionMessage =
driver.FindElement(By.XPath("/html/body/form/div[4]/div/div/div[2]/div[1]/section/div[1]/div/ul/li[5]"));
Assert.AreEqual("Please select appropriate question", appropriateQuestionMessage.Text);
}

[TestMethod]
public void QueryValMsgContactUsSIT2()
{
var queryMessage =
driver.FindElement(By.XPath("/html/body/form/div[4]/div/div/div[2]/div[1]/section/div[1]/div/ul/li[6]"));
Assert.AreEqual("Query is required", queryMessage.Text);
}

[TestMethod]
public void VerifyNoCaptchaErrorCuSIT2()
{

var captchaMessage =
driver.FindElement(By.XPath("/html/body/form/div[4]/div/div/div[2]/div[1]/section/div[1]/div/ul/li[7]"));
Assert.AreEqual("That was quick! To ensure you’re not a spambot, please wait 10 seconds and submit again.",
captchaMessage.Text);
}

[TestInitialize]
public void Setup()
{
driver.Navigate().GoToUrl("http://web2.femi.com/about/contact#.V0bjcoQrLcs");


//Using thread because the system is gettng slower (annoyingly)
Thread.Sleep(5000);
var submit = driver.FindElement(By.ClassName("js-duplicate-button"));
submit.Click();
}

[TestCleanup]
public void TearDown()
{
//close browser
driver.Quit();
}
}
}

Grzegorz Gałęzowski said...

Kam fet, I think you can always send me a private message via google plus.

The only thing I see in your tests that I find suspicious is that web driver is created through inline field initialization. In MsTest, this would cause a new driver created for each test. In NUnit, this will only create a single web driver for the entire test suite. This may be the reason why the second test in the suite doesn't run as you expected.

My advise would be: try moving the creation of web driver to [SetUp] method, before "driver.Navigate().GoToUrl" call.