Moving to NUnit from MsTest - experience report

Recently, I was a part of migration of some tests from MsTest to NUnit. I was in a situation where some of the tests were written in NUnit and some (legacy ones) in MsTest and it was too costly to maintain both tools, so the money were put on NUnit. The goal was to convert existing tests as fast as possible, leaving refactoring and cleanup of test and production code for later. I didn't know much about MsTest before, and here's what I learned about it and the transition:

Assertions

Much to my astonishment, the number of assertions MsTest supports is rather small, compared to NUnit. You cannot assert something is greater than something else, cannot assert a value is in a certain range etc. This way, a lot of assertions looked like this (remember this was ugly legacy test code :-)):

Assert.True(numberOfMessages <= 1000 
  && numberOfMessages >= 950, "numberOfMessages")

This, at failure, gives you the a message like: "Expected true, got false". Pretty helpful, isn't it (sarcasm)? Sure, you can make the message better, but you have to work hard on it yourself. The NUnit's (better) equivalent is:

Assert.That(numberOfMessages, Is.InRange(950, 1000));

Usually, when writing good unit tests, I don't really need this kind of assertions, but this was legacy code and it tested many kinds of weird situations over a large set of objects put together.

On the other hand, MsTest has a strongly typed assertions in form of Assert.AreEqual<T>() which are missing from NUnit. This, however, is not a big deal, because creating a custom wrapping assertions which will give you this benefit in NUnit is trivial. For example:

public class XAssert
{
  public static void AreEqual<T>(T expected, T actual)
  {
    NUnit.Framework.Assert.AreEqual(expected, actual);
  }
}

Lessons learned:

1. If you ever use MsTest, do yourself a favor and pick an external assertion library.
2. If you use NUnit, writing strongly typed wrappers over common assertions may be a good idea.

Deployment Items in .testsettings file

MsTest has this little feature of deployment items that can be configured in solution-wide .testsettings file to be copied to the output directory before a test is run (usually these are some extra files etc.).

But wait! Why on earth would someone be willing to use that kind of feature in unit tests at all?

Unfortunately, this was my case - the deployment items were used to load an XML configuration :-(. The problem with Deployment Items is that some 3rd party runners support it, and some don't (namely: NCrunch and Gallio, which makes running tools like Sonar over such tests pretty difficult). Also, some 3rd party unit testing tools do not run tests from the build output directory, but create their owne custom directories somewhere on the hard drive (e.g. NCrunch does), but the paths coded in Deployment Items are not adjusted and get gopied to the same directories as always, leading to failed runs.

Lessons learned:

1. If you're using Deployment Items in MsTest for unit tests, you don't even give me a chance to think you're serious.
2. When Deployment Items are used to copy some additional files to the build output, this mechanism can be replaced with including these config files in test project as links and setting their "Copy Local" property to copy them to the build output directory.
3. Do not write or load files in unit tests!!!

PrivateType and PrivateObject

I was shocked when I discovered that MsTest includes helpers for accessing private instance/static methods: PrivateType and PrivateObject. Even for legacy code, there are better strategies to apply. Anyway, what shocks me more is that anyone can even think that using these may be a good idea (it is not). Thus, when converting to NUnit, I found such constructs in the code. They lead to brittle tests that are accessing members using strings and reflection. It seems like MsTest allows generating accessors that are strongly typed and use strings and reflection under the hood, but this is still coupling to implementation details, plus I saw PrivateType and PrivateObject being used directly many times.

Lessons Learned:

1. When writing new tests, forget that PrivateType and PrivateObject exist. Look for the real problem.
2. PrivateType and PrivateObject are just helpers over a reflection API, so they don't require being executed under MsTest to work properly. During transition to NUnit, you can leave references to MsTest assembly and change all code references to include namespace e.g. from PrivateObject to Microsoft.VisualStudio.TestTools.UnitTesting.PrivateObject. The code will still work.

Cost of Transition

In general, it's a lot simpler to migrate from MsTest to NUnit than the other way round. This is because NUnit requires only that an assembly references nunit.framework.dll and marks certain methods with [Test] attribute. On the other hand, MsTest requires a special type of project (a Test Project) to hold the tests. Also, as I said, MsTest's assertions are for the most part a subset of NUnit's. Anyway, to convert a project from MsTest to NUnit, the following steps are usually sufficient:

1. Reference nunit.framework.dll from the test project.
2. Convert Deployment Items into links in project copied to the build output folder.
3. Search-Replace the following:
   1. [TestClass] -> [TestFixture]
   2. [TestMethod] -> [Test]
   3. [TestInitialize] -> [SetUp]
   4. [TestCleanup] -> [TearDown]
   5. [TestClassInitialize] -> [TestFixtureSetUp]
   6. [TestClassCleanup] -> [TestFixtureTearDown]
   7. using Microsoft.VisualStudio.TestTools.UnitTesting; -> using NUnit.Framework;
4. Change Assert.IsInstanceOfType(x, typeof(X)) to Assert.IsInstanceOf<X>(x);
5. Change Assert.AreEqual<X>(x, y) to Assert.AreEqual(x, y)
6. Fix references to some MsTest-specific classes (like PrivateObject)

Summary

And that's it. These were the lessons learned and tips for dealing with the transition as fast as possible. I advise to do a real cleanup of the test and production code sooner than later, but the tips I outlined here let you defer this cost.