Techniques for testing concurrent code in NUnit
Last time I blogged about a way to make concurrent unit tests deterministic. This time I’d like to focus on the technical aspects, namely NUnit. Let us start with the following test:
[Test]
public void TestThread()
{
int n = 0;
var thread1 = new Thread(() =>
{
n = 10;
Assert.Fail("This should fail!");
n = 20;
});
thread1.Name = "Broken thread";
thread1.Start();
thread1.Join();
Assert.AreEqual(10, n);
}
Can you tell what will happen when you run it? When you start a thread, it will execute lambda function code, assign 10 to n, and then… Fail? Nope. Assert.Fail(“This should fail”) will only throw an exception. Who is going to catch it? How NUnit can relate that exception to the unit tests running in another thread or, strictly speaking, not running any longer (in case you forgot to join() to it)? So the correct answer is test will print some warnings and will eventually pass:
Thread Name: Broken thread
NUnit.Framework.AssertionException: This should fail!
at NUnit.Framework.Assert.Fail(String message, Object[] args)
at NUnit.Framework.Assert.Fail(String message)
...
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
1 passed, 0 failed, 0 skipped, took 0.89 seconds (NUnit 2.5).
Any solutions? Sure:
- Use PNUnit, available as part of NUnit as of 2.5.0.9117. It allows you to run parallel (and even distributed!) tests, while capturing exceptions properly. The downside of this approach is that you cannot really use threads, instead you have to break your test into separate tests and make them synchronize to each other using PNUnit services. I am not a huge fan of this approach, since it introduces dependencies between tests and the whole rig becomes a bit more complex.
- Use CrossThreadTestRunner by Peter Provost. It simply starts a delegate in a thread, waits for the thread to complete, captures any exceptions thrown by the thread and then uses a neat trick to re-throw an exception in unit test’s primary thread. Here is an example:
[Test]
public void TestThread()
{
int n = 0;
CrossThreadTestRunner c = new CrossThreadTestRunner(()=>
{
n = 10;
Assert.Fail("This should fail!");
n = 20;
});
c.Run();
Assert.AreEqual(10, n);
}
- Use Mike Peretz’s solution. Mike suggests using asynchronous delegate calls instead of threads to test parallel code. The example:
[Test]
public void TestThread()
{
int n = 0;
ThreadStart main = () =>
{
n = 10;
Assert.Fail("This should fail!");
n = 20;
};
var asyncResult = main.BeginInvoke(null, null);
main.EndInvoke(asyncResult);
Assert.AreEqual(10, n);
}
In both cases results are the same – test fails where it should fail, NUnit produces reasonable stack trace. In case of CrossThreadTestRunner the stack is more readable, though.
Back to the subject, CrossThreadTestRunner blocks the test’s execution so I cannot use it for parallel code testing. Mike Peretz’s way looks more appealing for testing parallel processes. Unfortunately it comes with a catch: .NET actually uses the thread pool behind the scenes when invoking delegates asynchronously, so it is not guaranteed to run threads in parallel. Here is a sample code to confirm this concern:
public delegate void T(int n);
[Test]
public void TestMany()
{
T main = (n) =>
{
Console.Out.WriteLine("Thread " + n + " starting");
Thread.Sleep(1000);
Console.Out.WriteLine("Thread " + n + " about to fail");
Assert.Fail("This should fail!");
Console.Out.WriteLine("Thread " + n + ": you should never see this");
};
IAsyncResult[] results = new IAsyncResult[100];
for (int i = 0; i < 100; i++)
{
results[i] = main.BeginInvoke(i,null, null);
}
Thread.Sleep(1000);
for (int i = 0; i < 100; i++)
{
main.EndInvoke(results[i]);
}
}
Here I start 100 concurrent delegates, each prints first line, waits 1 second, prints the second line and fails. After starting all delegates I wait for 1 second and end the invocations. Can you tell how many threads will run? If you said 100 you are wrong, here is the test output:
Thread 0 starting
Thread 2 starting
Thread 3 starting
Thread 1 starting
Thread 4 starting
Thread 5 starting
Thread 3 about to fail
Thread 0 about to fail
Thread 2 about to fail
Thread 1 about to fail
Thread 6 starting
Thread 7 starting
Thread 8 starting
Thread 9 starting
TestCase 'TestMany' failed: This should fail!
Server stack trace:
...
0 passed, 1 failed, 0 skipped, took 1.91 seconds (NUnit 2.5).
Reading Thread.CurrentThread.IsThreadPoolThread inside the delegate returns true, so I’d suggest we are running 6 threads at a time here. This is not good for parallel code testing, since it can eliminate race conditions. In practice you barely run 3+ threads scenarios, but load tests may actually use many threads. Still, this prevents tests to run in predictable manner.
Another sad side effect of this technique: your test will only fail if you call EndInvoke for all delegates you started. If you forget to do it, you may end up in a situation where NUnit thread already finished executing the test, but some asynchronous delegate calls are still sitting in the queue.
So it looks like I need a mixture of both approaches, but this is a topic for a separate post.