Unit tests should not Debug Assert
Preface
When we write unit tests we try to cover all edge cases, not just run through the cases that work. That means that in many cases the product code will assert before throwing an exception.
We have to separate cases here
1. We expected the assert and want to ignore it
2. We did not expect an assert, and in this case we don’t want to get the unit test stalled, but rather simply fail.
In this blog I’m suggesting a rather simple approach for dealing with this issue that will only require changes to the unit tests for case 1 above. The approach is relevant for MsTest based unit tests, although it can apply to other frameworks as well with some minor modifications.
Background
When we call Debug.Assert, Debug.Fail, Trace.WriteLine or similar methods, the resultant messages are routed to the static list of trace listeners (System.Diagnostics.Trace.Listeners). One of these listeners is the DefaultTraceListener that among other things pops up the Assert Dialog in debug mode.
For more information:
http://msdn.microsoft.com/en-us/library/system.diagnostics.tracelistener.aspx
http://msdn.microsoft.com/en-us/library/system.diagnostics.defaulttracelistener.aspx
Obviously when we run unit tests we don’t want this dialog to come up, because it will stall the overall run. If we are running hundreds of unit tests and a few of them pop asserts, we will be sitting for a long time waiting for results to come back (or interactively keep clicking ignore).
Thankfully we can modify this static list on run time to suit our needs.
A Basic solution
1: TraceListener removeListener = null;
2: foreach (TraceListener listener in Debug.Listeners)
3: {
4: if (listener is DefaultTraceListener)
5: {
6: removeListener = listener;
7: break;
8: }
9: }
10: Debug.Listeners.Remove(removeListener);
In this solution we simply iterate through the list of listeners (Debug.Listeners) and find the DefaultTraceListener and remove it.
Note that I’m not assuming it’s the first listener, since I don’t want to make assumptions that this list was never manipulated.
This code can be added before encountering an assert in the body of the test, but it poses a few limitations
1. It will remove the listener for the tests to follow
2. You have to put it in any test that fires asserts because there is no guarantee in what order or even if all tests are run
Fortunately we can make this a more global approach
1: [TestClass]
2: public class UnitTestInitializer
3: {
4: private static bool _initialized = false;
5: private static object _lock = new object();
6:
7: [AssemblyInitialize]
8: public static void Initialize(TestContext context)
9: {
10: lock (_lock)
11: {
12: if (_initialized)
13: {
14: return;
15: }
16:
17: TraceListener removeListener = null;
18: foreach (TraceListener listener in Debug.Listeners)
19: {
20: if (listener is DefaultTraceListener)
21: {
22: removeListener = listener;
23: break;
24: }
25: }
26: Debug.Listeners.Remove(removeListener);
27: }
28: }
29: }
In this approach I create another test class with the sole purpose of initializing the complete test assembly. It will now run for every test regardless if that test is run on it’s own or as part of a group run.
Note that the Initialize method is decorated with [AssemblyInitialize] and is within a [TestClass]
Still with this approach asserts will now be ignored, and the only way to see that they got fired is the Trace section of the test result, which will typically be ignored if the test passed.
Fail on assert (well sometimes…)
Most of the time we want asserts to fail the tests but not always, sometimes when testing edge cases we do expect the product to assert, and we don’t want the test to fail in these cases.
So the following FailOnAssert class got created, and we add it to the DebugListerner list (this line goes below line 26 in the above code snippet)
1: Debug.Listeners.Add(FailOnAssert.GetInstance());
FailOnAssert
1: public class FailOnAssert : TraceListener
2: {
3: [ThreadStatic]
4: private static bool _disable;
5:
6: private static FailOnAssert _instance = null;
7:
8: private static object _lock = new object();
9:
10: private FailOnAssert()
11: {
12:
13: }
14:
15: public static FailOnAssert GetInstance()
16: {
17: if (_instance != null)
18: {
19: return _instance;
20: }
21:
22: lock (_lock)
23: {
24: if (_instance == null)
25: {
26: _instance = new FailOnAssert();
27: }
28: }
29: return _instance;
30: }
31:
32: public static bool Disable
33: {
34: get
35: {
36: return _disable;
37: }
38: set
39: {
40: _disable = value;
41: }
42: }
43:
44: public override void Fail(string message)
45: {
46: if (!Disable)
47: {
48: Assert.Fail("Product raised an assert: " + message);
49: }
50: }
51:
52: public override void Fail(string message, string detailMessage)
53: {
54: if (!Disable)
55: {
56: Assert.Fail("Product raised an assert: " + message + "\n" + detailMessage);
57: }
58: }
59:
60: public override void Write(string message)
61: {
62: }
63:
64: public override void WriteLine(string message)
65: {
66: }
67: }
Fail on assert is a TraceListener, in order to do that I derived it from a TraceListener. And overridden the Fail, Write and WriteLine methods. Since the calls to Debug.Assert and Debug.Fail ultimately turn into calls to Fail, they call Assert.Fail and stop the test.
Note that Write and WriteLine are abstract in the TraceListener base class and have to be overridden although they don’t serve any purpose in this case.
The Disable property is used to temporarily allow asserts to fire without failing the tests, the _disable member is decorated with the [ThreadStatic] attribute since we don't want a test on one thread inadvertently changing this behavior for another thread.
The messages logged by any product trace calls are logged by the unit test framework listeners and hence I don't need to implement any logging in my code.
Using the Disabled Property:
1: try
2: {
3: FailOnAssert.Disabled = true;
4: MyTestAction();
5: }
6: finally
7: {
8: FailOnAssert.Disabled = false;
9: }
Conclusion
Unexpected asserts should not be ignored or stall the tests, and should trigger a test failure.
By simply adding this code to any existing unit tests project we where able to catch all asserts being fired without having to edit file by file, the code was generic enough to simply drop two files into each assembly and be mostly done.