NUnit Tutorial – Part IV: Running Unit Tests

Running NUnit Tests


We have seen so far, in this series of tutorials, many concepts about unit tests and its implementation, such as what are the benefits of unit testing, how to organize your testing projects and how to implement testing code. We will now address the dynamic aspects of unit testing, that is, those concepts related to running unit tests.

In this article, we will demonstrate how you can run unit tests, by means of a real example of NUnit unit tests written in C#. We will also see some of the testing execution tools available to you and what are the essential steps that make up the process of running unit tests, from the selection of the tests to run through to the analysis of the results obtained, so to drive us to the fix of the failures and errors encountered.

NUnit Tools and the Process of Running Unit Tests.


There are several different ways to run unit tests, each one being more (or less) appropriate for your specific needs, and it’s up to you to decide which tool to use to accomplish this task. Here are some of the most common ways of running unit tests. Feel free to comment, at the end of this article, another tools and approaches you know on how to run unit tests.

  • From the command line: you can run your unit tests via the command prompt, as described by the NUnit documentation.. This may be the best option if you have a continuous Integration server in which your unit tests are run during the build process of your software.
  • NUnit plugins and extensions:being very useful for programmers, this option allows you to run your unit tests from inside Visual Studio or your favorite IDE. Some of these are free, such as NUnit Test Adapter, while others need to be purchased, such as Telerik Unit Test Runner, for example.
  • NUnit GUI: this tool offers a graphical user interface with many features, commands and options that facilitate the selection of tests to be run, running them and viewing the results of the tests execution, as we will see in this article.

Regardless of which of these approaches or tool you choose, the process of running unit tests will be basically the same, consisting of steps similar to the following:

  1. Selecting the tests: the first thing to do is choosing which tests you want to run. You can, for example, select all of your tests or choose to run only those tests related to the functionalities in which you are currently working.
  2. Running the tests: after selecting the tests to be run, the next thing to do is running them.
  3. Analyzing the Tests Results: after running the tests, you analyze the results generated by NUnit for these tests. Some tools present the results directly into their graphical interfaces, while others generate files containing the test results. Among the information you can extract from the results of your tests, you will encounter which and how many tests failed, which and how many tests passed (i.e success result), details about the failures and erros found and what was the time taken to run your tests.
  4. Fixing the problems found: if your tests do not result in 100% of success, you will need to take some actions to fix the problemns found, which could be either failures or errors. We give an example on how to do this further in this article.
  5. Run the tests again: After working in the problems found, you run your tests again to verify if these problems have been actually fixed and to ensure that new bugs have not arisen due to side effects caused by the changes performed in the code, repeating this process again as many times as necessary to remove all failures and errors found in your code.

In this article we will follow this entire process, demonstrating each of the above steps, using as an example the NUnit GUI tool. The information given here should be enough for you to get started working with any other unit testing tool, because the concepts and process are quite similar, so you will have no problems to apply this information no matter which unit testing tool you choose to use.

The NUnit GUI Tool


NUnit GUI (GUI = Graphical User Interface) can be found in the NUnit Framework installation directory and corresponds to the file nunit.exe. After installing NUnit Framework (see how to do this in “Adding the NUnit Framework to our Testing Project” in Part II of this tutorial), NUnit GUI tool will be available in a path similar to “C:Program Files (x86)NUnit 2.6.3binnunit.exe” on your computer. Note that this folder may vary depending on your operating system and the version of NUnit that’s installed in your system.

When you run this application (nunit.exe), NUnit GUI initially looks like as shown below:

Nunit IDE

From this picture we can identify some of the features this tool offers us:

  • Two buttons called Run and Stop, which respectively run and stop the execution of unit tests.
  • A tab entitled “Errors and Failures”, in which are listed the erros and failures that occurr during the execution of your unit tests.
  • A tab titled “Tests Not Run”, in which are listed the tests which were selected to be run but were not executed.
  • A tab titled “Text Output”, which displays all log commands found in the test code. These logs can be written by using System.Diagnostics.Debug.Write() and System.Console.WriteLine() in your test code.
  • A tab titled “Tests”. Here is where all your unit tests are listed, in the form of a tree, as we shall see further in this article.
  • A label titled “Test Cases” and whose text is currently showing “0″ (zero). This label will show the number of unit tests that were selected to be run.

Other resources available in NUnit GUI tool can be accessed by the application’s menu options. Some of these menu options are:

  • File Menu: This menu presents a range of options related to unit testing projects. Here you can create a new project, open an existing project or close the currently open project. You can also save the project which is currently open and reload it, as well as see a list of projects in which you have worked recently. This menu also lets you close the application by selecting the option File -> Exit.
  • View Menu: here you find features related to the application user interface, such as changing the font size or hide and show some parts of the GUI according to your personal preference.
  • Project Menu: This menu allows you to modify your project settings and add new test assemblies to your project.
  • Tests Menu: This menu displays options which allow you to run and stop the execution of unit tests.
  • Tools Menu: from the perspective of this tutorial the most important option in this menu is “Save Results as XML …”, which allows you to save the results of the tests as a file for future analysis. (The other operations available in this menu are related to NUnit GUI only, and not with unit tests, which is why we will not go into more detail on this menu in this tutorial.)
  • Help Menu: allows you to access an offline version of NUnit’s documentation and shows information about the version of NUnit GUI which you are using.

Compiling the Code


The first thing to do before running the unit tests is to compile our code, both the production code and the test code. This is necessary so that our test project assembly (*.dll) will be generated with the latest changes we made in our code.

In this article, we will use a modified version of the Calculator project that was presented at Part II of this tutorial series. If you have not already created this project, we suggest you to read Part II of this series of tutorials before proceeding.

Open the project and modify the Calculator and CalculatorTest classes as shown below:

namespace SimpleCalculator
{
    // The class which represents our calculator.
    public class Calculator
    {
        public int Add(int a, int b)
        {
            return a + b;
        }

        public int Subtract(int a, int b)
        {
            return a - b;
        }

        public int Multiply(int a, int b)
        {
            // Intentional failure: the multiplication test will fail because this function is adding instead of multiplying.
            return a + b;
        }

        public int Divide(int a, int b)
        {
            return a / b;
        }

        public double Exponentiation(int a, int b)
        {
            // Intentional failure: the exponentiation test will fail because this function is always returning zero.
            return 0;
        }
    }
}

using SimpleCalculator;
using NUnit.Framework;

namespace SimpleCalculatorTests
{
    [TestFixture]
    public class CalculatorTests
    {
        // The calculator object that will be used by the tests to verify the Calculator class funcionalities.
        private Calculator _calculator;

        [TestFixtureSetUp]
        public void Initialize()
        {
            _calculator = new Calculator();
        }

        [Test]
        public void Add_Test()
        {
            // Setup the values to be used by this test.
            int a = 10;
            int b = 20;
            int expectedSum = (a + b);

            // Exercises the sum functionality of the calculator.
            int actualSum = _calculator.Add(a, b);

            // Checks if the sum was performed correctly.
            Assert.AreEqual(expectedSum, actualSum, "The sum function of the calculator did not return the expected value.");
        }

        [Test]
        public void Subtraction_Test()
        {
            // Defines the values ​​to be used by this test.
            int a = 10;
            int b = 7;
            int expectedDifference = (a - b);

            // Exercises the subtraction functionality of the calculator.
            int actualDifference = _calculator.Subtract(a, b);

            // Checks if the subtraction was performed correctly.
            Assert.AreEqual(expectedDifference, actualDifference, "The subtraction function of the calculator did not return the expected value.");
        }

        [Test]
        public void Multiplication_Test()
        {
            // Defines the values ​​to be used by this test.
            int a = 2;
            int b = 3;
            int expectedProduct = (a * b);

            // Exercises the multiplication functionality of the calculator.
            int actualProduct = _calculator.Multiply(a, b);

            // Verifies if the subtraction was performed correctly.
            Assert.AreEqual(expectedProduct, actualProduct, "The multiplication function of the calculator did not return the expected value.");
        }

        [Test]
        public void Divide_Test()
        {
            // Setup the values ​​to be used by this test.
            int a = 50;
            int b = 10;
            int expectedResult = (a / b);

            // Exercises the Divide functionality of the calculator.
            int actualResult = _calculator.Divide(a, b);

            // Checks if the division was performed correctly.
            Assert.AreEqual(expectedResult, actualResult, "The divide function of the calculator did not return the expected value.");
        }

        [Test]
        public void Exponentiation_Test()
        {
            Assert.AreEqual(1, _calculator.Exponentiation(5, 0), "The exponentiation function of the calculator is not working correctly.");
            Assert.AreEqual(5, _calculator.Exponentiation(5, 1), "The exponentiation function of the calculator is not working correctly.");
            Assert.AreEqual(25, _calculator.Exponentiation(5, 2), "The exponentiation function of the calculator is not working correctly.");
            Assert.AreEqual(0.2, _calculator.Exponentiation(5, -1), "The exponentiation function of the calculator is not working correctly.");
        }
    }
}

Note that we have inserted some deliberate failures in the code for didactic reasons. We ask you to keep the code with these failures, which we will fix while demonstrating to you the process of running unit tests.

With your code updated with the above changes, compile the projects SimpleCalculator and SimpleCalculatorTests by selecting the menu option Build -> Build Solution (or by pressing the shortcut key F6), as shown in the picture below.

Menu Build Solution

Building your solution will generate the binary files of your program in the bin directory of your projects. Depending on the configuration of your solution you can find the generated binaries in one of these directories: bindebug or binrelease, so be sure to check both of these directories in case you don’t find the *.dll file in the first one you look at.

Loading Unit Tests in NUnit


To run your unit tests, you need to create a test project in NUnit GUI tool and add to this project the compiled *.dll file generated from the build of your unit test project. After doing that, NUnit will analyze the *.dll file and load the test classes and test methods you have defined in your test code.

Follow the steps below to create a test project in NUnit GUI:

  1. In NUnit GUI, select the menu option File -> New Project.
  2. Choose a name for the project and select the directory in which you want to save your project. Click Save. NUnit GUI will set up your test project as a file with the *.nunit extension, saving this file in the folder you have selected.
  3. Select the menu option Project -> Add Assembly…
  4. Menu Project Add Assembly

  5. In the Add Assembly dialog box, navigate to the bin directory of the test project we created in Visual Studio (SimpleCalculatorTests), select the *.dll file of the same name of the project (SimpleCalculatorTests.dll) and click Open.
Adding Test Assembly to NUnit Project

Adding Test Assembly to NUnit Project

By doing this, NUnit GUI will load all unit tests that we have implemented in the test project of our calculator, presenting our tests as a tree structure organized and divided into classes and test methods, as you can see in the picture below.

List of Loaded Unit Tests

List of Loaded Unit Tests

Running Unit Tests


To run the unit tests, select the node CalculatorTests in the tests tree and click the Run button. With that NUnit will start running the tests and will show a progress bar to track the test execution progress. While running the calculator tests, you will see that the progress bar turned red during and after the tests execution. This occurred because there are tests that are failing, and the red color is a feedback to alert the developer that some of the features being tested need to be fixed.

Below the test progress bar, NUnit GUI is shows that 3 of our tests have passed (run successfully) and two tests have failed. We can also see, on the left side of the NUnit GUI window, that some of the tests in the tree are highlighted with green, and other tests with red, indicating respectively which tests passed and which tests failed. For the current version of our calculator, the division, addition and subtraction tests have passed successfully, while the Exponentiation and Multiplication tests have resulted in failures.

Test Results with Failures

Test Results with Failures

As we said earlier, and as we can better see now, the results presented by the NUnit GUI after the execution of unit tests brings us important information about the code being tested, such as:

  • How many test cases were executed.
  • How many errors and failures occurred during the tests run.
  • The total time duration taken to run the unit tests.
  • How many tests resulted in success.
  • A message describing the test failures, which we define in the test code as a parameter passed to the assertion methods. (Learn more about assertions at Part III of this tutorial series on unit testing.)

Analyzing and Fixing the Problems Found by the Tests


With the implementation of our unit tests we were able to clearly identify the functionalities that need to be fixed in our code. Many failures such as these can occur unnoticed in your code, and here is where a suite of unit tests becomes essential for you to find these bugs and increase the reliability of your code.

To better understand why our tests have failed we can analyze the results generated by NUnit after running them. Beneath the progress bar, inside the Errors and Failures tab, NUnit provides a list composed by the name, error message and the reason why each one of the tests failed, as shown in the following picture:

Tests Faliures Feedback List

Tests Faliures Feedback List

Analyzing the first item on the list, SimpleCalculatorTests.CalculatorTests.Exponentiation_Test(), we find that the exponentiation test failed because an assertion was expecting the value 1, but the value was verified as 0, that is, the Eponentiation function did not behave as it should. Going further in our analysis of this test failure, you can click the button “Display real stack trace” Stack Trace real to view the Stack Trace that corresponds to the failure found during the test execution. For example, for the Exponentiation test failure, NUnit GUI presents a stack trace similar to the following:


at SimpleCalculatorTests.CalculatorTests.Exponentiation_Test() at D:UlyssesDropboxBlog C# ProgrammerConteudo das PublicacoesTutoriaisTestes Unitarios com nUnitCodigoSimpleCalculatorSimpleCalculatorTestsCalculatorTests.cs:line 81

Note that this stack trace tells us exactly the file and the line of code where the failure occurred, as highlighted in bold in the preceding paragraph. This gives us one more tip to help us in finding the source of the problem. If we open the specified file in the given line, we will see that the failure occurred in the following section of our test code:

Assert.AreEqual(5, _calculator.Exponentiation(5, 1), "The exponentiation function of the calculator is not working correctly.");

You can also check the fault line without leaving the NUnit GUI. Just click the button “Display Source Code Context”  Display Source Code Context . When you click this button NUnit will show you the corresponding failure found in the source code of your test, highlighting in red the line of code where the failure occurred, as we see in the following picture:

Source Code Context

Source Code Context

The failure in this line of code leads us to the conclusion that the method SimpleCalculator.Calculator.Exponentiation() did not calculate in the right way the value of 5 raised to 0 (5 ^ 0). Instead of returning 1 (which would be the correct result for this mathematical calculation) the result was 0, which is what the NUnit GUI reported in the tests failures list, with the message “Expected 1 But was 0.0d.”

This clearly demonstrates that unit tests fulfill their role in allowing us to identify not only what are the features of our software which are not working correctly, but also where those problems are located in our source code. This greatly facilitates the tasks of developing and maintaining software, especially for those large systems where there are literally hundreds or even thousands of lines of code working together for the good (or bad) functioning of your system.

For a more complete list of benefits that unit tests can bring to your software development process, read Party I of this tutorial series on unit testing in C# with NUnit.

Fixing the Problems Found


As we saw in the previous section, the SimpleCalculator.Calculator.Exponentiation() function is not returning the expected value for the calculations, so this is the point of our code that must be fixed. If we analyze the code of this method, we see that it is currently implemented as follows:

        public double Exponentiation(int a, int b)
        {
            // Intentional failure: the exponentiation test will fail because this function is always returning zero.
            return 0;
        }

As we see, this method is always returning 0 and actually it’s never performing the exponentiation calculation of the parameters passed to it, being this thus the reason for the failure of our test. To fix the problem, you need to modify the method so that it uses an exponentiation algorithm of your choice. You can also use the method System.Math.Pow() as we have done here:

        public double Exponentiation(int a, int b)
        {
            // Bug fixed: now our code is actually calculating the exponentiation of the given values​​.
            return System.Math.Pow(a, b);
        }

Following the same process we can also verify what caused the failure of the Multiplication test: the programmer (perhaps tired at the end of his long workday) didn’t notice when he used the addition operator (+) instead of the multiplication operator (*), thus obtaining the sum of the informed parameters, instead of returning the result of multiplying these values. Let’s modify the Calculator.Multiply() method to correct this, as shown in the snippet below:

        public int Multiply(int a, int b)
        {
            // Bug fixed: the sum operator (+) was removed and replaced by multiplication operator (*).
            return a * b;
        }

Running the Tests Again


After making the necessary changes to fix the flaws found in our code, we must build our solution and run the tests again. This will show us if the changes we did in fact removed the problems found during the first test run, and will also let us know if these changes did not cause side effects in other parts of our system.

We have made changes to the code since the last execution of the tests, so we need to access the option File -> Reload Tests (CTRL + R) to make NUnit GUI reload the test Assembly with the changes we have made to our code​​. After doing that, run the tests again and you will see that the progress bar has turned green this time, indicating that all executed tests have passed successfully, as shown in the picture below:

Green Progress Bar: 100% of Success in the Executed Tests!

Green Progress Bar: 100% of Success in the Executed Tests!

The green progress bar gives us an indication that all the features that were tested are working the way it should, and that no failures nor erros are occurring in that area of our software. If your tests are correct and if you have good test coverage for your code, this can be a good indicator of the quality of your code and, thus, a good indication of the quality of your software.

Conclusion – Part IV


We saw in this tutorial how you can run your unit tests and how the tests results can help you in finding what are and where are the flaws of your code, giving you a simple but effective way to keep bugs away from your software.

We hope the information presented here will help and encourage you to implement and execute unit tests in your own projects, and to analyze the test results to make it easier for you to fix the problems found in your code.

In the next part of this tutorial, which will be published soon, we will demonstrate in a practical way how to build software using a technique called Test Driven Development (TDD). Subscribe to our Newsletter or to our Feed to be notified when this and other articles became available in this blog.

Thank you for reading this article, and don’t forget to apply unit testing in your projects to increase the quality of your code and of your softwares.

If you enjoyed reading this article, remember to subscribe to our Newsletter and/or to subscribe to our Feed in order to stay updated about the future publications of C# Programmer.

Join the discussion. Your contribution is very important to our community.

Your email address will not be published. Required fields are marked *