Being human, you will make mistakes. Period. Software developers are human so we have to protect ourselves from making mistakes. With software, fixing a mistake (probably a bug) can be as easy as changing some lines of source code. When the bug is discovered during the development process, the impact will be low. Is the bug discovered by the end user or even worse - causing disruption of the business - the impact will be high.
We all know that testing software before releasing it to the customer is a good idea. This post is all about software testing; the types, levels, ways of testing, the process, frameworks and tools. I went deep into this subject because I'm using VSTS and soon start with unit testing in my current project and was asked to investigate TestDriven.NET.
Testing vs. Debugging
Software testing should not be confused with debugging. Debugging is the process of analyzing and locating bugs when software does not behave as expected. Debugging supports testing, off course. But testing is a more methodical approach of identifying bugs. This post is not about debugging, but about software testing.
Software testing is used in association with verification and validation:
- Validation: Are we doing the right job?
- Verification: Are we doing the job right?
Black-box- vs. White-box testing
A software system can be tested in two ways. It depends on your point of view. It can be with or without technical knowledge of the system. Or, tested by the developer or end-user. Or, tested by the manufacturer or customer. Black-box or white-box.
Without technical knowledge
Black-box tests can be functional or non-functional, though usually functional. The tester selects valid and invalid input for the test and determines if the output is correct. The tester doesn't need to have knowledge of the internal structure of the system. Typical black-box test design techniques include:
- Equivalence partitioning;
To reduce the number of test cases and select test cases that cover all possible scenarios
- Boundary value analysis
Validates input and checks if the input is in the valid range, i.e. if (month > 0 && month < 13)
- Decision table testing
Are about if- and switch-statements. Decision tables model conditional logic.
- Pairwise testing
Test each pair of input parameters to a method. Simple bugs are triggered by a single parameter, next simplest category of bugs consists of those dependent on interactions between pairs of parameters.
- State transition tables
Shows in what state a system moves to, based on the current state and input parameters.
- Use case testing
Users work through use cases with the aid to verify that a UI fulfills the needs of its users, as described in the use case model. The tester identifies which use case(s) to test, the actors (users), input, output and system effects and the flows of interest between the use cases.
- Cross-functional testing
The work of one person is reviewed by the team as a whole.
With technical knowledge
The white-box tests are done by somebody with technical knowdlegde of the system and with programming skills. Typical white-box test design techniques include:
- Control-flow testing;
The order in which the individual statements, instructions or function calls are executed or evaluated.
- Data-flow testing
Looks at the lifecycle of a particular piece of data, i.e. a variable.
Most used tests in the white-box category are unit tests.
Levels of software testing
- Unit testing;
Each unit (component, module, class) of the software is tested individually. Unit testing is done during white-box testing.
- Integration testing (also called as I&T);
Individual software modules are combined and tested as a group.
- System testing;
All software groups are integrated to the overall product and the complete product is tested. The purpose of system testing is to identify defects that will only surface when a complete system is assembled.
- Acceptance testing
The customer tests the system before he/she accepts the system and transfers the ownership. Each acceptance test represents some expected result from the system. Customers are responsible for verifying the correctness of the acceptance tests and reviewing test scores to decide which failed tests are of highest priority. The customer performs only black-box testing.
Test-driven development
With TDD, the developer writes a test case and implements only the code necessary to pass the test. Testing is part of the development process from day 1. Its a good way of designing software, not only for testing.
It is very important that the developer has a clear understanding of the specfications and requirements. Off course this is important with or without TDD, but TDD forces this understanding because its necessary in order to write the test. The specifications and requirements (use cases) can be found in the Functional Design.
With TDD a developer creates the test before he or she implements the code. The developer creates test units. A unit is the smallest testable part of an application. Unit testing is a method to test a piece of software code independent of the rest. Unit-testing frameworks support the developer with running the tests and managing and reporting the results.
The concept of TDD consists of 5 steps:
- Add a test;
Write the test code/unit test. This test must fail.
- Run all tests and see the new one fail;
This is to validate that the testing framework works as expected. The test should fail for the expected reason.
- Write code;
Implement the code. Write only the code that is required to pass the test. Later steps allows the developer to improve the code.
- Run the tests again and see it succeeding;
All tests should pass now. The programmer can be confident that the code meets all the tested requirements.
- Refactor the code
Refactoring is the process of changing code to improve readability or simplify the structure without changing the results. Tests should succeed after the refactoring.
The schema below shows the above 5 steps in a different way, but with the same idea.
Benefits of TDD
- Less need to use a debugger. Most mistakes are caught by the testing framework;
- In combination with a version control system like TFS, developers can see which code change caused the test failure;
- Software development can be better and faster. Not only because the corectness is validated, but also because the developer is actively involved in thinking how the functionality is used by the user;
- Gives a greater level of trust;
- Limit the number of defects (aka mistakes/bugs) in the code. Most defects are discovered and fixed during the development cycle;
- Modularized, flexible, and extensible code.
Unit testing
As said before, a unit is the smallest testable part of an application. Most of the time, this will be a class (in C# or Java). Good unit test design produces test cases that cover all paths through the unit with attention paid to loop conditions. The four most important benefits are:
- Facilitates change;
Code can be refactored, implementation can be changed.
- Simplifies integration;
In a bottom-up testing approach, testing the units first before testing the sum of its parts make integration easier.
- Documentation;
Developers looking to learn how to use a module can look at the unit test to determine how to use the module.
- Separation of interface from implementation (same benefit as SOA's)
Create a mock object for situations in which a scenario references multiple classes. A unit test should never go outside of its own class boundary. A mock object is an object that mimic the behavior of a real object in a controlled way.
Unit-testing frameworks
Unit-testing frameworks help simplify the process of unit testing. And I love things that help simplify. Now days, there are many, many frameworks available. Because I'm a .NET developer, I'll limit myself to UTFs for the .NET platform. The most popular UTF for .NET is NUnit. I also looked at VSTS (Visual Studio Team System).
Two editions of Microsoft's VSTS include unit testing:
NUnit
NUnit is a unit-testing framework for all .NET languages. It is written in C# and takes advantage of many .NET language features, for example custom attributes and other reflection related capabilities. NUnit is free. The official web site is: http://www.nunit.com/ . NUnit works with .NET 1.0 to 3.5 on Windows and Linux (using Mono).
Example
This example is copied from the NUnit Quick Start site:
Suppose we are writing a bank application and we have a basic domain class – Account. Account supports operations to deposit, withdraw, and transfer funds. The Account class may look like this:
public class Account
{
private float balance;
public void Deposit(float amount)
{
balance+=amount;
}
public void Withdraw(float amount)
{
balance-=amount;
}
public void TransferFunds(Account destination, float amount)
{
}
public float Balance
{
get{ return balance;}
}
}
The class below is a test class using NUnit: AccountTest
using NUnit.Framework;
[TestFixture]
public class AccountTest
{
[Test]
public void TransferFunds()
{
Account source = new Account();
source.Deposit(200.00F);
Account destination = new Account();
destination.Deposit(150.00F);
source.TransferFunds(destination, 100.00F);
Assert.AreEqual(250.00F, destination.Balance);
Assert.AreEqual(100.00F, source.Balance);
}
}
NUnit ships with a Console Runner and GUI Runner. The GUI Runner shows the tests in an explorer-like browser window and provides a visual indication of the success or failure of the tests.
Reporting/Results
NUnit stores the results of a test run in a file named TestResult.xml. Using XSLT-stylesheets, this can be transformed into a HTML-document or any other format.
VSTS Unit testing
Visual Studio 2005 is the first Microsoft IDE with built-in support for unit testing. Benefits of VSTS are:
- Code generation of test method stubs;
- Running tests within the IDE;
- Incorporation of test data loaded from a database;
- Code coverage analysis once the tests have run.
Especially the integration within the IDE is very useful. You don't need to switch from application to perform unit tests.
_thumb.gif)
_thumb.gif)
The unit test tool of VSTS can also be used from the command-line. The name of the tool is MSTest.exe. MSTest does require VSTS to be installed to work. More information about unit testing from the command-line can be found here.
Please note that unit testing in VSTS has another goal than NUnit. NUnit is dedicated to unit testing, unit testing in VSTS was born from a desire to integrate into the process of creating software.
There is a good article about unit testing available on the Microsoft site: A Unit Testing Walkthrough with Visual Studio Team Test
Reporting/Results
VSTS is able to report the results of a test run to a Team Foundation Server. TFS is the heart of the Team System and runs on a dedicated server. Components of TFS include Windows Server, SQL Server, Analysis Services, Reporting Services and SharePoint. The Visual Studio IDE is integrated with TFS. Results can be directly published to a centralized location.
If your organization doesn't have TFS, or doesn't want to use TFS, there are other ways for you to get insight in the results. A VSTS/MSTest test run stores the results in a local TRX-file. This TRX-file with XML-format can also be transformed into a HTML-document. This isn't supported out of the box, but there are free tools on the net, like trx2html and the VSTS TestRun Report Viewer.
Compatibility between NUnit and VSTS Unit Testing
The UTF's NUnit and VSTS are not compatible with each other, but their approach is very similar. It is possible to convert a NUnit project to a VSTS test project.
Follow these steps in order to do so:
- Add a reference to Microsoft.VisualStudio.QualityTools.UnitTesting.Framework;
- Change the using NUnit.Framework declaratives to using Microsoft.VisualStudio.QualityTools.UnitTesting.Framework;
- Search and replace the following:
[TestFixture] => [TestClass]
[TestFixtureSetUp] => [ClassInitialize]
[TestFixtureTearDown] => [ClassCleanup]
[SetUp] => [TestInitialize]
[TearDown] => [TestCleanup]
[Test] => [TestMethod]
This can be done with a using alias declarative as well.
- Open the *.*proj file using Notepad equivalent and add the following to the element
{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC} to the node that contains the assembly name.
The modification to the ProjectTypeGuids sets the project to be a VSTS Test Project. VSTS uses the ProjectTypeGuids to identity what type of project it is loading. Note: this change is not supported by MS. Another approach is to create a test project and add the files into the test project. (source)
TestDriven.NET
The NUnit frameworks lacks in support to integrate with Visual Studio. This is where TestDriven.NET gets in. TestDriven.NET makes it easy to run unit tests with a single click, anywhere in your Visual Studio solutions. TestDriven.NET is a Visual Studio add-in for integrating existing unit testing frameworks into the IDE. It works with NUnit, VSTS unit testing and many others.
Although unit testing of VSTS is integrated in Visual Studio, it has its limitations. People complain about the high number of UI steps required to perform an action. Because of that, professional unit testers prefer TestDriven.NET.
Cool features of TestDriven.NET are:
- Ability to run your test on a single method, within a class, or within a namespace;
- Jump into the debugger by defining breakpoints within your test fixtures;
- Convenience of not having to switch to an external program to run my unit tests (nunit gui, etc.);
- Failing tests show up in your Task List and are just a double-click away
A complaint of TestDriven.NET is that some developers prefer to use a separate application because they believe an add-in slows down their IDE.
TestDriven.NET is not a free product. The current price is $ 95 (€ 65, june '07). It is not recommended to use TestDriven.NET in conjunction with Visual Studio Express.
Conclusion
Software needs to be tested in one way or another. The sooner a bug is detected, the better. With TDD, testing is part of the overall development process. And by letting the developer writing unit tests, he or she gains more knowledge about the functional requirements.
NUit and VSTS are both excellent frameworks for unit testing. NUnit is free and saves licensing costs. I recommend to use TestDriven.NET because the integration saves time and effort.