« Testing is Not a Service
Product management vs. Project management »

Testing Early and Often Can be a Wasted Effort

September 27, 2007 | Author: admin | Filed under: Cost Management, Project Lifecycle Phases, Project Management Best Practices

Testing Early and Often Can be a Wasted Effort
By Michael Sheeley

We have all heard the saying, “test early, test often” but in many software development efforts this can be a very costly mistake. It is imperative for software project managers to understand the full added cost each type of testing will have on the final delivery date. In many cases, it is the timing of when to write these tests that matter the most.

There are three levels of testing, unit testing, integration testing, and system testing. Integration testing and system testing are not as flexible, so I will not discuss these two efforts in this post. Unit testing does have some flexibility.

Unit testing is a required means to assure that the smallest possible elements have been tested. Unit testing is a required process in testing a system, no question. But the timing of these tests and the level of detailed required need to be considered. Writing highly detailed unit tests early in a project adds cost to the current development phase but, more importantly, adds cost to future phases as well. Maintaining these tests can be extremely time-consuming. In iterative process models, units are rewritten and redesigned over and over again. Each change to unit requires changes to the corresponding unit test. The more detailed the unit test the more time and effort required to maintain and update.

Project management needs to be aware of the added costs and implications to the project’s schedule. The nature of the project, the type of development process, and the size of the system being developed all need to be considered in deciding the unit testing level of detail and what phase or iteration these levels should be enforced.

Unit Testing Levels of Detail

First let’s clearly define each levels of testing. They are numbered and described below from the least complex level to the most complex.

  1. Algorithm Accuracy testing - Testing the accuracy of an algorithm. In object oriented languages, this is testing the return of a method based on a few basic inputs.
  2. Full statement testing - Making sure the unit tests execute every line of code
  3. Full branch testing - All decision points (i.e. if statements) are executed for all outcomes
  4. Every loop tested at 0,1 and many - Testing each loop (i.e while and for loops) when the loop is executed more than once at a time (many), only once (1), and when the condition of the loop keeps the loop from executing its block of code at all (0)
  5. Boundary value tests - Testing boundary values for boundary condition statements (i.e. if(size > 0 && size <= 10) Boundary value tests would test this statement when size equals -1, 0, 10, and 11 )
  6. Full predicate testing - Similar to level 2, Full branch testing, adding the requirement for each condition for each predicate must be tested. (i.e if(boolOne == 1 && boolTwo == 2) Full predicate testing would tests all combinations when boolOne equals 1 and does not equal 1 and boolTwo equals 2 and does not equal 2)
  7. Path testing - This level of testing implies all the previous levels but each branch, loop, boundary, and predicate test must be done independently of the others.

Each level of detail adds more benefit to the reliability, accuracy, and even the complexity of the system. There is no question about this. Determining what level of detail is required by the stakeholders needs to be done regardless of the development process chosen, but understanding the cost of each level is important in estimating cost and schedule.

Software Development Processes

We’ll now describe the basic types of development processes. Depending on the project’s process, the costs of testing differ greatly.

  • Waterfall process – a sequential effort of phases (requirements analysis, design, implementation, testing (validation), integration, and maintenance). Each phase occurs only once. Traditionally, testing occurs once the implementation phase finishes but many times unit testing occurs during implementation. Either way, these efforts only occur once.
  • Iterative process – a non-sequential effort where phases can be revisited after there initial completion. A common iterative process, Agile process, comprises of short iterations. Each iteration visits or revisits each software phase. Think of an iteration as a small waterfall. Iterative processes reduce the level of risk to a project by giving the stakeholders of the project small releases of the product at the end of each iteration. The stakeholders are able to evaluate these releases and add feedback, suggest changes, and foster new ideas for the product. Stakeholders can better evaluate whether to continue on with the project or not. If continued, the project can redesign, and recode the product using the stakeholders’ feedback. This method ensures that the project and the stakeholders are communicating towards build a better product throughout the effort. This is contrary to the waterfall processes that only allow stakeholders to evaluate working examples of the final product at the very end of the effort.

If you are using the waterfall development process, once the implementation phase is done, you will not revisit this phase. Therefore, determining the cost attributed to each level of detail is simple. If an iterative process in being used, each level of detail will increase by a multiple of the iterations in the process. In many iterative development efforts, the code written in the first and even the second iterations will greatly change once the stakeholders give new feedback and make new requests. Writing highly detailed unit tests for these iterations will most likely be a wasted effort. Proceeding iterations will also have the added burden of editing and changing these tests.

Making sure these tests are written efficiently can cut down on the complexity and the level of effort needed to maintain these tests. Tools such as the Spring Framework can help to cut down drastically on the complexity of the setup code needed for each unit test and should always be used on medium and large efforts.

Levels of details such as levels 4, 5 and 6 should be post postponed until the system and software requirements have been mainly worked out. These levels add a large amount of extra code to the test suite. Not until the stakeholders seem to settle on the majority of requirements and details of the system should these levels of detail be added to the test suite. Although these tests greatly increase the quality of the software, their cost to write and maintain surpass their benefits to the early iterations.

Level 7, path testing, is a level of detail that adds great quality to a system, but it is extremely expensive. A testable section of code’s complexity can great affect the cost to create and maintain tests at this level. Some argue that requiring engineers to write full path tests influence engineers to keep the complexity at a minimum. I agree with this point, but in the early iterations of a development effort code complexity is irrelevant. These iterations are done to prove a concept to the stakeholders and to iron out the integration effort problems and system level issues. The last iterations before a system is released should be saved for this level of detail. The added costs in adding these tests to the system will be outweighed by the time and effort required to maintain these tests.

Conclusion

Every software project is unique and it is up to the management to understand the costs involved in each decision they make. Test adds quality but also adds cost. Test wisely adds quality but can reduce cost. As a project manager, make sure you understand the complexity your decisions add to future phases and iterations of the project. More detailed testing can be postponed until later interactions where there is less future maintenance needed to update the tests.

Michael Sheeley is an entrepreneur in the software industry. His experiences range for starting businesses to developing new software as a software engineer. His current position is at BAE Systems developing and enhancing new technologies for the armed forces. Michael’s blog can be found at http://www.sheeleytech.com/.

Share this article:
  • StumbleUpon
  • Digg
  • del.icio.us
  • Technorati
  • Reddit
  • YahooMyWeb
  • blogmarks

    • This entry was posted on Thursday, September 27th, 2007 and is filed under Cost Management, Project Lifecycle Phases, Project Management Best Practices. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

    No comments yet.

    feel free to leave a comment

    Comment Guidelines: Basic XHTML is allowed (a href, strong, em, code). All line breaks and paragraphs are automatically generated. Off-topic or inappropriate comments will be edited or deleted. Email addresses will never be published. Keep it PG-13 people!

    XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>

    All fields marked with " * " are required.

    Project Management Categories