« Documenting Requirements - A Sample Requirements Specification
Bug’s life – Test Management »

Managing Testing Resources: Five Suggestions for the Project Manager - Lay Out Criteria for Important Milestones, and Stick to Them

April 14, 2008 | Author: PM Hut | Filed under: Quality Management, Time Management

Managing Testing Resources: Five Suggestions for the Project Manager - Lay Out Criteria for Important Milestones, and Stick to Them (#4 in the series Managing Testing Resources: Five Suggestions for the Project Manager)
By Cem Kaner and Johanna Rothman

An important tool for managing the project, and your relationship with the test group, is a milestone criteria chart. This lays out criteria for moving the project through different development milestones. How complete is the project supposed to be at a certain point? How should we measure that completeness? These are project-wide issues, and though a good test manager or test lead will gladly help you develop them, these are ultimately your issues to clarify.

Brian Lawrence and Bob Johnson present an excellent set of milestone definitions and checklists at their website, http://www.coyotevalley.com. We don’t agree with all of their definitions, but that’s not the point of their lists. The lists are intended as a starting point and a model. You can customize them for your project. For an alternative set of milestone-specific ideas, see Chapter 12 in Kaner, Falk & Nguyen’s Testing Computer Software. You can probably find additional milestone lists and definitions–none will be perfect for your project, but the mix might help you settle on a specific set of criteria that work for you.

One of us (Johanna) helped a client develop a set of project criteria and is in a position to publish some of the details. We can’t reveal the true name of the company or the product. Here, we call it the “Messenger” product. Johanna was retained mid-project to manage Messenger and get it released. Johanna wrote in more detail about “Messenger” in “Achieving Repeatable Processes” in the June 1998 Software Development.

Messenger had been in the market already. This upgrade was of particular interest to specific, important customers. We had negotiated specific, firm delivery dates for interim versions of the product. The product had to go beta in April and it had to ship in July. These customers’ acceptance of the final version of the new product would depend on our meeting of these dates with the agreed features at the agreed level of reliability.

As part of our planning, we developed the criteria listed below. (Note: these are fairly generic criteria. There were some very specific other ones, too, but we’re not in a legal position that lets us tell you about those. In your project, you will include items that are more specific than these.)

Messenger’s Milestone Criteria

System Test Entry Criteria

  • All code must compile and build for all platforms.
  • All developer tests must pass.
  • The code is frozen.
  • All features except tokens are complete.

Beta Criteria

  • All features complete in code, with developer tests.
  • All code must compile and build for all platforms.
  • All developer tests must pass.
  • All available tests for beta customer must run and pass.
  • All current bugs are entered into the bug-tracking system.
  • First draft documentation is available and shippable to customers.
  • The code is frozen.
  • Technical support training plan is in place, and the people are in place.
  • There are less than 36 open high priority bugs.
  • Ship Beta before April 30.

Release Criteria

  • All code must compile and build for all platforms.
  • Zero high priority bugs.
  • Document workarounds for all open bugs in release notes.
  • All planned SQA tests run, > 90 percent pass.
  • Number of open bugs decreasing for last three weeks.
  • All Beta site reports obtained and evaluation documented.
  • Reliability criterion: Simulate one week of usage …
  • Final documentation ready to print.
  • A working demo runs on previous release.
  • (Performance criterion)
  • At least two Beta site references.
  • Ship release before July 15.

Note that the testing group is vitally interested in these criteria, but the ultimate decision-maker for them was me (Johanna) because I was the project manager. The testing group helped me understand whether the project was meeting the criteria, they helped me understand how to word the criteria and what else should be included. They played an important role. But the list was mine.

Looking at Messenger’s product release criteria, we see a focus on getting to the ship date with a certain set of features, but not a particularly low defect rate. More precisely, several classes of defects didn’t matter (as far as the executives and the market researchers were concerned). The overall project reliability had to be understood and reasonable. Specific aspects of the product had to be immaculate. Errors in those areas became high priority quickly. Additionally, anything that genuinely interfered with early use of the product became high priority. Getting ourselves clear on these expectations was essential for successful testing. How could a test group provide support for these very specific objectives if they didn’t understand them?

Messenger’s system test entry criteria were chosen as the minimum set that would allow system test to start, even if not all the features were complete. Allowing testers to start early meant that they could find problems early. Every study that we (Johanna and Cem) have seen says that the sooner that you find and fix bugs, the cheaper. We wanted every opportunity to make our dates, and that meant getting good information as soon as possible.

The project was a success. We released the beta version two weeks late. We might have released an earlier version, that had been a beta candidate, but it didn’t meet our beta criteria. However, we were able to explain our criteria to our customers who had insisted on beta copies, and to explain our status against those criteria pretty precisely. This level of control made them comfortable and they accepted the late release without protest. We also had disputes with the testing group, who wanted to continue testing in areas that we had made clear were not relevant to the customers’ acceptance criteria. The clarity of the written, approved beta and release criteria went a long way toward keeping those difficult discussions focused and within sensible time bounds. We shipped the product on time and the key customers were extremely happy with it.

Original article can be found at: http://www.jrothman.com/Papers/Managingtestingresources.html

Johanna Rothman consults, speaks, and writes on managing high-technology product development. Johanna is the author of Manage It!’Your Guide to Modern Pragmatic Project Management’. She is the coauthor of the pragmatic Behind Closed Doors, Secrets of Great Management, and author of the highly acclaimed Hiring the Best Knowledge Workers, Techies & Nerds: The Secrets and Science of Hiring Technical People. And, Johanna is a host and session leader at the Amplifying Your Effectiveness (AYE) conference (http://www.ayeconference.com). You can see Johanna’s other writings at http://www.jrothman.com.

Share this article:
  • Digg
  • del.icio.us
  • Facebook
  • Google
  • blogmarks
  • LinkedIn
  • Reddit
  • StumbleUpon
  • TwitThis
  • Yahoo! Buzz

Related Articles

This entry was posted on Monday, April 14th, 2008 and is filed under Quality Management, Time Management. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

No comments yet.

feel free to leave a comment

Comment Guidelines: Basic XHTML is allowed (a href, strong, em, code). All line breaks and paragraphs are automatically generated. Off-topic or inappropriate comments will be edited or deleted. Email addresses will never be published. Keep it PG-13 people!

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

All fields marked with " * " are required.

Project Management Categories