Monday, February 9, 2009

Traceability

At the beginning of February, there was a discussion on the agile testing yahoo group on traceability. The original question was “How do you know what piece of code affects what user stories and what documentation and what tests?” since agile values working code over documentation. In our book, Lisa and I have less than a page on traceability and the first paragraph recognizes that traceability matrices were pretty much the only way to know what had changed and when in a traditional project. In agile those restrictions aren’t there. We have many other methods that provide an automatic trace from story to story test to unit test to code. These tests become part of the regression suite so that every time they are run, the story is tested.


If we think about that, why do we need to link the code changes to user stories? User stories, requirements, features evolve over time. Each story or feature builds on previous ones so the likelihood of a test being in sync with a user story is low. The need no longer exists. If you are using a continuous integration tool, you can see what code is being changed on every check-in if you really want to. It is easy enough to add in defect numbers in check in comments so that even defects could be traced if needed.

In my experience I have found the request for traceability often comes because history has shown that side effects occur because the developers haven't told the testers what exactly was changed and the final testing often showed up unanticipated problems. In agile, each story is tested as it is developed, with unit tests and acceptance tests and any other tests that it required (including performance, etc). If there are issues you find them right away with your suite of automation. I used to be a fan of traceability because it was the only way to understand what was changed. I have found I don't need it anymore.


Sometimes auditors will request traceability for regulated industries. If they don’t understand how version control works, or the power of automated tests, you may want to show them how you have automated traceability, and explain the cost of maintaining any type of manual traceability.

A related discussion about 7months ago on "Traceability Matrix in an Agile Project" and was summarized by Vikas Hazrati at http://www.infoq.com/news/2008/06/agile-traceability-matrix. One of the main points I got from that discussion is the difference between traceability and a traceability matrix. A matrix is just one implementation of the need for traceability. I also liked Michael Bolton’s reference “Transparency over Traceability" in his 2007 article entitled "Lean Traceability: a smattering of strategies and solutions" at http://www.cmcrossroads.com/content/view/9089/264/.

Before you blindly start a traceability matrix, ask the reason for the request and question the cost of maintenance against the benefit. In regulated environments look for the simplest thing that meets the need to provide evidence that the software works as specified. It may not need a full traceability matrix. Use your source control to your advantage. Use naming conventions on your tests that help identify what is being tested, or perhaps even integrate your requirments into your automation framework. Think simple to keep maintenance as low as possible.

2 comments:

Brad Appleton said...

Hi Janet!
You cite Michael Bolton as the author of "Lean Traceability: a smattering of strategies and solutions", but the author of that article is me (Brad Appleton), along with Rob Cowham, and Steve Berczuk.

Janet Gregory said...

Brad, Thank you for the correction. I must have been reading something of Michael's where he made a reference to your article, but I'm not sure since it was a while ago. My apologies for the mistake.
Janet