Story Maps - A Testing Tool After All

The following was recently published as a sidebar in Lisa Crispin and Janet Gregory's new book More Agile Testing. The book is full of great advice from the authors as well as other contributors. I encourage you to take a look.

So you’re an agile tester and wonder why you should care about story maps. You may already be convinced that modelling your backlog in two dimensions is useful for helping the whole team visualize the big picture. However, story maps are also a valuable testing tool, providing two additional testing avenues. In the first case, the map itself offers the ability to test the validity of a solution. In the second, a story map improves a team’s ability to identify story slices and then test them.

User Story Mapping Series

* How to Create a User Story Map
* How to Prioritize a User Story Map
* Tips for Facilitating a User Story Mapping Session
* Don't Etch your User Story Map in Stone
* User Story Mapping Tool Review – SmartViewApp
* Story Maps - A Testing Tool After All

Testing What to Build

User story maps are a representation: they provide a means to visualize a system that might be built and are useful for testing the validity of that system before investing significant time and money. A story shared at a recent Agile Winnipeg event demonstrated this principle well. The company involved used story mapping to test an idea before building any software. The team had a project idea that they thought would serve their client well. After quickly building a story map around that idea, they presented the map to their client at the next customer conference. Although it soon became clear that the idea missed the mark, the customer was able to collaborate with the team on the spot, to adjust the map until it represented what they actually wanted built. The map itself was the tool that allowed for the idea to be tested (and then adjusted) and moved the project forward.

Testing Application Slices

As Crispin and Gregory demonstrated in their first book Agile Testing, identifying thin slices and small chunks is important for testing agile projects. Story maps help identify those slices but, perhaps more importantly, they help us understand how those thinly sliced stories might fit together to form a thin slice of the whole application. When undertaking an agile project, testers are required to make a vital shift in thinking; only test small pieces at a time. Despite this fundamental change, it is also important to ensure that the first few pieces fit together, enabling end to end testing as early as possible. The story map helps to identify and prioritize that first application slice. It may be based on a user scenario or just a string of stories that represent the smallest stories that allow left to right movement on the map.

Visualizing testing slices in your map

As that the team identifies that first slice, utilizing excellent testing skills is crucial. By looking at the map, you can identify areas that will be difficult to test, areas where the test variations are still relatively unknown, or areas that represent higher risk. This activity can help identify stories that should be included in the first application slice.

When coding and testing begins, personas and user scenarios that were created can be revisited, helping to flesh out the map and application slices. Testing with a persona in mind helps ensure that the targeted customer will be satisfied with the solution. It may not be possible or wise to test if the application works well for everyone but testing should evaluate whether the targeted personas can use the application easily, and that the new functionality fits into, or adds to their current processes without getting in the way.

Story Maps—A Testing Tool After All

At first glance, the story map doesn’t appear to be an obvious asset for testing, but upon closer inspection, it proves its value in any testing toolbox. The map itself is a reliable way to test that the right system is being built before any code is written. The map also provides a visual aid for testing in horizontal application slices, allowing for early confirmation that a project is on the right track.

Subscribe to Winnipeg Agilist by Email

Build Quality In: Applicable in Football too

One of the lean software principles is "Build Quality In". In short, we try to build it right the first time in order to reduce or eliminate re-work and defects. I turns out that this same rule is applicable when training a running back not to fumble.

Here is a story in Yahoo about the Patriot's running back Green-Elllis who at the time of the article had an incredible streak - no fumbles in four seasons. His college coach said:

"Far too often, he has seen players fumble and then heard the coach afterward demand that his players protect the ball by not carrying it carelessly into traffic. This always struck him as backward. Why talk about ball security after a fumble? Why not before? What are the preventative measures? You’ve got to put running backs in a position where they can eliminate fumbles." 

To put this statement in the context of software development:

"Far too often, he has seen the number of failed tests rise and then heard the project manager and testers afterward demand that the software developers write better quality code. This always struck him as backward. Why talk about tests after development? Why not before? What are the preventative measures? You've got to put software developers in a position where they can eliminate defects."

Test first my friends, test first.

Is agile suitable for all projects?

At some point in your agile journey you begin to ask and/or hear this question: "Is agile suitable for all projects?"

Here are some of the responses I've heard from both those who are still learning about agile and those who have a lot of agile experience:

• Agile isn't necessary when you have a known problem and a known solution.
• Agile doesn't work in regulated environments.
• Agile doesn't work in government.
• Agile can't work in a large enterprise.
• etc.

In my opinion and experience all of these answers are borne out of legitimate concerns but none of these responses are valid. Let me explain why.

One of the significant differences between traditional projects and agile projects are the short feedback loops. These short feedback loops allow teams to validate assumptions sooner, identify and deal with risks and issues sooner, find and correct defects sooner, and deliver a solution sooner. Agile will be implemented differently based on the project types identified in the responses above, but I believe that all projects will benefit from agile's short feedback loops. To say otherwise assumes that we always get it right the first time - even in the uncommon situation where the solution is 100% known, we get everything right the first time about 0% of the time. Agile helps us discover that sooner so that we can correct our mistakes.

Additionally, any project can benefit from increased trust amongst team members, increased trust between teams and customers, increased trust between teams and management, and increased trust of project status.

In my opinion and experience, the only question you need to ask has nothing to do with the type of project, the type of problem, or the type of solution. Instead, here is the key question:

Does the team want to use agile and have the support to do it?

If the answer is yes, then it can work. If the answer is no, it is not likely to work.

P.S. Here are some examples of agile being used in projects and organizations where "agile won't work":

Agile in a regulated environment.
Abbot Labs experience report - Development of a nucleic acid purification platform and companion real-time analyzer
Agile in a Regulated Environment - Linkedin Group

Agile in government.
Public sector case study - Social Security domain, software to support change in legislation.
Manitoba Parks Reservation Service - Agile case study as told by Terry Bunio.

Agile in a large enterprise
The agile experience at John Deere's Intelligent Solutions Group.- presentation by Chad Holdorf at Much Ado About Agile VI - Vancouver 2011.
Plus an article talking about the John Deere agile experience. "I figure if John Deere can test working software every two weeks on a tractor in a field, then Agile will work anywhere."

Agile with a known solution and a known problem
I recently completed a project with a known solution and a known problem - converting a VB6 application to dotnet "as is". We used agile and the project was very successful.

Agile outside of software
Wikispeed - Joe Justice and his volunteer team designed a 100+ MPG car using agile techniques."We can swap out an engine in the time it takes to change tires".

Want to receive future blog posts in your inbox? Enter your email address here.

An alternative to bug tracking tools

One of my pet peeves is working in and with bug tracking tools. I am well aware of some of the arguments for the importance of these tools and I am not trying to address those here. Instead, I'll show you an example of an alternative that I have found useful.

First, using an approach like Specification by Example can reduce the need for bug tracking tools because communication goes up and defect counts go down. But even using this technique, defects still occasionally occur. Here is an example of how to use Specification by Example not only for 'requirements', but also for defects.

In June of this year I spoke at the Prairie Developer's Conference in Regina, Saskatchewan. Some of the speakers and volunteers were involved in creating the website, services, and mobile application for that conference. Since I was doing a Specification By Example talk I decided to use the conference web services to illustrate how easy it is to create your first automated test against a web service using FitNesse. As I was working with the services I found a small defect. Instead of writing up a defect in a bug tracker with the steps to re-produce it, I wrote a test in FitNesse to confirm the defect:

This example calls a service that returns a list of sessions and does a few basic C# calculations on that list. It counts the number of sessions (allSessions.Count) in the conference and FitNesse maps that to the NumberOfSessions variable above. It then counts the number of unique abstracts (allSessions.Abstracts.Distinct.Count) and FitNesse maps that to the Number of Unique Abstracts. FitNesse then compares the numbers and displays the results. In this case, 63 does not match 62 so it displays an error with the expected and actual results as above.

Once the test confirmed the defect, I simply communicated the failing test to the developers. When the developers reviewed the test they could clearly see what the problem was. No back and forth was required to understand the issue or to confirm the steps required to reproduce it. No one had to set the status of the defect to "working", "fixed", "duplicate", "resolved", "more information required", or anything else. One of the developers fixed the issue and even added an additional service that we could call to address the root cause - Are Session Abstracts Unique? I added the new test, ran all my tests again and was pleased to see them all go green.

This process improved communication between tester and developer, ensured that the defect would always be tested and re-tested for, and kept us from spending unnecessary time in a bug tracking tool.

Agile Testing - a response to The Golden Rules of Testing

Today someone sent me a link to a Software Test Professionals (STP) article on the Golden Rules of Testing as applied to an agile project. I'm pleased that the testing community is embracing agile more and trying to figure out how to fit in. However, I was troubled by some of the statements I read. It appears I'm the "that's not how we do it in agile" guy who has some objections to his views. Commenting on the article directly required giving my name, address, occupation etc which I was unwilling to do so I'm posting my comments here. Interestingly, the author wrote a post on his personal blog “Am I an agile tester?” that is much closer to the views that I hold.

Ray's words are in normal text below. My replies are italicized and in blue.
 
***************************

Switching from Waterfall to Agile is known to directly impact testers.

Yes. But not just testers - everyone.

It's true; the change can be difficult for some of us. However, fear not, some things never change regardless of the development approach.

Agreed.

I've put together what I think are the golden rules of testing that still apply. So when someone says "that's not how we do it in Agile" (and believe me - they will) don't take none of it and stick with the basics. 

<spidey senses tingling>Depending on how you interpret this statement, this is either ok, or a recipe for failure.</spidey senses tingling>

Read these simple golden rules for software testing based on my own experiences.

It’s all about finding the bug as early as possible:

Close. It is actually about preventing the defect from being found rather than finding it as early as possible. Finding it earlier is better, preventing it is best. For more on this topic, check out this fantastic go-to article on agile QA: http://bit.ly/aOfJM5

• Start the software testing process by analyzing requirements long before development. I object to the word “long” here. It implies that we do big requirements up front. It also more than implies a process smell - the long gap between anaylsis and implementation. Rather, let’s take a look at a story together as a team right before development begins on that story to analyze the requirements and create our tests before we start coding. Then, repeat for the next story.

Make sure you have these 3 software testing levels:

• Integration testing (performed by IT) performed by the team.
• System testing (performed by professional testers) performed by the team.
• Acceptance testing (performed by business users) performed by the team.

Don’t expect too much of automated testing:

• First let me state this: Automated testing can be extremely useful and can be a real time saver. But it can also turn out to be a very expensive and an invalid solution. I tried to find more some information from Ray on what he means by automated testing but couldn’t find any additional info despite the fact that he has written a few blog posts about automated testing. This statement is usually delivered by someone who has attempted and struggled with automated UI testing. Automated UI testing can be more difficult and more expensive, but I'm not sure how it is an "invalid solution". However, automated service testing is comparably simple, not expensive, not invalid, and a consistent time saver. Automated UI testing can still be valuable, but the ratio of service to UI tests should be heavily weighted toward service testing IMO.

Deal with resistance:

• If you like to be instantly popular, don’t become a software tester! You’ll find out that you are going to meet a great deal of resistance. It is very likely that you will end up being the sole defender of quality at a certain point. Other participants in the project will be tempted to go for the deadline, whatever the quality of the application is. This is one of the reasons the agile testing community preaches a whole team approach to quality. Being the sole defender of anything on a project is a problem. We want our teams to own the budget and schedule, not just the PM. We want our teams to own quality, not just the tester, etc.

FitNesse and today's date with .net

I have an acceptance test that says I need to validate the age of majority in each of the different states and provinces.  Here is a simple example:

Given Mary who is born January 5, 1995 and lives in Manitoba

When she asks if she is the age of majority

Then return no

The test above is faily simple and I could write it like this in the wiki as a Column Fixture (using the fitSharp.dll to test C# code)

!|Check Age of Majority|

|Province State|Birth Date|Am I Underage?|

|MB            |5-Jan-1995|Yes           |

The problem of course is that this test will start failing on January 5, 2013 when Mary turns 18.  Also, it does not perform the boundary testing that I would like it to do in order to test someone who is 18 today vs. someone who will turn 18 tomorrow.  In order to improve this test, I investigated some other date functions in FitNesse and a plugin by James Carr that allowed you to add days to the current date.  These work ok for smaller calculations like "Given document ABC, When it is 30 days old, Then archive it".  However, this would be a little more cumbersome for birth dates when adding 18 years (esp. with leap year calculations) and the !today function in FitNesse does not work in ColumnFixture wiki tables.  So, I found a simple way to meet my requirement.

First, I wrote a class in C# that accepts two parameters to Add or Subtract Years and Days to the current date.  The class uses C#'s simple DateTime addition to add or subtract the years/days from today and returns the result.  You could easily extend this to add months or add other functionality required in your tests:

namespace FitNesseTutorial.Tests

{

      public class GetDateBasedOnToday : ColumnFixture

      {

        public int AddYears;

        public int AddDays;

        public DateTime ResultingDate()

            {

            return DateTime.Today.AddYears(AddYears).AddDays(AddDays);

            }

      }

}

Then in FitNesse at the top of my script for this story I call GetDateBasedOnToday and store the resulting values in FitNesse variables.  Finally,  I use the variable names through my script to reference the underage and of age birth dates.  Here is an example:

The FitNesse script:

''Get underage and of age dates for 18 and 19 year olds''

!|Get Date Based On Today          |

|Add Years|Add Days|Resulting Date?|

|-18      |1       |>>UNDERAGE_18  |

|-19      |1       |>>UNDERAGE_19  |

|-18      |0       |>>OFAGE_18     |

|-19      |0       |>>OFAGE_19     |

!|Check Age of Majority|

|Province State|Birth Date   |Am I Underage?|

|MB            |<<OFAGE_18   |Yes           |

|MB            |<<UNDERAGE_18|No            |

|BC            |<<OFAGE_19   |Yes           |

|BC            |<<UNDERAGE_19|No            |

In FitNesse, the final result including the acceptance criteria above looks like this:

 (Note: The example above should probably be written as a unit test because it is fairly straightforward, but it simply illustrates how to use the date logic that I'm using as part of larger acceptance tests.)

The last shall be first

At what point in the project do you start your testing effort?  In many project plans that I have seen, QA and/or UAT is added to the end of the project.  The development team hands over the code to the testing team who then writes and executes the scripts.  The bugs are passed to the developer and the test & fix cycle begins.  What fun.

Here is an alternative to the test & fix cycle:

1. Write test scripts before development starts on any particular feature. Test scripts help confirm the requirements with the client and allow developers to have a full understanding of how the code must work.  The test scripts function as executable requirements and answer "How will I know when I'm done".

2. Minimize the time between when a features has been developed and when it is tested. This allows you to find defects or requirement misunderstanding early so that developers do not re-build the same defects or misunderstandings into future features. This helps increase the quality of the system while decreasing the time spent in the test-fix-test-fix cycle.  Complete testing of any feature should occur in the same iteration that the code is written and should be part of your 'done' criteria.
 
Both of these two simple steps are based on the Lean Principles of Eliminate Waste and Build Quality In.  For more information, check out this article from NetObjectives.

FitNesse gets the gold!

I've been using FitNesse for just over 3 weeks and I am pleased with the value it is adding to our project even though I'm only using the basic features at this time. As a former developer I'm finding it fun to use because I have to write a little bit of code in order to create an interface between FitNesse and each service that I'm testing. Most of our testing on the project was going well and we were avoiding major errors - until last Thursday...

A change in the code from a newly completed work item resulted in 81 of our 191 tests failing. Imagine how long it would take to re-run all 191 tests manually to find out that 81 had failed. Imagine how long it would take to re-test all 191 tests manually to make sure they were fixed. As you can see from the image below, it took us 39 minutes to find, fix and re-test all 191 tests. Thanks FitNesse.

** Update 6/28:  This project had the highest quality metric that I've ever been a part of in terms of defects / month / developer.  FitNesse was a big part of that success.