Effort estimation is a necessary evil and will never go away…

The reason for any tester to get sucked into test estimation is a simple one. Your manager at some point in your career, will ask you how much time it will take to test the next release, project, product. You’ll probably do what I did: check what is – planned to be – in it, compare it to the previous thing you tested and come up with a number. A bit less than before if you’re more comfortable with the way of working and the system under test. A bit more if you’re feeling unsure about the quality. As a result, your manager won’t like it and you’ll probably get less than you’re asking for. Maybe, because it is an experienced manager that tricks you into believing this release is less complex or critical. Maybe because you’re not able to substantiate your estimate. Call it character, call it ambition, when this happened to me, I was not happy so I started my “quest” for a more reliable effort estimate. This quest is still going on…

I do realize that there is a discussion “against” test effort estimation in the traditional form, maybe even in “any” form. The arguments are valid as many start with “I don’t know…” (see one of my previous posts). On the dots you can put any item: my team, the product, the development team, the business, the business case, the target, the aspects to tests, where the defects are, what the deadlines are, … This is a non-limitative list and I know you can come up with many more items.

The awful truth is that we have to come up with an estimate anyway. Either we need to tell our manager what time we need to test or we need to show him what we can do within the available time. Including a high-level description of how, when, what, …. Whether we want to or not, because if we don’t do it, he will. Since I believe that craftsman are able to better estimate their work and to discuss, argument and agree about it, I think testers should do it themselves. It’s better than being told and sigh…

Especially for a tester, an estimate is an approximation, nothing more or less. We know that all kinds of things will happen that will disrupt progress, require additional time to investigate or explore, lead to a major change in the scope of the project and/or the product. I know we don’t control most of the factors that influence the required effort. But that doesn’t mean we can’t take them into account! In my opinion, we need to “collect” these factors and show the impact on the test estimate.

And by the way, against all current discussions and opinions, I do believe there is a “best practice” in estimation: you need to discuss, monitor, update, change and adapt your effort estimate continuously.

Advertisement

Why improvement usually fails…

If you want to improve you have to invest time and money. What usually happens, is that you look for what everyone at that moment considers to be the right thing to do. So you tend to adopt the test approach, including tools and templates, that is “hot”.

Let’s say, you decide to implement risk based testing. You learn how to translate business risks into an appropriate test approach, including techniques, and create a beautiful plan and strategy.  And then? Your (pilot) project starts and fails…
Why, you wonder. You applied everything you learned, went to conferences, listened to the gurus, read the books, hired expensive consultants, …

As a test consultant I had the privilege to work in different organisations and projects and discovered a trend in “failures” and “successes”.  It boils down to one common denominator: did you or didn’t you involve the people who do the “real thing” in the test process? Yes, the ones that do the actual work a.k.a. the testers.  Somehow, they do not get involved in the early stages of test improvement.

We tend to start with test management, make the test organisation more professional, develop a test strategy, follow a lifecycle (plan, prepare and execute), use the right tools, select adequate test design techniques … According to the book! Don’t we make the same mistake at the start of a project? Forget to involve those same testers in the early stages?  When we define a test strategy and write a test plan, we talk to the business, the developers, all of our stakeholders, …

Don’t forget: the testers are the most important part of your test process and improvements, they have to learn to apply the new methods, tools and templates. In fact, they are the ones that have to change the most!

I found out the hard way that a few days training – even from the guys that wrote the book – is not enough for that senior business tester that has been there for over 10 years and knows the system inside out… And what about the “fire fighters” that have become heroes because they find, fix and retest defects quickly? They will not be convinced, they won ’t even attend!

So it’s all about people and how you involve them.  Take into account the capabilities and potential of your test engineers. Listen to what they have to say.  On the short term, you might end up at a maturity level you do not like. Or your approach is not exactly what you want it to be. But you know what, if you help them manage the transition, they will reward you with the highest level possible in the shortest possible time.  You will become truly successful.

It’s not your test management approach that decides how successful you are, your test engineers make it or break it. TMap? ISTQB? RST? As long as your people are able and capable to apply it in their day to day work.

The awful truth about estimation revisited

Last Friday I submitted my presentation The Awful Truth About Test Estimation – Have I been wrong all along? to EuroSTAR. Turns out there is a lot to say, discuss, consider, think, explore,… about estimation and that 45 minutes, including loads of interaction, is pretty short.

Awful truth?

According to wikipedia, “Estimation is the process of finding an estimate, or approximation, which is a value that is usable for some purpose even if input data may be incomplete, uncertain, or unstable. The value is nonetheless usable because it is derived from the best information available.” To make life a little easier, wikipedia explains that “An approximation is anything that is similar but not exactly equal to something else. The term can be applied to various properties (e.g. value, quantity, image, description) that are nearly but not exactly correct; similar, but not exactly the same.”

So in fact, test effort estimation is a the process of defining a number – hours, man days, lapse time, … – that is nearly but not exactly correct AND is based on incomplete, uncertain and unstable information.
For me that is awful yet true.

Should we stop doing it?

Unfortunately, for most of us that is not an option. We need to provide some insight into the necessary effort for testing, because (project)management needs it to establish a budget request, (line)management wants insight into the resources needed for a project, et cetera.

…if you don’t know what to do

…if you don’t know how to do it

…if you don’t know who will do it

…if you don’t know where to do it

…if you don’t know with what to do it

…if you don’t know when to do it

…if you don’t know when to stop

…if you don’t know what to do first, last, in between

We should however always treat it as an estimate: nearly but not exactly correct, based on incomplete, uncertain and unstable information. And share the uncertainties, instabilities and incompleteness!

You should never stop doing it!

One of the most important lessons to learn about estimation is that you’re never done with it. Each day one of the uncertainties can become a certainty, decisions are made that remove instabilities, etcetera. As a tester you should continuously look at your estimates and adjust them to the new situation.

Biggest challenge: you need to convince (project)management that an estimate is an incomplete, uncertain and unstable number. And that fixed deadlines sometimes just mean that not all of the testing work is done… You could compromise but my suggestion is to do the best you can within the given restraints. And be open and honest about it. One of the best results I got was in a project where the project manager decided to stop detailing the requirements and start coding immediately to save time and effort on the development side (approximately 200 man hours). I did not complain or blame. I just told him that I expected that as a consequence the test effort would go up by at least 300 man hours due to the fact that we needed to find out about the requirements ourselves now and we expected more defects… After a prolonged discussion, he withdrew his decision.

See you in Gothenburg?

If you are coming to Gothenburg and are interested in attending my talk, please register here

You can also do so by visiting this page on the EuroSTAR website: “W9 – The Awful Truth About Test Estimation“

Triggers for Improvement? Opposites attract!

The result of the poll I held amongst fellow testers and on this blog, is in line with my expectations: most test improvement initiatives start when quality issues occur. So in that sense, management treats testing like any other discipline: why invest in change or improvements when there is no (obvious) reason for it.

In my experience, motivated testers continuously look for ways to change and improve themselves and the way they work. And they are not afraid to look for help. They’re active on the internet, participate in Special Interest Groups, and meet with each other quite frequently. That’s why most improvement initiatives start at the operational level: the testers want to improve.

The first hurdle is to convince management that it is worthwhile to invest in change and improvement, a.k.a. business case. Unfortunately, currently the focus is on numbers: what are the benefits (in Euro’s please)? what are the costs? And by the way, we want Return on Investment within 6 months. Again, management treats testing like anything else: most companies  only start a project if it is beneficial within 6 months.

Testers focus on qualitative benefits:
– the right test at the right moment;
– deliver what was promised;
– the right coverage at the right spot.

Management on quantitative benefits:
– reduced time-to-market;
– deliver what is necessary;
– increased productivity.

Funny thing is that also in this situation opposites attract: management will never get what they want without achieving what testers strive for; testers will never be able to reach their goals without meeting management needs. I’ll even go one step further: by helping management to achieve its goals, testers will achieve their own goals as well. Without the right test with the right coverage at the right moment at the right spot, reduced time-to-market and increased productivity is impossible. The only thing we can and have to discuss is scope: deliver what is promised or deliver what is necessary?

The awful truth about estimation – EuroSTAR 2013

I’m invited by EuroSTAR this year to give a challenging talk about estimation.

The Awful Truth About Test Estimation – Have I been wrong all along?

Abstract
Throughout my career, I tried to make effort estimation reliable.
With colleagues, I created a formal approach linked to function point analysis. That didn’t catch on. Was it too difficult? I worked on an informal approach based on questions about size, strategy, resources, and etcetera. Still only small successes. Does “Garbage in, garbage out” apply? Have I been asking the wrong questions?

Please join me in my quest for the right questions to ask. Help me to deal with this once and for all and solve this issue through our collective experience and do what testers are good at: asking questions in search for an answer.

Video message
A new phenomenon is that you promote your talk in a video message. For those of you who do not follow EuroSTAR or visit the website regularly, please check my message here:

 

See you there!

If you are coming to Gotenborg and are interested in attending my talk, please register here

You can also do so by visiting this page on the EuroSTAR website: “W9 – The Awful Truth About Test Estimation”

Hope to see you in Gotenborg!

What would trigger you to perform a Test Process Improvement assessment?

Triggered by a discussion with some colleagues, I decided to add a poll to this blog.

 

 

Hope you share your opinion with the rest of us!

In-house assessments have major benefits!

Digging through my personal archive, I found “Is There an Assessment in the House? Diagnosing Test Process Ailments in House”, an article Better Software published in December 2006. Re-reading the article, I decided it contained tips about performing an in-house assessment that are still very true.

Performing an assessment in house is a less expensive initiative than hiring outside consultants. Maybe not as inexpensive as the Low Budget approach I described in my previous post, but still. And it has other substantial benefits. An in-house assessment team knows the organization, its people, its processes, and its habits in substantially more detail than an “outsider” could learn during the relatively short assessment period. And the process improvement suggestions might be more realistic, better tuned to the situation, and more achievable.

Here are some – slightly updated – tips for leading a successful in house assessment:

• Define why you are performing the assessment and what you hope to accomplish.
• Choose a test process assessment model that suits you and your organization. Use it to make sure your assessment is objective and your conclusions are legitimate.
• Create questionnaires to ensure that you cover all major topics when talking to people.
• Familiarize yourself with all parts of the organization and how it actually tests software. Look for discrepancies between process information – the way they claim to do things – and project information – what they really do.
• Make sure you choose a representative sample of people to talk to. It’s easier to sit down with friends, coworkers, and people nearby, but they may not be the best sources of information.
• Don’t just talk to testers. Testing is part of software development, decisions made by any discipline significantly impact testers. So be sure to include people from these areas.
• If you organize a group interview, make sure they are peers. Otherwise, you’re bound to get “right” answer – what the manager wants said – instead of the truth.
• Arrange to meet in their offices. People feel more comfortable on their home turf and may be more open to your questions.
• Be an effective listener. You may be more of an expert in an area than the person you are talking to, but don’t jump to conclusions. Don’t fill in the blanks or use your words rather than theirs. It’s crucial not to interpret – just listen, learn, and record.
• Ask what problems exist in the software testing process and how they think those problems could be solved. You will discover a wealth of information.
• Keep confidences. You may hear some interesting things during your interviews. Don’t attach names to negative comments. Remember, you are trying to gather information, not set people up for punishment later.
• Take detailed notes even if you’re getting the answers you expected. You won’t be able to remember all the details later, and your notes will be vital.
• Don’t become enamored with certain people. They may be handsome or beautiful or powerful or famous or charismatic, but that doesn’t mean they know what they’re talking about.
• Work with a partner. While you are asking the questions, he records the answers. After a while, switch roles. I’ve always found my partner notices things I miss and asks questions I tend to forget.
• Stop regularly to evaluate what you’ve learned and what is still unknown. Use that knowledge to guide your next interviews.
• Present your findings to all who contributed to the assessment.
• Tell the truth. As testers, the only lasting power we have is our integrity. Don’t do anything to damage yours.

If you ensure that you’re not biased and you make everyone share their weak spots, you’ll achieve wonderful things through an in-house assessment.