The awful truth about estimation – EuroSTAR 2013

I’m invited by EuroSTAR this year to give a challenging talk about estimation.

The Awful Truth About Test Estimation – Have I been wrong all along?

Abstract
Throughout my career, I tried to make effort estimation reliable.
With colleagues, I created a formal approach linked to function point analysis. That didn’t catch on. Was it too difficult? I worked on an informal approach based on questions about size, strategy, resources, and etcetera. Still only small successes. Does “Garbage in, garbage out” apply? Have I been asking the wrong questions?

Please join me in my quest for the right questions to ask. Help me to deal with this once and for all and solve this issue through our collective experience and do what testers are good at: asking questions in search for an answer.

Video message
A new phenomenon is that you promote your talk in a video message. For those of you who do not follow EuroSTAR or visit the website regularly, please check my message here:

 

See you there!

Register for my track at EuroSTAR

If you are coming to Gotenborg and are interested in attending my talk, please register here

You can also do so by visiting this page on the EuroSTAR website: “W9 – The Awful Truth About Test Estimation

Hope to see you in Gotenborg!

Advertisements

Assessments by external consultants have major benefits too!

Although I personally believe that doing it yourself is one of the best ways to successfully improve, in-house assessments have severe drawbacks as well.

SEPNumber one: you are part of the team that is being assessed and are therefore biased. No matter how rational and objective you try to be. During one of my assignments, I came across an effect called “Somebody Else’s Problem (SEP)”. I think this describes best what the consequences of being biased can be. Maybe we even use a so called SEP-field generator to help overlook issues within our own approach and team.

Number two: your coworkers are hesitant to share their weak spots with you because they fear you might use anything they say against them in the next project. As a consequence you will not find the root cause of recurring issues or the real issues preventing things to improve.

Number three: experienced consultants have performed assessments many times before, understand the assessment models, and are efficient in surveying organizations. Because of their independence, they can dare to ask questions about “everything you’ve always wanted to know but were afraid to ask.”

Number four: external consultants bring a wealth of experience form other organizations. They have helped other organizations improve their testing and usually have access to several good practices that might help you make that next step. I’ve also learned that as an external consultant it’s easier to point out that certain existing habits should change.

Number five: when you’re leading an improvement initiative yourself, you might overdo it and strive to become the best in test. But your companies focus is on delivering the right product and/or service to your customers. Which implies that being efficient in testing might be enough. External consultants can help you decide what level is enough based on their experience at similar organizations.

Number six: even if you’re a craftsman, well-trained, experienced and knowledgeable, it is always good to have someone else come in and discuss with you what you’re doing and how you’re doing it.

CloutNumber seven: your findings and recommendations, no matter how well reasoned and well-presented they are, do not have the “clout” that an outside assessor brings. This is a strange phenomenon that I’ve encountered quite often: an in-house group has performed an assessment and has made recommendations to management that are simply ignored. Later, when we are asked as external consultant to perform an assessment, we make in essence the same recommendations, management responds positively to our report, and the recommendations are implemented.

The hints, tips and tricks I listed in my previous blog also apply to anyone performing an assessment. One additional, crucial tip when using an external assessment team. Make sure that you do not focus on knowledge of and experience with process improvement models, but please check if the assessment team incorporates a wealth of experience in testing at any level.

In-house assessments have major benefits!

Digging through my personal archive, I found “Is There an Assessment in the House? Diagnosing Test Process Ailments in House”, an article Better Software published in December 2006. Re-reading the article, I decided it contained tips about performing an in-house assessment that are still very true.

Better Software - December 2006Performing an assessment in house is a less expensive initiative than hiring outside consultants. Maybe not as inexpensive as the Low Budget approach I described in my previous post, but still. And it has other substantial benefits. An in-house assessment team knows the organization, its people, its processes, and its habits in substantially more detail than an “outsider” could learn during the relatively short assessment period. And the process improvement suggestions might be more realistic, better tuned to the situation, and more achievable.

Here are some – slightly updated – tips for leading a successful in house assessment:

  • Define why you are performing the assessment and what you hope to accomplish.
  • Choose a test process assessment model that suits you and your organization. Use it to make sure your assessment is objective and your conclusions are legitimate.
  • Create questionnaires to ensure that you cover all major topics when talking to people.
  • Familiarize yourself with all parts of the organization and how it actually tests software. Look for discrepancies between process information – the way they claim to do things – and project information – what they really do.
  • Make sure you choose a representative sample of people to talk to. It’s easier to sit down with friends, coworkers, and people nearby, but they may not be the best sources of information.
  • Don’t just talk to testers. Testing is part of software development, decisions made by any discipline significantly impact testers. So be sure to include people from these areas.
  • If you organize a group interview, make sure they are peers. Otherwise, you’re bound to get “right” answer – what the manager wants said – instead of the truth.
  • Arrange to meet in their offices. People feel more comfortable on their home turf and may be more open to your questions.
  • Be an effective listener. You may be more of an expert in an area than the person you are talking to, but don’t jump to conclusions. Don’t fill in the blanks or use your words rather than theirs. It’s crucial not to interpret – just listen, learn, and record.
  • Ask what problems exist in the software testing process and how they think those problems could be solved. You will discover a wealth of information.
  • Keep confidences. You may hear some interesting things during your interviews. Don’t attach names to negative comments. Remember, you are trying to gather information, not set people up for punishment later.
  • Take detailed notes even if you’re getting the answers you expected. You won’t be able to remember all the details later, and your notes will be vital.
  • Don’t become enamored with certain people. They may be handsome or beautiful or powerful or famous or charismatic, but that doesn’t mean they know what they’re talking about.
  • Work with a partner. While you are asking the questions, he records the answers. After a while, switch roles. I’ve always found my partner notices things I miss and asks questions I tend to forget.
  • Stop regularly to evaluate what you’ve learned and what is still unknown. Use that knowledge to guide your next interviews.
  • Present your findings to all who contributed to the assessment.
  • Tell the truth. As testers, the only lasting power we have is our integrity. Don’t do anything to damage yours.

If you ensure that you’re not biased and you make everyone share their weak spots, you’ll achieve wonderful things through an in-house assessment.

Test Process Improvement for a Low Budget

At the start of the current economic situation (some would call it an economic crisis), we were asked to create and present a workshop on test process improvement “on a shoestring” a.k.a. for a low-budget. Companies nowadays do not have the time and the means for extensive improvement initiatives. However, they are in dire need of improvements that help them in achieving cost reduction, shortening time-to-market and achieve a higher quality level. In other words: do more with fewer resources, in a shorter period of time and with higher quality, without (almost) any investment. Challenging, yet not impossible.

Do what is necessary When you have been working in an organization for a while, several actions and activities turn into routine: you perform them without really thinking about. For the majority of actions, that’s fine because they are necessary steps you routinely perform to get where you need to be. For some actions however, you need to fight the routine and perform them consciously every time. Because sometimes they’re not necessary, over the top or counterproductive. Most of us know that, yet do not take the time to discuss and implement changes to the way of working. Because “no one challenges the Way We Work” and this is, by the way, part of human nature. We do comment on the way we work and criticize things we want to be different, but don’t always act.

Another part of human nature however is that we knowingly and unknowingly collect ideas for improvement and ways to solve recurring issues. They just don’t always take the time or have the platform to turn those ideas into actions and go further than fighting the symptoms. By just doing that: creating a platform for collecting ideas to turn them into actions and for solving the root cause of the issues, you can improve your testing.

I’ve learned that an interactive workshop to gather improvement suggestions and select the most suitable measures with all involved parties, enables an organization to get the best out of their test team and create support for change and effectively improve the test process. All involved parties are needed, not just testers or test managers. So invite anyone who is working together with testing, from business to operations, from project management to developers. Crucial advantage: commitment is guaranteed by involving all parties.

Stop nagging

Gather ideas in small multidisciplinary teams. Ground rule: any idea or suggestion is welcome. Bad ideas don’t exist, even the suggestion to stop testing is fine! So, anything is OK.

Next step is to collectively cluster the ideas and set priorities: When do improvement ideas have effect? And what is the impact? What are the related costs (and benefits)? Last but not least, the easier it is to implement the measures the more successful it will be. The search is for “silver bullets”: Free measures that have immediate effect, high impact and are easy to implement. The workshop wraps up with an action list where each selected measure directly leads to concrete actions on the list.

Test Improvement for a low-budget does help, but is limited. Renowned test improvement models like TI4Automation®, TI4Agile®, TPI® Next and TMMi® as well as extensive experience in test improvement projects, still is needed to achieve improvements beyond picking the “low hanging fruit”. And by the way, any improvement initiative can only be successful if the people who need to do it are not only involved but are heard…

Test Improvement Revisited

I don’t know exactly when I got involved in test improvement for the first time, but it was way before models were introduced for it. Maybe it was even before people started calling it Test Improvement… I’m actually in testing long enough to have been involved in the pioneering stages of it and, as an author, played a part in defining the profession. What I do know is that helping people and/or organizations to get better at (software) testing still is a major motivator for me. I think it is (one of) the most rewarding jobs within the testing craft there is. That’s why I decided to start a series of blogs on test improvement to share my experiences.

MY FIRST MESSAGE

Truly helping a person and/or organization is not forcing them to do it your way. It is all about enabling them to do the best they can in their context: give them the means they can use in their situation. Wherever you put the emphasis “means they can use” has to be true. THEY have to use it, so it has to be in line with their skills, not yours. Whether they CAN use it depends on their abilities and if they are allowed to use it. I’ve worked in organizations where, due to regulations, the tests had to be designed and approved before execution (otherwise the test results were considered void and the product was not allowed to be shipped). Exploratory Testing was not considered an option by management. If people choose to USE something it’s because they think/know/feel/hope/expect/… that using it enables them to do it better, faster and/or cheaper. So the MEANS have to help them achieve what they want. In my opinion, “fit for context” says it all.

That’s also why I’m convinced that “one model to rule them all” does not apply to Test Improvement. I learned “through damage and shame” (sorry for the Dunglish) that the characteristics of each test improvement model define it’s added value. And it’s limitations. If you want to help improve testing and use a model, the improvement model has to be “fit for context”.

Context Driven TPI

Keep you posted!

I promised myself to improve my blogging and share my experiences in the area of test improvement through this blog. I already started working on the next post…

STAREast 2013: April 28th – May 3rd

Looking forward to the STAREast conference where I’ve been invited to present.

STAREast 2013

Track: Reports of the Death of Testing Have Been Greatly Exaggerated

Due to the success of Martin Pol‘s lightning strikes the keynote contribution, I’ve taken on the challenge on presenting a track that is rowing against the current. The title does cover my message. Hope to see you there and discuss about the future of testing!

Tutorials

I’ve also been invited to present the tutorials Test  Management for Cloud-based Applications and High-flying Test Design Techniques again.