How to sell software testing?

In the last two months I was testing software for Enrise via Polteq as a part of their development team The Impediments. Testing for them has been nice and instructive. The people at Enrise did not stop asking questions. Most of the questions I answered immediately, but some took a while to find the right answer. The question that lingered the most was: “How do we sell software testing to our customers?” I have to say that I’m very glad that Enrise asks this question! It shows me that Enrise sees the value and the importance of testing. The high quality software that Enrise delivers, can only be delivered when there is time set aside for software testing.

So how do you sell software testing?

• Sell a project (testing is part of the project).
• Only provide guarantee for production incidents when the customer pays for testing.
• Make the customer aware why you value testing.

Basically, you do not sell testing, you sell quality!

Software development is practiced in teams. Testing is a set of tasks that need to be executed in the teams, so there is implicit time for testing. You need to see that testing is essential to quality software and the time needed to test has to be allocated. The software may appear more expensive to your customer, but this will pay off! In the long-term, you will see less disturbances of production when the software is professionally tested. Investing in quality up front is worth it.

You’re probably thinking: “How will testing add to the quality?” Well, in many ways… I’ll give you three:

1. If you plan to test, you will need testable requirements. Ask questions on user stories until you have clear what your customer actually wants to have. Developers usually make more assumptions and think about the solution, where testers think about value for the customer. More questions in the beginning will result in an easier development process and a smaller chance on defects.
2. A tester tests software instead of checking if the software works. A customer will use the software to see if it works, where the tester tries to find the edge cases and out of the box situations. A negative age is an unlikely situation, but a typo in a birthdate is quite plausible… The tester thinks of these situations, so when these typos arise in production the situation has already been covered.
3. Testing is about mitigating risks. This implies that the risks need to be identified. Testing will create a better view on risks. So even risks that are not mitigated, can be communicated to the customer. Then it’s up to the customer to decide what to do: Invest more to mitigate these risks, or live with the risks?

In short: if you want quality products, you really need to test!

*This blog is also posted at Enrise

Advertisement

Noordertest 2012

Last week the Noordertest conference took place in Groningen. With three groups of five parallel presentations and one keynote, each of the 160 attendees could find something interesting.

Polteq provided two presentations for the conference. I presented about Test Improvement for Agile together with Edze Knol and Ruud Teunissen presented about how to properly do test automation. Edze and I were in the first group of presentations, so after a general introduction to the conference we had to kick-off.

Test Improvement for Agile proved to be a hit, since we had more people than we had seats 🙂 We started with a short introduction to Agile and SCRUM, followed by a short introduction on test improvement. After the introduction we provided some more depth information on three of our key areas:

• Teamwork
• Test management
• Defect management

In teamwork we seek for collaboration, trust and the willingness to work outside your comfort zone. About test management you can read more in my previous post and defect management should be to support the team in stead of the business.

After our presentation I attended “The fragility of agility” by Lloyd Roden. It was nice to see that he pointed out the same groups of improvable items that we dealt with in our presentation.

Parallel was the presentation about proper test automation by Ruud Teunissen. He told the audience that we need to make sure that test automation bridges the gap between testware and the system under test. Make decisions on what you want to achieve and not on e.g. the tools that are already present in an organization. Remember that test automation is a form of development and should be treated as such.

Next was a workshop on how to ask questions. Main points were to make sure that you ask the question you really want to ask and then listen to the answer. Don’t add your own information while asking questions, so you will get the real answer in stead of what you want to hear.

Finally the keynote – also by Lloyd Roden – about challenges in software testing. Here Lloyd presented eight challenges in the software testing world. Learning, skills and communication where of course part of the challenges.

I really enjoyed the conference!

Is a test manager still needed in Agile development?

This is the translation of my article for Polteq in TestnetNieuws 2012-2.

Within Agile the responsibilities shift more to the executing teams. For instance the planning and estimation of the work that used to be done by management, now lies completely within the teams. So is there enough work left to justify the function test manager? To be able to judge this we need to do a couple of things. First determine what a test manager did and which parts of his work will stay within Agile. Next is to determine if new responsibilities and tasks arise fir the test managers.

Who does what?

The first tasks that come in mind when I think of the traditional test manager are:

• Create and update test plans;
• Set the test policy;
• Initiate and execute product risk analysis;
• Estimate the work;
• Create planning;
• Report on risks, progress and quality;
• Manage the test team;
• Manage the test project.

Mostly organizing tasks dealing with testing. When we start working Agile, many of these task will be picked up by the Agile teams. These teams will have the responsibility to estimate and plan the work that is provided to them. A clear indication the the work of the test manager will be less… Whenever the Agile teams execute their planning sessions they need to think about risks, because these risks will influence the amount of work. Therefore the product risk analysis will be split up into smaller parts and will be executed every iteration by the teams.

Next the test plans, test policy and reporting. One of the Agile principles states: “Working software over comprehensive documentation”. So we need to find out how much of the plans and reports is still valid in Agile and how we can minimize this. Test plans cannot become comprehensive documentation! These plans usually leave little room for flexibility and flexibility is just what the Agile teams need. They need to be able to adjust their plans every iteration to fit their needs. The policy will be dealt with later on in this article.

Reporting is also placed in the Agile teams. The teams report on their status every day at the daily stand-up; everyone talks about what they will do and which impediments they have. The burn-down chart – a graph to show how many work is still left in the current sprint – should also be part of the stand-up. It helps to track the progress and well functioning teams will adjust their plannings according to the information of the burn-down chart.

The Agile test manager

The Agile test manager will focus more on people management and therefore has the responsibility to pick the right people for the different teams. The test manager needs to make sure that they hire people fit for Agile, testers have the opportunity to attend essential courses and that sufficient growth possibilities are present within the organization. Beware that not every good (or even excellent) tester is fit for Agile!

Managing the test team has become more complex. At first we had a test team that worked together, but the testers are now divided over different teams. In stead of keeping one plate in the air, a multitude of plates should be kept in the air. It is reasonable to assume that this aspect of test management takes more time than before.

Another important aspect is facilitating knowledge sharing. Of course the knowledge should be shared within the Agile team to enable good and efficient work of the team. But when there are multiple teams, these teams can also learn a lot from each other. The test manager should have a test meeting once every couple of weeks to let the testers of the different teams learn from each other. In this way the teams can benefit from problems and best practices from other teams.

Thinking about risks within the team is very important, but this should also be done at higher levels within the organization. During creation of the product backlog and the release planning (cross-iteration activities), the test manager should make sure that a risk analysis is performed. The impact of different backlog items on each other should become clear. Some items are prerequisite for others and should be planned like that. In projects with multiple teams it often occurs that multiple teams effect the same parts of the system, this risk needs to be addressed during release planning and the sprint planning sessions. In short, we need a helicopter view on the project with a focus on quality and risks.

The business needs a view on product quality. With Agile testing we do not need to reinvent the wheel, a structured testing approach still offers insight into quality. The Agile test manager needs to create a generic test approach that deals with several subjects. Think of how to register defects, dealing with non-functionals and how to do end-to-end testing. The amount of freedom that the teams get to fill in how they deal with these topics depends on a number of factors like risks, the maturity of testing, the maturity of the team, etc. The approach should offer space for the teams to make their own decisions.

Conclusion

Small project with only one or two teams don’t need a full-time test management job, one of the more experienced testers in the teams will be able to fill the role of test manager. Organizations with large and/or a lot of projects have enough room for a full Agile test manager function. So the role of a test manager will not disappear!

The focus of test management is more on people management. Next to that on facilitating and controlling the quality of testing. We need generic guidelines to manage the testing within the organization and the test manager needs to supply these. These guidelines should not limit the freedom of the Agile teams too much.

Thinking about risks still is important. This should not only be done within the Agile teams, but also cross-iteration and even cross-project. The test manager can offer the helicopter view and look at risks on several levels. It’s very important to realize the risks with multiple parallel Agile teams.

So: Yes, in the Agile world we need test management!

The Certified Agile Tester exam

I received some questions about the Certified Agile Tester exam and decided to answer them in a blog post, so that more people could benefit from this. The course will prepare you for all the parts of the exam, but if you will pass will depend on a lot of factors. At the moment of writing, the exam is available in English and in German. You must be able to answer the questions in these languages too. For the normal courses it is no problem to have some spelling or grammar errors as long as the correctors can understand what you mean, but the demands are higher for the train-the-trainer courses.

Theory

i have some questions about the nature of the examination for CAT as the description is pretty vague . What does an open questions exam mean exactly? Can you detail it a bit ? Like how many questions in the 3h, what s the general scope of the questions ? How your knowledge of written English language affects your mark(and your ability to express yourself in English for non native English speakers) ? Maybe an example of similar question to one at the exam?

The theoretical part of the exam consists of 10 open questions, directly related to the contents of the course and 3 scenarios that also have a couple of questions (this number varies). During the course you will get example questions that can be discussed with the class. Giving an example of a scenario is quite hard, but the open questions could look like:

“Name 1 statement from the agile manifesto and describe how it affects the testers in an agile team”

You will then be awarded four points for an answer containing one of the statements of the manifesto and at least three explanations of the effect on testers.

Practice

Second part of the examination aka a practical section where the examinee’s testing skills are put to the test, how exactly is that done ? There are 100 ways to test practical knowledge in my mind , how exactly it is done here ? Again maybe you can give a theoretical example in the style of the exam.

The practical part consists of testing a website which is described by several user stories. You act as if you are the tester in a team and start working on planning your time, deciding what and how to test followed by actually testing the website. The increments to the website can be deployed, so you will see some bug fixes and some new bugs throughout the exam. Note that you have to write down a lot, to give the corrector a good impression on what you planned to do, what you actually did and why you did or did not do certain things. During the course we will work on a website too, this will prepare you for the exam.

Focus

The certification is focus on the Scrum “flavor” of agile development or it s general including Extreme Programming ,Crystal etc? I m asking mostly because from what i ve read already some of the “flavors” (in this case Extreme Programming) diminish the role of the professional tester to the point that it s just a hat the developers wear at some points in a iteration.

The main focus of the course is on the Scrum flavor. We do talk a bit about some other flavors and some of the questions in the theoretical part can deal with another flavor than Scrum. By the way, I do not agree that with Extreme Programming the professional testing is just a hat the developers wear. Never completely rely on the books you read, in practice you will see more use for testers!

Growing need for technical testers

This is a translation of my article for the “Testkrant Jaargang 2012 Editie 1“. The article is based on my experiences at Polteq.

The world changes and this sets new requirements for testers, especially the technical aspect of testing. Cloud computing enables us to use functionality from the cloud. For cloud computing we need to test if the functionality of the cloud service correctly supports the business processes, but we also need technical tests to check if the service correctly integrates with the existing systems. We also need to monitor the service in production by using test automation, which needs technical skills.

The other big “hype” that requires technical testers is Agile. By using multi disciplinary teams, the tester gets closer to the developer. Direct communication between these two functions will increase since less documentation is used. To enable good communication, the testers will need to know something about development and vice versa. The iterative and incremental character of Agile requires to execute regression tests more often. This demands for more test automation. This article will describe some situations for cloud, Agile and test automation that will clarify the need for technical testers.

Cloud

A growing number of companies uses services from the cloud (e.g. e-mail, CRM or environments). All services are provided via the internet and should be integrated in the current business processes. Testing the communication with the service is a technical task. The focus will be on the messages that are used to communicate with the service which requires looking below the GUI.

Using cloud services has an impact on internal development projects. Whenever we need to test chains of events after internal changes a connection to the (live) service is often unwanted. So we need to use a stub or mock to simulate the cloud service in order to test our processes without touching the production version of the cloud service.

A service comes with agreements and contract with the supplier about performance and other aspects of the service in production. We need to check in production if the supplier will keep up to the agreements. There is the need for frequent (or even continuous) tests that will monitor the service. This implies test automation which will be covered further on in this article.

Agile

Agile uses four simple basic priciples:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

The importance of interaction can be found within the multi disciplinairy teams. Less documentation can only work of the team members communicate well on what they are doing. To understand what the other team members are doing, you will need to know about the other disciplines. We expect from our testers that they have a basic knowledge of programming and have some basic programming skills. This will help to discuss about software architecture and possible solutions, which will be defined by the team.

The short development cycles force us to test incomplete products. The tester will need to apply the use of drivers, stubs and mocks to be able to test. Creating or setting up these facilities to support testing needs to be signaled (and preferably also executed) by the testers.

The working software that gets delivered every iteration will need to work together with the previously delivered items. Each iteration the functionality will increase which also reflects on the size of the regression test set. This set grows, but the time to execute it is only limited. The frequent execution of the regression test set, together with the time pressure, make a good business case for test automation. You should not think to lightly of test automation within Agile.

Another requirement as a result of the frequent deliveries is good version and configuration management. The testers will need to help to make this a succes. When and how to integrate? Which demands do we have for the different environments? Technical knowledge will help to get a better and clearer view of what should happen.

Test automation

No need to explain that test automation has technical aspects. Since test automation is an important part of testing with cloud computing and within Agile, test automation deserves some special attention.

Test automation can occur on different levels. The foundation for test automation is set with the unit test. These unit tests are in general created by the developers, but this does not mean that testers are not involved. We can help the developers apply test design techniques, since we have more knowledge of testing. Pair programming of a tester with a developer can be of value here. The tester can assist the developer in thinking about which test cases need to be created. In addition to this assisting, we can also review the unit tests when we have enough knowledge and skills with respect to the programming language.

Functional test automation is closer to the testers. With Agile development the graphical user interface keeps changing to meet the (changing) requirements, so succesfull automation needs to be at a layer below the GUI. So once again we need to know how to manipulate the code to be able to test the functionality. When automation is used that does use the graphical user interface, this GUI must be manipulated in some manner. When we use Fitnesse or Cucumber to achieve this, it seems like little or no technical work, but we do need some programming. Both Fitnesse and Cucumber need an interpretation layer to talk to the test object. Allthough both come with a lot of standard functions and options, they almost never fit completely for your specific situation. Therefore we need to add some functionality, which needs to be programmed. More and more it is expected from the tester that they can do this.

Multiple contexts require continuous testing. Within Agile for the integration and regression, with cloud for monitoring the supplier, etc. The only way to achieve continuous (or at least very frequent) testing is by automation. Thinking about how to realise this and actualy implementing it requires knowledge of architecture, programming and testing.

Conclusion

The software we create is getting more and more complex. This does not only require more technical knowledge from the developers, but also from the testers. In test jobs we see requests for programming skills (preferably in multiple programming languages) and people expect the testers to be able to discuss about architectural problems found during development. The integration of different systems, as seen with cloud computing, requires the testers to be able to read technical logging, use drivers, stubs and mocks, etc. The closeness that we find in the Agile teams implies more technical discussions and communication.

The tester needs to keep up in the quickly changing world, where Agile and cloud are becoming the standard. The only way to keep up with the speed of development is to use test automation. The tools that will be needed for this should not only be used, but also understood and implemented. So yes, the need for technical testers grows.

Keep learning and broaden your knowledge, so that we – as a test community – can fill this need!