Test improvement in Agile/Scrum*

This is the translation of my article for Polteq in TestnetNieuws 2012-1.

Testnet exists fifteen years, in this time testing and the context of testing has changed a lot. This should have influenced test improvement models, but are the current test improvement models fit for Agile/Scrum? Most of the models are based on the tradition development and test methodologies  (waterfall) . The key areas that get addressed by these models depend largely on the phases in these development models. One change we see within Agile/Scrum is the phases that are used… Above all, we see that not only the phases have changed, but the role of testing too. A test manager needs a new point of view on Agile projects and for the testers in Scrum teams we see new demands. Testing has become a role in stead of a function! To assess the maturity of testing, we need to change the models and incorporate Agile/Scrum into the models.

Maturity

To be able to improve, we need to asses the current maturity of Agile/Scrum and of testing. Testing is not “just a phase” but part of the whole. We are thinking about the following levels of maturity:

  • No Agile/Scrum (it’s called Agile, but it is waterfall);
  • Agile/Scrum process in place (all useful meetings and artifacts are introduced, we see little waterfalls with more and more Agile elements)
  • Agile/Scrum is understood and applied (good incremental/iterative development incorporating the Agile principles)

To achieve optimal improvement, not only testing has to be addressed. However in this article the focus will be on some important testing aspects.

Test knowledge

Evolution of testingThe first prerequisite is a vast knowledge of testing. The evolution of testing is depicted in the following figure. At first testing was at the pioneering phase; everything had to be invented. After that the structured approaches started to emerge. The logical next step was to let go of the structure and use the gained (structured) knowledge in an Agile way. When starting to work Agile before learning the structure, the testing basics will miss and we keep reinventing “test wheels”.

The tester

Agile/Scrum has different demands for testers than waterfall projects. The test needs to do more tasks, which previously were dealt at higher levels in the organization. For instance planning and estimation are now dealt with at team level. We need more proactive testers. Since test basis usually does not exists when a sprint starts, the tester needs to actively help to get it. Don’t wait behind your desk until the specifications are ready! Help the team by creating clear Definitions of Done per backlog item.

Test management

The test manager will have a facilitating role in Agile/Scrum. The role might be a little less test manager and somewhat more team manager. So supply the teams with people that have enough test expertise! However there will be enough work in the area of testing for the test manager. When the product backlog gets assembled, the test manager should think and communicate about risks. This will help to find the dependencies between backlog items as early as possible and will help to decide what to do in the regression tests and/or end-to-end tests. The test manager has the overall view on risks! When multiple teams exist, this is a very important aspect. Another tasks for the test manager is creating generic test approach. For Agile/Scrum this can consist of how to deal with critical issues, demands for test automation and minimal mandatory reporting. Make sure to know the context, it can be organisation wide or only for the Agile projects. Creating a generic approach will give the business more insight and the business still feels like there is some control.

Planning

Sad enough, testing is often only partly involved. Planning sessions take place without involving testers or with not enough respect for the testers. Resulting in incorrect estimations and too little thought about the risks. Testers need to be involved in all phases within Agile/Scrum. Preparations like assembling and prioritizing the product backlog also needs input from testing.

Test automation

An important element of testing within Agile/Scrum is test automation. This is necessary to keep up in the incremental process, since the regression test will become bigger and bigger. Within test automation there are a lot of improvement possibilities. Good automation isn’t as easy as it seems. The foundation of good automation is a sound architecture (test framework). Make sure that the test automation is kept up-to-date.

Business commitment

The organization as a whole needs to support the new way of working. Controlling based on reports is no longer possible, so the business needs to actively communicate with the teams. This will lead to guiding the teams in the right direction while they still have enough freedom to be Agile. Teams can still do some reporting to help involve the business. Most tools that teams use to keep track of progress and managing defects have reporting facilities. Make sure that the reports deliver what the business needs to know!

Conclusion

It is important to realize that the focus and approach of testing within Agile/Scrum is different. The team is responsible for quality, so testing has become a team responsibility. All people in the team will be testing, but the testers still have the testing expertise. So especially the testers can help to improve the testing within Agile/Scrum.

*) Agile/Scrum is Agile with Scrum

Testing gets disqualified

SoftwareTestingLast week I read an article (in Dutch) which presented some negative views on testing and IT people in general. “Just like other professionals they don’t like to take the blame.” Technology is unreliable and the -superior- human always has to fix it. The writer states that the root cause in 80 percent of the incidents is human failure. This seems strange; since the technology is developed and used by humans I’d say that this should be 100 percent… But the reasoning of the writer has some more strange ways. “From humans we can expect that a failure is made every now and then, but from machines we may expect more.” I think that the writer underestimates the humans in this case. Humans can recognize when something goes wrong. When they see something goes wrong, they can correct it, where a machine cannot think.

“All people involved seem to be stuck in sub optimal processes and don’t make use of modern technology” Some people indeed are stuck in processes and in some areas I think that is a good thing. But more and more development is Agile and will adapt. The next point actually felt insulting. “Instead of ‘fooling around’ IT people rather use the word ‘testing’.” So here we have a ‘manager’ disqualifying the work I love to do. I don’t have the feeling that he has any idea of what he is talking about.

“Let IT do what it’s good in, automate.” This is the only paragraph of the article where I had some recognition. Automation, more specific test automation, is used too little in the software development. I also see that in a lot of development processes people say that automation is too difficult, or too expensive. By now we must understand that it will cost you to invest in (test) automation, but if chosen well on what to automate, it will pay back.

I read the article just before I went away for a couple of days. I actually expected a lot of responses about how wrong the author was and that he made some statements that are clearly his opinion, but not based on any thorough research. Strange enough, it got positive responses… So I just had to write about it.

Building blocks of a good (software testing) course

PracticeIn my opinion a good course will contain theory, experience based stories and a practical part. Each of these will deliver value to the participants in its own way, therefore I think all these elements need to be present in every course. No software testing course can be complete without some actual testing in it.

Theory

Theory is considered to be boring by most of the participants, but it’s necessary! The theory will provide some common understanding of terms and definitions that can be used throughout the course. Of course the theory is best accepted by participants when it’s backed by experience based stories, but these stories alone will not suffice to bring the message. It’s up to the teachers to deliver a good mix of theory and experience based stories! The theoretical part of the course is actually a guideline for the teacher, so no relevant subjects for the course are forgotten. Since a course is usually not limited to a single teacher, the theoretical part will make sure that the minimal need information to make the course worthwhile is delivered every time.

Experiences

In the previous paragraph I mentioned that it’s up to the teacher to deliver a mix of theory and experience based stories. Beware for courses where the teacher lacks personal experience in the provided content. However, the teacher is the only one that provides experience stories. A good course will use some form of discussions to let all the attendees share their knowledge. Stimulate all attendees to participate in discussions. Where one person sees problems, the other already tackled these. A good teacher will learn from these discussions and will be able to provide more information every next course. It’s important to acknowledge that you, as a teacher, do not know everything. Also give credit to the attendees and tell them that you will use their input to improve the course.

Practice

To improve the chance that the information in a course sticks with the attendees; always make sure to have some exercises. In the case of software testing, make sure there is some actual testing in the course! Challenge all the people to actively try to make the topic work for them. To really learn something, people need to apply the learned in their own context. Soft skills are important for testing too! Help people develop their soft skills e.g. by letting them present a part of the course and than using the group to give feedback.

What makes a good course great, is a follow up session after the course. Let the attendees try to work with the material in their context and then let them tell how it worked out. As a teacher you’ll learn how the provided material will work out in different contexts, preparing you even better for next courses.

Conclusion

Theory is needed to provide the context for the course. Add stories based on experience to emphasize how the theory can help in real world situations. Let people practice with the provided material, enabling them to apply the material in their own context.

Test automation in Agile

test automatedIn Agile, Scrum in particular, the whole team is responsible for delivering a high quality product at the end of a sprint! At the end of a sprint defects should not only have been found, but also fixed appropriately. It’s important to work efficiently, since creating software, testing it, fixing it and retesting it can be quite time consuming. Recently Martijn de Vrieze had two posts on test automation that provide relevant information on how to deal with test automation within Agile/Scrum (All my automated tests are passing, what’s wrong? and Throw-away test automation).

Martijn describes a couple of types of automation:

  • Test driven tests (supposed to fail until that part is developed)
  • Automated regression tests (supposed to pass)
  • Automation to support testing

Test driven tests

The test driven tests are an important part of the development. For this post I would like to stretch this definition a bit to “automated tests needed within the sprint”. To be more clear: unit tests, system tests and functional tests. When the team uses test driven development, developers are required to start with creating unit tests before creating functionality. Test driven development has been described a lot, so I won’t go in depth on this. Automating system tests can be realised by using the same style as the unit tests, dealing with “more than units”. It’s important to realise that automating these tests needs to be done with a risk based approach. Automating everything will create overhead, since not all of these tests need to be repeated a lot. Be aware that for automating functional tests, it is advisable to automate just below the user interface, since the user interface has the largest probability to change.

Every test written within the sprint should be created to help achieve the Definition of Done.  Therefore, at the end of the sprint, all these tests should pass! During the sprint, tests from previous sprints will start failing, since new functionality is added and the expected result will change. If you already created new tests for these situations, the older ones can be removed. Be aware that all results the old test was checking are still checked in the new context. In other cases, you need to update the old test for the new context.

Automated regression tests

Regression tests are needed to check whether the new software doesn’t break any existing software. Since these tests are run at least towards the end of every sprint (but preferably more often) these tests are a good candidate for automation. Of course the focus of the regression tests is different form the test driven tests. The focus is more on connectivity and interaction with other systems and even the quality in an end-to-end context. When a regression test fails, there are a couple of options:

  • The context has changed and the test is no longer valid, then update or delete the test.
  • The newly created software has a defect which needs to be fixed this sprint.
  • The existing software shows a previously undiscovered defect (which needs to be fixed).
    • Product owner needs to decide who fixes, when to fix and if the feature that shows the defect should go live.

Automation to support testing

To enable faster and more effective testing, testers may require automation. This type of automation does not need to be very robust. Think of a simple recorded script to get you through the necessary steps of logging in to an application. This script only works with the credentials you used while recording, but that might be just what you need. A simple automation like this can save you a lot of time, making sure that you can start testing the functionality that comes available after logging in. Another helpful part of automation can be to extract system information that is needed for defect logging. Delivering quick and dirty automation solutions to help testers is good enough, since the time invested in this automation needs to be earned back.

Agile will fail

No Agile hereWhat quite some Agile consultants tend to do, is implement Agile/Scrum by the book without looking at the actual context. Most of these consultants define an approach that will need to have all layers of an organisation involved. On all these layers, a change process is needed to make a good transition to Agile. Guidance, for the large changes that are needed, is usually only supplied at the team level. This leaves middle and upper management, after a couple of awareness sessions, unable to efficiently deal with Agile/Scrum. After the Agile Consultants have left (most of the time they are too expensive to stay for a long period of time), the company has to carry on working with the mess that is left behind: a process with little Agility due to the remains of the “old process”.

Teams know what they should do, but cannot act since backlog items are not provided in the correct manner or upper management still has a very strict release calendar. Freedom of choice is not present and the team members are forced back into functions instead of roles. Management still thinks in these functions too and will judge people based on their functions instead of based on their performance and value added as a team. People who are not able to work in an Agile setting are kept on board for their “critical knowledge”, but they hinder the Agility of the process and the team by not sharing their knowledge.

Another mistake often made by companies is removing management (project management, test management, …). Especially for larger projects, management and a senior architect are still needed. Successful projects need people that keep a “helicopter view” to make sure that all the parts will work together. You need to know all the connections with other (parts of the) systems to incorporate the risks involved. Large parts of risk mitigation need to happen at this level and will fail without proper management.

Yet another strange phenomenon is not adapting the organisation to the Agile manifesto. The four rules are so clear, but in the transition to Agile the required documentation by various parts of the organisation is usually not re-evaluated. Most maintenance departments still demand exhaustive documentation. Yes, of course there needs to be some documentation, but in most cases the amount of documentation can be much lower while still delivering the essential knowledge.

For Agile to succeed properly, test automation should be an essential part of every sprint. An often heard claim about test automation: “We don’t do it, because it is too difficult in our setting.” This usually proves to be false… The teams do not realise that the changed context has impact on previous decisions on test automation. People often just do not want to invest in test automation, since the benefits are not made clear enough. On short term the business will only see costs without the benefits. If you want test automation to be supported, define a business case! Describe when test automation will start to pay off and what the initial costs will be. Above all make a plan on how to introduce test automation. It is not just using a tool, but a selection process on what to automate and which tool will be most suitable in your context.

Of course I don’t think that Agile will fail, but this is merely a reflection of what I see happening in many companies: a rigid, by the book, implementation with no room for context or Agility. Luckily I have also seen quite some good implementations of Agile/Scrum.

Do you recognize these problems or have problems to add?