Challenging areas in Agile testing maturity

First posted on the Polteq website.

A couple of years ago, I had to assess the testing maturity of a company that was practicing Agile/scrum. The maturity model that I needed to use was TPI Next. At the end of the assessment the model showed a lot of areas for improvement. The improvements that would result in a more mature organization according to the model, would in my opinion not benefit the organization. So why did my opinion and the model differ so much? The reason was the Agile/scrum context. The model steers an organization to more structure, where Agile/scrum asks for flexibility. The context of this company required a different view on what needed to improve. That’s when Polteq decided to put effort in creating a test improvement model for this specific context: TI4Agile .

Since the introduction of TI4Agile a couple of years ago, it has been successfully applied many times in different contexts. I have analyzed the results of these assessments and found common challenges with Agile testing maturity that are interesting to share. It appears that many organizations have a low maturity in the following areas, which are a subset of the twelve key areas of TI4Agile:

• Test management
• Test process
• Test automation
• Interaction

Test management and test process share some problems in the transition to Agile development. Both areas were introduced into the traditional testing world to provide more structure to testing. The flexibility and adaptability that is needed in the current Agile context, requires large changes in how to approach test management and test processes.

Test automation has gained a more prominent role in the Agile context, to facilitate iterative and incremental development with fast feedback loops. The increase in importance is recognized by organizations, but often lacks professionalism. Therefore this area lacks maturity.

And last but not least, self-organizing teams require more and better interaction between the team members. Scrum facilitates the interaction in the different meetings that form the basis of the development process. To gain most value out of these meetings, the purpose of these meetings must be clear to all participants.  The team members must be able to switch between very technical topics and complex business situations.

In future posts I will dive deeper into each of the areas to provide more insight in the specific challenges of Agile testing maturity.

Advertisement

Transitioning to Agile/SCRUM: the impact on testing

This article has been published in Testing Circus – Volume 4 – Edition 12 – December 2013.

An increasing number of companies are using Agile/SCRUM to implement their software. It is quite a shift from traditional/waterfall development to the more flexible, less documented Agile/SCRUM approach. The transition often proves to be a challenge at some point.

In the book “Scaling Software Agility: Best practices for large Organisations” by Dean Leffingwell you can find some important processes that will change. Going through a couple of these processes, I will describe the impact of the transition on testing.

Changes caused by the transition

Measures of success

In traditional development the main measure of success for a project is usually on time delivery, but in Agile this changes to working code. This is a fundamental difference in measuring success. Instead of time driven, the project will be quality driven. Of course this has its impact on testing. When working code is the measure of success, we need to put more effort in providing working code. The only way for the team to find out that it is working is by testing.

In traditional development the focus for the different disciplines (such as business analysts, developers and testers) is on different aspects. In Agile the complete cross functional team needs to have its main focus on quality. To make this happen, it is important to spread the knowledge of testing across the whole team. This can be achieved by:

• using pairing to pair testers with people in other roles to facilitate implicit knowledge sharing;
• providing basic test training for the team members to explicitly focus on testing aspects of their roles.

Both of these methods can be executed by the tester in the team, but the latter of the two can also be executed by people outside the team. Testers need to communicate with all different roles in a project and help them to understand and apply testing in their context.

Management culture

Another important aspect that needs to change is the management culture. Where the keywords in traditional development are command and control, the culture needs to shift towards collaborative leadership and empowerment of teams. The impact on testing as a craft seems minimal, but the impact on the traditional test functions, such as test managers and testers, is large.

Test managers used to be responsible for test strategy, product risk analysis, test plans, test estimation, resourcing, etc. But how will this work in Agile/SCRUM?

• Planning and estimation are a team responsibility.
• Detailed product risk analysis upfront is not possible.
• Teams need a degree of freedom, so extensive strategy and plans are uncalled for.

In short, the role of test management changes. The human resources aspect of management is more important. How to get the right tester in the right team and keep the testing knowledge of the testers up-to-date? This is done by knowing the testers and their needs. Test management needs to find ways to get the necessary information out of the different (SCRUM) teams in order to have a bird’s eye view on the testing process.

Keep in mind that a lot of the previous management responsibility shifts to the teams. This requires a high rate of trust in the people and keeping away from micro management. So management needs to let go some of the control and the people in the teams have more responsibilities and need to deal with this. Mind that not everyone will feel comfortable with this, so make sure to pick the right people for the different roles in the team. Next to that: not every team needs the same type of tester.

Requirements and design

The change in requirements and design is very big and testing needs to find a way to cope with this. Where we had big upfront design, we now have continuously changing, emergent, just in time documents. This impact on testing is felt at management level and at engineering
level.

In Agile, test management cannot identify the detailed risks, since there only is a high level, global set of requirements. To retain a risk based testing approach, we need different levels of product risk analysis, abstract up front and more detailed in the teams when more detail is known. So (test)management should be able to do a high level risk assessment at product backlog level, where the team will do detailed risk assessments at the sprint backlog level.

One of the main complaints by traditional trained testers in an Agile environment is the lack of upfront requirements and designs to use as a basis for their testing. Test cases need to emerge from discussions at the grooming or planning session. Testers need to start creating test cases based on these discussions before the requirements and designs are properly documented. By having testers and designers review each other’s products you have quality control early in the process. Test cases and designs prove to have a better match and any differences can be discussed with the product owner.

Note that a good product owner is indispensable for Agile projects. Only by good product ownership, the right product gets build.

Coding and implementation

Everybody is aware of the different phases in a traditional project. Testing takes place after coding, which in its place is after design. A good practice in Agile/SCRUM is the use of test driven development (TDD). Coding and testing then go hand in hand. This increases the quality and maintainability of the code. The shift to TDD is not always that easy. Developers often don’t like to write unit tests and did not receive proper training in how to do right TDD. When used incorrectly, TDD will take a lot of time and have no or little benefits. Testing can support once again by pairing to support the developers with applying white box testing techniques.

The short development cycles and the incremental approach require a lot of regression testing. Since regression testing takes place in every sprint, test automation will save a lot of time. The impact is that people with test automation skills are needed it the team and that we need to plan for the automation. If the testers are not able to automate the tests themselves, they at least need to know what should be automated and communicate this with the automation specialist.

Overall impact on testing

Basically, we still need to test. The craft of testing is still in place and we must not forget what we learned in the past. But we need to adjust and adapt to our new context: Agile. The quick and changing world of Agile development requires a more pragmatic approach to testing. No large upfront planning and documenting, but small pieces of functionality. Pieces that are manageable by the teams. Testing goes beyond the tester; it is part of the complete team.

Automation has become an essential part of testing to keep up with the development pace in Agile. This requires the testers to have more technical knowledge and better communication skills. An early start with automation, usually results in a better maintainable product, so
enough reason to automate!

Last but not least, it’s all about people! Investing in people and skills is needed to perform well in an Agile context. Provide training in testing for all team members and don’t forget to provide training in other disciplines for the testers as well.

Noordertest 2012

Last week the Noordertest conference took place in Groningen. With three groups of five parallel presentations and one keynote, each of the 160 attendees could find something interesting.

Polteq provided two presentations for the conference. I presented about Test Improvement for Agile together with Edze Knol and Ruud Teunissen presented about how to properly do test automation. Edze and I were in the first group of presentations, so after a general introduction to the conference we had to kick-off.

Test Improvement for Agile proved to be a hit, since we had more people than we had seats 🙂 We started with a short introduction to Agile and SCRUM, followed by a short introduction on test improvement. After the introduction we provided some more depth information on three of our key areas:

• Teamwork
• Test management
• Defect management

In teamwork we seek for collaboration, trust and the willingness to work outside your comfort zone. About test management you can read more in my previous post and defect management should be to support the team in stead of the business.

After our presentation I attended “The fragility of agility” by Lloyd Roden. It was nice to see that he pointed out the same groups of improvable items that we dealt with in our presentation.

Parallel was the presentation about proper test automation by Ruud Teunissen. He told the audience that we need to make sure that test automation bridges the gap between testware and the system under test. Make decisions on what you want to achieve and not on e.g. the tools that are already present in an organization. Remember that test automation is a form of development and should be treated as such.

Next was a workshop on how to ask questions. Main points were to make sure that you ask the question you really want to ask and then listen to the answer. Don’t add your own information while asking questions, so you will get the real answer in stead of what you want to hear.

Finally the keynote – also by Lloyd Roden – about challenges in software testing. Here Lloyd presented eight challenges in the software testing world. Learning, skills and communication where of course part of the challenges.

I really enjoyed the conference!

Is a test manager still needed in Agile development?

This is the translation of my article for Polteq in TestnetNieuws 2012-2.

Within Agile the responsibilities shift more to the executing teams. For instance the planning and estimation of the work that used to be done by management, now lies completely within the teams. So is there enough work left to justify the function test manager? To be able to judge this we need to do a couple of things. First determine what a test manager did and which parts of his work will stay within Agile. Next is to determine if new responsibilities and tasks arise fir the test managers.

Who does what?

The first tasks that come in mind when I think of the traditional test manager are:

• Create and update test plans;
• Set the test policy;
• Initiate and execute product risk analysis;
• Estimate the work;
• Create planning;
• Report on risks, progress and quality;
• Manage the test team;
• Manage the test project.

Mostly organizing tasks dealing with testing. When we start working Agile, many of these task will be picked up by the Agile teams. These teams will have the responsibility to estimate and plan the work that is provided to them. A clear indication the the work of the test manager will be less… Whenever the Agile teams execute their planning sessions they need to think about risks, because these risks will influence the amount of work. Therefore the product risk analysis will be split up into smaller parts and will be executed every iteration by the teams.

Next the test plans, test policy and reporting. One of the Agile principles states: “Working software over comprehensive documentation”. So we need to find out how much of the plans and reports is still valid in Agile and how we can minimize this. Test plans cannot become comprehensive documentation! These plans usually leave little room for flexibility and flexibility is just what the Agile teams need. They need to be able to adjust their plans every iteration to fit their needs. The policy will be dealt with later on in this article.

Reporting is also placed in the Agile teams. The teams report on their status every day at the daily stand-up; everyone talks about what they will do and which impediments they have. The burn-down chart – a graph to show how many work is still left in the current sprint – should also be part of the stand-up. It helps to track the progress and well functioning teams will adjust their plannings according to the information of the burn-down chart.

The Agile test manager

The Agile test manager will focus more on people management and therefore has the responsibility to pick the right people for the different teams. The test manager needs to make sure that they hire people fit for Agile, testers have the opportunity to attend essential courses and that sufficient growth possibilities are present within the organization. Beware that not every good (or even excellent) tester is fit for Agile!

Managing the test team has become more complex. At first we had a test team that worked together, but the testers are now divided over different teams. In stead of keeping one plate in the air, a multitude of plates should be kept in the air. It is reasonable to assume that this aspect of test management takes more time than before.

Another important aspect is facilitating knowledge sharing. Of course the knowledge should be shared within the Agile team to enable good and efficient work of the team. But when there are multiple teams, these teams can also learn a lot from each other. The test manager should have a test meeting once every couple of weeks to let the testers of the different teams learn from each other. In this way the teams can benefit from problems and best practices from other teams.

Thinking about risks within the team is very important, but this should also be done at higher levels within the organization. During creation of the product backlog and the release planning (cross-iteration activities), the test manager should make sure that a risk analysis is performed. The impact of different backlog items on each other should become clear. Some items are prerequisite for others and should be planned like that. In projects with multiple teams it often occurs that multiple teams effect the same parts of the system, this risk needs to be addressed during release planning and the sprint planning sessions. In short, we need a helicopter view on the project with a focus on quality and risks.

The business needs a view on product quality. With Agile testing we do not need to reinvent the wheel, a structured testing approach still offers insight into quality. The Agile test manager needs to create a generic test approach that deals with several subjects. Think of how to register defects, dealing with non-functionals and how to do end-to-end testing. The amount of freedom that the teams get to fill in how they deal with these topics depends on a number of factors like risks, the maturity of testing, the maturity of the team, etc. The approach should offer space for the teams to make their own decisions.

Conclusion

Small project with only one or two teams don’t need a full-time test management job, one of the more experienced testers in the teams will be able to fill the role of test manager. Organizations with large and/or a lot of projects have enough room for a full Agile test manager function. So the role of a test manager will not disappear!

The focus of test management is more on people management. Next to that on facilitating and controlling the quality of testing. We need generic guidelines to manage the testing within the organization and the test manager needs to supply these. These guidelines should not limit the freedom of the Agile teams too much.

Thinking about risks still is important. This should not only be done within the Agile teams, but also cross-iteration and even cross-project. The test manager can offer the helicopter view and look at risks on several levels. It’s very important to realize the risks with multiple parallel Agile teams.

So: Yes, in the Agile world we need test management!

Testnet summer school

Last wednesday was the Testnet summer school. A nice day of workshops about several testing topics. There were three workshops on the Certified Agile Tester training, one of them was mine. Since I like to improve my teaching, I decided to follow the other two workshops too.

The first workshop by Bart Bouwers dealt with user stories. The important thing to take away from this workshop was the lifecycle of a user story. From the initial setup as a high level requirement to the go-live moment. The second workshop was my own workshop for Polteq on Agile test management. I shared with the attendees that the role of the test manager probably won’t disappear, but it will change to more people management and more facilitating. The last workshop I attended was by Cecile Davis. We were with a very small group, but had a lot of fun discussing about several Agile topics that came out of the group.

I really enjoyed myself at this great event and hope that Testnet will continue to organize the summer school.