Challenging areas in Agile testing maturity

First posted on the Polteq website.

A couple of years ago, I had to assess the testing maturity of a company that was practicing Agile/scrum. The maturity model that I needed to use was TPI Next. At the end of the assessment the model showed a lot of areas for improvement. The improvements that would result in a more mature organization according to the model, would in my opinion not benefit the organization. So why did my opinion and the model differ so much? The reason was the Agile/scrum context. The model steers an organization to more structure, where Agile/scrum asks for flexibility. The context of this company required a different view on what needed to improve. That’s when Polteq decided to put effort in creating a test improvement model for this specific context: TI4Agile .

Since the introduction of TI4Agile a couple of years ago, it has been successfully applied many times in different contexts. I have analyzed the results of these assessments and found common challenges with Agile testing maturity that are interesting to share. It appears that many organizations have a low maturity in the following areas, which are a subset of the twelve key areas of TI4Agile:

• Test management
• Test process
• Test automation
• Interaction

Test management and test process share some problems in the transition to Agile development. Both areas were introduced into the traditional testing world to provide more structure to testing. The flexibility and adaptability that is needed in the current Agile context, requires large changes in how to approach test management and test processes.

Test automation has gained a more prominent role in the Agile context, to facilitate iterative and incremental development with fast feedback loops. The increase in importance is recognized by organizations, but often lacks professionalism. Therefore this area lacks maturity.

And last but not least, self-organizing teams require more and better interaction between the team members. Scrum facilitates the interaction in the different meetings that form the basis of the development process. To gain most value out of these meetings, the purpose of these meetings must be clear to all participants.  The team members must be able to switch between very technical topics and complex business situations.

In future posts I will dive deeper into each of the areas to provide more insight in the specific challenges of Agile testing maturity.

Advertisement

Why testers do not automate their tests

Test automation exists for quite a while, but it isn’t practiced consistently. In the years that I am in software testing, I’ve heard a lot of arguments on why testers do not automate their tests. The ones I hear the most are:

• Test automation will take away my job.
• I don’t know where to start.
• I cannot write code.

Test automation will take away my job

You as a tester think that you will lose your job when we automate testing? In short: No, it won’t! Testing is an intellectual process. Before you can automate anything, you need to think of what you want to automate. The test automation can only check situations that you defined. Remember, the test automation cannot think for itself and additional manual testing is always needed.

I don’t know where to start

Well, at the beginning of course! Start with investigating on test automation, what can it do, what has worked before and might also work in your situation? Since automating test is an investment, you need to find out which parts are important enough to automate. Product risk analysis (a good practice of structured testing) will definitely help you to define the risk-full and important parts of the software. Often repeated manual test cases are also a very likely candidate for test automation. Watch out not to start too complex. Work on some simple test automation to prove your business case on test automation and then expand.

I cannot write code

This is no reason not to automate tests! Who said you have to do it yourself… You – as a tester – can help decide which test scripts need to be automated. If you do want to do it yourself, you need to invest. At least know some basics in programming. Most frameworks that can help you do test automation don’t require in depth programming skills, but the basics will help you enough to do some valuable automation.

What went wrong?

Recently I discovered some strange defects. The defects could not consistently be reproduced and seemed to appear randomly. So what could I log into our defect management tool? “Every now and then the software seems to be unstable and give these errors” After some discussion with the development team, we decided to log the errors and assign them to me. I tried to find the root cause of the problems, basically the question was: “What went wrong?”. Following I will describe my search for the strange errors, which all appeared to have the same root cause.

Bug : This process sometimes terminates unexpectedly

So how to reproduce the termination of the process. First execute the process and see if it will work. After running the process 10 times without problems, I got bored by manually testing the process and decided to automate the execution. A loop of 1000 times executing the process still did have an abnormally terminated process. A positive side-effect was that we could now easily generate test data, since this process was the starting point for some other processes. When other people observed the ease of data generation, they wanted to use my script too. From that point on, the defect started to re-appear more frequently.

Having more appearances of the defect, it is time to check the logging again. The logging stated: “Could not insert record into …”. Ok, this was as it was with the previous appearances, but when restarting the same process, it would work. So why couldn’t it insert the record in the first place?  Suddenly I had to think on a course I attended on concurrency. The database sometimes locks a table when writing in it, to make sure there is only one edit at the time. When I distributed my script to more people, the process started to run in parallel in stead of in a single thread.  This was the cause of the problem! This proved to be the cause of all the inconsistent errors and could easily be resolved now by correctly implementing a lock-key mechanism.

Experiences at EuroSTAR 2012 (part 2)

In the previous post I described my experiences at the Tuesday of EuroSTAR 2012. In this post I will continue my EuroSTAR experience with the Wednesday.

Changing Management Thinking – John Seddon

A nice quote from this talk about changing management thinking: “The primary cause of failing is management”. Managers tend to make decisions that are not in the benefit of projects. For example when you want to decrease the costs, managers start managing on costs… This actually increases the costs in most cases. However, when you manage on value, this will more often decrease the costs. A very useful story that John told us, was about chicken wings and spare ribs. Management of a large chain of restaurants decided to replace the spare ribs as a starter with chicken wings, since the chicken wing had a larger margin. Customers were disappointed and asked the waiters if they could get a small portion of ribs (still available as a main) as a starter. The waiters want to please their customers, so they say that it is possible. Then the fun part will start… The waiter needs to put the starter in the cash register and since there is no starter of spare ribs listed, they choose to file it under chicken wings. Since management reads the registers and sees that chicken wings are sold very often, they order more chicken wings for their restaurants. A fine example of failing management.

Adventures in Test Automation – Breaking the Boundaries of Regression Testing –

 John Fodeh

John provided information on automated monkey testing. The presentation was supported by using some scenes from the IT Crowd to inform us on automation.  Automated monkey testing p roved to be an easy to understand concept: by randomizing each step, you are simulating monkey testing. The problem of course is that it is easy to miss out on obvious defects, it does not effectively emulate real scenarios and debugging lng test runs can be quite a pain. They felt the need to create more intelligent monkeys by creating somewhat more expectable behavior via the use of state tables with probabilities per action.

Evolving Agile Testing – Fran O’Hara

After a short introduction on Agile and SCRUM, Fran started off on requirements. When we start to talk about user stories, we should try to find out about acceptance criteria for the story. This serves several goals:

• Define the boundaries for a user story/feature
• Help the product owner to find out what it is that delivers value
• Help the team gain understanding of the story
• Help developers and testers to derive tests
• Help developers when to stop adding functionality to a story

Fran reminds us to keep these acceptance criteria at a relatively high level, so do not lose yourself in too much details. Detailing will be done in e.g wireframes, mockups or validation rules. Another place where we find detailing is in the automated acceptance tests. Try to find examples that support your acceptance criteria.

Next Fran stresses the fact that we still need test strategy in Agile. We need to think about the minimal tests in the sprints (automated unit, automated acceptance, manual exploratory) a

nd sometimes need to do some additional testing e.g for non-functionals, feature integration or business processes. The testers themselves need to have broad knowledge (more than just testing) and deep knowledge in testing. This requires a ‘technical awareness’.

Testing of Cloud Services; The Approach: From Risks to Test Measures – Kees Blokland & Jeroen Mengerink

Kees and myself presented Cloutest® our approach to testing cloud services. We started off with an introduction to cloud computing to set the context. To properly introduce the concept, we decided to use the definition provided by NIST. After this into our approach. We identified 143 risks that arise when using cloud computing and grouped these risks into categories:

• Performance
• Security
• Availability & Continuity
• Functionality
• Manageability
• Legislation & Regulations
• Suppliers & Outsourcing

Since 143 risks is quite a lot, we decided to give a limited set of examples of risks and detail these. For instance there occurs a performance risk, since a cloud service usually has several customers. So it’s not only you as a customer that is putting load on the service, but also other users of other customers. This will influence the performance of the service. Imagine your webshop hosted at the same hosting provider that hosts WikiLeaks… The huge amounts of traffic that a new publication on WikiLeaks will generate, might result in your webshop not being available due to performance problems of the service.

With testing we provide methods to mitigate risks, so that is what we did too. The good news is that we can still use a lot of what we have learned over the years. Some techniques need to be tweaked to fit in the cloud context, but they are very useful. Next to the tweaked measures, we also describe some new measures that we have used at our clients. We grouped the measures too:

• Selection
• Performance
• Security
• Manageability
• Availability & Continuity
• Functional
• Migration
• Legislation & Regulations
• Production

How to test the scalability of a cloud service??? Providers promise scalable services and customers pay per use, so if you need more, you will get more. With traditional load testing, we can gradually increase the load and see how the system responds. This can be applied to the service too, but it will scale. You will see the point where the scaling starts in your response times, they will drop when more performance is added at the service. Check around the boundary of the scaling point to see if the billing is also scaled.

We see that test starts earlier, the scope is wider and testing will not stop in production.

Inspirational Talk: Sky is not the limit: Copenhagen Suborbitals – Peter Madsen

The inspirational talk was very nice, however not very test related. It showed us that with the right vision and perseverance you can reach goals that seem to be unreachable. Peter showed us how he built a homemade submarine and a homemade rocket.

Noordertest 2012

Last week the Noordertest conference took place in Groningen. With three groups of five parallel presentations and one keynote, each of the 160 attendees could find something interesting.

Polteq provided two presentations for the conference. I presented about Test Improvement for Agile together with Edze Knol and Ruud Teunissen presented about how to properly do test automation. Edze and I were in the first group of presentations, so after a general introduction to the conference we had to kick-off.

Test Improvement for Agile proved to be a hit, since we had more people than we had seats 🙂 We started with a short introduction to Agile and SCRUM, followed by a short introduction on test improvement. After the introduction we provided some more depth information on three of our key areas:

• Teamwork
• Test management
• Defect management

In teamwork we seek for collaboration, trust and the willingness to work outside your comfort zone. About test management you can read more in my previous post and defect management should be to support the team in stead of the business.

After our presentation I attended “The fragility of agility” by Lloyd Roden. It was nice to see that he pointed out the same groups of improvable items that we dealt with in our presentation.

Parallel was the presentation about proper test automation by Ruud Teunissen. He told the audience that we need to make sure that test automation bridges the gap between testware and the system under test. Make decisions on what you want to achieve and not on e.g. the tools that are already present in an organization. Remember that test automation is a form of development and should be treated as such.

Next was a workshop on how to ask questions. Main points were to make sure that you ask the question you really want to ask and then listen to the answer. Don’t add your own information while asking questions, so you will get the real answer in stead of what you want to hear.

Finally the keynote – also by Lloyd Roden – about challenges in software testing. Here Lloyd presented eight challenges in the software testing world. Learning, skills and communication where of course part of the challenges.

I really enjoyed the conference!