Testing in China

ChinaTest 2012Last 9 days I’ve been in Shanghai together with Martin Pol for Polteq. Luckily the typhoon didn’t reach the part of Shanghai we were in, we had some wind and rain, but nothing to extreme.Ā We’ve presented some tutorials and tracks at ChinaTest 2012, but also needed to visit a couple of companies to talk about testing. So there was a lot to do in very little time. At one company you speak with a selected group of a couple of people, where at other companies you have a room filled with about a hundred people and a video connection to other offices where also a hundred people are attending our presentation on testing. No matter what the setting was, we would get a lot of questions, since everyone wanted to learn as much as they could from our visit. This was also the case for the conference.

Together we presented a full day tutorial on structured testing. It was great to notice how people at first were a bit hesitant to ask question, but during the day dared more and more. Some concepts that require little explanation in Europe, required more in China and the other way around too. The next day of the conference Martin had one half day tutorial on compact test process improvement and I had two half day tutorials. Both of my tutorials dealt with Agile testing. The first one was on how to work as a tester in scrum teams. At some parts of the tutorial I had groups of people discuss together on some topic and then present their findings to the whole group. Usually I can follow what people talk about and try to guide them a bit in the direction I want them to think, but my Chinese is not good enough to understand the discussions šŸ˜‰ So I just had to trust that they were talking about the topics I wanted them to. When we started discussing about the topics in English it became clear that they did. The other tutorial was on how to improve testing in Agile (mostly scrum) settings. What struck me was that most questions dealt with automation and tools and not the actual job of testing.

The third day of the conference started with key notes from Martin Pol and Richard Bender. After these keynotes we had to do some more company visits. The last day of the conference we both needed to do some presentations. I presented on how to test mobile apps and on the Polteq approach for testing cloud computing: Cloutest. Both of these topics proved to be very popular! Martin presented on outsourcing for a crowd that seemed to grow during the presentation. The conference ended with a set of lighting talks, where Martin told that we need to save the testing skills and I provided some fast insight into Agile.

Usually I try to provide some information on the different presentations I attended at a conference, but since these were all in Chinese there is not much that I can say about them.

I’d like to thank the organizing committee for a great conference and hope to be able to come back for the 2013 edition!

Test conferences


The number of events that deal with software testing seem to increase. This globalisationĀ is great news for the testing community! More and more opportunities to share knowledge and learn from others. Last year my presentation on how to test mobile apps got to several stages and this year the release of the book Cloutest provides some opportunities for presentation slots. A presentation on Cloutest has been accepted for EuroStar and for ChinaTest. Maybe some more events will follow.

Next to Cloutest, my stage appearances will focus around Agile. China is starting to go Agile and ChinaTest has accepted two Agile proposals next to my Cloutest presentation. These will deal on how to work as a tester in Agile teams and how to improve testing in Agile settings. Ā The Testnet fall event also focuses on Agile, the proposal that got accepted for Testnet deals with Agile and outsourcing.

It’s great to have a lot of proposals accepted, now to prepare each presentation! For the Agile part, there are some basics that need to be dealt with in all the presentations, so let’s think about reusability šŸ™‚ Working at Polteq provides a great setting to discuss about several testing topics. It’s great to get a lot of support and help. For Agile we recently started a discussion group, helping to see common problems and different solutions in Agile settings. Every meeting is an eye opener for me and I hope we will have a lot of informative meetings to come.

Certified Agile Tester

CATLastĀ DecemberĀ I became a trainer for Certified Agile Tester (CAT). Following the course has been very intensive, but really great! Four full days of training, with a couple of hours every night to keep up with the provided material, followed by an exam. The effort paid off and I passed the exam, so since then Polteq can offer the training!

Giving this training feels great. Seeing how your students evolve in using Agile and getting more comfortable every day is very satisfying. This course actually has all the elements that I mentioned in my post on good courses. Every day starts with explaining the theory (and exam material) to the students alternated with discussions. All the students actively participate in the discussions, allowing them to learn from each other’s experience. Another part of the day is a practical exercise, where the students work together in teams to experience what working in Agile teams is like.

Not only the students are learning from this course. Every discussion we have, provides me with more examples that I will be able to use the next time. Tomorrow will be the last day of exam preparation and I want to wish my students all the luck with their exam on Tuesday.

Test improvement in Agile/Scrum*

This is the translation of my article for Polteq in TestnetNieuws 2012-1.

Testnet exists fifteen years, in this time testing and the context of testing has changed a lot. This should have influenced test improvement models, but are the current test improvement models fit for Agile/Scrum? Most of the models are based on the tradition development and test methodologies Ā (waterfall) . The key areas that get addressed by these models depend largely on the phases in these development models. One change we see within Agile/Scrum is the phases that are used… Above all, we see that not only the phases have changed, but the role of testing too. A test manager needs a new point of view on Agile projects and for the testers in Scrum teams we see new demands. Testing has become a role in stead of a function! To assess the maturity of testing, we need to change the models and incorporate Agile/Scrum into the models.


To be able to improve, we need to asses the current maturity of Agile/Scrum and of testing. Testing is not “just a phase” but part of the whole. We are thinking about the following levels of maturity:

  • No Agile/Scrum (it’s called Agile, but it is waterfall);
  • Agile/Scrum process in place (all useful meetings andĀ artifactsĀ are introduced, we see little waterfalls with more and more Agile elements)
  • Agile/Scrum is understood and applied (good incremental/iterative development incorporating the Agile principles)

To achieve optimal improvement, not only testing has to be addressed. However in this article the focus will be on some important testing aspects.

Test knowledge

Evolution of testingThe first prerequisite is a vast knowledge of testing. The evolution of testing is depicted in the following figure. At first testing was at the pioneering phase; everything had to be invented. After that the structured approaches started to emerge. The logical next step was to let go of the structure and use the gained (structured) knowledge in an Agile way. When starting to work Agile before learning the structure,Ā the testing basics will miss and we keep reinventing “test wheels”.

The tester

Agile/Scrum has different demands for testers than waterfall projects. The test needs to do more tasks, which previously were dealt at higher levels in the organization. For instance planning and estimation are now dealt with at team level. We need more proactive testers. Since test basis usually does not exists when a sprint starts, the tester needs to actively help to get it. Don’t wait behind your desk until the specifications are ready! Help the team by creating clear Definitions of Done per backlog item.

TestĀ management

The test manager will have a facilitating role in Agile/Scrum. The role might be a little less test manager and somewhat more team manager. So supply the teams with people that have enough test expertise! However there will be enough work in the area of testing for the test manager. When the product backlog gets assembled, the test manager should think and communicate about risks. This will help to find the dependencies between backlog items as early as possible and will help to decide what to do in the regression tests and/or end-to-end tests. The test manager has the overall view on risks! When multiple teams exist, this is a very important aspect. Another tasks for the test manager is creating generic test approach. For Agile/Scrum this can consist of how to deal with critical issues, demands for test automation and minimal mandatory reporting. Make sure to know the context, it can be organisation wide or only for the Agile projects. Creating a generic approach will give the business more insight and the business still feels like there is some control.


Sad enough, testing is often only partly involved. Planning sessions take place without involving testers or with not enough respect for the testers. Resulting in incorrect estimations and too little thought about the risks. Testers need to be involved in all phases within Agile/Scrum. Preparations like assembling and prioritizing the product backlog also needs input from testing.

Test automation

An important element of testing within Agile/Scrum is test automation. This is necessary to keep up in the incremental process, since the regression test will become bigger and bigger. Within test automation there are a lot of improvement possibilities. Good automation isn’t as easy as it seems. The foundation of good automation is a sound architecture (test framework). Make sure that the test automation is kept up-to-date.

BusinessĀ commitment

The organization as a whole needs to support the new way of working.Ā ControllingĀ based on reports is no longer possible, so the business needs to actively communicate with the teams. This will lead to guiding the teams in the right direction while they still have enough freedom to be Agile. Teams can still do some reporting to help involve the business. Most tools that teams use to keep track of progress and managing defects have reporting facilities. Make sure that the reports deliver what the business needs to know!


It is important to realize that the focus and approach of testing within Agile/Scrum is different. The team is responsible for quality, so testing has become a teamĀ responsibility. All people in the team will be testing, but the testers still have the testing expertise. So especially the testers can help to improve the testing within Agile/Scrum.

*) Agile/Scrum is Agile with Scrum

Test automation in Agile

test automatedIn Agile, Scrum in particular,Ā the whole team is responsible for delivering a high quality product at the end of a sprint!Ā At the end of a sprint defects should not only have been found, but also fixed appropriately. It’s important to work efficiently, since creating software, testing it, fixing it and retesting it can be quite time consuming. Recently Martijn de Vrieze had two posts on test automation that provide relevant information on how to deal with test automation within Agile/Scrum (All my automated tests are passing, whatā€™s wrong?Ā and Throw-away test automation).

Martijn describes a couple of types of automation:

  • Test driven tests (supposed to fail until that part is developed)
  • Automated regression tests (supposed to pass)
  • Automation to support testing

Test driven tests

The test driven tests are an important part of the development. For this post I would like to stretch this definition a bit to “automated tests needed within the sprint”. To be more clear: unit tests, system tests and functional tests. When the team uses test driven development, developers are required to start with creating unit tests before creating functionality. Test driven development has been described a lot, so I won’t go in depth on this. Automating system tests can be realised by using the same style as the unit tests, dealing with “more than units”. It’s important to realise that automating these tests needs to be done with a risk based approach. Automating everything will create overhead, since not all of these tests need to be repeated a lot. Be aware that for automating functional tests, it is advisable to automate just below the user interface, since the user interfaceĀ has the largest probability to change.

Every testĀ written within the sprint shouldĀ be created to help achieve the Definition of Done.Ā Ā Therefore, at the end of the sprint, all these tests should pass! During the sprint, tests from previous sprints will start failing, since new functionality is added and the expected result will change. If you already created new tests for these situations, the older ones can be removed. Be aware that all results the old test was checking are still checked in the new context. In other cases, you need to update the old test for the new context.

Automated regression tests

Regression tests are needed to check whether the new software doesn’t break any existing software. Since these tests are run at least towards the end of every sprint (but preferably more often) these tests are a good candidate for automation. Of course the focus of the regression tests is different form the test driven tests. The focus is more on connectivity and interaction with other systems and even the quality in an end-to-end context. When a regression test fails, there are a couple of options:

  • The context has changed and the test is no longer valid, thenĀ update or delete the test.
  • The newly created software has a defect which needs to be fixed this sprint.
  • The existing software shows a previously undiscovered defect (which needs to be fixed).
    • Product owner needs to decide who fixes,Ā when to fix and if the feature that shows the defect should go live.

Automation to support testing

To enable faster and more effective testing, testers may require automation. This type of automation does not need to be very robust. Think ofĀ a simple recorded script to get you through the necessary steps of logging in to an application. This script only works with the credentials you used while recording, but that might be just what you need. A simple automation like this can save you a lot of time, making sure that you can start testing the functionality that comes available after logging in. Another helpful part of automation can be to extract system information that is needed for defect logging. Delivering quick and dirty automation solutions to help testers is good enough, since the time invested in this automation needs to be earned back.