A couple of years ago, I had to assess the testing maturity of a company that was practicing Agile/scrum. The maturity model that I needed to use was TPI Next. At the end of the assessment the model showed a lot of areas for improvement. The improvements that would result in a more mature organization according to the model, would in my opinion not benefit the organization. So why did my opinion and the model differ so much? The reason was the Agile/scrum context. The model steers an organization to more structure, where Agile/scrum asks for flexibility. The context of this company required a different view on what needed to improve. That’s when Polteq decided to put effort in creating a test improvement model for this specific context: TI4Agile .

Since the introduction of TI4Agile a couple of years ago, it has been successfully applied many times in different contexts. I have analyzed the results of these assessments and found common challenges with Agile testing maturity that are interesting to share. It appears that many organizations have a low maturity in the following areas, which are a subset of the twelve key areas of TI4Agile:

Test management
Test process
Test automation
Interaction

Test management and test process share some problems in the transition to Agile development. Both areas were introduced into the traditional testing world to provide more structure to testing. The flexibility and adaptability that is needed in the current Agile context, requires large changes in how to approach test management and test processes.

Test automation has gained a more prominent role in the Agile context, to facilitate iterative and incremental development with fast feedback loops. The increase in importance is recognized by organizations, but often lacks professionalism. Therefore this area lacks maturity.

And last but not least, self-organizing teams require more and better interaction between the team members. Scrum facilitates the interaction in the different meetings that form the basis of the development process. To gain most value out of these meetings, the purpose of these meetings must be clear to all participants. The team members must be able to switch between very technical topics and complex business situations.

In future posts I will dive deeper into each of the areas to provide more insight in the specific challenges of Agile testing maturity.

Mar 2 2017

Testing needs to change

Abraham Lincoln once said:

“The dogmas of the quiet past are inadequate to the stormy present. The occasion is piled high with difficulty, and we must rise with the occasion. As our case is new, so we must think anew and act anew.”

To tackle the changes in the development processes, testing must grow along with these changes. The shift in most companies is from waterfall (or traditional) development to Agile development. To enable this growth, testing should go from an industrial, manufacturing model towards an agricultural model. You cannot predict the outcome of testing, you can just – like a farmer – create the conditions under which testing begins to flourish. Keep in mind that you can predict the outcome of a single test, but not of the complete testing process. To make testing flourish, you need to adapt testing to the context. In some settings it seems enough to reform your testing, but in most cases a transformation is needed.

I advise starting with a clean slate (transforming) over changing your current process (reforming). Aligning with the new context is easier when you are not bound by decisions made in the old context. Start off with thinking about what you need and not what you have! So what do you need? Your context can have a lot of (conflicting) demands, such as:

Short time to market
High quality
Low costs
High auditability

From these demands, look at efficient and effective ways to arrange your testing process along with the development process. You must not only find out when to test what, but also how this testing should take place. Do you still need the process driven testing, with testing techniques to get you test cases that can check if the product meets its requirements? Or do you need exploratory testing to validate that you built the right product? Or do you need both, or other approaches?

In my opinion you need to look at your product from multiple angles. So in your process, you should find a combination of approaches that best fits your needs. None of the approaches is completely wrong or right, they are just different. So why not try to take the best of all worlds?

Jun 27 2016

Learn Development Practices To Improve Your Test Automation Code

This was originally posted on testhuddle.

clean-code-bart-simpson Test automation is a prominent part of testing. To improve your test automation code, you should look at development practices. Creating clean code is a good practice to improve the maintainability and readability of the test automation.
Recognizing clean code is quite easy. If it reads like prose and you understand the intent (almost) immediately even if you are not the author, then it’s probably clean. However, writing clean code is difficult. You will need a lot of practice to properly create clean code, but the following topics will help you along the way. Keep in mind that test automation is software development. So why not use good practices from the field of software development?

Naming

Let’s start with naming. Use proper names for everything that can be named, for instance variables, methods, classes, and packages. The code you create will have to be maintained and read by others. So its intention should be clear. You can improve clarity comes by applying proper naming. If you want your naming to be clear to all the people you work with, use names from the problem domain. These make sense to the business and not only to the development team. When iterating over rows, name your loop variable to indicate what you are looping over:

WRONG:     
for (int i = 0; i < rows.length; i++)

CORRECT:
for (int row = 0; row < rows.length; row++)

Another example is the naming of your methods. When you write a method printRows, it should only print the rows and for instance not also alter them. When you read the method name, it should do what you expect it to do from the name, no more, no less. To achieve this so called single responsibility of a method, you usually need to apply refactoring.

As a final tip on naming, use a coding convention and automate the verification of this convention. This ensures that naming is consistent throughout the complete solution. Conventions around naming have changes over the years. In the early days, the Hungarian notation (prefixing variable names with an abbreviation of the type) was useful, but current integrated development environments (IDEs) show the type of a variable, making prefixing the type no longer necessary.

Refactoring

While creating your code, it usually evolves gradually. This evolution adds or improves functionality, but this does not mean the new version of your code is clean. Given that you want to create a method login that logs into the application with a given user. When you first run your login method, you notice that you need a registered user to be able to log in. Then you decide to alter your login method and you get something like this:

public void login(String user, String password) {
  driver.findElement(By.className("register")).click();
  driver.findElement(By.id("email")).sendKeys(user);
  driver.findElement(By.id("passwd")).sendKeys(password);
  driver.findElement(By.id("passwd2")).sendKeys(password);
  driver.findElement(By.id("SubmitRegistration")).click();

  driver.findElement(By.className("login")).click();
  driver.findElement(By.id("email")).sendKeys(user);
  driver.findElement(By.id("passwd")).sendKeys(password);
  driver.findElement(By.id("SubmitLogin")).click();
  Assert.assertTrue(driver.findElement(By.cssSelector("ul.myaccount_lnk_list"))
    .isDisplayed());
}

Obviously, you can refactor and take out the registration functionality into a different function as can be seen in the next piece of code. This appears to be quite nice, but now the method login, also does registration! In this case you can easily separate the responsibility to register from the responsibility to login. But the example shows how quickly the function name can become inconsistent with the function behavior.

public void login(String user, String password) {
  register(user, password);
  driver.findElement(By.className("login")).click();
  driver.findElement(By.id("email")).sendKeys(user);
  driver.findElement(By.id("passwd")).sendKeys(password);
  driver.findElement(By.id("SubmitLogin")).click();
  Assert.assertTrue(driver.findElement(By.cssSelector("ul.myaccount_lnk_list"))
    .isDisplayed());
}

The caller should decide if it is needed to register, so if we create a test that needs to register and login, that test calls the register method followed by the login method.

Conclusion

Test automation is software development! So you can improve your automation code by applying techniques that are already applied in software development. Proper naming keeps it easier to understand what is happening in the code. Refactoring is needed to keep the intent of the code clear. Separate the responsibilities into different functions, classes, and packages. Make sure that the caller of your code gets what they can expect from the naming, no more, no less.

Jan 7 2014

Transitioning to Agile/SCRUM: the impact on testing

This article has been published in Testing Circus – Volume 4 – Edition 12 – December 2013.

testingcricus

An increasing number of companies are using Agile/SCRUM to implement their software. It is quite a shift from traditional/waterfall development to the more flexible, less documented Agile/SCRUM approach. The transition often proves to be a challenge at some point.

In the book “Scaling Software Agility: Best practices for large Organisations” by Dean Leffingwell you can find some important processes that will change. Going through a couple of these processes, I will describe the impact of the transition on testing.

Changes caused by the transition

Measures of success

In traditional development the main measure of success for a project is usually on time delivery, but in Agile this changes to working code. This is a fundamental difference in measuring success. Instead of time driven, the project will be quality driven. Of course this has its impact on testing. When working code is the measure of success, we need to put more effort in providing working code. The only way for the team to find out that it is working is by testing.

In traditional development the focus for the different disciplines (such as business analysts, developers and testers) is on different aspects. In Agile the complete cross functional team needs to have its main focus on quality. To make this happen, it is important to spread the knowledge of testing across the whole team. This can be achieved by:

using pairing to pair testers with people in other roles to facilitate implicit knowledge sharing;
providing basic test training for the team members to explicitly focus on testing aspects of their roles.

Both of these methods can be executed by the tester in the team, but the latter of the two can also be executed by people outside the team. Testers need to communicate with all different roles in a project and help them to understand and apply testing in their context.

Management culture

Another important aspect that needs to change is the management culture. Where the keywords in traditional development are command and control, the culture needs to shift towards collaborative leadership and empowerment of teams. The impact on testing as a craft seems minimal, but the impact on the traditional test functions, such as test managers and testers, is large.

Test managers used to be responsible for test strategy, product risk analysis, test plans, test estimation, resourcing, etc. But how will this work in Agile/SCRUM?

Planning and estimation are a team responsibility.
Detailed product risk analysis upfront is not possible.
Teams need a degree of freedom, so extensive strategy and plans are uncalled for.

In short, the role of test management changes. The human resources aspect of management is more important. How to get the right tester in the right team and keep the testing knowledge of the testers up-to-date? This is done by knowing the testers and their needs. Test management needs to find ways to get the necessary information out of the different (SCRUM) teams in order to have a bird’s eye view on the testing process.

Keep in mind that a lot of the previous management responsibility shifts to the teams. This requires a high rate of trust in the people and keeping away from micro management. So management needs to let go some of the control and the people in the teams have more responsibilities and need to deal with this. Mind that not everyone will feel comfortable with this, so make sure to pick the right people for the different roles in the team. Next to that: not every team needs the same type of tester.

Requirements and design

The change in requirements and design is very big and testing needs to find a way to cope with this. Where we had big upfront design, we now have continuously changing, emergent, just in time documents. This impact on testing is felt at management level and at engineering
level.

In Agile, test management cannot identify the detailed risks, since there only is a high level, global set of requirements. To retain a risk based testing approach, we need different levels of product risk analysis, abstract up front and more detailed in the teams when more detail is known. So (test)management should be able to do a high level risk assessment at product backlog level, where the team will do detailed risk assessments at the sprint backlog level.

One of the main complaints by traditional trained testers in an Agile environment is the lack of upfront requirements and designs to use as a basis for their testing. Test cases need to emerge from discussions at the grooming or planning session. Testers need to start creating test cases based on these discussions before the requirements and designs are properly documented. By having testers and designers review each other’s products you have quality control early in the process. Test cases and designs prove to have a better match and any differences can be discussed with the product owner.

Note that a good product owner is indispensable for Agile projects. Only by good product ownership, the right product gets build.

Coding and implementation

Everybody is aware of the different phases in a traditional project. Testing takes place after coding, which in its place is after design. A good practice in Agile/SCRUM is the use of test driven development (TDD). Coding and testing then go hand in hand. This increases the quality and maintainability of the code. The shift to TDD is not always that easy. Developers often don’t like to write unit tests and did not receive proper training in how to do right TDD. When used incorrectly, TDD will take a lot of time and have no or little benefits. Testing can support once again by pairing to support the developers with applying white box testing techniques.

The short development cycles and the incremental approach require a lot of regression testing. Since regression testing takes place in every sprint, test automation will save a lot of time. The impact is that people with test automation skills are needed it the team and that we need to plan for the automation. If the testers are not able to automate the tests themselves, they at least need to know what should be automated and communicate this with the automation specialist.

Overall impact on testing

Basically, we still need to test. The craft of testing is still in place and we must not forget what we learned in the past. But we need to adjust and adapt to our new context: Agile. The quick and changing world of Agile development requires a more pragmatic approach to testing. No large upfront planning and documenting, but small pieces of functionality. Pieces that are manageable by the teams. Testing goes beyond the tester; it is part of the complete team.

Automation has become an essential part of testing to keep up with the development pace in Agile. This requires the testers to have more technical knowledge and better communication skills. An early start with automation, usually results in a better maintainable product, so
enough reason to automate!

Last but not least, it’s all about people! Investing in people and skills is needed to perform well in an Agile context. Provide training in testing for all team members and don’t forget to provide training in other disciplines for the testers as well.

Nov 21 2013

How to test refactoring?

This article has been published in Agile Record number 16.

A fundamental part of the Agile methodology is refactoring: rewriting small sections of code to be functionally equivalent but of better quality. Don’t forget to test the refactoring! What do you test? The answer is simple: you test if the code really is functionally equivalent.

To test the rewritten code, you use the unit tests that accompanied the original code. But does unit testing alone prove that you really have functionally equivalent code? No! While refactoring, developers often change more than just the complexity and quality of the code. A tester’s nightmare… It appears to be a small change, but the code is quite likely used in several parts of the solution. So you must perform a regression test after testing the changed code itself. First I will describe how to test the current and rewritten code with unit test. I have identified three scenarios that occur in practice. The code that needs refactoring has:

no unit tests;
bad unit tests;
good unit tests.

After these scenarios I will go into the regression test and explain the importance of proper regression testing while refactoring.

Unit test the current and rewritten code

Unit tests are tests to test small sections of the code. Ideally each test is independent and stubs and drivers are used to get control over the environment. Since refactoring deals with small sections of code, unit tests provide the correct scope.

Refactor code that has no existing unit tests

When you work with very old code, in general you do not have unit tests. So can you just start refactoring? No, first add unit tests to the existing code. After refactoring, these unit tests should still hold. In this way you improve the maintainability of the code as well as the quality of the code. This is a complex task. First you need to find out what the functionality of the code is. Then you need to think of test cases that properly cover the functionality. To discover the functionality, you provide several inputs to the code and observe the outputs. Functional equivalence is proven when the code is input/output conformant to the original code.

Refactor to increase the quality of the existing unit tests

You also see code which contains badly designed unit tests. For example the unit test verifies multiple scenarios at once. Usually this is caused by not properly decoupling the code from its dependencies (Code sample 1). This is undesired behavior; the test must not depend on the state of the environment. A solution is to refactor the code to support substitutable dependencies. This allows the test to use a test stub or mock object. As shown in Code sample 2, the unit test is split into three unit tests which test the three scenarios separately. The rewritten code has a configurable time provider. The test now uses its own time provider and has complete control over the environment.

Treat unit tests as code

The last situation deals with a piece of code which has good unit tests. Just refactor and then you are done, right? Wrong! When you refactor this code, the test will pass if you refactor right. But do not forget to check the validity of the tests. You might think the tests are good, but the unit tests are code too. Every refactor action incorporates a check, and possibly a refactor, of the unit tests.

Perform a regression test

After unit testing the code, you need to verify if the code works in the solution’s context. Remember: In Agile you must provide business value. To show the value, you need to perform a test that relates to the business. A regression test is designed to test the important flows through the solution. And these flows embody the business value. Do you run a complete regression test after each time you refactor? This depends on the risks and on the scalability of the regression test.

Create a scalable regression test

The use case is a common way to describe small parts of functionality. This is a great way to partition your regression test. Create a small set of regression test cases to cover a use case. When you use proper version management for the code, it is easy to see which part of the code belongs to which use case. Whenever a section of code is changed, you can see to which use case it belongs and then execute the regression tests for that use case. However, when code is reused (another good practice), you target a group of use cases. I generally use mindmaps for tracking dependencies within my projects. The mindmaps provide insight in which code is used by which use cases. This requires a disciplined development team. When you reuse existing code, you need to update the mindmap!

Expand the scope of the regression test

Do you test enough when you scale the regression test to the scope determined in the mindmap? No, the regression test serves a larger goal. You check if the (in theory) unaffected areas of the solution are really unaffected. So you test the part that is affected by the refactoring and you test the main flows through the solution. The flows that provide value to the customer are the most important.

Refactoring requires testing

Every change in the code needs to be tested. Therefore testing is required when refactoring. You test the changes at different levels. Since a small section of code is changed, unit testing seems the most fitting level. But do not forget the business value! Regression testing is of vital importance for the business.

Refactoring requires testing

Testing refactoring requires a good understanding of the code

A good understanding of the code requires a disciplined development team

A disciplined development team refactors

Code sample 1: Unit test depending on the environment

From http://xunitpatterns.com/

public void testDisplayCurrentTime_whenever() {
      // fixture setup
      TimeDisplay sut = new TimeDisplay();
      // Exercise sut
      String result = sut.getCurrentTimeAsHtmlFragment();
      // Verify outcome
      Calendar time = new DefaultTimeProvider().getTime();
      StringBuffer expectedTime = new StringBuffer();
      expectedTime.append("");
      if ((time.get(Calendar.HOUR_OF_DAY) == 0)
         && (time.get(Calendar.MINUTE) <= 1)) {
         expectedTime.append( "Midnight");
      } else if ((time.get(Calendar.HOUR_OF_DAY) == 12)
                  && (time.get(Calendar.MINUTE) == 0)) { // noon
         expectedTime.append("Noon");
      } else {
         SimpleDateFormat fr = new SimpleDateFormat("h:mm a");
         expectedTime.append(fr.format(time.getTime()));
      }
      expectedTime.append("");
      assertEquals( expectedTime, result);
}

Code sample 2: Independent unit tests