Monitor or test?

The webinar that Ruud Teunissen presented was based on the architecture of Cloutest. It was great to hear how Ruud told about the approach! I feel very proud to have contributed to this approach and to be one of the authors of the book. The cloud is hot, which was proven by the large number of questions that was asked at the end of the webinar. I want to compliment Ruud with how he handled the questions! As the result of this webinar, I hope that people realise that testing really needs to adapt to the way that software is used nowadays. Much uncertainty is introduced with the use of cloud computing, but by selecting the right service and supplier you can allready mitigate a lot of risks. Don’t forget to keep testing in production, to check if “everything still works”. One of the questions mentioned that this is monitoring and not testing. I think we need to broaden our definition of testing and make monitoring a part of testing. Realise yourself that the maintenance department might not monitor on the service, since it is maintained by the supplier… I think what needs to be done in production is testing! A continuous end-to-end test to check if the processes still work, is that monitoring or testing? Doesn’t the word test in E2E test imply a testing activity?

Edit: This topic is also active at the Cloutest site.

5 comments on “Monitor or test?

  1. When running tests in production continuously, what would you call it other than tests? Even when you report your test results into a monitoring system, what you are running is still tests, which result in a status which gets reflected in a monitoring system.
    In what way would you classify this different from running your tests within a CI environment?

    If proactive production monitoring is not testing, then what is a functional automated test integration within your CI system?

    In other words, I agree with you, testing, to be more precise, automated testing should happen a lot in any production environment, but especially in a cloud-based production environment.

  2. Hi Jeroen and Martijn, I agree that monitoring can of course be part of testing.
    But in my opinion there is some distinction.
    If you just want to see if the system is still up and running that is monitoring, it does not look for effects of specific situations or specific changes.
    If you want to know how the system behaves in changing situations and/or under changing conditions that fits within the definition of testing.
    In a simple batch-process there is a clear distinction. Nowadays, especially with real time distributed cloud computing the distinction between monitoring and testing fades.
    Still I think we must be careful what words we choose. We shouldn’t end up in a situation where people don’t test a system in a test environment before going live, just because they heard that you can do testing in production and they want to speed up the development process. We should emphasize that some aspects of quality can be tested in the live situation and that it is very wise to test some aspects continuously because of the ever-changing situation in cloud environments. But we should also make clear that other aspects should be tested “more traditionally”, before a new version of the system is put in the live environment.
    A phrase like “continue testing even after the system has gone live” could be used to make this clear.

    • Hi Rik,

      Thanks for your reply, I think we agree on the subject. As always we, as testers, need to inform the business about the risks, preferably after “more traditional” testing. The testing in production statement follows on the continuously changing world and monitoring is a useful tool to detect the changes. Since monitoring takes place by executing tests in the setting that Ruud was talking about, we decided to label it testing.


  3. I agree completely. “Software testing” encompasses a vast range of activities. The basic underlying property of all those activities is asking how software works. If we already knew we wouldn’t ask! Monitoring is included, without apologies. Developers want to know, and release management wants to know; so do teams and decision-makers in marketing, legal, finance, econometrics, sales, research, IT planning, … In the cloud, the need for testing skills does not decline, but only increases, though we may not recognize the name of the activity as “traditional”.

What do you think of this post?

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s