333-How much time do you need for testing?

About one third of the total test effort should be spent on determining how to test [1]. I’ve usually counted another third to prepare reusable test data for regression tests and yet another third to finally execute the test. That’s what the 333 stands for me. This is for the current sprint where new features are being developed.

 But I was often asked, how much time does my team need to perform a full manual regression test. While I could - of course - give a rough estimation, I liked to challenge the interviewer whether she can tell me the number of bugs that I am going to find with our tests. I asked that because I often experienced tests that were originally scheduled for 30-45 minutes, could easily take up to 2 hours and more depending on the number of defects we found during the tests.

How can this happen?

• You observe an anomaly and need to analyse it. Is it really a bug or a feature that you just don't remember working exactly this way?
  
  
• You need to find a reliable scenario to reproduce the anomaly on the target test environment
  
  
• Then you need to know, since when is this anomaly in the system? We need to know because this may impact the priority of the issue. Is it there since months and none noticed? Is this recently broken feature that worked fine in the previous version? To find answers to these questions you need to test the same scenario on a different test environment that runs an older version. Maybe you have to test it yet a third test environment to get more accurate information about the history of that anomaly.
  
  
• Then follows the documentation for developers. One needs to create screenshots that highlight the situation before/after, adding labels to the pictures so the bug becomes clear for everyone without losing time reading a lot of text.
  
  
• Sometimes, an issue cannot easily be reproduced and you may need to examine the exact previous steps you took. Easy if you followed a script, more challenging if you followed an exploratory approach while testing. The issue may occur in one particular specifity of the workflow while it may still work fine if you execute the workflow from a different starting point.

These are all tasks that impact how far the estimation deviates from the actual time needed to perform a regression test.

T. Zelger, August 2020