Difference between 1.9.9 and 1.9.3 'Not Run' report

[Zone99]

Hi,

I'm playing with upgrading TestLink from 1.9.3 to 1.9.9. I've followed all the instructions and it's solved some of the problems we were having with 1.9.3 (blank pages on assign test execution and Test result matrix).

One change, however, that isn't helpful is is the 'Not Run' report in 1.9.9. In 1.9.3, it seemed to be unique meaning that if I had a test case assigned to a build and it wasn't run for multiple builds, I'd get a report that would show a not run for ANY build in the test plan. So, if the test is run in a later build, it doesn't show in the report.

1.9.9 seems to show all test cases not run in any build and really gives us too much data.

For example, in 1.9.3:
- assign test case 1 to build 1
- don't run test case 1 in build 1
- not run report - test case 1 isn't run
- run test case 1 in build 2
- not run report - test case 1 isn't listed because it was run in build 2

In 1.9.9:
- assign test case 1 to build 1
- don't run test case 1 in build 1
- not run report - test case 1 isn't run
- run test case 1 in build 2
- not run report - test case 1 IS listed because it wasn't run in build 1

How do I get 1.9.9 to emulate the functionality of 1.9.3 in this area?

Thanks!

[fman]

As usual one of the problem is the use we have done of word ANY.
On help displayed to user when the report is generated we have written:

This report shows all test cases that have not been executed for any build within this test plan

I do not think the help was different on 1.9.3, then issue is what is the right definition of NOT RUN test case when you have multiple BUILDS.
The same question IMHO has to be done regarding 'FAILED' and 'BLOCKED'.

As implemented on 1.9.9 report meaning is more or less:
if you have multiple builds (B1,B2;B3) on test plan TPLAN-A, and you have assigned tester to test case TCA on all builds you will get TCA here if TCA
has not been run AT LEAST on one BUILD.
Then this report allow you to find what test case need still to be run.

and IMHO can be ok and useful.

IN ANY case (I need to test FAILED and BLOCKED) all these reports IMHO has to use same logic.

regarding your question about possibility to change this behaviour, there is no possibility. (you can have a picture of all possible configuration options
or at least 99% is you give a look to config.inc.php)

For failed test case report help said:
This report shows all test cases for each build and each platform (if used for this test plan), whose last execution result is "failed".

Then IMHO, NOT RUN uses same logic => is OK.

What do you need/want to have is a report of test cases THAT HAS NOT RUN ON ALL BUILDS and this do not exists.

[Zone99]

Hi,

Thanks for the quick response! I appreciate the help here.

For clarity, let me explain what I've done:
- Our TestLink 1.9.3 is running in a Virtual Machine
- I cloned the Virtual Machine and upgraded the clone to 1.9.9

The information for both instances is identical at this point since there are no changes so I'm dealing with the same data.

When I run a Not Run report in 1.9.3, I see:

Build: 2.1.2.101 (426 items)

When I run the same Not Run report in 1.9.9 I see:

Build: 2.1.2.101 (1055 items)

It's quite the difference!

Digging further:
- when I look for test case sc-3900 in 1.9.9, I find it marked as 'Not Run' in build 2.1.2.101
- when I look for test case sc-3900 in 1.0.3, it doesn't show up in the report

I then run a test report on that test suite and see:
- on 1.9.9, sc-3900 passed on build 2.1.2.111
- on 1.9.3, sc-3900 passed on build 2.1.2.111

So, given the data is the same, I would expect the reports to be the same but they are not.

I did the same for 'FAILED' and 'BLOCKED' and found that the report results for both versions are identical.

So, I have multiple builds on test plan TPLAN-A and I assigned a tester to test case sc-3900 on all builds (because when I add a build, I copy assignments forward) but on 1.9.3, it removes it when it has been run.

It is useful to have the understanding of what test cases haven't been run on what builds unless you have a lot of builds. Then it creates a lot of information you may not need since you may be setting up the test plan in the beginning and spanning the testing across multiple builds.

I think, however, what you are saying is the behavior in 1.9.9 did change for this area and we will work with it.

It does break one of our uses cases, finding unique tests that have not been executed and executing them, not multiple instances of the same test case.
In the example use case above, in 1.9.3, I had the ability to tell that sc-3900 was executed because it didn't show up in the report if it was run on any build.
In 1.9.9, I don't have that ability because, even if it was run, it still shows up as 'NOT RUN' if it wasn't run on any build.

Thanks again for the quick response.

Any other tips you may have for our situation would be appreciated.