The task engine incrementally schedules tasks and this breaks the isolation of tests; tests affect each other. For the index there is a hack in the testing language that clears the partition before analysis. A similar hack is required for the task engine.

Submitted by Gabriël Konat on 24 June 2013 at 21:41

Instead of a hack, the testing language could also make use of the possibilities, for example by encapsulating test cases in their own partition. BTW, shared names between test cases can be a means for simulating multiple files.

test FancyTest [[ ... ]] // file 1
test FancyTest [[ ... ]] // file 2

This can be extended, when we switch to hierarchical test names:

test FancyTest/foo/f1.cs [[ ... ]] // file foo/f1.cs
test FancyTest/bar/f2.cs [[ ... ]] // file bar/f2.cs

I am not sure if this is the right way, but it might be an easy one to get the testing language working again.

Putting them in their own partition does not help with isolation, if you have class A in file 1 and class A in file 2, that’s a duplicate. Tests should be isolated, unless this is not wanted. You can put them in a different index and task engine by putting them in different projects, but the second time the test is executed it would re-use the old results. So we still need this ‘hack’ of clearing the index and task engine before executing the test.

Tests with the same name for testing incrementality also doesn’t work because the execution order of tests can be arbitrary. I guess the execution order could be changed to execute tests with the same name in sequence, but a better approach would be to have a multi-staged test like:

test Incremental [[ 
    class [[A]] { [[A]] a; }
]] resolves #2 to #1 [[ 
    class B { [[A]] a; }
]] 1 error [[
    class [[A]] { [[A]] a; }
]] resolves #2 to #1

But that would probably require a lot of changes in the testing language.

Clarification: In my proposal, tests with the same name where not intended for testing incrementality, but for isolation (which tests should be considered as part of the same project).

I also don’t get why you need to clear the index if you have the tests isolated. Wouldn’t it be great if we had incremental testing?