Spock gripes
I’ve recently been using Spock, the Groovy test framework. I’ve got several gripes with it that I wanted to document. This is mostly for my future self, but may be interesting to others.
Subtests
@Unroll with a data provider is the way to run subtests. I have a major and a minor qualm with this:
-
Major: You can’t run a single subtest (for example with
./gradlew test --tests “*some test*”). -
Minor: For dynamic data providers you have to create a whole ‘nother class. It’s much more cumbersome than I’d prefer.
Fixed parallelism
I should be able to run any number of tests concurrently. Indeed with Spock I can do that as described here. But, despite the appearance of supporting fixed test parallelism, it actually supports fixed execution parallelism. This leads to very surprising behaviour, as documented at spockframework/spock/issues/2117.
Poor diff quality
Here’s a Groovy diff comparing a want proto to a got proto:
Gradle Test Executor 102 > EndToEndSpec > assembleVideoForAimLow FAILED
Condition not satisfied:
GrpcUtils.invokeFunction(stub, in) == want
| | | | | |
| | | | | is_completed: true
| | | | | is_successful: true
| | | | | success_response {
| | | | | class_name: "com.netflix.stratum.pod.functions.functions.protogen.HelloResponse"
| | | | | value {
| | | | | fields {
| | | | | key: "response"
| | | | | value {
| | | | | string_value: "Hello from POD world, my name is red. Here is CosmosPode2eFunction"
| | | | | }
| | | | | }
| | | | | }
| | | | | }
| | | | false
| | | {
| | | "fullFunctionName": "com.netflix.stratum.pod.functions.AimLowVideoAssembler.assembleVideoForAimLow",
| | | "arguments": [
| | | {
| | | "name": "listOfMountedEncodes",
| | | "cloudObjects": {
| | | "cloudObjects": [
| | | {
| | | "url": "baggins://reloaded-test-temp/cosmos-e2e-tests/1730401804568/AIMLOWVIDEOENCODE_chunk.al"
| | | }
| | | ]
| | | }
| | | },
| | | {
| | | "name": "assembeledVideoOutput",
| | | "direction": "OUTPUT_REF",
| | | "cloudObject": {
| | | "url": "baggins://reloaded-test-temp/cosmos-e2e-tests/1730401982908/assembledVideo.lo"
| | | }
| | | }
| | | ],
| | | "stack": ""
| | | }
| | <com.netflix.stratumintegtest.protogen.StratumIntegtestServiceGrpc$StratumIntegtestServiceBlockingStub@6f1dd321 channel=ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=19, target=localhost:53139}} callOptions=CallOptions{deadline=null, authority=null, callCredentials=null, executor=null, compressorName=null, customOptions=[[internal-stub-type, BLOCKING]], waitForReady=false, maxInboundMessageSize=null, maxOutboundMessageSize=null, streamTracerFactories=[]}>
| is_completed: true
| is_successful: true
| success_response {
| class_name: "com.google.protobuf.Empty"
| value {
| }
| }
class GrpcUtils
at EndToEndSpec.assembleVideoForAimLow(EndToEndSpec.groovy:73)
1 test completed, 1 failed
Can you tell the problem!? Nope! It’s inscrutable.
Here’s the two side-by-side, by the way:
---------- got:
is_completed: true
is_successful: true
success_response {
class_name: "com.google.protobuf.Empty"
value {
}
}
---------- want:
is_completed: true
is_successful: true
success_response {
class_name: "com.netflix.stratum.pod.functions.functions.protogen.HelloResponse"
value {
fields {
key: "response"
value {
string_value: "Hello from POD world, my name is red. Here is CosmosPode2eFunction"
}
}
}
}
See how much easier that is to understand? That Spock diff is inexcusably bad.
Poor logging support
Preface: This is not a Java-ecosystem issue that’s not specific to Spock.
So you’ve got bad diffs, but you think you’ll println(got, want) to get around them, right? Nope! Because printlns don’t show up when you run your tests. That’s right! You have to turn on info logging first (for example, ./gradlew test --info). But, the problem is, everything info logs.
I ran a single test, whose project has 6 ~small dependencies. It generated 689 lines of output. So, good luck finding your println in several hundreds (on the low end?) lines of output.
(FWIW for fun I ran the same test with --debug and got 32,313 lines. Nice!)
Dynamic property resolution
Preface: This is a Groovy issue, not Spock.
Groovy closures use dynamic property resolution. This has the effect of things that should be compilation issues only being surfaced during runtime.
For example, if you want to retry until something happens, you might use Spock’s PollingConditions, whose eventually takes a closure. If you have compilations in that closure, you’re going to compile, run, retry until you time out, and only after that will your compilation error be surfaced. This is an insane loss of productivity considering that Groovy is built on a strongly typed language!
Verbosity, magic, and assertions
Spock looks for any function that has an expect: or then: block, and calls it a test. This is needlessly magical, adds a lot of verbosity (what if I just want a 1==1 simple assertion? I need to go add expect: first), and restrain what you can do (it’s harder to have utility functions doing assertions, because it’d require an expect: block, which now means your utility function is a test!).
The assertions are not particularly enjoyable, either. Groovy looks for any free boolean: whether a direct comparison like 1 == 1, or a function that returns a boolean like myOptional.isPresent, and calls that an assertion. This is too magical and completely uncustomisable.
For example, let’s say that I spend 30s waiting for something to happen. The assertion would look like retryUntil(...).success, and failure in groovy is just going to be “false expected to be true”. What I want is something like, retryUntil(...).success << “waited 30s for <whatever>, but timed out without success” (using C++’s assertion annotation syntax there). That provides a meaningful message to the reader. But, you can’t do that in Spock!