I have some tests that give me on my local workstation the following Bazel WARNING:

WARNING: //okapi/rendering/sensor:film_test: Test execution time (25.1s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".

When I change the size of the test to small, e.g. buildozer 'set size small' //okapi/rendering/sensor:film_test

My CI job fails with a timeout:

//okapi/rendering/sensor:film_test
TIMEOUT in 60.1s
/home/runner/.cache/bazel/_bazel_runner/7ace7ed78227d4f92a8a4efef1d8aa9b/execroot/de_vertexwahn/bazel-out/k8-fastbuild/testlogs/okapi/rendering/sensor/film_test/test.log

My CI Job is running on GitHub via GitHub-hosted runners - those runners are slower than my local dev box.

What is the best practice here? Choose test size always according to CI and ignore Bazel warnings on local machine? Get a better CI?

CodePudding user response：

Get a better CI?

One of the main purposes of software testing is to simulate the software behavior in an environment that reliably represents the production environment. The better the representation, the easier it is to spot possible issues and fix them before the software is deployed. My opinion is that you are qualified more than any of us to say is the CI you are currently using, a reliable representation of the production environment of your software.

//okapi/rendering/sensor:film_test: Test execution time (25.1s excluding execution overhead)

You can always recheck if your test target is packed correctly, i.e. ask yourself do all of these tests really belong to a single test target. What will be gained/lost if those tests are divided into several test targets?

My CI Job is running on GitHub via GitHub-hosted runners - those runners are slower than my local dev box.

Have you tried employing the test_sharding?

Size vs timeout

When it comes to test targets and their execution, for Bazel it is the question of the underlying resources (CPU, RAM,..) and their utilization.

For that purpose, Bazel exposes two main test attributes, size and timeout

The size attribute is mainly used to define how many resources are needed for the test to be executed, but Bazel uses size attribute to determine a default timeout of a test. The timeout value can be overridden by specifying the timeout attribute.

When using the timeout attribute you are specifying both the minimal and maximal execution time of a test. In Bazel 6.0.0 those values in seconds are:

0 <= short <= 60
30 <= moderate <= 300
300 <= long <= 900
900 <= eternal <= 3600

Since at the time of writing this answer the BUILD file is not shown, I'm guessing that your test target has at least one of these (not that size = medium is the default setting if the attribute is not specified):

size = "medium" or timeout = "moderate"

"All combinations of size and timeout labels are legal, so an "enormous" test may be declared to have a timeout of "short". Presumably it would do some really horrible things very quickly."

There is another option that I don't see being used quite often but might be helpful in your case and that is to specify --test_timeout value as written here

"The test timeout can be overridden with the --test_timeout bazel flag when manually running under conditions that are known to be slow. The --test_timeout values are in seconds. For example, --test_timeout=120 sets the test timeout to two minutes."

One last employable option is, just like you wrote, ignoring Bazel test timeout warnings with --test_verbose_timeout_warnings