I'm working on a TCL project, for which version control is based on Git. So, in order to produce good-quality code, I set up put execution of the tests in pre-commit hook.

However, even if they are executed (trace is shown in command-line), and one of the tests is failed, Git performs the commit. So I launched the hook manually to check the error code, and I figured out that it is null, explaining why Git does not stop:

$ .git/hooks/pre-commit
     FlattenResult-test PASSED
(...)
==== CheckF69F70 FAILED
==== Content of test case:
(...)
==== CheckF69F70 FAILED

$ echo $?
0

(Launching the tests script with tclsh also results in $? to be 0.)

So my question is about this last line: why is $? equal to 0, when one of the tcl tests is failed? And how can I achieve a simple pre-commit hook that stops on failure?

I read and reread the tcltest documentation, but saw no setting or information about this error code. And I would really like not to have to parse the tcl tests output, to check if ERROR or FAILED is present...

CodePudding user response：

This depends on how you run your test suite. Normally you run a file called tests/all.tcl which may look something like this:

package require Tcl 8.6
package require tcltest 2.5

namespace import tcltest::*

configure -testdir [file dirname [file normalize [info script]]] {*}$argv
runAllTests

That final runAllTests returns a boolean indicating success (0) or failure (1). You can use that to generate an exit code by changing the last line to:

exit [runAllTests]

CodePudding user response：

I use this redefinition in some of my test scripts:

# Exit non-zero if any tests fail.
# tcltest's `cleanupTests` resets the numTests array, so capture it first.
proc cleanupTests {} {
    set failed [expr {$::tcltest::numTests(Failed) > 0}]
    uplevel 1 ::tcltest::cleanupTests
    if {$failed} then {exit 1}
}

CodePudding user response：

After some research, I could make it work, even though several factors were against me:

• I have to use an old TCL version (8.5) with tcltest version 2.3.4, in which runAllTests returns nothing;
• I forgot to write cleanupTests at the end of test scripts, as the documentation is not really clear about its usage. (It is not clearer now. I just figured out it is needed if you want to get your tests run by runAllTests, which is really not obvious).

And here is my solution, mostly based on Hai's DevBits blog post:

all.tcl

package require tcltest

::tcltest::configure (...)

proc ::tcltest::cleanupTestsHook {} {
  variable numTests
  set ::exitCode [expr {$numTests(Total) == 0 || $numTests(Failed) > 0}]
}

::tcltest::runAllTests
exit $exitCode

Some thoughts about it:

• I added $numTests(Total) == 0 as a failure condition: this means that no tests was found, which is clearly an erroneous condition;
• This doesn't catch exceptions in the configuration of the tests, for instance a source command that points to a non-existing file, revealing some failure in tests scaffolding. This would be catched as error in other test framewords (ah, pytest, I miss you!)