How to run test cases for a binary in Makefile-CodePudding

There is a small project which produces a binary application. The source code is C, I'm using autotools to create the Makefile and build the binary - it works as well.

I would like to run tests cases with that binary. Here is what I did:

SUBDIRS = src
dist_doc_DATA = README

TESTS=
TESTS =tests/config1.conf
TESTS =tests/config2.conf
TESTS =tests/config3.conf
TESTS =tests/config4.conf
TESTS =tests/config5.conf
TESTS =tests/config6.conf
TESTS =tests/config7.conf
TESTS =tests/config8.conf
TESTS =tests/config9.conf
TESTS =tests/config10.conf
TESTS =tests/config11.conf

I would like to run these cases as argument with the tool. When I run make check, I got:

make[3]: Entering directory '/home/airween/src/mytool'
FAIL: tests/config1.conf
FAIL: tests/config2.conf
FAIL: tests/config3.conf

which is correct, because those files are simple configurations files.

How can I solve that make check runs my tool with the scripts above, and finally I get a list with number of success, failed, ... tests, like in that case:

============================================================================
Testsuite summary for mytool 0.1
============================================================================
# TOTAL: 11
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  11
# XPASS: 0
# ERROR: 0

Edit: so I would like to emulate these runs:

for f in `ls -1 tests/*.conf; do src/mytool ${f}; done

but - of course - I want to see the summary at the end.

Thanks.

CodePudding user response：

The Autotools' built-in test runner expects you to specify the names of executable tests via the make variable TESTS. You cannot just put random filenames in there and expect make or Automake to know what to do with them.

The tests can be built programs, generated scripts, static scripts distributed with the project, or any combination of the above.

How can I solve that make check runs my tool with the scripts above, and finally I get a [test summary report]?

You have acknowledged that your configuration files are not scripts, so stop calling them that! This is in fact the crux of the problem. The easiest solution is probably to create actual executable scripts, one for each case, and name those in your TESTS variable. Each one would run the binary under test with the appropriate configuration file (that is, you're responsible for making them do that if those are the tests you want to perform).

CodePudding user response：

Okay, the solution from here:

tests/Makefile.am:
==================

TEST_EXTENSIONS = .conf
CONF_LOG_COMPILER = ./test-suit.sh


TESTS=
TESTS =config1.conf
TESTS =config2.conf
TESTS =config3.conf
TESTS =config4.conf
TESTS =config5.conf
TESTS =config6.conf
TESTS =config7.conf
TESTS =config8.conf
TESTS =config9.conf
TESTS =config10.conf
TESTS =config11.conf

test/test-suit.sh:
==================

#!/bin/sh

CONF=$1

exec ../src/mytool $CONF

And the result:

make check
...
PASS: config1.conf
PASS: config2.conf
PASS: config3.conf
PASS: config4.conf
PASS: config5.conf
PASS: config6.conf
PASS: config7.conf
PASS: config8.conf
PASS: config9.conf
PASS: config10.conf
PASS: config11.conf
============================================================================
Testsuite summary for mytool 0.1
============================================================================
# TOTAL: 11
# PASS:  11
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

This is what I expected.