Gather uncaught/untested warnings that were fired during testthat::test

I want to run all the tests and obtain the test results and produced warnings to programmatically create a markdown report showing test outcomes and potential warnings that occurred in the tested code.

But it seems there is no way to obtain or capture warnings during the test run! I understand that tests are executed in a closed environment, but is there really no way to let testthat provide me the thrown warnings?

In the following setup, the warn_list variable is always empty.

Three files for the minimal example:

./tests/testthat.R

library(testthat)
 
warn_list <- list()
outcome <- withCallingHandlers(

    testthat::test_dir(testthat::test_path(), stop_on_failure = FALSE),
   
    warning = function(w) {
        warn_list <<- c(warn_list, list(msg = w$message))
    }
)

rmarkdown::render(input = './tests/create_test_report.Rmd')

Note that the outcome (and warn_list) variable is used in the Rmd file.

./tests/testthat/test_thrown_warn.R

test_that("Throws Warning", {
   
    testthat::expect_gt(sum(3, 2), 3)
    
    testthat::expect_equal(
        {
            warning('Example warning fired inside test!')  # WHERE WARN IS THROWN
            5
        }, 5)
   
})

./tests/create_test_report.Rmd

---
title: "test_results_overview"
output: md_document
---
 
## Test outcomes overview
```{r test_outcome_summary, echo=FALSE}
# the test result
outcome2 <- as.data.frame(outcome)
outcome2 <- outcome2[, names(outcome2)[!names(outcome2) %in% c('result')]]
 
knitr::kable(t(data.frame(passed   = sum(outcome2$passed),
                          warnings = sum(outcome2$warning),
                          failed   = sum(outcome2$failed),
                          error    = sum(outcome2$error))))
```
 
## Full test outcomes:
 
```{r test_outcome, echo=FALSE}
knitr::kable(outcome2)
```

## Produced warnings during the tests:
 
```{r warnings_during_testing, echo=FALSE}
knitr::kable(warn_list)                     # WHERE I TRY TO SHOW IT
```

CodePudding user response：

It seems the SummaryReporter reporter object records warnings. As you mention in your comment, these are very minimally documented, but this seems to do it:

library(testthat)
summary <- SummaryReporter$new()
test_file("somewarnings.R", reporter = summary)
summary$warnings$as_list()

Each of the warnings results in an entry in the list from the last statement. They are stored as condition objects, so you can do stuff like

awarning <- summary$warnings$as_list()[[1]]
getSrcFilename(awarning$srcref)

CodePudding user response：

An alternative solution is actually available within the testthat_results object. It does include specific test case responses like warnings, skipped tests or errors of crashed tests, but the content is not nicely structured.

E.g. in the testcase shown in the example, it will show three result fields, while there are only two 'expect_...' statements. The middle one will be of class c('expectation_warning', 'expectation', 'condition') and contain the warning object.

The alternative solution is to obtain those warnings (and optionally error and skips as a bonus) from the outcomes like this:

outcome <- testthat::test_dir(testthat::test_path(), stop_on_failure = FALSE)

create_warn_df <- function(res) {
    data.frame(test = res$test, warning_msg = res$message)
}
 
warn_df <- data.frame()
for (i_test in seq_along(outcome)) {
    tst <- outcome[[i_test]]
   
    for (i_res in seq_along(tst$results)) {
        res <- tst$results[[i_res]]
       
        if (is(res, "expectation_warning")) {
            warn_df <- rbind(warn_df, create_warn_df(res))
        } 
    }
}
 
knitr::kable(warn_df)

Note: that 'create_warning_str' will format the table and could obtain much more information, like stack etc. from the warning object.

The test results are located in outcome[[X]]$results[[Y]] where X is the file that is being processed and Y is the test case (or warning, as described earlier). The number of Y elements are not equal to the number of test cases.

A more advanced example:

This one includes the line number of where it occurred and includes errors and skips in addition. It also shows the 'file' instead of the 'test' (name).

create_warn_df <- function(file, src_ref, warn_msg) {
    data.frame('file' = file, 'line' = getSrcLocation(src_ref), 'warning_msg' = warn_msg)
}
 
warn_df <- data.frame()
for (i_test in seq_along(outcome)) {
    tst <- outcome[[i_test]]
   
    for (i_res in seq_along(tst$results)) {
        res <- tst$results[[i_res]]
       
        if (is(res, "expectation_warning")) {
            warn_df <- rbind(warn_df, create_warn_df(tst$file, res$srcref, res$message))
        } else if (is(res, "expectation_error")) {
            warn_df <- rbind(warn_df, create_warn_df(tst$file, res$srcref, paste('An error crashed this test:', res$message)))
        } else if (is(res, "expectation_skip")) {
            warn_df <- rbind(warn_df, create_warn_df(tst$file, res$srcref, paste('A test is skipped,', res$message)))
        }
    }
}
 
knitr::kable(warn_df)