there is a function to tune threshold for say a binary classification described here: https://mlr3pipelines.mlr-org.com/reference/mlr_pipeops_tunethreshold.html

Here's my failed attempt:

  RF_lrn <- lrn("classif.rfsrc", id = "rf", predict_type = "prob")
  RF_lrn$param_set$values = list(na.action = "na.impute", seed = -123)
  single_pred_rf = po("subsample", frac = 1, id = "rf_ss") %>>%
    po("learner", RF_lrn) %>>% po("tunethreshold")

That did not work in my mlr3 pipeline and I did not find any solution explained anywhere so here is my code:

   xgb_lrn <-
    lrn("classif.xgboost", id = "xgb", predict_type = "prob")
  single_pred_xgb = po("subsample", frac = 1, id = "xgb_ss") %>>%
    po("learner", xgb_lrn)
  
    lrnrs <- list(
      RF_lrn,
      xgb_lrn)
    
    lrnrs <- lapply(lrnrs, function(x) {
      GraphLearner$new(po("learner_cv", x) %>>% po("tunethreshold",
                                                   param_vals = list(
                                                     measure = "classif.prauc"
                                                   )
      ))
    })
    library("GenSA")
    lrnrs$RF_lrn <- auto_tuner(
      learner =  RF_lrn,
      search_space = ps(
        ntree = p_int(lower = 20, upper = 300),
        mtry = p_int(lower = 2, upper = 5),
        nodesize = p_int(lower = 1, upper = 7)
      ),
      resampling = rsmp("bootstrap", repeats = 2, ratio = 0.8),
      measure = msr("classif.prauc"),
      term_evals = 100,
      method = "random_search"
    )

which somehow works but I want the decision threshold to be tuned as a parameter the same way I tune other hyperparameters using the random search in benchmarking/cross validation. Any solution? Thanks in advance

CodePudding user response：

the solution is to use po("threshold") instead of po("tunethreshold") as suggested in the comments and this mlr gallery example