Change a fixed value for a value coming from a column result in error: overloaded method value

I have a ready function used to do some calculations

def calculateFund(cColName: String, inst: Int)(df: DataFrame): DataFrame = {
  val colsToSelect = df.columns.toSeq
  
  val days = 30
  val cSeq = Seq("c_01", "c_02")
  
  df
    .withColumn("amount_ta", lit(1.0) - col(cColName))
    .withColumn("high_pmr", round(pow(lit(1.0)   $"funding_ca"   $"funding_pa", lit(1.0/12)) - lit(1.0), 4))
    .withColumn("rate_mpo", $"high_pmr"   lit(1.0))
    .withColumn("amount_pi", $"amount_ta"/lit(inst))
    .withColumn("c_01", lit(-1.0)*when(lit(1) <= lit(inst), (pow($"rate_mpo", (1.0*days-1)/days) - lit(1.0))*$"amount_pi").otherwise(0.0))
    .withColumn("c_02", lit(-1.0)*when(lit(2) <= lit(inst), (pow($"rate_mpo", (2.0*days-1)/days) - lit(1.0))*$"amount_pi").otherwise(0.0))
    .withColumn(cColName, round(cSeq.map(col).reduce(_   _), 4))
    .select(colsToSelect.map(col):_*)
}

Using the function example

df.transform(calculateFund("credit_col", 1))

It's working fine, but I need to change a fixed value in c_01 and c_02 withColumn condition to use a value coming from a column called set_days

Before: (1.0*days-1)/days)

After: (1.0*days - $"set_days")/days)

So, in this subtraction I'm trying to use the values coming from set_days column instead of using de fixed 1 value

I'm getting this error when I tried to put the column directly, like I did in the example above

error: overloaded method value - with alternatives:
  (x: Double)Double <and>
  (x: Float)Double <and>
  (x: Long)Double <and>
  (x: Int)Double <and>
  (x: Char)Double <and>
  (x: Short)Double <and>
  (x: Byte)Double
 cannot be applied to (org.apache.spark.sql.Column)

I already tried to put on a val, on the function call, but still doesn't work. Could anyone help me, please?

CodePudding user response：

The whole expression used in pow should either be a fixed value or a Column.

(1.0*days - $"set_days")/days expression is a mix of Column and fixed value, more specifically you're using operation - between a fixed value and a Column, this is not supported. That's what the error message says: - cannot be applied to Column as 2nd operand.

You should wrap the fixed value in a Column using lit:

(lit(1.0*days) - $"set_days") / lit(days)