Spark: How to skip second condition in OR construction-CodePudding

When I try to check OR condition in Spark where function, the second condition is executed even thought first condition is true.

How can I skip the check of second condition?

df.
...
.where(
  (
    lit(lastLoadingDate).isNull
    .or(
         col(srcDTTM) > lastLoadingDate.format(formatterDTTM)
       )
  )
  && col(SrcDTTM) <= currentLoadingDate.format(formatterDTTM)
)

I tried even check next expression:

df.
...
.where(
  (
    lit(true)
    .or(
         col(srcDTTM) > lastLoadingDate.format(formatterDTTM)
       )
  )
  && col(SrcDTTM) <= currentLoadingDate.format(formatterDTTM)
)

But second condition:

col(srcDTTM) > lastLoadingDate.format(formatterDTTM)

is always executed.

CodePudding user response：

Skip the check of second condition may result in incomplete data, because it is or judgment. If the second condition is true and the first condition is false, the amount of data in the result set will increase.

CodePudding user response：

Checking the second condition in OR judgement wont make any difference when the first condition is true. Assume adding another condition or using any other function to skip the second condition check. If first check is false then condition to check if first one is true or false and then going to the second part of OR judgement. It will be like 3 conditions instead of 2. Its better to use OR judgement as it is.