I am trying some way to get the newoutput column with logic whenever I divide the column E/C i.e 40/5 it give 8 as a divided and I need to take column E digit(excluding end zero) * by dividend. i.e 4,4,4,4,4,4,4,4.

Please find screen shot of a data frame or table.

CodePudding user response：

You can use array_repeat and array_join after identifying the value to repeat and number of times to repeat.

Working Example

from pyspark.sql.functions import array_repeat, array_join, col as c, floor, lit, when
from pyspark.sql import Column

data = [(5, 40, ), (10, 80, ), (20, 120, )]

df = spark.createDataFrame(data, ("C", "E", ))

def repetition(col_c: Column, col_e: Column) -> Column:
    times_to_repeat = floor(c("E") / c("C")).cast("int")
    value_to_repeat = when(c("E") % lit(10) == 0, (c("E") / lit(10)).cast("int")).otherwise(c("E"))
    value_repeat = array_repeat(value_to_repeat, times_to_repeat)
    return array_join(value_repeat, ",")

df.withColumn("Newoutput", repetition(c("C"), c("E"))).show(100, False)

Output

 --- --- ----------------- 
|C  |E  |Newoutput        |
 --- --- ----------------- 
|5  |40 |4,4,4,4,4,4,4,4  |
|10 |80 |8,8,8,8,8,8,8,8  |
|20 |120|12,12,12,12,12,12|
 --- --- -----------------