Create a range of dates in a pyspark DataFrame-CodePudding

I have the following abstracted DataFrame (my original DF has 60 billion lines )

Id    Date          Val1   Val2
1     2021-02-01    10     2
1     2021-02-05    8      4
2     2021-02-03    2      0
1     2021-02-07    12     5
2     2021-02-05    1      3

My expected ouput is:

Id    Date          Val1   Val2
1     2021-02-01    10     2
1     2021-02-02    10     2
1     2021-02-03    10     2
1     2021-02-04    10     2
1     2021-02-05    8      4
1     2021-02-06    8      4
1     2021-02-07    12     5
2     2021-02-03    2      0
2     2021-02-04    2      0
2     2021-02-05    1      3

Basically, what I need is: if Val1 or Val2 changes in a period of time, all the values between this two dates must have have the value from previous date. (To be more clearly, look at ID 2).

I know that I can do this in many ways (window function, udf,...) but my doubt is, since my original DF has more than 60 billion lines, what is the best approach to do this processing?

CodePudding user response：

I think the best approach (performance-wise) is performing an inner join (probably with broadcasting). If you worry about the number of records, I suggest you run them by batch (could be the number of records, or by date, or even a random number). The general idea is just to avoid running all at once.