Calculate stdev for a row and previous row in pandas without series error-CodePudding

Here is my dataset

Date,p1Close,p2Close, spread, movingAverage
2022-02-28,5,10,2,NaN
2022-03-01,2,6,3,2.5
2022-03-02,4,8,2,2,5
2022-03-03,2,8,4,3

I am trying to create a new column in pandas data frame that is equal to the standard deviation between 'spread' from previous row and current row.

    df['standardDeviation'] = statistics.stdev(df['spread'], df['spread'].shift(1))

I keep getting this error:

    File "/usr/lib/python3.9/statistics.py", line 797, in stdev
    var = variance(data, xbar)
  File "/usr/lib/python3.9/statistics.py", line 740, in variance
    T, ss = _ss(data, xbar)
  File "/usr/lib/python3.9/statistics.py", line 684, in _ss
    T, total, count = _sum((x-c)**2 for x in data)
  File "/usr/lib/python3.9/statistics.py", line 166, in _sum
    for n, d in map(_exact_ratio, values):
  File "/usr/lib/python3.9/statistics.py", line 248, in _exact_ratio
    raise TypeError(msg.format(type(x).__name__))
TypeError: can't convert type 'Series' to numerator/denominator

I believe it is because I am using shift(1) and on the first calculation it doesn't have a shift(1) value so it errors out. Not sure how to approach solving this one.

CodePudding user response：

You can actually just use <column>.rolling(2).std():

df['standardDeviation'] = df['spread'].rolling(2).std()

Output:

>>> df
         Date  p1Close  p2Close  spread  movingAverage  standardDeviation
0  2022-02-28        5       10       2            NaN                NaN
1  2022-03-01        2        6       3            2.5           0.707107
2  2022-03-02        4        8       2            2.0           0.707107
3  2022-03-03        2        8       4            3.0           1.414214

CodePudding user response：

@richardec's answer is the best solution but for your specific question, statistics.stdev expects an iterator, so you'll need to pass in pairs:

df['stdev'] = [statistics.stdev(pair) for pair in zip(df['spread'], df['spread'].shift())]

Output:

         Date  p1Close  p2Close  spread  movingAverage     stdev
0  2022-02-28        5       10       2            NaN       NaN
1  2022-03-01        2        6       3            2.5  0.707107
2  2022-03-02        4        8       2            2.0  0.707107
3  2022-03-03        2        8       4            3.0  1.414214