Adding and removing values from list python-CodePudding

I have continous stream of data coming in from raspberry pi and I am appending them in a list. I am predicting the output in realtime, the thing now is Memory Management, I am afraid if keep the device running continously it will fill the list and eventually I may run out of memory. Is there any way I can append the value predict them and keep removing all the values from the list that are 100th prediction (index) above my current prediction(index).

i2c = board.I2C()
mpu = adafruit_mpu6050.MPU6050(i2c)
mpu_accelerometer_range = adafruit_mpu6050.Range.RANGE_4_G

data1 = []
j = 0
data2 = []
while(True):
    
    
    j =1
    ax,ay,az = mpu.acceleration
    magnitude = np.sqrt(np.square(ax) np.square(ay) np.square(az))
    data1.append([magnitude])
    df = pd.DataFrame(data1)

CodePudding user response：

I think a solution is to predict the values for the first 100 incoming pieces of data and then start the 'while True'-loop in which you delete the last item of the list at the end of the loop. I'm not familiar with the library and functions you use, but I think it might look like this:

i2c = board.I2C()
mpu = adafruit_mpu6050.MPU6050(i2c)
mpu_accelerometer_range = adafruit_mpu6050.Range.RANGE_4_G

data1 = []
j = 0
data2 = []

for i in range(100):
    j =1
    ax,ay,az = mpu.acceleration
    magnitude = np.sqrt(np.square(ax) np.square(ay) np.square(az))
    data1.append([magnitude])
    df = pd.DataFrame(data1)

while(True):
    j =1
    ax,ay,az = mpu.acceleration
    magnitude = np.sqrt(np.square(ax) np.square(ay) np.square(az))
    data1.append([magnitude])
    df = pd.DataFrame(data1)
    
    data1.pop(0)

Using the pop function and specify the first (in python index 0) element of the list we can remove the element. This might seem confusing, but look at it this way: the oldest item of the list is always the first one. And because we did the loop 100 times at the beginning, you are now constantly deleting the 100st item of the list.

Let me know if this wasn't a solution or if you have any questions.

CodePudding user response：

You currently do this:

data1 = []
j = 0
while(True):
    j =1
    magnitude = ... #something
    data1.append([magnitude])
    df = pd.DataFrame(data1)

Instead, if there's a maximum number n of values you need to store, you could do this:

data1 = [None] * n
j = 0
while True:
    j =1
    magnitude = ... #something
    data1[(j - 1) % n] = [magnitude]
    df = pd.DataFrame(data1[j % n:]   data1[:j % n if j % n else n])

This will keep the memory footprint at a fixed size on the order of n, and at each iteration will deliver a dataframe with the n most recent values of your measurement in chronological order.

The above approach could also optimize the dataframe update logic to simply do this:

# before entering loop:
df = pd.DataFrame(data1)

# inside loop:
df.iloc[(j - 1) % n, 0] = magnitude

CodePudding user response：

If you are concerned about memory using a array may be better.