Somehow an error is not being picked up by try... except-CodePudding

I am using a try... except loop to deal with opening a file that is updated throughout the day. Every now and then it would throw an error "pickle data is truncated", so I introduced the loop to try at least 100 times:

import pandas as pd

for i in range(100):
    try:
        df = pd.read_pickle('data')
        break
    except EOFError:
        time.sleep(0.01)

Somehow an error was thrown in the middle of the loop though? Its a problem because it stops my entire process which runs throughout the day.

Is there a better way of trying to open the file? Its updated every few milliseconds throughout the day, and all I want to do is retry a few milliseconds later if there is a problem.

EDIT: as suggested by Davis Herring perhaps how I am saving over the data file is incorrect:

import pandas as pd

df.to_pickle('data')

should instead be:

import pandas as pd
import os
df.to_pickle('tmp')
os.rename('tmp','data')

CodePudding user response：

As suggested by Davis Herring in comments, please see the edit for the answer.