I have a couple of python scripts that automatically run daily within an ETL tool (pentaho) on a remote server. The first script creates a series of files (10 files) and the second script sends each of these file to a specified list of emails. The script has been running for several months but in the past few weeks there have been a few days where the script stalled half way through without returning any error (where it stops is random and I can infer it by the number of files created/sent). If I then manually stop the script and run it again it runs without issues.

I'm looking for suggestions on how to find out what is causing this. The problem is random and does not happen when running the script manually. Is there a way to insert some debugging lines (maybe outputted on a txt file) daily so that when it will stop again I can check what alert is being flagged - if any?

CodePudding user response：

Is there a way to insert some debugging lines (maybe outputted on a txt file) daily so that when it will stop again I can check what alert is being flagged - if any?

Yes, simplest way is appending to text file, let say it is named log.txt then

with open("log.txt","a") as f:
    print("done something",file=f,flush=True)

Append done something to log.txt file, note flush=True which does provide applying changes to file as soon as possible.

script stalled half way through without returning any error

Then look for blocking actions depending on external conditions, for example fetching resource via net and server is overloaded that might result in snail download speed and thus look like stalled.