I have the following code snippet as a python script. I get the proper job output files with no errors. However the stdout
subprocess
log files (i.e. COSMOSIS_output_XXX.txt
etc)don't store the expected runtime logs. Instead, these files have <_io.TextIOWrapper name=9 encoding='UTF-8'>
as the only output written in the files. What is it I am doing wrong?
import subprocess
from subprocess import Popen
jobname = "cosmosis"
arg2 = "output.filename="
Vector = (
" " ini " " arg2 COSMOSIS_PATH os.path.splitext(ini)[0] ".txt"
)
job3 = subprocess.Popen(
["cosmosis" Vector],
shell=True,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
file = open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "a")
sys.stdout = file
print(job3.stdout)
file_e = open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "a")
sys.stdout = file_e
print(job3.stderr)
try:
outs, errs = job3.communicate(timeout=2000)
except TimeoutExpired:
outs, errs = job3.communicate()
job3.kill()
file.close()
file_e.close()
CodePudding user response:
Passing a list as the first argument with shell=True
is system-dependent. I'm guessing you really mean
with open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "ab") as file,
open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "a") as file_e:
try:
job3 = subprocess.run(
["cosmosis", ini, arg2, COSMOSIS_PATH, os.path.splitext(ini)[0] ".txt"],
stdout=file, stderr=file_e, check=True,
timeout=2000)
except TimeoutExpired:
pass
There is no way for job3.stdout
or job3.stderr
to contain anything because we redirected them to files. (We open the file handles with binary mode so we don't need to specify a text encoding.) There is also no way for the process to return a useful result if it is killed by a timeout, and obviously no need to kill it when it was already killed.
As the subprocess
documentation already tells you, you should prefer subprocess.run
or one of the legacy high-level wrappers instead of Popen
when you can. Perhaps see also Actual meaning of shell=True
in subprocess which also explains why you want to avoid using a shell when you can.
On the other hand, if (as indicated in comments) you want the process to run in the background until completion while your Python script proceeds to perform other tasks and take out the timeout, you do need Popen
.
with open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "ab") as file,
open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "a") as file_e:
job3 = subprocess.Popen(
["cosmosis", ini, arg2, COSMOSIS_PATH, os.path.splitext(ini)[0] ".txt"],
stdout=file, stderr=file_e)
...
# time passes and your script does other things ...
job3.wait()
You might want to periodically poll
the job to see if it has finished.
CodePudding user response:
First of all you are just opening the file, you are not reading anything from it, you are not storing the information of the file anywhere, so it will just create that <_io.TextIOWrapper name=9 encoding='UTF-8'>
which is very easy reproducible:
file = open("testtextf.txt","a")
print(file)
You have to read it somehow with for example with .read()
:
data = file.read()
Also i do not recommend using "open" unless you want to deal with files being left open if anything goes wrong in between the lines before you close it again.
I highly recommend to use "with open" instead.
CodePudding user response:
Ok, the following snippet is sufficient solution to the question posted above.
with open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "wb") as file, open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "wb") as file_e:
job3 = subprocess.Popen(
["cosmosis" Vector],
shell=True,
text=True,
stdout=file,
stderr=file_e,
)