How to correctly format list of dictionaries from os.popen output?-CodePudding

I'm trying to separate the output of ps -eo pid,command,size --sort -size into a list of dictionaries. I have managed to do so but when there is a space in the output of 'COMMAND', the rest of the output gets transferred to 'SIZE'. I've been stuck trying to figure out how to remove the arguments after the command.

Here's my code:

#!/usr/bin/env python3
import os
import pprint

output = os.popen('ps -eo pid,command,size --sort -size').readlines()
headers = [h for h in ' '.join(output[0].strip().split()).split() if h]
raw_data = map(lambda s: s.strip().split(None, len(headers) - 1), output[1:])
processes = [dict(zip(headers, r)) for r in raw_data]

pprint.pprint(processes)

And here's the output of the script:

[{'COMMAND': '/media/data/opt/xmrig/xmrig', 'PID': '110348', 'SIZE': '3275716'},
 {'COMMAND': '/usr/lib/firefox/firefox', 'PID': '67036', 'SIZE': '1195252'},
 {'COMMAND': '/usr/lib/firefox/firefox', 'PID': '85671', 'SIZE': '-c 693132'},
 {'COMMAND': '/usr/lib/firefox/firefox', 'PID': '207544', 'SIZE': '-c 621752'},
 {'COMMAND': '/usr/lib/firefox/firefox', 'PID': '87797', 'SIZE': '-c 541708'},
 {'COMMAND': '/usr/lib/firefox/firefox', 'PID': '188105', 'SIZE': '-c 483948'},
 {'COMMAND': '/usr/lib/firefox/firefox', 'PID': '67141', 'SIZE': '-c 413528'},
 {'COMMAND': '/usr/lib/firefox/firefox', 'PID': '205146', 'SIZE': '-c 378096'},
 {'COMMAND': '/usr/lib/firefox/firefox', 'PID': '200542', 'SIZE': '-c 289744'},
 {'COMMAND': '/usr/lib/firefox/firefox', 'PID': '96557', 'SIZE': '-c 285156'},
...

Output of 'ouput' in my script before formatting:

['    PID COMMAND                      SIZE\n',
 '  67036 /usr/lib/firefox/firefox    1226276\n',
 '  85671 /usr/lib/firefox/firefox -c 718468\n',
 '  87797 /usr/lib/firefox/firefox -c 488584\n',
 '  67141 /usr/lib/firefox/firefox -c 396152\n',
 ' 261912 /usr/lib/firefox/firefox -c 359196\n',
 ' 205146 /usr/lib/firefox/firefox -c 358444\n',
 ' 200542 /usr/lib/firefox/firefox -c 288724\n',
 '  67185 /usr/lib/firefox/firefox -c 282588\n',
 ' 261847 /usr/lib/firefox/firefox -c 270268\n',
 '  96557 /usr/lib/firefox/firefox -c 266768\n',
 '   4714 picom                       249176\n',
 '   3074 /usr/bin/dockerd -H fd://   247576\n',
 '   3232 containerd --config /var/ru 227116\n',
 '   4733 conky                       111668\n',
...

CodePudding user response：

You should probably use psutil https://psutil.readthedocs.io/en/latest/ instead of trying to parse the output of ps.

To get the (almost) the same information as your ps command in the question you can try

import psutil
# not entirely sure about the memory thing - replace with whatever you actually need
processes = [
    {
        "COMMAND": " ".join(p.cmdline()),
        "PID": p.pid,
        "SIZE": p.memory_info().data / 1024
    }
    for p in psutil.process_iter()
]

CodePudding user response：

You can use regex to get the answer from the output of readlines():

import re

pattern = re.compile(r'(?<=\s)-{1,2}. \s(?=\d)', re.UNICODE)
raw_data = [pattern.sub('', row.strip()).split(None, len(headers)-1) for row in output[1:]]

Which gives the following output:

[['67036', '/usr/lib/firefox/firefox', '1226276'],
 ['85671', '/usr/lib/firefox/firefox', '718468'],
 ['87797', '/usr/lib/firefox/firefox', '488584'],
 ['67141', '/usr/lib/firefox/firefox', '396152'],
 ['261912', '/usr/lib/firefox/firefox', '359196'],
 ['205146', '/usr/lib/firefox/firefox', '358444'],
 ['200542', '/usr/lib/firefox/firefox', '288724'],
 ['67185', '/usr/lib/firefox/firefox', '282588'],
 ['261847', '/usr/lib/firefox/firefox', '270268'],
 ['96557', '/usr/lib/firefox/firefox', '266768'],
 ['4714', 'picom', '249176'],
 ['3074', '/usr/bin/dockerd', '247576'],
 ['3232', 'containerd', '227116'],
 ['4733', 'conky', '111668']]

Then, continuing as you were:

processes = [dict(zip(headers, r)) for r in raw_data]

gives the output:

[{'PID': '67036', 'COMMAND': '/usr/lib/firefox/firefox', 'SIZE': '1226276'},
 {'PID': '85671', 'COMMAND': '/usr/lib/firefox/firefox', 'SIZE': '718468'},
 {'PID': '87797', 'COMMAND': '/usr/lib/firefox/firefox', 'SIZE': '488584'},
 {'PID': '67141', 'COMMAND': '/usr/lib/firefox/firefox', 'SIZE': '396152'},
 {'PID': '261912', 'COMMAND': '/usr/lib/firefox/firefox', 'SIZE': '359196'},
 {'PID': '205146', 'COMMAND': '/usr/lib/firefox/firefox', 'SIZE': '358444'},
 {'PID': '200542', 'COMMAND': '/usr/lib/firefox/firefox', 'SIZE': '288724'},
 {'PID': '67185', 'COMMAND': '/usr/lib/firefox/firefox', 'SIZE': '282588'},
 {'PID': '261847', 'COMMAND': '/usr/lib/firefox/firefox', 'SIZE': '270268'},
 {'PID': '96557', 'COMMAND': '/usr/lib/firefox/firefox', 'SIZE': '266768'},
 {'PID': '4714', 'COMMAND': 'picom', 'SIZE': '249176'},
 {'PID': '3074', 'COMMAND': '/usr/bin/dockerd', 'SIZE': '247576'},
 {'PID': '3232', 'COMMAND': 'containerd', 'SIZE': '227116'},
 {'PID': '4733', 'COMMAND': 'conky', 'SIZE': '111668'}]