I'm trying to find out memory leaks in a huge project using tracemalloc for extracting code frames. However, I cannot get more than one deepest frame, which is useless for large projects with lots dependencies and myriads of nested calls. According to the docs, I tried to use:

1. tracemalloc.start limit argument
2. PYTHONTRACEMALLOC env variable
3. -X tracemalloc command line argument

And the tracemalloc.get_traceback_limit() shows correct number that I've set. However, I still get only one frame per Traceback object. It works the same on Python versions 3.8.5 and 3.9.7 on different machines. What's wrong here?? How can I fix this?

Here is minimal example:

import os
import tracemalloc

def get_top_malloc(trace_number):
    snapshot = tracemalloc.take_snapshot()
    snapshot = snapshot.filter_traces((
        tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
        tracemalloc.Filter(False, "<unknown>"),
    ))
    top_stats = snapshot.statistics("lineno")
    msg = []
    if trace_number > 0:
        msg.append(f"Top malloc {trace_number} lines")
        for index, stat in enumerate(top_stats[:trace_number]):
            msg.append(f"#{index}: {stat.size // 1024} KB, {stat.count} times")
            print(stat.traceback._frames)  # Only single frame is here!
            print(dir(stat.traceback))
            msg.extend(line for line in stat.traceback.format(limit=16))
        other = top_stats[trace_number:]
        if other:
            size = sum(stat.size for stat in other)
            msg.append(f"{len(other)} other: {size // 1024} KB")
    total = sum(stat.size for stat in top_stats)
    msg.append(f"Total allocated size: {total // 1024 // 1024} MB")
    return "\n".join(msg)


storage = {}

def func2():
    def func3():
        value = '3.1415926535897932384626433832795028841971'
        value = value * 4096
        storage['pi'] = value
    func3()

def func1():
    func2()

if __name__ == "__main__":
    tracemalloc.start(4)
    print(f"\n-- Limit: {tracemalloc.get_traceback_limit()}")
    func1()
    print(f"\n-- {get_top_malloc(1)}")
    tracemalloc.stop()

CodePudding user response：

I've found the solution, it is trivial yet not apparent. The statistics(...) method of snapshot objects takes a key_type argument which is used to group the traces, either by file ("filename"), file line ("lineno"), or by whole traceback ("traceback"). I used the default value "lineno" which filters out everything excluding the last frame per each traceback, so switching to the "traceback" option solved the problem and allowed me to get all the frames I needed :)