Python sort list of ctype structure-CodePudding

I have a ctype structure as bellow:

class BOX(Structure):
    _fields_ = [("x", c_float),
                ("y", c_float),
                ("w", c_float),
                ("h", c_float)]

class DETECTION(Structure):
    _fields_ = [("cl", c_int),
                ("bbox", BOX),
                ("prob", c_float)]

and ML model passing detection result via share libs

lib = CDLL('./libdetector.so', RTLD_GLOBAL)
detect          = lib.detect
detect.argtypes = [...]
detect.restype  = POINTER(DETECTION)

dets = detect(...)

My code work fine, I checked the result and it is correct. But I want to sort detection by its confidence score, so I use this line:

_dets = sorted(dets, key=lambda det: det.prob, reverse=True)

In this step, my computer's memory keep going and when it is full, only "Killed" message was printed to my console. So, does python sorted not work with a list of ctype structure? I can simply use for loop to sort, but everyone want to keep code clean and beauty, don't we? :3

CodePudding user response：

So, if it is a POINTER(DETECTION) result, it is not a list. But using sorted creates a list, what's more, is composed of ctypes.c_float objects, which require 128 bytes per instance, plus don't forget the 8 bytes for the pointer. So if that underlying array is already substantially large, this could balloon your memory requirements...

import numpy.ctypeslib

arr = numpy.ctypeslib.as_array(dect, shape=(n,))
arr.sort(order='prob')

where n has to be the size of the resulting array.

Note, that the resulting numpy structured array object is simply a wrapper around the underlying buffer. It requires a small, constant amount of auxiliary space -- although watch out, some numpy functions/methods might copy, but it won't be as bad as creating a list of ctypes objects!

CodePudding user response：

dets is an pointer to a detection structure, but has no associated size. When passed to sorted() Python will start iterating the pointer forever adding objects to the list it wants to sort, or even crashing if it accesses uncommitted memory.

Example:

from ctypes import *

class BOX(Structure):
    _fields_ = [("x", c_float),
                ("y", c_float),
                ("w", c_float),
                ("h", c_float)]

class DETECTION(Structure):
    _fields_ = [("cl", c_int),
                ("bbox", BOX),
                ("prob", c_float)]

    def __init__(self,prob):  # Add some data to differentiate objects
        self.prob = prob

    def __repr__(self):       # Add a display representation
        return f'D(prob={self.prob})'

# Create a ctypes array of three DETECTION objects
a = (DETECTION * 3)(DETECTION(1.25), DETECTION(3.5), DETECTION(2.25))

# Create a pointer to the first element of the array
# simulating OP's return value from ctypes.
dets = cast(byref(a),POINTER(DETECTION))

print(dets)
print(dets[0])
print(dets[2])
_dets = sorted(dets, key=lambda det: det.prob, reverse=True) # HANGS
print(_dets)

Output:

<__main__.LP_DETECTION object at 0x000002032B7FDC40>
D(prob=1.25)
D(prob=2.25)
# On my system hangs here like OP described.

The fix is you need to know the size of the returned array, and to slice the returned pointer to that known size. The array size in this case was three:

_dets = sorted(dets[:3], key=lambda det: det.prob, reverse=True)

Output:

<__main__.LP_DETECTION object at 0x000001F31763DC40>
D(prob=1.25)
D(prob=2.25)
[D(prob=3.5), D(prob=2.25), D(prob=1.25)]