时间戳

首页 > 解决方案 > gc 上的分段错误，使用 ctypes

python - gc 上的分段错误，使用 ctypes

问题描述

我正在使用 ctypes 调用本机 c 库，传入一个专门的结构并取回另一个结构。调用 gc 时出现段错误，在 gdb 跟踪中找不到太多详细信息：

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffff7a5ca44 in visit_decref (op=0xf5ee10, data=0x0) at Modules/gcmodule.c:374
374 Modules/gcmodule.c: No such file or directory.
gdb) where
#0  0x00007ffff7a5ca44 in visit_decref (op=0xf5ee10, data=0x0) at Modules/gcmodule.c:374
#1  0x00007ffff798bb0c in subtype_traverse (self=0x7fff7744fd90, visit=0x7ffff7a5ca40 <visit_decref>, arg=0x0) at Objects/typeobject.c:1011
#2  0x00007ffff7a5bd77 in subtract_refs (containers=<optimized out>) at Modules/gcmodule.c:399
#3  collect (generation=generation@entry=0, n_collected=n_collected@entry=0x7fffffffb708, n_uncollectable=n_uncollectable@entry=0x7fffffffb710, nofail=nofail@entry=0) at Modules/gcmodule.c:956
#4  0x00007ffff7a5c95d in collect_with_callback (generation=0) at Modules/gcmodule.c:1128
#5  0x00007ffff7a5d1eb in collect_generations () at Modules/gcmodule.c:1151
#6  _PyObject_GC_Alloc (basicsize=<optimized out>, use_calloc=0) at Modules/gcmodule.c:1726
#7  _PyObject_GC_Malloc (basicsize=<optimized out>) at Modules/gcmodule.c:1736

代码示例是这样的：

_C_DOUBLE_P = POINTER(c_double)
_C_INT_P = POINTER(c_int)


class _IN_DATA(Structure):
    _fields_ = [('in_kps', _C_DOUBLE_P),
                ('in_desc', _C_DOUBLE_P)
                ]
    def __init__(self):
        t = c_int
        self.test = ctypes.cast(t, POINTER(c_int))  

class _OUT_DATA(Structure):
    _fields_ = [
        ('num_out_kps', ctypes.c_int),
        ('out_kps', _C_DOUBLE_P),  
        ('out_desc', _C_DOUBLE_P)
    ]

class _IN_DATA_LIST(ctypes.Structure):
    _fields_ = [
        ('num_crops', c_int),
        ('crops', POINTER(_IN_DATA))
    ]


    def __init__(self, crops_raw_kps: List[np.ndarray], crops_raw_descriptors: List[np.ndarray]):
        num_crops = len(crops_raw_kps)
        self.num_crops = num_crops
        crops = (POINTER(_IN_DATA) * num_crops)()
        self.crops = ctypes.cast(crops, POINTER(_IN_DATA))
        for i in range(num_crops):
            self.crops[i].in_kps = crops_raw_kps[i].ctypes.data_as(_C_DOUBLE_P)
            self.crops[i].in_desc = crops_raw_descriptors[i].ctypes.data_as(_C_DOUBLE_P)

class _OUT_DATA_LIST(ctypes.Structure):
    _fields_ = [
        ('crops_data', ctypes.POINTER(_OUT_DATA)),
        ('num_results', c_int)
    ]  

class SPPostWrapper:

    def __init__(self):

        self._post_processor_lib = ctypes.cdll.LoadLibrary("multitracker/custom_features/build/ffme/libffme.so")
        self._post_processor_lib.py_postprocess.restype = _OUT_DATA_LIST
        self._post_processor_lib.py_postprocess.argtypes = [POINTER(_IN_DATA_LIST)]

    def post_process_multi(self, crops_raw_kps: List[np.ndarray], crops_raw_descriptors: List[np.ndarray]):
        num_crops = len(crops_raw_kps)
        adjusted_kps = [np.asarray(np.squeeze(kp), np.double) for kp in crops_raw_kps]
        adjusted_desc = [np.asarray(np.squeeze(desc), np.double) for desc in crops_raw_descriptors]
        crops_struct = _IN_DATA_LIST(adjusted_kps, adjusted_desc)
        out_result= self._post_processor_lib.py_postprocess(ctypes.byref(crops_struct))

我在python中分配IN_DATA，在c代码中分配OUT_DATA，我尝试在c中使用缓存（假设python不会清理内存）或为每次调用分配新内存（假设python确实释放了out_data内存）-两者调用 python 的 gc 时方法失败。

更新： 为了更好地隔离问题，我删除了 out_data 的使用，将方法设置为 void，但问题仍然存在。我还尝试将 in 数据保留为成员，我认为这可以防止数据发生，直到进程关闭。所以它一定与我分配给输入/输入列表的内存有关。

更新 2： 我能够验证仅当我在 IN_DATA_LIST 中传递超过 1 个项目（超过 1 个作物）时才会出现问题。这确实很奇怪……

标签： pythoncsegmentation-faultctypes

解决方案

在验证即使 c 代码什么也不做也不返回任何内容并且无法解决问题之后，我最终使用了更浅的输入（和输出），这似乎运行正常，没有 seg 错误：

class _IN_DATA(Structure):
    _fields_ = [
                ('num_crops', c_int),
                ('in_kps', _C_DOUBLE_P),
                ('in_desc', _C_DOUBLE_P)
                ]
    def __init__(self, kps_flat, desc_falt, num_crops):
        self.num_crops = num_crops
        self.in_kps = kps_flat.ctypes.data_as(_C_DOUBLE_P)
        self.in_desc = desc_falt.ctypes.data_as(_C_DOUBLE_P)


class _OUT_DATA(Structure):
    _fields_ = [
        ('num_results', ctypes.c_int),
        ('results_per_crop', _C_INT_P),
        ('out_kps', _C_DOUBLE_P),  # x,y, score
        ('out_desc', _C_DOUBLE_P)
    ]

推荐阅读