python - Python多处理不要等待所有元素完成

Question

我有以下代码

global total_pds
total_pds = []

ksplit = wr.s3.list_objects(pred_path)
ksplit = list(ksplit)

def process(x):
    dk = wr.s3.read_parquet(path = pred_path+x,dataset=False)
    return dk

def log_result(result):
    print(len(total_pds), end = ' ')
    total_pds.append(result)

def error_back(error):
    print('error', error)

pool = mp.Pool(processes=4,maxtasksperchild=10)
dcms_info = [pool.apply_async(process, args=(spl,), callback = log_result, error_callback = error_back) for spl in ksplit] 

for x in dcms_info:
    x.wait()

pool.close()
pool.join()

dataset = pd.concat(total_pds, ignore_index=True)

最后一个元素给我这个错误：

error("'i' 格式需要 -2147483648 <= number <= 2147483647"

谢谢

标签： pythonmultiprocessingpython-multiprocessing