python - 切换使用的进程数是否有效

Question

我对线程编程相当陌生。我写了一个小脚本来并行下载照片。如果我遍历一个对象列表，每个对象的照片数量从 0 到 20+ 不等，在线程池中拥有静态数量的线程或根据图像数量变化是否更有效？一些初步测试表明，静态数量是有意义的，对数据进行了一些统计，几何平均值约为 15.5 张图像，平均值为 23 张。

def processObj(obj):
  # this is function called per object
  from multiprocessing import Pool, cpu_count
  from functools import partial
  # .... processing object that may have multiple photos
  # numPhotos is the num of photos to download
  if len(allPhotos) < 10:
      numProcess = 6
    else:
      numProcess = 12
    getLogger('info').info('\t Extracting Photos')
    pool = Pool(numProcess)
  # processPhoto downloads photo
  download_func = partial(processPhoto, folderName = folderName)
  # photoList is a list of urls to photos
  results = pool.map(download_func, photoList)
  pool.close()
  pool.join()

标签： pythonmultiprocessing