python - Process a list with a limted number of threads

Question

I want to process a list with a limited number of threads, I want them to take turns, until the list is empty.

I tried the following code:

import threading


class testclass(threading.Thread):

    def __init__(self, name):
        threading.Thread.__init__(self)
        self.name = name

    def run(self):

        while List:
            threadLock.acquire()
            print ('This is thread :', self.name)
            testclass.test()
            threadLock.release()


    def test():
            print (List[0])
            List.pop(0)


List = [1, 2, 3, 4, 5]

threadLock = threading.Lock()

test1 = testclass('test1')
test2 = testclass('test2')
test1.start()
test2.start()

test1.join()
test2.join()

But only thread1 gets to process the whole list.

root@user:/home/user# python3.7 test.py
This is thread : test1
1
This is thread : test1
2
This is thread : test1
3
This is thread : test1
4
This is thread : test1
5
This is thread : test2
Exception in thread test2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "test.py", line 15, in run
    testclass.test()
  File "test.py", line 20, in test
    print (List[0])
IndexError: list index out of range
5

How can I make them take turns on the list and not lock it until the list is empty ?

Thank you in advance !

标签： pythonmultithreading

Answer 1

您可以花很少的时间等待，以便其他线程有时间“获取”资源：

import threading
import time


threadLock = threading.Lock()


class testclass(threading.Thread):

    def __init__(self, name):
        threading.Thread.__init__(self)
        self.name = name

    def run(self):

        while List:
            threadLock.acquire()
            print ('This is thread :', self.name)
            testclass.test()
            threadLock.release()
            time.sleep(0.01)


    def test():
            print (List[0])
            List.pop(0)


List = [1, 2, 3, 4, 5]


test1 = testclass('test1')
test2 = testclass('test2')
test1.start()
test2.start()

test1.join()
test2.join()

输出：

This is thread : test1
1
This is thread : test2
2
This is thread : test1
3
This is thread : test2
4
This is thread : test1
5

但如果你想处理你的列表，我会推荐使用Pool.map：

from multiprocessing.pool import Pool
import multiprocessing

List = [1, 2, 3, 4, 5]

def process_worker(x):
    print(f'[{multiprocessing.current_process().name}]process: {x}')

p = Pool(2) # set the amount of workers

p.map(process_worker, List)

输出：

[ForkPoolWorker-7]process: 1
[ForkPoolWorker-8]process: 2
[ForkPoolWorker-7]process: 3
[ForkPoolWorker-8]process: 4
[ForkPoolWorker-7]process: 5