python - Python - 多处理，给每个处理器一个来自文本文件的电子邮件

Question

所以我一直在玩多处理，我正在考虑升级我的知识，我可以从文本文件中读取进程 1 的第一句话，然后是进程 2 的第二句等等......

txt file:

[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]

这就是代码的样子：

def info(thread):
    global prod
    prod = int(thread) + 1
    runit()


def runit():

    log("Profile-" + str(prod) + Fore.GREEN + ' - ' + email)
    #From here I can then use the email for each worker basically. Or thats the plan atleast. Theplan is that every worker will have its own email that can be used in here.
    sys.exit()

def main():
    user_input = 0
    while True:
        try:
            user_input = int(input(Fore.WHITE + 'How many tasks do you wanna run? [NUMBERS] \n' + Fore.RESET))
        except ValueError:
            print(Fore.RED + "Stop being stupid" + Fore.RESET)
            continue
        else:
            with open('email.txt') as f:
                content = f.readlines()
            content = [x.strip('\n') for x in content]

            try:
                for i, email in enumerate(content):
                    print(email)

            except ValueError as e:
                print(e)

            HowManyThread = user_input
            i = 0
            jobs = []
            for i in range(HowManyThread):
                p = multiprocessing.Process(target=info, args=(str(i),))
                jobs.append(p)
                time.sleep(.5)
                p.start()

            for p in jobs:
                p.join()

            sys.exit()

---

日志基本上只是一条日志消息，没什么特别的

Fore.COLOR <-- Colorama

但是，我完全不知道应该怎么做才能让每个进程都占用每个电子邮件行。所以基本上......

Process-1 to take [e-mail]
Process-2 to take [e-mail]
Process-3 to take [e-mail]
Process-4 to take [e-mail]
Process-5 to take [e-mail]

关于如何做到这一点有什么建议？我完全离开了，完全不知道如何前进。

---

更新

from multiprocessing import pool, Process, Queue
from tqdm import tqdm

with open('email.txt') as f:
    content = f.readlines()

global email_list
email_list = [x.strip('\n') for x in content]


def info(thread):
    global prod
    prod = int(thread) + 1
    runit()


def runit(email_index):
    email = email_list[email_index]

    log("Profile-" + str(prod) + Fore.GREEN + ' - ' + email)
    sys.exit()



def main():
    wipe()
    text()
    Infotext = "First name : Last name : Email: : Random char + Street"
    with open('data.json', 'w') as f:
        json.dump(Infotext, f)
        f.write("\n")

    with Pool(8) as pool:
        result_list = list(tqdm(pool.imap_unordered(, range(len(email_list)), chunksize=5), total=len(email_list))))


if __name__ == '__main__':
    try:
        main()

    except Exception as e:
        print(e)
        print(traceback.print_exc())
        print(traceback)

标签： pythontextmultiprocess

Answer 1

以下方法将多处理委托给一组工作人员，每个工作人员接收一大块索引并一次处理这些索引一行（此处的选择poolsize=8和chunksize=5此处是任意的，可以根据您的要求进行调整）。

然后将所有工人的结果收集到最终列表中。请注意，imap_unordered仅当您不关心处理行的顺序时才适用（即result_list不保持content.

from multiprocessing import Pool
# progress bar to track your multiproc
from tqdm import tqdm

with open('email.txt') as f:
    content = f.readlines()

# this list will be accessed by each worker
global email_list
email_list = [x.strip('\n') for x in content]

# define function that worker will apply to each email
# it gets sent an index for the list of emails
# it accesses the email at that index, performs its function and returns
def runit(email_index):
    email = email_list[email_index]
    # do the stuff you're interested in for a single email

# run the multiprocessing to get your results
# this sends the indexes for the emails out to the workers
# and collects the results of runit into result list
with Pool(8) as pool:                                                
  result_list = list(tqdm(pool.imap_unordered(runit,    
                          range(len(email_list)), chunksize=5),                 
                          total=len(email_list)))