python-3.x - 减少对'_thread.lock'对象的{方法'acquire'}python的调用次数

Question

嗨，我正在努力使用我的 I/O 绑定应用程序，以使其对潜在用户来说足够快

我正在获取 X 个 URL，例如 10 个，使用 MULTI THREADING ，每个 URL 有 1 个线程

但这花费了太长时间，我在我的代码上运行了 Cprofile，我发现瓶颈在于 “{method 'acquire' of '_thread.lock' objects}”

在 Cprofile 结果中，我注意到方法“获取”被称为每个线程 9 次

任何人都可以阐明我如何减少每个线程的调用次数 这是一个示例代码：

url_to_get = ["https://api.myip.com","https://api.myip.com","https://api.myip.com","https://api.myip.com",
              "https://api.myip.com","https://api.myip.com","https://api.myip.com","https://api.myip.com",
              "https://api.myip.com","https://api.myip.com"]

def fetch(url):
    with requests.get(url,proxies=proxy) as response:
        print(response.text)
        
def main():
    with ThreadPoolExecutor(max_workers=10) as executor:
        executor.map(fetch, url_to_get)
       
        

if __name__ == '__main__':
    import cProfile, pstats
    profiler = cProfile.Profile()
    profiler.enable()
    main()
    profiler.disable()
    stats = pstats.Stats(profiler).sort_stats('tottime')
    stats.print_stats(10)

Cprofile 结果：

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       90    3.581    0.040    3.581    0.040 {method 'acquire' of '_thread.lock' objects}
       10    0.001    0.000    0.001    0.000 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:1177(_make_invoke_excepthook)
       10    0.001    0.000    0.001    0.000 {built-in method _thread.start_new_thread}
       10    0.000    0.000    0.028    0.003 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\concurrent\futures\thread.py:193(_adjust_thread_count)
       20    0.000    0.000    0.025    0.001 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:270(wait)
       21    0.000    0.000    0.000    0.000 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:222(__init__)
       10    0.000    0.000    0.028    0.003 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\concurrent\futures\thread.py:158(submit)
       32    0.000    0.000    0.000    0.000 {built-in method _thread.allocate_lock}
       10    0.000    0.000    0.001    0.000 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:761(__init__)
       10    0.000    0.000    0.025    0.002 C:\Users\MINOUSAT\AppData\Local\Programs\Python\Python38-32\lib\threading.py:540(wait)

太感谢了

标签： python-3.xmultithreadingthreadpoolpython-multithreadingconcurrent.futures