python - 如何使用线程处理多个 GET 请求并进行比较

Question

我一直在试图弄清楚如何加快速度并获得一些线程知识。

我一直在尝试创建一个放置两个 GET 请求的函数。对于每个链接，我会抓取一些数据，然后将其保存到返回的列表中，然后我将使用它来比较这些链接之一中是否出现了新链接：

"""
def getScrapeLinks(self, siteURL):
    response = requests.get(
                    siteURL,
                    timeout=5
                )

    if response.ok:
        bs4 = soup(response.text, 'lxml')

        links = ['{}'.format( raw_product.find('a').get('href')) for
                    raw_product in bs4.find_all('div', {'class': 'test'})]

        return links

"""

def pollNewProducts(self, storeClass):

    # storeClass.siteCatalog = ["https://www.google.com", "https://www.facebook.com"]

    LinksLists = reduce(operator.add, [getScrapeLinks(getLinks) for getLinks in storeClass.siteCatalog])

    while True:

        newLinksLists = reduce(operator.add,
                                 [getScrapeLinks(getLinks) for getLinks in storeClass.siteCatalog]
                                 )

        for URL in newLinksLists:
            if URL not in LinksLists:
                print("New link")
                print(URL)
                LinksLists.append(URL)
        else:
            print("Sleep to see new links!")
            time.sleep(random.randint(2, 4))

现在我的问题是我使用“减少”功能，它首先对 etc Google 进行第一次请求，然后获取所需的数据，然后一旦完成，我就会执行第二个请求，即 Facebook。我在这里要做的是加速它，使每个链接都有自己的线程，这样它就可以同时运行，而不是依赖于每个链接。

我想知道，如果 GET 请求中出现新 URL，我怎样才能自己运行每个链接，并且仍然能够比较并获取新 URL？

标签： pythonmultithreadingfor-loop

Answer 1

改编自我对这个问题的回答。

---

您应该研究异步编程。与线程不同，异步代码在同一个线程中运行，但它在事件循环中运行。当存在 Python 关键字时，此事件循环会自动在不同操作之间切换上下文await。

换句话说，将抓取网站视为以下内容：

client sends request -> ... waiting for server reply ... <- server replies

发送请求是一项耗时极少且几乎不消耗资源的操作。实时消费者正在等待服务器响应，然后处理服务器的回复。相反，如果我们执行类似于以下的操作：

client sends request -> switch operation -> ... wait ... <- server replies
client sends request -> switch operation -> ... wait ... <- server replies
client sends request -> switch operation -> ... wait ... <- server replies
...

然后我们可以最大限度地减少等待服务器回复的时间，而不是已经在拍摄下一个请求。换句话说，我们可以有效地做的是告诉 Python 发送请求，然后立即切换到我们的代码的不同部分发送另一个请求，然后另一个部分发送另一个请求，依此类推。发送完所有请求后，我们可以返回并开始解释各个服务器的回复。

网上有很多关于如何在 Python 中异步编程的参考资料（使用内置asynchro模块 + PyPi 可安装aiohttp模块），我建议谷歌搜索一下。这是一个代码示例，它将花费不到 4 秒的时间来抓取 100 多个网站（请注意，这非常好，4 秒实际上是由于打印语句......没有，它实际上更接近 2 秒）：

import asyncio
import aiohttp
import time


websites = """https://www.youtube.com
https://www.facebook.com
https://www.baidu.com
https://www.yahoo.com
https://www.amazon.com
https://www.wikipedia.org
http://www.qq.com
https://www.google.co.in
https://www.twitter.com
https://www.live.com
http://www.taobao.com
https://www.bing.com
https://www.instagram.com
http://www.weibo.com
http://www.sina.com.cn
https://www.linkedin.com
http://www.yahoo.co.jp
http://www.msn.com
http://www.uol.com.br
https://www.google.de
http://www.yandex.ru
http://www.hao123.com
https://www.google.co.uk
https://www.reddit.com
https://www.ebay.com
https://www.google.fr
https://www.t.co
http://www.tmall.com
http://www.google.com.br
https://www.360.cn
http://www.sohu.com
https://www.amazon.co.jp
http://www.pinterest.com
https://www.netflix.com
http://www.google.it
https://www.google.ru
https://www.microsoft.com
http://www.google.es
https://www.wordpress.com
http://www.gmw.cn
https://www.tumblr.com
http://www.paypal.com
http://www.blogspot.com
http://www.imgur.com
https://www.stackoverflow.com
https://www.aliexpress.com
https://www.naver.com
http://www.ok.ru
https://www.apple.com
http://www.github.com
http://www.chinadaily.com.cn
http://www.imdb.com
https://www.google.co.kr
http://www.fc2.com
http://www.jd.com
http://www.blogger.com
http://www.163.com
http://www.google.ca
https://www.whatsapp.com
https://www.amazon.in
http://www.office.com
http://www.tianya.cn
http://www.google.co.id
http://www.youku.com
https://www.example.com
http://www.craigslist.org
https://www.amazon.de
http://www.nicovideo.jp
https://www.google.pl
http://www.soso.com
http://www.bilibili.com
http://www.dropbox.com
http://www.xinhuanet.com
http://www.outbrain.com
http://www.pixnet.net
http://www.alibaba.com
http://www.alipay.com
http://www.chrome.com
http://www.booking.com
http://www.googleusercontent.com
http://www.google.com.au
http://www.popads.net
http://www.cntv.cn
http://www.zhihu.com
https://www.amazon.co.uk
http://www.diply.com
http://www.coccoc.com
https://www.cnn.com
http://www.bbc.co.uk
https://www.twitch.tv
https://www.wikia.com
http://www.google.co.th
http://www.go.com
https://www.google.com.ph
http://www.doubleclick.net
http://www.onet.pl
http://www.googleadservices.com
http://www.accuweather.com
http://www.googleweblight.com
http://www.answers.yahoo.com"""


async def get(url):
    try:
        async with aiohttp.ClientSession() as session:
            async with session.get(url=url) as response:
                resp = await response.read()
                print("Successfully got url {} with response of length {}.".format(url, len(resp)))
    except Exception as e:
        print("Unable to get url {} due to {}.".format(url, e.__class__))


async def main(urls, amount):
    ret = await asyncio.gather(*[get(url) for url in urls])
    print("Finalized all. ret is a list of len {} outputs.".format(len(ret)))


urls = websites.split("\n")
amount = len(urls)

start = time.time()
asyncio.run(main(urls, amount))
end = time.time()

print("Took {} seconds to pull {} websites.".format(end - start, amount))

输出：

Successfully got url http://www.google.com.br with response of length 12188.
Successfully got url http://www.google.it with response of length 12155.
Successfully got url https://www.t.co with response of length 0.
Successfully got url http://www.msn.com with response of length 46335.
Successfully got url http://www.chinadaily.com.cn with response of length 122053.
Successfully got url https://www.google.co.in with response of length 11557.
Successfully got url https://www.google.de with response of length 12135.
Successfully got url https://www.facebook.com with response of length 115258.
Successfully got url http://www.gmw.cn with response of length 120866.
Successfully got url https://www.google.co.uk with response of length 11540.
Successfully got url https://www.google.fr with response of length 12189.
Successfully got url http://www.google.es with response of length 12163.
Successfully got url http://www.google.co.id with response of length 12169.
Successfully got url https://www.bing.com with response of length 117915.
Successfully got url https://www.instagram.com with response of length 36307.
Successfully got url https://www.google.ru with response of length 12128.
Successfully got url http://www.googleusercontent.com with response of length 1561.
Successfully got url http://www.xinhuanet.com with response of length 179254.
Successfully got url http://www.google.ca with response of length 11592.
Successfully got url http://www.accuweather.com with response of length 269.
Successfully got url http://www.googleadservices.com with response of length 1561.
Successfully got url https://www.whatsapp.com with response of length 77951.
Successfully got url http://www.cntv.cn with response of length 3139.
Successfully got url http://www.google.com.au with response of length 11579.
Successfully got url https://www.example.com with response of length 1270.
Successfully got url http://www.google.co.th with response of length 12151.
Successfully got url https://www.amazon.com with response of length 465905.
Successfully got url https://www.wikipedia.org with response of length 76240.
Successfully got url https://www.google.co.kr with response of length 12211.
Successfully got url https://www.apple.com with response of length 63322.
Successfully got url http://www.uol.com.br with response of length 333257.
Successfully got url https://www.aliexpress.com with response of length 59742.
Successfully got url http://www.sohu.com with response of length 215201.
Successfully got url https://www.google.pl with response of length 12144.
Successfully got url https://www.googleweblight.com with response of length 0.
Successfully got url https://www.cnn.com with response of length 1138392.
Successfully got url https://www.google.com.ph with response of length 11561.
Successfully got url https://www.linkedin.com with response of length 71498.
Successfully got url https://www.naver.com with response of length 176038.
Successfully got url https://www.live.com with response of length 3667.
Successfully got url https://www.twitch.tv with response of length 61599.
Successfully got url http://www.163.com with response of length 696338.
Successfully got url https://www.ebay.com with response of length 307068.
Successfully got url https://www.wordpress.com with response of length 76680.
Successfully got url https://www.wikia.com with response of length 291400.
Successfully got url http://www.chrome.com with response of length 161223.
Successfully got url https://www.twitter.com with response of length 291741.
Successfully got url https://www.stackoverflow.com with response of length 105987.
Successfully got url https://www.netflix.com with response of length 83125.
Successfully got url https://www.tumblr.com with response of length 78110.
Successfully got url http://www.doubleclick.net with response of length 129901.
Successfully got url https://www.yahoo.com with response of length 531829.
Successfully got url http://www.soso.com with response of length 174.
Successfully got url https://www.microsoft.com with response of length 187549.
Successfully got url http://www.office.com with response of length 89556.
Successfully got url http://www.alibaba.com with response of length 167978.
Successfully got url https://www.reddit.com with response of length 483295.
Successfully got url http://www.outbrain.com with response of length 24432.
Successfully got url http://www.tianya.cn with response of length 7941.
Successfully got url https://www.baidu.com with response of length 156768.
Successfully got url http://www.diply.com with response of length 3074314.
Successfully got url http://www.blogspot.com with response of length 94478.
Successfully got url http://www.popads.net with response of length 14548.
Successfully got url http://www.answers.yahoo.com with response of length 104726.
Successfully got url http://www.blogger.com with response of length 94478.
Successfully got url http://www.imgur.com with response of length 4008.
Successfully got url http://www.qq.com with response of length 244841.
Successfully got url http://www.paypal.com with response of length 45587.
Successfully got url http://www.pinterest.com with response of length 45692.
Successfully got url http://www.github.com with response of length 86917.
Successfully got url http://www.zhihu.com with response of length 31473.
Successfully got url http://www.go.com with response of length 594291.
Successfully got url http://www.fc2.com with response of length 34546.
Successfully got url https://www.amazon.de with response of length 439209.
Successfully got url https://www.youtube.com with response of length 439571.
Successfully got url http://www.bbc.co.uk with response of length 321966.
Successfully got url http://www.tmall.com with response of length 234388.
Successfully got url http://www.imdb.com with response of length 289339.
Successfully got url http://www.dropbox.com with response of length 103714.
Successfully got url http://www.bilibili.com with response of length 50959.
Successfully got url http://www.jd.com with response of length 18105.
Successfully got url http://www.yahoo.co.jp with response of length 18565.
Successfully got url https://www.amazon.co.jp with response of length 479721.
Successfully got url http://www.craigslist.org with response of length 59372.
Successfully got url https://www.360.cn with response of length 74502.
Successfully got url http://www.ok.ru with response of length 170516.
Successfully got url https://www.amazon.in with response of length 460696.
Successfully got url http://www.booking.com with response of length 408992.
Successfully got url http://www.yandex.ru with response of length 116661.
Successfully got url http://www.nicovideo.jp with response of length 107271.
Successfully got url http://www.onet.pl with response of length 720657.
Successfully got url http://www.alipay.com with response of length 21698.
Successfully got url https://www.amazon.co.uk with response of length 443607.
Successfully got url http://www.sina.com.cn with response of length 579107.
Successfully got url http://www.hao123.com with response of length 295213.
Successfully got url http://www.pixnet.net with response of length 6295.
Successfully got url http://www.coccoc.com with response of length 45822.
Successfully got url http://www.taobao.com with response of length 393128.
Successfully got url http://www.weibo.com with response of length 95482.
Successfully got url http://www.youku.com with response of length 762485.
Finalized all. ret is a list of len 100 outputs.
Took 3.899034023284912 seconds to pull 100 websites.

如您所见，在我的互联网连接（迈阿密，佛罗里达州）上使用 aiohttp 在大约 4 秒内成功访问了来自世界各地的 100 个网站（使用或不使用 https）。请记住，以下内容可能会使程序减慢几毫秒：

• 打印语句（是的，包括上面代码中的语句）。
• 到达远离您的地理位置的服务器。

上面的示例具有上述两个实例，因此可以说它是执行您所要求的最优化的方式。但是，我相信这对于您正在寻找的东西来说是一个很好的开始。