时间戳

首页 > 解决方案 > 如何使用 aiohttp 将一组请求发布到 2 个 url

python-3.x - 如何使用 aiohttp 将一组请求发布到 2 个 url

问题描述

我有 2 个 URL 和 60k+ 请求。基本上，我需要将每个请求都发布到两个 URL，然后比较它们的响应，而不是等待响应发布另一个请求。

我试着用aiohttp和asyncio

import asyncio
import time
import aiohttp
import os

from aiofile import AIOFile

testURL = ""
prodURL = ""

directoryWithRequests = ''
directoryToWrite = ''

headers = {'content-type': 'application/soap+xml'}

i = 1


async def fetch(session, url, reqeust):
    global i
    async with session.post(url=url, data=reqeust.encode('utf-8'), headers=headers) as response:
        if response.status != 200:
            async with AIOFile(directoryToWrite + str(i) + '.xml', 'w') as afp:
                await afp.write(reqeust)
                i += 1
        return await response.text()


async def fetch_all(session, urls, request):
    results = await asyncio.gather(*[asyncio.create_task(fetch(session, url, request)) for url in urls])
    return results


async def asynchronousRequests(requestBody):
    urls = [testURL, prodURL]
    global i
    with open(requestBody) as my_file:
        body = my_file.read()

    async with aiohttp.ClientSession() as session:
        htmls = await fetch_all(session, urls, body)
        # some conditions

async def asynchronous():
    try:
        start = time.time()
        futures = [asynchronousRequests(directoryWithRequests + i) for i in os.listdir(directoryWithRequests)]
        for future in asyncio.as_completed(futures):
            result = await future

        print("Process took: {:.2f} seconds".format(time.time() - start))

    except Exception as e:
        print(str(e))

if __name__ == '__main__':
    try:
        # AsyncronTest
        ioloop = asyncio.ProactorEventLoop()
        ioloop.run_until_complete(asynchronous())
        ioloop.close()
        if i == 1:
            print('Regress is OK')
        else:
            print('Number of requests to check = {}'.format(i))

    except Exception as e:
        print(e)

我相信上面的代码有效，但它创建了 N 个期货，其中 N 等于请求文件的数量。这带来了某种 ddos​​，因为服务器无法同时响应该数量的请求。

标签： python-3.xpython-asyncioaiohttp

解决方案

找到合适的解决方案。基本上它只是 2 个异步任务：

tasks = [
                postRequest(testURL, client, body),
                postRequest(prodURL, client, body)
            ]
await asyncio.wait(tasks)

它与问题中的代码具有可承受数量的请求的性能不同，但至少它不会对服务器造成太多影响。

推荐阅读