python - Python asyncio 任务列表进度记录
问题描述
我需要处理大量数据行,只有异步执行此操作才有意义。
我需要查看列表处理状态,即Done processing 1/3
,但是当我增加计数器时,它始终保持在 1。这是有道理的,因为我将计数器发送到函数中。我需要这样做,因为没有它,我会得到:
UnboundLocalError: local variable 'processed' referenced before assignment
使用 Python 3.8
任何帮助,将不胜感激!
这是一个测试链接:https ://ideone.com/gRjrf2
我在下面抽象了我的代码:
import os, logging
import asyncio
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s', datefmt='%d-%b-%y %H:%M:%S')
logger = logging.getLogger(__name__)
items = [{"name": "A"}, {"name": "B"}, {"name": "C"}]
processed = 0
async def increment(item):
count = item.get('count', 0)
count += 1
return count
async def get_and_update(item, processed):
item['count'] = await increment(item)
# Show progress now, but how?
processed += 1
logger.info(f"You can't see me {processed}")
async def run():
logger.info(f"Processing {len(items)} items...")
await asyncio.gather(*[
asyncio.create_task(
get_and_update(item, processed)
) for item in items
])
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
我得到的输出是:
28-Aug-20 11:19:22 INFO [prog.py:23] Processing 3 items...
28-Aug-20 11:19:22 INFO [prog.py:20] You can't see me 1
28-Aug-20 11:19:22 INFO [prog.py:20] You can't see me 1
28-Aug-20 11:19:22 INFO [prog.py:20] You can't see me 1
解决方案
您的基本问题是,通过将 声明processed
为参数get_and_update
,您正在隐藏全局processed
变量。您需要删除参数,然后processed
在该函数中声明为全局,如下所示:
import os, logging
import asyncio
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s', datefmt='%d-%b-%y %H:%M:%S')
logger = logging.getLogger(__name__)
items = [{"name": "A"}, {"name": "B"}, {"name": "C"}]
processed = 0
async def increment(item):
count = item.get('count', 0)
count += 1
return count
async def get_and_update(item):
global processed
item['count'] = await increment(item)
# Show progress now, but how?
processed += 1
logger.info(f"You can't see me {processed}")
async def run():
logger.info(f"Processing {len(items)} items...")
await asyncio.gather(*[
asyncio.create_task(
get_and_update(item)
) for item in items
])
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
上面的输出是:
28-Aug-20 08:15:00 INFO [asynctest.py:25] Processing 3 items...
28-Aug-20 08:15:00 INFO [asynctest.py:22] You can't see me 1
28-Aug-20 08:15:00 INFO [asynctest.py:22] You can't see me 2
28-Aug-20 08:15:00 INFO [asynctest.py:22] You can't see me 3
推荐阅读
- php - 每个用户的 PHP 循环 HTML 表
- html - 如何以响应方式浮动按钮?
- javascript - 使用 jQuery 更改类但元素不改变颜色?
- javascript - 如何让 chrome 使用代码键入密钥?
- node.js - 了解节点包管理器以及应用程序如何启动/运行
- python - 在不激活虚拟环境的情况下运行 Python 脚本
- javascript - 用 vanilla javascript 制作表单
- html - 在 Angular CLI 中提交后 ReactiveForms 刷新页面
- firebase - 更新firestore中文档的createdAt和updatedAt日期字段的正确方法是什么?
- angular - 角度 7 中的单击事件未触发输入类型复选框