时间戳

首页 > 解决方案 > 芹菜工人不读取热解图会话文件

celery - 芹菜工人不读取热解图会话文件

问题描述

我正在尝试通过 celery 任务（调度等）执行 pyrogram 函数

该函数在通过 shell 运行时起作用：

from app_name.users.tasks import establish_session
establish_session()

通过将其发送到芹菜establish_session.delay()是问题出现的地方。

通过 celery 执行时完全相同的功能无法读取会话所需的会话文件。

我已经确认在这两种方法中都可以看到会话文件，并且具有 os.R_OK、os.W_OK、os.F_OK 的权限。

users.tasks

@shared_task
def establish_session():
    from utils.telegram import get_new_session
    user_bot = get_new_session()
    print(user_bot)

utils.telegram

def get_new_session():
    import os
    cwd = os.getcwd()
    print(cwd)
    print(os.access('user.session', os.R_OK)) # Check for read access
    print(os.access('user.session', os.W_OK)) # Check for write access
    print(os.access('user.session', os.X_OK)) # Check for execution access
    print(os.access('user.session', os.F_OK)) # Check for existence of file
    user_bot = Client("user", api_id=ID, api_hash=HASH)
    user_bot.start()
    user_bot.stop()
    return user_bot

输出差异：

建立会话（）

INFO 2021-01-23 18:07:21,379 connection Connecting...
INFO 2021-01-23 18:07:21,382 connection Connected! Production DC5 - IPv4 - TCPAbridgedO
INFO 2021-01-23 18:07:21,383 session NetworkTask started
INFO 2021-01-23 18:07:21,435 msg_id Time synced: 2021-01-23 10:07:21.439058 UTC
INFO 2021-01-23 18:07:21,439 session NextSaltTask started
INFO 2021-01-23 18:07:21,439 session Next salt in 33m 13s (at 2021-01-23 18:40:35)
INFO 2021-01-23 18:07:21,524 session Session initialized: Layer 122
INFO 2021-01-23 18:07:21,524 session Device: CPython 3.8.6 - Pyrogram 1.1.10
INFO 2021-01-23 18:07:21,524 session System: Linux 5.8.0-33-generic (EN)
INFO 2021-01-23 18:07:21,524 session Session started
INFO 2021-01-23 18:07:21,540 session PingTask started
INFO 2021-01-23 18:07:21,620 dispatcher Started 6 HandlerTasks
INFO 2021-01-23 18:07:21,632 syncer Synced "user" in 11.2832 ms
INFO 2021-01-23 18:07:21,639 syncer Synced "user" in 7.18236 ms
INFO 2021-01-23 18:07:21,640 dispatcher Stopped 6 HandlerTasks
INFO 2021-01-23 18:07:21,640 session PingTask stopped
INFO 2021-01-23 18:07:21,640 session NextSaltTask stopped
INFO 2021-01-23 18:07:21,640 connection Disconnected
INFO 2021-01-23 18:07:21,641 session NetworkTask stopped
INFO 2021-01-23 18:07:21,641 session Session stopped

对比

建立会话延迟（）

[2021-01-23 18:07:35,832: INFO/ForkPoolWorker-2] Start creating a new auth key on DC2
[2021-01-23 18:07:35,832: INFO/ForkPoolWorker-2] Connecting...
[2021-01-23 18:07:36,105: INFO/ForkPoolWorker-2] Connected! Production DC2 - IPv4 - TCPAbridgedO
[2021-01-23 18:07:37,592: INFO/ForkPoolWorker-2] Done auth key exchange: 
[2021-01-23 18:07:37,592: INFO/ForkPoolWorker-2] Disconnected
[2021-01-23 18:07:37,605: WARNING/ForkPoolWorker-2] Pyrogram v1.1.10, Copyright (C) 2017-2021 Dan <https://github.com/delivrance>
[2021-01-23 18:07:37,605: WARNING/ForkPoolWorker-2] Licensed under the terms of the GNU Lesser General Public License v3 or later (LGPLv3+)
[2021-01-23 18:07:37,605: INFO/ForkPoolWorker-2] Connecting...
[2021-01-23 18:07:37,875: INFO/ForkPoolWorker-2] Connected! Production DC2 - IPv4 - TCPAbridgedO
[2021-01-23 18:07:37,875: INFO/ForkPoolWorker-2] NetworkTask started
[2021-01-23 18:07:38,459: INFO/ForkPoolWorker-2] Time synced: 2021-01-23 10:07:38.353224 UTC
[2021-01-23 18:07:38,732: INFO/ForkPoolWorker-2] NextSaltTask started
[2021-01-23 18:07:38,732: INFO/ForkPoolWorker-2] Next salt in 44m 58s (at 2021-01-23 18:52:37)
[2021-01-23 18:07:39,096: INFO/ForkPoolWorker-2] Session initialized: Layer 122
[2021-01-23 18:07:39,096: INFO/ForkPoolWorker-2] Device: CPython 3.8.6 - Pyrogram 1.1.10
[2021-01-23 18:07:39,096: INFO/ForkPoolWorker-2] System: Linux 5.8.0-33-generic (EN)
[2021-01-23 18:07:39,096: INFO/ForkPoolWorker-2] Session started
[2021-01-23 18:07:39,099: WARNING/ForkPoolWorker-2] Enter phone number or bot token:
[2021-01-23 18:07:39,099: INFO/ForkPoolWorker-2] PingTask started
[2021-01-23 18:07:39,100: INFO/ForkPoolWorker-2] PingTask stopped
[2021-01-23 18:07:39,100: INFO/ForkPoolWorker-2] NextSaltTask stopped
[2021-01-23 18:07:39,100: INFO/ForkPoolWorker-2] Disconnected
[2021-01-23 18:07:39,101: INFO/ForkPoolWorker-2] NetworkTask stopped
[2021-01-23 18:07:39,101: INFO/ForkPoolWorker-2] Session stopped

非常感谢任何帮助！

标签： celerydjango-celerycelery-taskpyrogram

解决方案

我做了很多工作来让 pyrogram 在芹菜下工作。这并不理想，但它适用于我的情况。也许这也可以帮助你。

我正在使用最新版本的 pyrogram(1.3.5) 和 celery(5.2.3)


# first need to create a client, save session file in memory
tg_client=Client(":memory:",APP_ID=123,APP_HASH="abc")

# create celery app
app = Celery('tasks', broker=BROKER)

@app.task
def some_task():
    print(tg_client.get_me())

# define celery startup 

def run_celery():
    # pool must be threads
    argv = [
        "-A", "tasks", 'worker', '--loglevel=info',
        "--pool=threads"]
    app.worker_main(argv)


if __name__ == '__main__':
    tg_client.start() # <-- I think you can also put it in the first line of `run_celery`
    threading.Thread(target=run_celery, daemon=True).start()
    idle()
    celery_client.stop()

要点是：

  1. 需要在与主线程不同的线程中启动 celery worker，因为 pyrogram 是异步库，它依赖于主线程，而 celery 阻塞了主线程
  2. 芹菜池必须是线程或独奏

除此之外，您还可以在任务中使用 with

@app.task
def some_task()
    with tg_client():
        print(tg_client.get_me())

一些参考资料： https ://github.com/pyrogram/pyrogram/issues/480

https://github.com/tgbot-collection/ytdlbot/blob/master/ytdlbot/tasks.py

推荐阅读