python - 如何避免芹菜截断“django_celery_results_taskresult”中的“task_args”文本

Question

在我的 Django 项目中，我使用 Celery 异步运行我的任务。

我面临的问题是“”中的“ task_args”列django_celery_results_taskresult被截断（即文本后跟省略号）。

例如，我期望“ ”中的整个“ process_4_items”值task_args。然而，事实并非如此。我可以在执行中的 Celery worker 的命令提示符中看到相同的截断文本（见下文“ [CELERY WORKER COMMAND PROMPT FOR ONE ITERATION]”）。

我真的不确定我需要做什么才能获得整个 " " 中的process_4_items" 值task_args。"

我在下面详细说明了我认为我需要复制这个项目的所有内容。

最重要的是，当我调试项目并逐行执行时，“ process_4_items”值的全部内容出现在“ task_args.”中。

[celeryconfig.py]

broker_url = 'amqp://localhost:5672//'

imports = ('operation_hawkeye.tasks',)

result_backend = 'django-db'

accept_content = ['json']

task_always_eager = False

[views.py]

from celery import group

process_4_items = [

{

"service_id": "7f7962b082cb8da55e01f4f013d8fefbc97948ec03400c93423b050d",

"service_id_unique": "c30ed891987f4d8d3debcaef7edb0f582e6b634f1ebed00fb88f0ec7",

"service_type": "buzz_collect_compare",

"service_file_pf_folder": "",

"service_file_nanonsportsproductivity_folder": "",

"service_file_tba_report": "",

"service_file_buzz_collect_compare": "services/buzzcollect_xml_compare/2021/06/01/file_2_name_Update_COOKINGHD_May_2021_Listin_yWoWXo1.xml",

"service_file_type": "XML 1.0 document, ASCII text",

"service_file_name": "file_2_name_Update_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml",

"service_file_size": 396557,

"service_status": "In Queue for Upload",

"service_status_remarks": {

"source_information": {

"source_id": "5191",

"source_name": "10NEWS2",

"file_1_name": "Original_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml",

"file_2_name": "Update_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml"

}

},

"service_creation_date": "2021-06-01T18:18:40.968Z",

"service_owner": "[e-mail]"

},

{

"service_id": "7f7962b082cb8da55e01f4f013d8fefbc97948ec03400c93423b050d",

"service_id_unique": "2edaf2535945992d13b9ee85e908b21cbd520a02642bd7faf7f229bc",

"service_type": "buzz_collect_compare",

"service_file_pf_folder": "",

"service_file_nanonsportsproductivity_folder": "",

"service_file_tba_report": "",

"service_file_buzz_collect_compare": "services/buzzcollect_xml_compare/2021/06/01/file_1_name_Original_COOKINGHD_May_2021_List_P2PlYY7.xml",

"service_file_type": "XML 1.0 document, ASCII text",

"service_file_name": "file_1_name_Original_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml",

"service_file_size": 396409,

"service_status": "In Queue for Upload",

"service_status_remarks": {

"source_information": {

"source_id": "5191",

"source_name": "10NEWS2",

"file_1_name": "Original_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml",

"file_2_name": "Update_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml"

}

},

"service_creation_date": "2021-06-01T18:18:40.935Z",

"service_owner": "[e-mail]"

}

]

jobs_4 = group(tasks.buzz_collect_compare_main.s(process_4_items))

result_4 = jobs_4.apply_async()

[tasks.py]

@app.task(bind=True)

def buzz_collect_compare_main(self, items):

process_items = items

...

[CELERY WORKER COMMAND PROMPT FOR ONE ITERATION]

[2021-06-01 09:37:29,972: DEBUG/ForkPoolWorker-32] (0.010)

UPDATE

"django_celery_results_taskresult"

SET

"task_id" = 'daca83c1-3297-4e64-8a5e-f4b61aef84e2',

"task_name" = 'operation_hawkeye.tasks.buzz_collect_compare_main',

"task_args" = '([{''service_id'': ''dc30fc45948975044cb00c39a4d5a0db8e28bc34f0b4d52fd4dcddef'', ''service_id_unique'': ''2004e35ae5dcc2586ae5fca16ab8a5b2708ffeca233d3bb234feb347'', ''service_type'': ''buzz_collect_compare'', ''service_file_pf_folder'': '''', ''service_file_nanonsportsproductivity_folder'': '''', ''service_file_tba_report'': '''', ''service_file_buzz_collect_compare'': ''services/buzzcollect_xml_compare/2021/06/01/file_2_name_Update_COOKINGHD_May_2021_Listin_726wZgI.xml'', ''service_file_type'': ''XML 1.0 document, ASCII text'', ''service_file_name'': ''file_2_name_Update_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml'', ''service_file_size'': 396557, ''service_status'': ''In Queue for Upload'', ''service_status_remarks'': ''{"source_information": "{\"source_id\": \"27227\", \"source_name\": \"COOKINGHD\", \"file_1_name\": \"Original_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml\", \"file_2_name\": \"Update_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml\"}"}'', ''service_creation_date'': ''2021-06-01T09:36:56.825Z'', ''service_owner'': ''p...'', ...],)',

"task_kwargs" = '{}',

"status" = 'PROGRESS',

"worker" = 'celery@AMD-RYZEN-UBUNTU',

"content_type" = 'application/json',

"content_encoding" = 'utf-8',

"result" = '{"pending": false, "current": 1254, "total": 1254, "percent": 100.0, "description": "Building row 1254 of 1254"}',

"date_created" = '2021-06-01T09:36:58.465884+00:00'::timestamptz,

"date_done" = '2021-06-01T09:37:29.962267+00:00'::timestamptz,

"traceback" = NULL,

"meta" = '{"children": []}'

WHERE

"django_celery_results_taskresult"."id" = 808;

args=('daca83c1-3297-4e64-8a5e-f4b61aef84e2', 'operation_hawkeye.tasks.buzz_collect_compare_main', '([{\'service_id\': \'dc30fc45948975044cb00c39a4d5a0db8e28bc34f0b4d52fd4dcddef\', \'service_id_unique\': >\'2004e35ae5dcc2586ae5fca16ab8a5b2708ffeca233d3bb234feb347\', \'service_type\': \'buzz_collect_compare\', \'service_file_pf_folder\': \'\', \'service_file_nanonsportsproductivity_folder\': \'\', \'service_file_tba_report\': \'\', \'service_file_buzz_collect_compare\': \'services/buzzcollect_xml_compare/2021/06/01/file_2_name_Update_COOKINGHD_May_2021_Listin_726wZgI.xml\', \'service_file_type\': \'XML 1.0 document, ASCII text\', \'service_file_name\': \'file_2_name_Update_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml\', \'service_file_size\': 396557, \'service_status\': \'In Queue for Upload\', \'service_status_remarks\': \'{"source_information": "{\\"source_id\\": \\"27227\\", \\"source_name\\": \\"COOKINGHD\\", \\"file_1_name\\": \\"Original_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml\\", \\"file_2_name\\": \\"Update_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml\\"}"}\', \'service_creation_date\': \'2021-06-01T09:36:56.825Z\', \'service_owner\': \'p...\', ...],)', '{}', 'PROGRESS', 'celery@AMD-RYZEN-UBUNTU', 'application/json', 'utf-8', '{"pending": false, "current": 1254, "total": 1254, "percent": 100.0, "description": "Building row 1254 of 1254"}', datetime.datetime(2021, 6, 1, 9, 36, 58, 465884, tzinfo=<UTC>), datetime.datetime(2021, 6, 1, 9, 37, 29, 962267, tzinfo=<UTC>), '{"children": []}', 808)

[$ celery -A hawkeye report]

software -> celery:4.4.7 (cliffs) kombu:4.6.11 py:3.8.6

billiard:3.6.3.0 py-amqp:2.6.1

platform -> system:Linux arch:64bit, ELF

kernel version:5.8.0-53-generic imp:CPython

loader -> celery.loaders.app.AppLoader

settings -> transport:amqp results:django-db

accept_content: ['json']

broker_url: 'amqp://guest:********@localhost:5672//'

imports: ('operation_hawkeye.tasks',)

result_backend: 'django-db'

task_always_eager: False

[更新]

通过定义 中的argsrepr_maxsize值解决了这个问题celery.py。虽然我之前知道这个解决方案，但我没有意识到它argsrepr_maxsize在celery.ampq.

[celery.py]

import os

from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'hawkeye.settings')

os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')

app = Celery('hawkeye')

default_config = 'hawkeye.celeryconfig'

app.config_from_object(default_config)

app.amqp.argsrepr_maxsize = 10485760

标签： pythonjsonasynchronouscelerydjango-celery