python - 启用延迟确认后，超过软时间限制后 Celery 任务重试

Question

我有一个运行大约两分钟的任务来执行一些数据库操作。我启用了延迟确认，但我发现在超过软时间限制后任务从未重试。

因此，我添加了autoretry_for 和 retry_backoof，如我正在使用的 celery 版本的文档中所述。为迟到的任务重试是否是一种好习惯？

如果我没有设置，autoretry_for那么任务将不会从它终止的地方运行。

任务.py

@shared_task(soft_time_limit=20, autoretry_for=(SoftTimeLimitExceeded, ), retry_backoff=True)
def test_func_celery(current_time):
    fake = Faker()
    logger = logging.getLogger()
    time1 = time.perf_counter()
    main_time_end = current_time + 25
    logger.info('Task started here \n \n')

    while time.time() < main_time_end:
        logger.info('Before doing main operations')
        time.sleep(10)
        logger.info(f'Inside Main operation ===>>> {time.time()}')

        while True:
            name = fake.first_name()
            logger.info(f'Inside inner operation for {name}')
            logger.info(f'inner operation before starts for {name}')
            time.sleep(12)
            logger.info(f'Inside inner {name}===>>> {time.time()}')
            logger.info(f'inner operation after finishes for {name}')
            break
        logger.info('Completed Main operation')
        break

    if time.time() >= main_time_end:
        logger.info('Out from main operations')
    time2 = time.perf_counter()
    logger.info(f'Total time =======>>>>>>> {time2 - time1}')
    return None

视图.py

def home(request):
    current_time = time.time()
    test_func_celery.apply_async(args=(current_time, ), acks_late=True)
    return HttpResponse("Celery being tested here")

芹菜日志

[2020-01-21 15:28:30,708: INFO/ForkPoolWorker-7] Task started here 
 

[2020-01-21 15:28:30,708: INFO/ForkPoolWorker-7] Before doing main operations
[2020-01-21 15:28:40,715: INFO/ForkPoolWorker-7] Inside Main operation ===>>> 1579599820.7153742
[2020-01-21 15:28:40,716: INFO/ForkPoolWorker-7] Inside inner operation for Joe
[2020-01-21 15:28:40,716: INFO/ForkPoolWorker-7] inner operation before starts for Joe
[2020-01-21 15:28:50,680: WARNING/MainProcess] Soft time limit (20s) exceeded for home.tasks.test_func_celery[4234d05e-672e-4565-bdb6-cd3d234c82c4]
[2020-01-21 15:28:50,726: INFO/MainProcess] Received task: home.tasks.test_func_celery[4234d05e-672e-4565-bdb6-cd3d234c82c4]  ETA:[2020-01-21 15:28:51.683861+05:45] 
[2020-01-21 15:28:50,728: INFO/ForkPoolWorker-7] Task home.tasks.test_func_celery[4234d05e-672e-4565-bdb6-cd3d234c82c4] retry: Retry in 1s: SoftTimeLimitExceeded()
[2020-01-21 15:28:51,938: INFO/ForkPoolWorker-7] Task started here 
 

[2020-01-21 15:28:51,938: INFO/ForkPoolWorker-7] Before doing main operations
[2020-01-21 15:29:01,948: INFO/ForkPoolWorker-7] Inside Main operation ===>>> 1579599841.948412
[2020-01-21 15:29:01,949: INFO/ForkPoolWorker-7] Inside inner operation for Lisa
[2020-01-21 15:29:01,949: INFO/ForkPoolWorker-7] inner operation before starts for Lisa
[2020-01-21 15:29:11,915: WARNING/MainProcess] Soft time limit (20s) exceeded for home.tasks.test_func_celery[4234d05e-672e-4565-bdb6-cd3d234c82c4]
[2020-01-21 15:29:11,965: INFO/MainProcess] Received task: home.tasks.test_func_celery[4234d05e-672e-4565-bdb6-cd3d234c82c4]  ETA:[2020-01-21 15:29:12.917995+05:45] 
[2020-01-21 15:29:11,967: INFO/ForkPoolWorker-7] Task home.tasks.test_func_celery[4234d05e-672e-4565-bdb6-cd3d234c82c4] retry: Retry in 1s: SoftTimeLimitExceeded()
[2020-01-21 15:29:13,136: INFO/ForkPoolWorker-7] Task started here 
 

[2020-01-21 15:29:13,136: INFO/ForkPoolWorker-7] Out from main operations
[2020-01-21 15:29:13,136: INFO/ForkPoolWorker-7] Total time =======>>>>>>> 0.00026752900157589465
[2020-01-21 15:29:13,138: INFO/ForkPoolWorker-7] Task home.tasks.test_func_celery[4234d05e-672e-4565-bdb6-cd3d234c82c4] succeeded in 0.05194566200225381s: None

有没有更好的解决方案来描述上述场景？这将是一个非常大的帮助。谢谢你。

标签： pythondjangocelery