logging - How to show tqdm progress in MetaFlow?

Question

When running MetaFlow Flows, tqdm progress bars do not get displayed until the final iteration, which defeats the purpose of measuring progress. Is there a way to force MetaFlow to print out tqdm updates?

标签： loggingtqdmnetflix-metaflow

Answer 1

The problem is that tqdm writes to stderr, but MetaFlow hides output to stderr. The solution requires two tricks: using a context manager to redirect tqdm outputs to a logger, and setting that logger to write to stderr.

Example:

import logging
import sys
from time import sleep

import tqdm
from metaflow import FlowSpec, step
from tqdm.contrib.logging import tqdm_logging_redirect

# stream to stdout needed because MetaFlow hides output to stderr :C
logging.basicConfig(level=logging.INFO, stream=sys.stdout)


class TQDMFlow(FlowSpec):
    @step
    def start(self):
        print("Training...")
        with tqdm_logging_redirect():  # this context manager redirects tqdm output to logging
            for _ in tqdm.tqdm(range(20)):
                sleep(0.25)
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    TQDMFlow()

I also tried redirecting output directly to stdout (without the tqdm_logging_redirect context manager or logging) using tqdm(range(n), file=sys.stdout) but that did not work.