python - Flask gevent threads are stuck on kafka consumer

Question

We have a flask app, deploying using gunicorn in gevent worker mode. The app itself listen to few REST apis but also have a kafka consumer (using confluent client) it listen to. When we deploy the app, kafka consumer work as expected but when submitting REST request to the app, they are not being handled by the app.

Seem like the app threads are stuck on the kafka consumer. If we remove the consumer code the app work as expected.

Any advice?

标签： pythonflaskgunicorngeventconfluent-platform

Answer 1

The issue was that the way we use kafka consumer is in a "while True"

while True:
    msg = self.client.poll(1.0)
    if msg is None:
        continue
    if msg.error():
        print("Consumer error: {}".format(msg.error()))
        continue
    self.handle_msg(msg.value())

Which cause the gevent to be stuck on it and not switching to other threads. The solution was to add a sleep command in the consumer loop which free the thread to handle REST requests as well