时间戳

首页 > 解决方案 > 在 Apache Flink 的flapMap Operator 中发现“缓冲池被破坏”问题

apache-flink - 在 Apache Flink 的flapMap Operator 中发现“缓冲池被破坏”问题

问题描述

当我尝试在 flatMap 运算符中写入 OUT 集合时，我得到非法状态异常（仅在高负载下）：缓冲池被破坏 我在这里做错了什么？flink 什么时候抛出 Buffer pool 错误？

java.lang.RuntimeException:  Buffer pool is destroyed.

    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45)
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:718)
    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:696)
    at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)

..........
Caused by: java.lang.IllegalStateException:  Buffer pool is destroyed.
    at org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
    at org.apache.flink.runtime.io.network.buffer.BufferBuilder.append(BufferBuilder.java:79)
    at org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.copyToBufferBuilder(SpanningRecordSerializer.java:95)
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.copyFromSerializerToTargetChannel(RecordWriter.java:150)
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:128)
    at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:101)
    at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:81)
    at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:107)```

标签： apache-flink

解决方案

什么是缓冲池？

缓冲池是网络堆栈的固定大小的内存段实例池，这意味着您在发送数据时将段放入其中，并在通过网络接收时从其中获取段。

为什么会被摧毁？

由于网络环境关闭，缓冲池被破坏。

为什么 NetworkEnvironment 会关闭？

这可能是我不太确定的具体情况。我建议你阅读更多关于这个缓冲池异常的日志。

推荐阅读