首页 > 解决方案 > Apache flink: How to forward elements in timeWinow to process function when window closes?

问题描述

My Flink pseudo-code

timeWindow(Time.seconds(10))
.trigger(onElement -> if total size of elements in window > 100KB: FIRE_AND_PURGE or CONTINUE)
.process(print(input_collection))

I have timewindow of 10 seconds. I have trigger that sends (FIRE_AND_PURGE) contents of window(when total size of all elements in window reaches 100KB) to window process function But after 10 seconds when time window is closed , if I have 80KB of data in window then that residual data is not sent to window process function. I am loosing that 80KB of data … Whats right approach for it so that I dont loose that 80KB of remaining elements in timeWindow when timeWindow is closed at 10th second (edited)

For example, if I generate 230KB of data in 10 seconds then first 2 100KBs are sent to process function(by trigger by FIRE_AND_PURGE) but when timeWindow closes at 10th second, remaining 30KB is lost (edited)

Whats right approach to forward that 30KB to process function when time window closes ? (edited)

标签: apache-flink

解决方案


When the event or processing time window expires, don't you get a call to your Trigger's onEventTime() or onProcessingTime() method that you could use to purge contents, the same as what you're doing in your onElement() method?


推荐阅读