Flink Watermark传播机制

flink应用程序会以数据流图的形式来调度执行。Watermark如何在不同算子传播呢下面分单并行数据流和多并行数据流来分别说明。Watermark传播策略满足以下规律上游算子中的Watermark以广播形式发送到下游算子。可以在Output接口的emitWatermark(Watermark mark)的注释看到。当调用该方法发送水位线时将以广播Watermark方式发送到下游算子。/** * A {link org.apache.flink.streaming.api.operators.StreamOperator} is supplied with an object of * this interface that can be used to emit elements and other messages, such as barriers and * watermarks, from an operator. * * param T The type of the elements that can be emitted. */PublicEvolvingpublicinterfaceOutputTextendsCollectorT{/** * Emits a {link Watermark} from an operator. This watermark is broadcast to all downstream * operators. * * pA watermark specifies that no element with a timestamp lower or equal to the watermark * timestamp will be emitted in the future. */voidemitWatermark(Watermarkmark);/** * Emits a record to the side output identified by the given {link OutputTag}. * * param record The record to collect. */Xvoidcollect(OutputTagXoutputTag,StreamRecordXrecord);voidemitLatencyMarker(LatencyMarkerlatencyMarker);}2.当数据流中Watermark的值为Long.MAX_VALUE时表示数据流结束不会有后续数据产生相当于一个终止标志。可以在Watermark源码看到定义的MAX_WATERMARKLong.MAX_VALUE标识事件时间的终止end-of-event-time。/** The watermark that signifies end-of-event-time. */publicstaticfinalWatermarkMAX_WATERMARKnewWatermark(Long.MAX_VALUE);对同一条数据流输入管道不管是多并行还是单并行它被下游算子接收后会取当前流最大的Watermark作为当前管道的Watermark输入。一般来说StreamTaskInput接口代表算子的输入其中StreamTaskNetworkInput是它的实现类代表从网络中获取的输入。StreamTaskNetworkInput类中的processElement方法