Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size randomtimestamp-1.0.0-py3-none-any.whl (15.8 kB) | File type Wheel | Python version py3 | Upload date | Hashes |
Filename, size randomtimestamp-1.0.0.tar.gz (2.7 kB) | File type Source | Python version None | Upload date | Hashes |
Algorithm | Hash digest |
---|---|
SHA256 | ee41c62ccab7ddfdabc7bdb8d4597b90b65e5d1d0689192aac511f571c7eeb5c |
MD5 | 0eacd9e1a0404ede5915111568847dd4 |
BLAKE2-256 | ae8fac1c8e1ea0e15fe8e77654b9bec938d1bdf50bf9543975eec577af09adb5 |
Algorithm | Hash digest |
---|---|
SHA256 | 5bcd08c790b777419b382e788e99d3753c3ee58df3ec979b5726aa7c7af7cd99 |
MD5 | 4fbc316d34da1a87cacaf30eca57857b |
BLAKE2-256 | 08198e72289ef87daa333c796e2ac4d5c08845f10e5cfa6dacd3a476efa6ab8f |
TimestampAssigner
.WatermarkGenerator
.WatermarkStrategy
that contains both aTimestampAssigner
and WatermarkGenerator
. A number of common strategiesare available out of the box as static methods on WatermarkStrategy
, butusers can also build their own strategies when required.WatermarkStrategy
for common watermark strategies orto bundle together a custom TimestampAssigner
with a WatermarkGenerator
.For example, to use bounded-out-of-orderness watermarks and a lambda function as atimestamp assigner you use this:TimestampAssigner
is optional and in most cases you don’tactually want to specify one. For example, when using Kafka or Kinesis youwould get timestamps directly from the Kafka/Kinesis records.WatermarkGenerator
interface later in WritingWatermarkGenerators.WatermarkStrategy
can beused: 1) directly on sources and 2) after non-source operation.WatermarkStrategy
directly on thesource usually means you have to use a source specific interface/ Refer toWatermark Strategies and the KafkaConnector for how this works ona Kafka Connector and for more details about how per-partition watermarkingworks there.WatermarkStrategy
after arbitrary operations)should only be used if you cannot set a strategy directly on the source:WatermarkStrategy
this way takes a stream and produce a new streamwith timestamped elements and watermarks. If the original stream had timestampsand/or watermarks already, the timestamp assigner overwrites them.WatermarkGenerator
also does not get any new informationon which to base a watermark. We call this an idle input or an idle source.This is a problem because it can happen that some of your partitions do stillcarry events. In that case, the watermark will be held back, because it iscomputed as the minimum over all the different parallel watermarks.WatermarkStrategy
that will detect idlenessand mark an input as idle. WatermarkStrategy
provides a convenience helperfor this:TimestampAssigner
is a simple function that extracts a field from an event, we therefore don’t need to look at them in detail. A WatermarkGenerator
, on the other hand, is a bit more complicated to write and we will look at how you can do that in the next two sections. This is the WatermarkGenerator
interface:onEvent()
and then emits a watermark when the framework calls onPeriodicEmit()
.onEvent()
and wait for specialmarker events or punctuations that carry watermark information in thestream. When it sees one of these events it emits a watermark immediately.Usually, punctuated generators don’t emit a watermark from onPeriodicEmit()
.ExecutionConfig.setAutoWatermarkInterval(..)
. Thegenerators’s onPeriodicEmit()
method will be called each time, and a newwatermark will be emitted if the returned watermark is non-null and larger thanthe previous watermark.BoundedOutOfOrdernessWatermarks
, which is a WatermarkGenerator
that workssimilarly to the BoundedOutOfOrdernessGenerator
shown below. You can readabout using that here.TimestampAssigner
in the example, the timestamps of the Kafka recordsthemselves will be used instead.WindowOperator
willfirst evaluate all windows that should be fired, and only after producing all ofthe output triggered by the watermark will the watermark itself be sentdownstream. In other words, all elements produced due to occurrence of awatermark will be emitted before the watermark.TwoInputStreamOperator
. However, in this case thecurrent watermark of the operator is defined as the minimum of both of itsinputs.OneInputStreamOperator#processWatermark
,TwoInputStreamOperator#processWatermark1
andTwoInputStreamOperator#processWatermark2
methods.WatermarkStrategy
,TimestampAssigner
, and WatermarkGenerator
, Flink usedAssignerWithPeriodicWatermarks
and AssignerWithPeriodicWatermarks
. You willstill see them in the API but it is recommended to use the new interfacesbecause they offer a clearer separation of concerns and also unify periodic andpunctuated styles of watermark generation.