Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions

Pinterest has introduced an event time-based partition finalization mechanism for its next-generation database ingestion framework, allowing downstream jobs to determine when streaming data is sufficiently complete to consume. The approach uses Apache Flink to measure event time progress and Apache Iceberg metadata to publish a non-regressing finalization watermark, addressing a challenge in continuous Change Data Capture (CDC) ingestion where data can arrive after a partition’s wall clock interval has ended.

Pinterest’s ingestion framework uses CDC, Kafka, Flink, Spark and Iceberg to move database changes from MySQL, TiDB and KVStore into analytical tables. Pinterest says the platform supports petabyte-scale data and thousands of pipelines, with online changes typically available within 15 minutes to an hour. Unlike the previous batch-based approach, where job completion indicated that a partition was ready, streaming ingestion continuously commits changes without an equivalent completion point.

Pinterest architecture showing Kafka CDC events flowing through Flink into a CDC Iceberg table (Source: Pinterest Blog Post)

Nitesh Singhal, Architect at Google, highlighted the design in a LinkedIn post, writing,

The non-regressing safe-to-read watermark is the elegant bit here.

The new mechanism runs inside the Flink to Iceberg sink. For each CDC record, the system extracts a configured event time. During Flink checkpoints, it collects minimum, maximum, and percentile event time statistics. These are stored in the Iceberg snapshot summary, with a t-digest representing the distribution in a few hundred bytes.

After an Iceberg commit, the system calculates a partition finalization watermark from recent checkpoint statistics. The default policy uses the minimum event time across a configurable number of recent commits and rounds it down to the relevant hour. The watermark is monotonic, preventing late-arriving records from moving an already published value backward. Percentile-based policies can trade some completeness for fresher data.

Pinterest describes the change by noting the watermark is stored as an Iceberg table property, allowing downstream sensors to read it without a separate coordination service. Spark also propagates the metadata to the base table during its CDC upsert commit and emphasizes

Finalization is no longer a byproduct of a job finishing.

The design builds on existing Flink and Iceberg capabilities. Flink watermarks track event time progress in out-of-order streams, while Iceberg snapshots provide persistent table metadata. Pinterest uses these capabilities together to derive a table-specific completeness signal and store it alongside the data.

Timeline showing late-arriving CDC events, Flink checkpoints, event time progress and the monotonically increasing finalization watermark. (Source: Pinterest Blog Post)

Tharun Kumar Reddy Polu, Engineer at Salesforce, identified the streaming and batch tradeoff in a LinkedIn comment, writing,

The streaming vs batch tradeoff framing is what makes this land. Getting freshness without losing the guarantees batch gave us is the core challenge.

Pinterest implemented the mechanism using extensibility points in its Flink-to-Iceberg sink. Accumulators collect record statistics, while commit processors execute finalization logic around Iceberg commits. Separately, an Apache Iceberg issue discusses extending IcebergSink for downstream integrations, including custom committable metadata such as watermark information.