Data Engineering

Streaming ETL with Apache Flink: A 2026 Reference Architecture

5 min read

A production-grade streaming ETL reference architecture using Apache Flink, with state, checkpointing, and exactly-once semantics.

  1. 01Kafka source with watermark strategy

    Bounded out-of-orderness; idle source detection.

  2. 02Keyed state with RocksDB backend

    Spillable state for high-cardinality keys.

  3. 03Checkpointing at 60s with EXACTLY_ONCE mode

    S3 incremental checkpoints.

  4. 04Sink with two-phase commit

    To Iceberg or Kafka transactional sink.

Compliance-Grade RAG PipelineSOC 2 + audit-trail architectureDocuments14K pagesChunkingHierarchicalEmbeddingsVoyage-3pgvectorOn RDSRerankCohere v3Claude 3.7Citation-disciplinedAUDIT TRAIL · Every query → S3 Object Lock · 7-year retentionQuestion · Retrieved chunks (hashed) · Prompt version · Model version · Final answerZero audit findings · Q1 2026 SOC 296.4% answer satisfaction · 2,400+ queries/month · p95 retrieval 180ms
The compliance-grade RAG architecture Ohveda deployed for a regulated FinTech in Q4 2025.

Ready to optimize your cloud or AI footprint?

Book a free 30-minute architecture review. We will deliver a written cost-and-architecture audit within 48 hours.

Book a free architecture review · sales@ohveda.com

Need help with streaming ETL Flink?

Ohveda runs free 30-minute architecture reviews. We will identify your top opportunities in writing within 48 hours — at no cost.

Book a Free Architecture Review →