About

Streaming real-time (Kafka) data to Iceberg is a core case in modern data infrastructure. However, achieving latency, storage efficiency and ops overhead targets is still tough for data engineering teams.

Takeaways

• Accumulate then write – this approach buffers real-time data until it’s ready to write to Iceberg.
• Write then compact – write data immediately and fix using Iceberg’s compaction feature.
• Lambda 2.0 – A novel approach that tiers Iceberg storage into a physical cold set and a logical hot set.