Batch processingand stream processing are two very different models for processing data. Both have their strengths but suit different use cases. In this post we cover the differences, provide examples of use cases, and look at the ways the two models can work together.
A common requirement in the area of data engineering is to first process existing historical data before processing continuously live data. Processing existing data first is also referred to as bootstrapping the system. How to easily achieve this with Apache Flink? In this blog-post we will look at Flink's HybridSource which is specifically designed for such a task. If you want to clone the repository with the code from this blog post, use
Since its inception, Apache Flink has undergone significant evolution. Today, it not only serves as a unified engine for both batch and streaming data processing but also paves the way toward a new era of streaming data warehouses. Apache Flink has the concept of Dynamic Tables, which bear resemblance to materialized views in databases. However, unlike materialized views, Dynamic Tables are not directly queryable. Recognizing the need to support querying of intermediate tables
Every year, Apache Flink® sets new records in its development journey. Standing as a testament to its growing popularity, Flink now boosts over 1.6k contributors, 21k GitHub stars, and 1.4M downloads. In operational environments, Flink clusters are reaching impressive scales, with some individual clusters surpassing 2000 nodes. The largest known Flink infrastructure in production boasts over 4 million CPU cores, processing a staggering 4.1B events per second. If scalability is a concern
In October, at Flink Forward 2023, Streamhouse was officially introduced by Jing Ge, Head of Engineering at Ververica. In his keynote, Jing highlighted the need for Streamhouse, including how it sits as a layer between real-time stream processing and Lakehouse architectures, and discussed the business value it provides.
In this blog post, you will learn how to build a real-time data view on top of your Streamhouse using Apache Paimon table format. If you are coming from the Data Management world, you might know that Data engineers are generally concerned about implementing a data analytics pipeline, minimizing compute-infrastructure cost, and achieving the smallest end-to-end latency for the target users.
推荐系统是一种信息过滤系统，用于预测用户偏好，从大量的信息中筛选出用户可能感兴趣的内容进行个性化推荐。一个完整的推荐系统流程主要包括了 多路召回 -> 素材补全 -> 精排过滤 -> 混排 ->适配输出 等处理节点。混排作为结果输出前的最后一层处理，主要作用是将不同来源的推荐结果进行归一化的组合排序，一方面是为了获取对于用户推荐效果最优的排序序列，另一方面也能提高推荐的多样性、个性化以及覆盖范围。
