Key concepts of Stream Processing
There are the following concepts that a user should know about stream processing:
It is essential as well as the most confusing concept. In stream processing, most operations rely on time. Therefore, a common notion of time is a typical task for such stream applications.
Kafka Stream processing refers to following notions of time:
There are different states maintained in the stream processing applications.
The states are:
A Table is a collection of records which is uniquely identified by a primary key. Queries are fired to check the state of data at a specific point of time. Tables do not contain history, specifically unless we design it. On the other hand, streams contain a history of changes. Streams are the strings of events where each event causes a change. Thus, tables and streams are two sides of the same coin. So, to convert a table into streams, the user needs to capture the commands which modify the table. The commands such as insert, update, and delete are captured and stored into streams. Also, if the user wants to convert streams into a table, it is required to convert all changes which a stream contains. This process of conversion is also called materializing the stream. So, we can have the dual process of changing streams into tables as well as tables to streams.
The term time windows means windowing the total time into parts. Therefore, there are some operations on streams which depend on the time window. Such operations are called Windowed operations. For example, join operation performed on two streams are windowed. Although people rarely care about the type of window they need for their operations.