The Stream Processing Loop

The internal logic of a stream in the Event Stream Processor can be represented as a loop with states that correspond to ways the Event Stream Processor handles data.

A normal processing sequence proceeds as follows:
  1. INPUT

    The stream waits for the input queue to become non-empty, then picks a transaction from the head of the input queue. The transaction is visible as inTrans, the current input transaction. The transaction is processed row-by-row.

    The current output transaction is set to be empty, prepared to collect the results of processing.

  2. COMPUTE

    The next record is selected from the current input transaction. It becomes visible as inRow, the current input row. In some cases, the current input record may actually be two records, combined into an UPDATE_BLOCK.

    If this is not the first iteration of the loop, the records produced from processing the previous input record are still visible as outRow.

    If there are any input breakpoints defined on the stream, they are evaluated against the current input record, which may trigger an Event Stream Processor pause.

    The check as to whether the Event Stream Processor is paused is performed. If paused, the stream pauses here and waits for permission to continue.

    Finally, the actual computation is performed on the current input record. It produces zero or more output records. These records become visible as outRow, and are also appended to the end of outTrans. These records follow certain internal rules, and are not exactly the same as when they are published externally. For example, the update records at this point usually have the operation type UPSERT, and the delete records are SAFEDELETE.

    Any output breakpoints defined on the stream are evaluated against the current input record, which may trigger an Event Stream Processor pause.

    If there are more records left in the input transaction, the compute loop continues; otherwise, the stream proceeds to put the calculated data into the store, unless an exception such as division by zero has happened, in which case it proceeds to the BAD_ROW processing.

  3. PUT

    The Event Stream Processor is checked to see whether paused. If so, the stream pauses here and waits for permission to continue.

    The new result is placed into the stream's store. This is not a simple process, as the result transaction gets cleaned and transformed according to the information already in the store. Because of this, the current output transaction is invisible after this point. There is no current output row either. Some of these transformations are:
    • SAFEDELETEs: are either thrown away (if there was no such record in the store) or converted to DELETEs (filled with all the data that they had in the store before being deleted).
    • UPSERTs: They are transformed into either INSERTS or UPDATE_BLOCKs. Any remaining UPDATEs are converted to UPDATE_BLOCKs , or may be discarded if no data is changed in the record from its previous state. An UPDATE_BLOCK is a pair of records; the first one has the operation type UPDATE_BLOCK and contains the new values, while the second one has the operation type DELETE and contains the old values. When an UPDATE_BLOCK is published to outside the Event Stream Processor, the second record is discarded and the first one is converted to an UPDATE. Inside the Event Stream Processor, the entire update block is visible.

    The PUT may trigger an exception too, for example, when trying to insert a record with a key that is already in the store. In this case, the entire transaction is aborted and the stream moves to the BAD_ROW location.

    The current input transaction and current output transaction (already transformed) are inserted into the stream's history. The input transaction is added to the end of inHist, the output transaction appended to the end of outHist. Since the processing is done now, inTrans and inRow become invisible, outTrans and outRow are already invisible by this time.

  4. OUTPUT

    The result transaction is queued to be published to the clients. If some clients are too slow and the output buffer fills, the stream waits for buffer space to become available.

    The result transaction is delivered to any streams that have this stream as their input. Again, if any of their input queues become full, this stream waits for them to become available.

    The stream then goes to INPUT the next transaction.

Besides this main loop, there are side branches. For the streams with expiry, the following side branch occurs every second:
Parent topic: Introduction