A log store allows data recovery inside a window if a server fails or is shut down.
Properly specified log stores recover window elements on failure, and make
sure data gets restored correctly if the server fails and restarts. You can use log
stores with windows that have no retention policy; you cannot use log stores with
stateless elements.
When using log stores:
- Log stores only store window contents.
- Log stores do not directly store intermediate state, such as variables.
- Local Flex stream variables and data structures are not directly stored. However, they may be regenerated from source data if the source data is in persistent storage.
- Log stores do not preserve opcode information. (During periodic log store compaction and checkpointing, only the current window state is preserved. Records are then restored as inserts.)
- Row arrival order is not preserved. In any stream, multiple operations may be collapsed into a single record during log store compaction, changing arrival order. Inter-stream arrival order is not maintained.
- You can define one or more log stores in a project. When using multiple stores make sure you prevent the occurrence of log store loops. A log store loop is created when, for example, Window1 in Logstore1 feeds Window2 in Logstore2, which feeds Window3 in Logstore1. Log store loops cause compilation errors.
- The contents of memory store windows that receive data directly from a log store window are recomputed once the log store window is restored from disk.
- The contents of memory store windows that receive data from a log store window
via other memory store windows are also recomputed, once the input window's
contents have been recomputed.
- In the case of partitioning, if the input of the partition target is a stream,
which is a stateless element, then operations such as filter, compute,
aggregate, and join are not supported.
- If the the input of a partitioned target is on a memory store and the target is
on a log store, this is supported only if the memory store (input element) can
recover its data from an element that is on a log store.
Note: If a memory store window receives data from a log store window via a stateless element, for example, a delta stream or a stream, its contents are not restored during server recovery.
When you shut down the server normally, it performs a quiesce and
checkpoint before it shuts down. It is therefore able to store all data currently in the
project, as the data has been fully processed and is in a stable state. When an abnormal
system shutdown occurs between checkpoints, there is no way of knowing the state of the
system or where the uncheckpointed data was. Therefore, the uncheckpointed data on the
input windows attached to log stores is replayed by streaming events down the project as
though they were going through the input windows for the first time. The uncheckpointed
data is replayed in an attempt to attain a state as close as possible to the state of
ESP before the abnormal shutdown.
Log stores are periodically compacted, at which point all data accumulated in the
store is checkpointed and multiple operations on the same key are collapsed. After a
checkpoint, the store continues appending incoming data rows to the end of the store
until the next checkpoint.
Note: The recovery of data written to the
store, but not yet checkpointed, is available for input windows only.
SAP recommends that when you assign a window to a
log store, you also assign all of its input windows to a log store. Otherwise, data
written to the window after the last checkpoint is not restored.
Unlike memory stores, log stores do not extend automatically. Use the CCL maxfilesize property to specify log store size. The size of a log store is extremely important. Log stores that are too small can cause processing to stop due to overflow. They can also cause significant performance degradation due to frequent cleaning cycles. A log store that is too large can hinder performance due to larger disk and memory requirements.