Guaranteed Delivery and Persistent Subscribe Pattern

Guaranteed delivery (GD) and persistent subscribe pattern (PSP) are delivery mechanisms that support high availability. They ensure that data continues to be processed from a stream or window even if the Server fails, the destination (third-party server) fails, or the destination does not respond for a period of time.

SAP recommends that you use guaranteed delivery rather than persistent subscribe pattern if possible. GD uses CPU and memory resources more efficiently and is more flexible from a development standpoint because it does not force you to decide how many subscribers will be supported when you set it up. However, you might prefer PSP if:

You need to guarantee delivery of data from a stream, from a delta stream, or from a window assigned to a memory store.
You do not want the guaranteed delivery store to be a log store for performance reasons. Using a memory store allows recovery when the client restarts, but not when the project restarts.

Guaranteed Delivery

Guaranteed delivery (GD) uses log stores to ensure that a GD subscriber registered with a GD window receives all the data processed by that window even if the client is not connected when the data is produced. GD is supported only on windows (not on streams or delta streams) and each GD window requires a log store.

You can specify GD support for a window in Studio or in the CCL. (See the Programmers Reference for CCL details.) A GD window supports multiple GD subscribers as well as both GD and non-GD subscriptions. To use GD, you must also:

Assign a log store to every window in the project that cannot be recovered by an upstream provider.
Do at least one of the following:
- Enable GD on any bindings for the project.
- Enable GD on project adapters that support it.
- Use the C++ SDK, the Java SDK, or the .NET SDK to configure publishers sending data to your project to retransmit any data for which they do not receive a commit confirmation.
- Use the C++ SDK, the Java SDK, or the .NET SDK to set up GD subscriptions for client applications. For more information, see the instructions on subscribing with guaranteed delivery in the C SDK Guide, the Java SDK Guide, and the .NET SDK Guide.

An ESP project that has at least one GD-enabled window provides information on GD in two metadata streams:

_ESP_Streams - tells you whether GD is enabled for the stream
_ESP_GD_Sessions - lists active and inactive GD sessions, identifies the sequence number of the last event committed for each stream a GD session subscribes to and the last update time for an entry, and identifies the user associated with this session (that is, the user who initiated the subscription).

For details on these metadata streams, see the Configuration and Administration Guide.

You can subscribe to the metadata streams using the C++ SDK, the Java SDK, or the .NET SDK. You can also monitor the streams yourself using esp_subscribe (see the Utilities Guide) or the Studio Server view. ESP stores data from the _ESP_GD_Sessions metadata stream in a special metadata log store so it will be available after a crash.

Consistent recovery mode ensures that if the server restarts, it recovers the state of all windows in a project to the last successful checkpoint state—provided you have followed the rules related to assigning windows to a log store. Consistent recovery is achieved by checkpointing all log stores atomically. If any checkpoints fail (which happens, for example, when the server shuts downs in the middle of a checkpoint or there is not enough space in the log stores), Event Stream Processor rolls all the log stores back to the last successful checkpointed state. See the Programmers Guide for more information on consistent recovery.

Use the Auto Checkpoint project option to set the number of input transactions that trigger a checkpoint. More frequent checkpoints reduce the risk of data loss; less frequent checkpoints reduce the burden on system resources and may improve performance. Note that the frequency N you set with this option only ensures that a checkpoint happens at least every N transactions. Checkpoints might happen at other times if the system decides that it is necessary or if a publisher issues a commit when the server is running in consistent recovery mode.

Note: SAP does not recommend using consistent recovery mode in projects where active-active HA mode is also enabled. Because ESP is nondeterministic, enabling consistent recovery mode on the instances of an active-active project cannot be guaranteed to produce the same data in the primary and secondary instances if there is a failure.

Persistent Subscribe Pattern

Input adapters support persistent subscribe pattern (PSP) using facilities provided by the data source. Output adapters use PSP directly.

For each subscriber, PSP involves a combination of an element such as a stream or window, where you enable PSP; a control window (also known as a truncate window); and a Flex operator with a log store. The element on which PSP is enabled and the control window plug into the Flex operator. The PSP-enabled element enters data into the Flex operator. The Flex operator generates a sequence number and opcode from the data and places them at the beginning of each row of data. The Flex operator sends this data to the adapter that is attached to it. The adapter passes the information to the control window. Finally, the control window informs the Flex operator of the data that has been processed by the adapter, and the Flex operator removes this data from the log store.

Note: The WebSphereMQ Input and Output adapters, all JMS Input and Output adapters, and the TIBCO Rendezvous adapter all support PSP. These adapters have specific PSP and GD parameters that are unique to them. For more information, see the Adapters Guide.

PSP Overview