Guaranteed delivery (GD) and persistent subscribe pattern (PSP) are delivery mechanisms that support high availability.
GD and PSP ensure that data continues to be processed when:
- The ESP Server fails.
- The destination (third-party server) fails.
- The destination (third-party server) does not respond for a period of time.
SAP recommends that you use guaranteed delivery rather than persistent subscribe pattern if possible. GD uses CPU and memory resources more efficiently and is more flexible from a development standpoint because it does not force you to decide how many subscribers will be supported when you set it up. However, you might prefer PSP if:
- You need to guarantee delivery of data from a stream, from a delta stream, or from a window assigned to a memory store.
- You do not want the guaranteed delivery store to be a log store for performance reasons. Using a memory store allows recovery when the client restarts, but not when the project restarts.
Guaranteed Delivery
Guaranteed delivery (GD) uses log stores to ensure that a GD subscriber registered with a GD window receives all the data processed by that window even if the client is not connected when the data is produced. GD is supported only on windows (not on streams or delta streams) and each GD window requires a log store.
You can specify GD support for a window in Studio or in the CCL. (See the Studio Users Guide for Studio details or the Programmers Reference for CCL details.) A GD window supports multiple GD subscribers as well as both GD and non-GD subscriptions. To use GD:
- Assign a log store to every window in the project that cannot be recovered by an upstream provider.
- Do at least one of the following:
- Enable GD on any bindings for the project.
- Enable GD on project adapters that support it.
- Use the C++ SDK, the Java SDK, or the .NET SDK to configure publishers sending data to your project to retransmit any data for which they do not receive a commit confirmation.
- Use the C++ SDK, the Java SDK, or the .NET SDK to set up GD subscriptions for client applications. For more information, see the instructions on subscribing with guaranteed delivery in the C SDK Guide, the Java SDK Guide, and the .NET SDK Guide.
- _ESP_Streams - tells you whether GD is enabled for the stream
- _ESP_GD_Sessions - lists active and inactive GD sessions, identifies the sequence number of the last event committed for each stream a GD session subscribes to and the last update time for an entry, and identifies the user associated with this session (that is, the user who initiated the subscription).
You can subscribe to the metadata streams using the C++ SDK, the Java SDK, or the .NET SDK. You can also monitor the streams yourself using esp_subscribe (see the Utilities Guide) or the Studio Server view. ESP stores data from the _ESP_GD_Sessions metadata stream in a special metadata log store so it will be available after a crash.
Consistent recovery mode ensures that if the server restarts, it recovers the state of all windows in a project to the last successful checkpoint state—provided you have followed the rules related to assigning windows to a log store. Consistent recovery is achieved by checkpointing all log stores atomically. If any checkpoints fail (which happens, for example, when the server shuts downs in the middle of a checkpoint or there is not enough space in the log stores), Event Stream Processor rolls all the log stores back to the last successful checkpointed state. See the Programmers Guide for more information on consistent recovery.
Use the Auto Checkpoint project option to set the number of input transactions that trigger a checkpoint. More frequent checkpoints reduce the risk of data loss; less frequent checkpoints reduce the burden on system resources and may improve performance. Note that the frequency N you set with this option only ensures that a checkpoint happens at least every N transactions. Checkpoints might happen at other times if the system decides that it is necessary or if a publisher issues a commit when the server is running in consistent recovery mode.
Persistent Subscribe Pattern
Input adapters support persistent subscribe pattern (PSP) using facilities provided by the data source. Output adapters use PSP directly.
The WebSphereMQ Input and Output adapter, all JMS Input and Output adapters, and the TIBCO Rendezvous adapter all support guaranteed delivery (GD). These adapters have specific PSP and GD parameters that are unique to them. Examples for enabling GD in one of the JMS CSV Output adapters and the WebSphere Output adapter are located <ESP_HOME>/examples/ccl/JmsOutBoundAdapterWithGDSupport and <ESP_HOME>/examples/ccl/WsmqOutBoundAdapterWithGDSupport respectively.
For each subscriber, PSP involves a combination of an element such as a stream or window, where you enable PSP; a control window (also known as a truncate window); and a Flex operator with a log store. The element on which PSP is enabled and the control window plug into the Flex operator. The PSP-enabled element enters data into the Flex operator. The Flex operator generates a sequence number and opcode from the data and places them at the beginning of each row of data. The Flex operator sends this data to the adapter that is attached to it. The adapter passes the information to the control window. Finally, the control window informs the Flex operator of the data that has been processed by the adapter, and the Flex operator removes this data from the log store.
