Published Data Lost When Node Fails

Problem: A subscriber receiving data through a stream does not receive all the data sent by the publisher before the ESP node shuts down unexpectedly.

Solution 1: Provide the Subscriber with a Window with a Long Retention Time

Replace the stream feeding the subscriber with a window configured with a retention time long enough to allow the subscriber to reconnect when the node comes back up or (if the node is in active-active HA mode) fails over to its secondary instance. For example, if it takes the subscriber a minute to reconnect, you might configure a 2-minute retention time to ensure that no data is lost.

Note: The subscriber must filter out any delete operations coming from the window and view only inserts.

See the Programmers Guide or the Studio Users Guide for information on data retention policies for windows.

Solution 2: Provide the Subscriber with a Guaranteed Delivery Window

Replace the stream feeding the subscriber with a window on which guaranteed delivery (GD) is enabled.

Guaranteed delivery (GD) uses log stores to ensure that a GD subscriber registered with a GD window receives all the data processed by that window even if the client is not connected when the data is produced. GD is supported only on windows (not on streams or delta streams) and each GD window requires a log store.

Note: SAP does not recommend using GD-enabled windows on a node configured for active-active HA mode. The shared disk requirements for GD log stores are not compatible with the continuous synchronization that enables an active-active primary instance to fail over quickly to its secondary instance.

For a window assigned to a memory store, a stream, or a delta stream, consider using persistent subscribe pattern (PSP) instead of GD.

See the Programmers Guide or the Studio Users Guide for information on guaranteed delivery. See the Studio Users Guide for information on PSP.