Determining Stream Memory Usage

When the server is running at log level 7 and it is shutdown cleanly, it reports the amount of memory consumed by every stream and any aggregation indices in the server log file.

The log level is a project configuration option you can set on the Advanced tab of the Project Configuration editor in Studio.

You do not have to constantly run the server at log level 7 to print memory usage statistics; the statistics are printed as long as the level is set at 7 when the server is shutting down. To change the log level at run time, use the esp_client tool and execute:

esp_client -p [<host>:]<port></workspace-name/project-name> -c <username>:<password> "loglevel 7"

Not all of the memory consumed by the server is reported when the server shuts down. Therefore, the total of the reported memory will not be equal to the memory reported by system utilities (Task Manager in Windows or top in Linux) for the ESP Server. Components that potentially consume significant amounts of memory that are not reported include:

Input queues for streams.
SPLASH data structures such as vectors and dictionaries used.
Indices required for handling Retention and Aging.
Subscriber queues.
Memory required to process SQL subscriptions that require aggregation and pulsed subscriptions.

The following sample illustrates the memory usage statistics reported in the log file:

[SP-6-131039] (189.139) sp(21115) CompiledSourceStream(W1): Collecting statistics (this could take awhile).
[SP-6-131040] (190.269) sp(21115) CompiledSourceStream(W1): Memory usage: 1,329,000,000 bytes in 3,000,000 records.
[SP-6-114012] (190.269) sp(21115) Platform(cepqplinux1)::run() -- cleaning up CompiledAggregateStream(grpbyout).
[SP-6-131039] (191.065) sp(21115) CompiledAggregateStream(grpbyout): Collecting statistics (this could take awhile).
[SP-6-124001] (191.065) sp(21115) CompiledAggregateStream(grpbyout)::Memory usage: 1,545,000,000 bytes in aggregation index.
[SP-6-131039] (195.957) sp(21115) CompiledAggregateStream(grpbyout): Collecting statistics (this could take awhile).
[SP-6-131040] (196.267) sp(21115) CompiledAggregateStream(grpbyout): Memory usage: 1,020,000,000 bytes in 3,000,000 records.
[SP-6-114012] (196.267) sp(21115) Platform(cepqplinux1)::run() -- cleaning up CompiledAggregateStream(grpbyout2).
[SP-6-131039] (197.038) sp(21115) CompiledAggregateStream(grpbyout2): Collecting statistics (this could take awhile).
[SP-6-124001] (197.039) sp(21115) CompiledAggregateStream(grpbyout2)::Memory usage: 1,545,000,000 bytes in aggregation index.
[SP-6-131039] (202.184) sp(21115) CompiledAggregateStream(grpbyout2): Collecting statistics (this could take awhile).
[SP-6-131040] (202.496) sp(21115) CompiledAggregateStream(grpbyout2): Memory usage: 1,122,000,000 bytes in 3,000,000 records.
[SP-6-114012] (202.496) sp(21115) Platform(cepqplinux1)::run() -- cleaning up CompiledStream(coutputwin).
[SP-6-131039] (202.496) sp(21115) CompiledStream(coutputwin): Collecting statistics (this could take awhile).
[SP-6-131040] (203.654) sp(21115) CompiledStream(coutputwin): Memory usage: 651,000,000 bytes in 3,000,000 records.