Monitoring with Metadata Streams

Metadata streams are automatically created by Event Stream Processor. Query and subscribe to these streams to obtain important health and performance information about the currently running project.

Some metadata streams contain static information that never changes while the project is running, for example, _esp_streams. Other streams continuously update at various periods or on various events. You can subscribe, query, and view metadata streams in the same way as regular streams. For example, you can use the esp_subscribe utility. This command subscribes to the streams _ESP_Connectors and _ESP_Streams from the project default/prj1, which is running on a cluster manager on localhost:11180, and prints all stream data in XML format on standard output.

esp_subscribe -c user-id:password -s _ESP_Connectors,_ESP_Streams -p localhost:11180/default/prj1

For details on esp_subscribe, see the Utilities Guide.

Note: The schema for metadata streams can change between releases as the set of statistics the streams report expands. New columns are added to the end of the schema for existing metadata streams. Keep this in mind when coding.

Cases where metadata streams differ from general streams:

Metadata streams have reserved names. No other objects can use these names.
Metadata streams store their records in a special store called ESPMetadataStore. No other streams can use this store.
Metadata streams cannot be used in CCL or serve as an input for a stream in a project. For example, the following usage is not possible:
```
INSERT INTO myStream SELECT * FROM _ESP_Connectors WHERE latency > 1
```

_ESP_Adapter_Statistics
Reports statistics unique to each adapter. Both internal and external adapters can publish statistics to this stream.
_ESP_Clients
Contains information about all the currently active gateway client connections.
_ESP_Clients_Monitor
Contains information about the performance of all currently active gateway client connections and a copy of data from the _ESP_Clients stream. Monitoring data is available only if the time-granularity option in the project configuration (CCR) file is set to greater than 0. The frequency of updates corresponds to the value of the time-granularity option. For example, if set to 1, an update is published every second, if set to 30, an update is published every 30 seconds, and if set to 0, reporting is disabled.
_ESP_Clockupdates
Delivers notifications of changes in the logical clock of the project.
_ESP_Columns
Contains information about all columns of all streams.
_ESP_Config
Contains the current CCX of the project.
_ESP_Connectors
Contains information about all internal adapters defined in the project.
_ESP_Keycolumns
Contains information about the primary key columns of all the streams. If a stream has a primary key, the columns that make up the key are listed in this stream.
_ESP_Project_Monitor
Contains information on project CPU usage, memory consumption, and number of threads. Monitoring data is available only if the time-granularity option in the project configuration (CCR) file is set to greater than 0. The frequency of updates corresponds to the value of the time-granularity option. For example, if set to 1, an update is published every second, if set to 30, an update is published every 30 seconds, and if set to 0, reporting is disabled.
_ESP_RunUpdates
Delivers notifications of changes during debugging. The Server sends notifications only when the project is in trace mode.
_ESP_Streams
Contains information about all streams, delta streams, and windows.
_ESP_Streams_Monitor
Contains information about the performance of streams, and a copy of data from the _ESP_Streams stream. Monitoring data is available only if the time-granularity option in the project configuration (CCR) file is set to greater than 0. The frequency of updates corresponds to the value of the time-granularity option. For example, if set to 1, an update is published every second, if set to 30, an update is published every 30 seconds, and if set to 0, reporting is disabled.
_ESP_Streams_Topology
Contains pairs of names for streams that are directly connected.
_ESP_Subscriptions
Contains information about all currently active subscriptions. A dropped connection is considered unsubscribed from everything to which it was subscribed.
_ESP_Subscriptions_Ext
Contains information about all currently active subscriptions.
_ESP_Tables
An internal stream that contains a copy of information contained in the _ESP_Streams stream.

Parent topic: Monitor a Cluster