esp_kdbout

Feeds streaming data from the Event Stream Processor to a KDB database table.

By default, the adapter matches the field names (in a case-insensitive manner) to determine the mapping between and the Event Stream Processor stream and the KDB table. You can also explicitly specify the mapping.

Syntax

esp_kdbout -H [kdbhost:]kdbport -p [host:]port -q source -s table -c user[:password] [OPTION...]

Required Arguments

-c <user:password> – Passes authentication credentials to the Event Stream Processor. If you do not provide either the password or the -k option, you are prompted for the password. If the Event Stream Processor successfully authenticates with these credentials, the connection is maintained, otherwise, the Event Stream Processor immediately closes the connection.
-p [<host>:]<port>/workspace-name/project-name – Together, the host:<port>/<workspace name>/ <project name> arguments specify the URI to connect to the ESP server (cluster manager). For example, if you have started your ESP cluster server in the port 19011, the host name is set as localhost, and you are running a project called prj1 in the default workspace, specify -p as: -p localhost:19011/default/prj1
-H kdbhost:kdbport – Specifies the port number, or the host name and port number, on which KDB is listening. The default host name is localhost.
-q <query> – Specifies either the name of a stream on the Event Stream Processor or a valid SQL query to retrieve the data to write to the KDB table.
-s <stream> – Specifies the name of the KDB table to which the data will be written.

Options

-a – Uses asynchronous mode transmission; where the adapter does not wait for acknowledgment from Event Stream Processor that it received the data.
-B – Uses droppable subscriptions. Event Stream Processor drops the subscription if the adapter cannot keep up with the data.
-b <blocksize> – Allows you to set the maximum number of records to include in a single batch write to KDB table. The default is 5000.
-d – Logs debug messages.
-e – Uses encrypted OpenSSL sockets for all communications between the adapter and the Event Stream Processor (which requires that it be started in encrypted mode). When this option is not present, no encryption occurs.
-h – Prints a list of possible options on the screen along with a brief explanation for each option.
-I – Specifies a comma-separated list of KDB field names whose values will be ignored. These fields are included in the message, but are always populated with NULL.
-k <privateRsaKeyFile> – Authenticates using the RSA private key file mechanism instead of a password. The privateRsaKeyFile must specify the pathname of the private RSA key file. With this option enabled, the user name must be specified with the -c option, but the password is not required. In addition, the ESP Server must have been started with the -k option specifying the directory in which to store the RSA keys.
-L <interval> – Uses pulsed subscribe when connecting to the Event Stream Processor. The pulse interval is specified in seconds.
-l – Uses "lossy" subscribe.
-M <permutation> – Specifies a mapping between the column name in the target stream and the column name in the KDB database table. The mapping is a colon-separated series of SPColumn=KDBColumn statements. If this parameter is not provided, the connector absorbs data only for those columns where the target stream column name matches the source table column name (in a case-insensitive manner).
-m – Writes the data to the table using the "upsert" operation. If this option is not used, the adapter works in streaming mode and uses the u.upd operation to write data to the KDB database.
-n – Transactions only, no base data is received. Default value is false.
-O – Specifies a comma-separated list of KDB field names to omit from the message. Unlike ignored fields which are part of the message but always NULL, these fields are not included in the message.
-Q – Runs in quiet mode, no messages will be displayed.
-R – Subscribes with shine through (if possible) so that previously received information is retained for any fields for which the update contains no new data.
-T <numtries> – Specifies the number of times to attempt to reconnect to the KDB database if the connection breaks during operation. The default is 1.
-t – Targets KDB table.
-u <user:password> – Passes authentication credentials to the KDB database.
-v – Prints the esp_kdbout utility version.

Examples

To subscribe to the SpTrades stream of project default/prj1 in the Event Stream Processor on the server myserver, where the cluster manager is on port 1221, and stream the data out to KdbTrades in the KDB database on the server outputserver, where KDB is listening on port 9200, use:

esp_kdbout -p myserver:1221/default/prj1 -H outputserver:9200 -q SpTrades -s KdbTrades

To populate fields XXX and YYY in a KDB table may with NULL (because they have no corresponding data in the Event Stream Processor), use:

esp_kdbout -p myserver:1221/default/prj1 -H outputserver:9200 -q SpTrades -s KdbTrades -I XXX,YYY

Sometimes a table in KDB may compute fields (for example, XXX and YYY) which should not be specified in the data update message. Doing so usually results in a length error in the database. To omit these fields from the update message altogether, use:

esp_kdbout -p myserver:1221/default/prj1 -H outputserver:9200 -q SpTrades -s KdbTrades -O XXX,YYY