esp_upload

The format of the data is zero or more occurrences of <Stream Handle><Raw Binary Record>. <Stream Handle> is an uint32_t indicating the destination stream for the record. This tool is typically used at the end of a pipeline with the esp_convert tool.

Syntax

esp_upload -p [<host>:]<port>/workspace-name/project-name -c user[:password] [OPTION...]

Required Arguments

• -c user[:password] – Authenticates with a user ID and, optionally, a password. If you do not provide either the password or the -k or -G option, you are prompted for the password. If the Event Stream Processor successfully authenticates with these credentials, the connection is maintained, otherwise the Event Stream Processor immediately closes the connection.

Options

• -b – Sets byteswap mode. The raw records fed into esp_upload (via esp_convert -b) and the server to which esp_upload is sending data have different byte orders than the architecture on which the esp_upload client is running (the byte order of the data must always match the byte order of the server).
• -d N – Inserts a delay of N microseconds between records or transaction blocks.
• -e – Encrypts traffic with the Event Stream Processor via openSSL sockets.
  Note: Ensure that the Event Stream Processor is started in encrypted mode to use this option.
• -f timeout:finalizer – Sets a finalizer to be run. The ESP Server runs the SQL finalizer statement (a combination of insert, update, or delete statements, separated by semicolons) if no message is received from esp_upload within timeout milliseconds. The SQL statement is also run when esp_upload stops.
• -G – (optional) authenticates access to the Event Stream Processor with credentials held within a Kerberos authentication ticket. Environment variables determine where the system will look to find authentication tickets (See Administrators Guide > Security in the Event Stream Processor > Authentication > Setting Environment Variables for Kerberos for more information). If the user name differs from the default principal name in the ticket cache, specify an alternate user name with the –c option to use the corresponding authentication ticket.
• -h – Prints a list of possible options on the screen along with a brief explanation for each option.
• -k privateRsaKeyFile – Performs authentication by using the RSA private key file mechanism instead of password authentication. The privateRsaKeyFile must specify the pathname of the private RSA key file. Ensure that the ESP Server has been started with the -k option specifying the directory in which to store the RSA keys.
  Note: With this option enabled, the user name must be specified with the -c option, but the password is not required.
• -m conn_name – Sets a symbolic tag name for the connection. This allows the esp_upload to look up the connection easily in the _ESP_Clients metadata stream. To kill the connections by tag name, use the esp_client command.
• -p [<host>:]<port>/workspace-name/project-name – (required) Together, the host:<port>/<workspace name>/ <project name> arguments specify the URI to connect to the ESP Server (cluster manager). For example, if you have started your ESP cluster server in the port 19011, the host name is set as localhost, and you are running a project called prj1 in the default workspace, specify -p as: -p localhost:19011/default/prj1
• -r N – Uploads records or transaction blocks at a rate of N per second. The default is to upload as fast as the server can absorb the data.
• -s N – Synchronizes the source streams every N records or transaction blocks. This guarantees that the records have been absorbed by the source streams. By default, source streams are not synchronized.
• -t size – Runs in transaction mode. Each record that esp_upload reads is buffered on a per-stream basis. When a buffer reaches the indicated number of records, it is wrapped as a single transaction and sent to the Event Stream Processor. If all records read are from one stream, this effectively buffers the stream into size record chunks and commits them as transactions. Any buffered records are sent as a single transaction per stream when an EOF is read.
• -w size – Runs in envelope mode. Each record that esp_upload reads is buffered on a per-stream basis. When a buffer reaches the indicated number of records, it is wrapped in a single envelope and sent to the Event Stream Processor. If all records read are from one stream, this effectively buffers the stream into size record chunks. Any buffered records are sent as a single envelope per stream when an EOF is read.
• -x – Upon receiving an EOF on the standard input, sends an <END OF STREAM> marker to each stream for which data has been uploaded. If all source streams of the Event Stream Processor receive an <END OF STREAM> marker, the Event Stream Processor shuts down and exits.
• -X – Forces the Event Stream Processor to exit when upload completes.
• -Y <beat> – Forces the Event Stream Processor to exit if no data is received in <beat> microseconds.
• -v – Prints the esp_upload utility version.

Examples

For a description of the format of the CSV and XML input files, see esp_convert.

cat foo.xml | esp_convert -p localhost:11180/default/prj1 | esp_upload -c user:pass -p localhost:11180/default/prj1
    

cat foo.csv | esp_convert -d "," -p localhost:11180/default/prj1 | esp_upload -c user:pass -p localhost:11180/default/prj1