esp_playback

The formats that are currently supported are ESP XML, comma-separated values, ESP binary, and ODBC sources.

This tool can also play data at a user-specified rate, which can be specified either in rows/millisecond, or at a rate determined by the values in a timestamp/datetime column in input data. If the latter mechanism is used, the user can specify a timescale rate, which can be used to speed up or slow down the playback.

This tool is intended to replace esp_upload and esp_convert, which work only on Sybase file sources.

Syntax

esp_playback -p [<host>:]<port>/workspace-name/project-name -c user[:password] [OPTION...]

Options

• -a – (optional) uses asynchronous publishing. In this mode, the publication does not wait for ESP Server to acknowledge the received data. The default is synchronous publishing.
• -p [<host>:]<port>/workspace-name/project-name – (required) together, the host:<port>/<workspace name>/ <project name> arguments specify the URI to connect to the ESP Server (cluster manager). For example, if you have started your ESP cluster server in the port 19011, the host name is set as localhost, and you are running a project called prj1 in the default workspace, specify -p as: -p localhost:19011/default/prj1
• -c user[:password] – (required) authenticates with a user ID and, optionally, a password. If neither the password nor the -k option is provided, the user is prompted for the password. If the Event Stream Processor successfully authenticates with these credentials, the connection is maintained, otherwise the Event Stream Processor immediately closes the connection.
• -h – (optional) prints a list of possible options on the screen along with a brief explanation for each option.
• -i – (optional) turns on the shine-through flag. In this mode, if there are any missing columns in an update, they are filled in with the previous values for the updated row. The default behavior is to fill any missing columns with NULLS.
• -e – (optional) specifies that communications to ESP Server should be encrypted. Note that Event Stream Processor must be started in encrypted mode for this to work.
• -k privateRsaKeyFile – (optional) authentication is performed using the RSA private key file mechanism instead of password authentication. The privateRsaKeyFile must specify the pathname of the private RSA key file.
  Note: Ensure that the ESP Server has been started with the -k option specifying the directory in which to store the RSA keys.
• -C connStr – (dependent required) specifies the connection string for the source. The connection string varies from source to source.

  Supported sources and their connection string formats include:

  • ODBC:{0}odbc:<dsnName>:<sql>|<file>:<fileName>|<query>
  • ESP XML: espxml:<inputFile>
  • ESP DLM: espdlm:<inputFile>[:<delimiter>]
  • Binary: binary:<inputFile>
  • Recorder: recorder:<inputFile>
  • KDB: kdb:<host>:<port>:<sql>|<file>:<filename>|<query>:[<user>:<password>]
  Note: This parameter is (optional) if you use the ESP XML or ESP DLM sources.

• odbc – the ODBC source string identifier. An ODBC 3.0 compliant driver for the required source must be installed and configured on the machine running this utility to use this source option.
• dsnName – the ODBC data source name.
• file|sql – specifies whether the following argument is a file name or a SQL query. One of these values is required.
• fileName|query – the fileName is the name of the file containing a SQL statement. The query is the SQL query that needs to be executed to retrieve the data. One of these values is required.

espxml:inputFile
            

where:

• espxml – is an identifier for an ESP XML file source.
• inputFile – is the full path and name of the file containing data in ESP DLM format.

espdlm:inputFile[:delimiter]
             

where:

• espdlm – is an identifier for ESP DLM file source.
• inputFile – is the full path and name of the file containing data in ESP DLM format.
• delimiter – this optional parameter is a single-character field delimiter; the default is a comma.

espxml:inputFile
           

where:

• espxml – is an identifier for an ESP XML file source.
• inputFile – is the full path and name of the file containing data in ESP DLM format.

binary:inputFile
          

where:

• binary – is an identifier for an ESP binary file source. The advantage to this format is that the loads are faster because there is no conversion required because the data is already in a format that the Event Stream Processor can absorb. The disadvantage to this format is that it is machine-architecture specific.

  One can use esp_convert to convert data from either XML or DML to binary format.

• inputFile – is the full path and name of the file containing data in binary format.

recorder:inputFile
          

where:

• recorder – is an identifier for a file generated by the recorder. The recorder can be started using the ESP Studio or the through the recorder examples in $ESP_HOME/client/pubsub/ folder.
• inputFile – is the full path and name of the file containing data generated by the recorder.

• -R playRate – specifies how fast/slow the data must be played back. If this parameter is not supplied, the record plays as fast as possible. The following examples show the two ways that the playback rate can be supplied.

records:milliseconds
          

where:

• records – is the number of records to publish in the given number of milliseconds. This value can be 0 only if the millisecond component is also 0. A value of 0 indicates play as fast as possible.
• milliseconds – is the number of milliseconds to play back the given number of records.

  This property is not supported for the source type recorder.

columnName

where:

• columnName – is the column name in the target stream that controls how fast the record is played back. The column name is case-sensitive and an error is reported if the provided column name does not exist in the target stream.

  The columnName property is ignored for the source type Recorder and is currently not supported for binary file sources.

• timeScaleRate – specifies a multiplication factor for the delta between the times for two consecutive records. This is used with the playback rate, when the playback rate is controlled by a column in the source. It takes an integer value between -N to +M. A positive value greater than +1 speeds up the time and a negative value less than -1 slows down the rate. A value of +1 or -1 plays back at the rate specified in the column and a value of 0 specifies that the column values are ignored. Default value is 1.
• -r interval – specifies the minimum number of seconds to wait between each reporting of the published statistics. The default is 5 seconds. A value of 0 indicates no reporting. The time interval check is triggered when there is a record to be published. This means that even when the reporting interval has been exceeded the statistics are reported if there are no records to trigger the check.
• -B bufferSize – specifies the internal read and write buffer sizes. The default buffer size is 32K, which is also the maximum allowed value. Smaller buffer sizes use less memory but may run slower in some situations.
• -t size – specifies that transaction blocks must be used to publish data to ESP Server, with each block size large. ESP Server processes data faster if transaction blocks deliver the data. The performance boost is achieved in two ways: the network is used more efficiently because a larger number of records are packed into a single network packet; and because ESP Server treats the records as a single block, which fails or passes in its entirety. This results in less processing overhead. Depending on the nature of the application, this option may not be suitable.
• -w size – specifies that envelopes must be used to publish data to the Event Stream Processor, with each envelope of size size. If neither the -t nor -w option is specified, the default value is -w64. The value of size must be between 1 and 1024. Using this option ensures network efficiency in delivering data to ESP Server by modifying it to treat the records as individual records.
• -H [ hostname :] port – specifies the hostname and port, or simply the port of the Command and Control interface of the hot spare instance of ESP Server. The default hostname is localhost.
• -s streamName – (required) specifies the target stream name for this utility. This option is required for ODBC sources. All other sources have the target stream-name/stream-ID embedded in them.

  The streamName variable is case-sensitive.

• -S maxStringSize – specifies the maximum string size that can be processed. This option is used in ODBC sources. The default value is 1024.

  This value is global for all strings in the source. Specifying a large value for this option may result in increased memory usage depending on the number of strings in the record and the value specified with the -B option.

• -m dateMask – the date mask to use for XML and delimited files. The default is ''%Y-%m-%dT%H:%M:%S'. The date mask is common for all date columns. This option is meaningful only for the ESP XML and delimited file sources.

Examples

• Reading and playing back data – For example, to read data from an ODBC source with a DSN name of foo into a stream called foobar, using a command line SQL statement:

  esp_playback -c user:pass -p localhost:19022/w1/p1 -C 'odbc:foo:query:select * from foobar_db' -s foobar
      

  For example, to read data from an ESP XML source foo.xml, plays the data at a rate of 10,000 rows per second, and reports the progress every 15 seconds:

  esp_playback -c user:pass -p localhost:19022/w1/p1 -C 'espxml:/tmp/foo.xml' -s bar -R 10000:1000 -r 15