esp_playback

Use esp_playback to test a project. It can send data with a play rate, and upload data to a project from different sources.

[1329258744] started
[1329258744] complete: 0 (success) 0 (errors) 0 (%)
[1329258746] complete: 1 (success) 0 (errors) 33 (%)
[1329258748] complete: 2 (success) 0 (errors) 66 (%)
[1329258750] complete: 3 (success) 0 (errors) 100 (%)
[1329258750] stopped

You can also slow down or speed up the rate of the playback. Use the -R option to specify the desired rate, either in rows/millisecond, or at a rate determined by the values in a timestamp/datetime column in input data. If you use the latter method, you can specify a timescale rate to speed up or slow down the playback.

Because esp_playback is a testing tool, there are various reasons for slowing down or speeding up the playback. Typically, you adjust the playback rate to verify your calculations and collection of data. When you're confident that your project is configured correctly, you can speed up the playback rate to simulate the anticipated incoming rate of actual data to ensure your project can handle the throughput.

Note that the input speed may influence your calculations. The example below demonstrates how modifying input speed can change the results of calculations due to timing discrepancies. The CCL code below sums the number of shares, numShares, and averages the price, price, of the input from the cbret_006 window.

create schema sharesSchema (
    tradeId integer,
    symbol string,
    tradeDate date,
    numShares integer,
    price float
);

CREATE  MEMORY  STORE "store" PROPERTIES  INDEXTYPE ='tree',  INDEXSIZEHINT =8;

create input window cbret_006a
    schema sharesSchema
    primary key(tradeId)
    STORE "store"
;
create output window cbret_006
    primary key deduced
    as
    select a.tradeId, a.symbol, a.tradeDate,
        sum(a.numShares) as numShares,
        avg(a.price) as price
        from cbret_006a a
        keep 1 seconds per (symbol)
        group by a.symbol
;

A sample input of three lines of XML records to be read. The ESP_OPS value i indicates that all three records have an insert opcode.

<cbret_006a ESP_OPS="i"  tradeId="1" symbol="EBAY" tradeDate="2000-05-04T12:00:01" numShares="500" price="150.0" />
<cbret_006a ESP_OPS="i"  tradeId="2" symbol="EBAY" tradeDate="2000-05-04T12:00:02" numShares="100" price="50.625" />
<cbret_006a ESP_OPS="i"  tradeId="3" symbol="EBAY" tradeDate="2000-05-04T12:00:04" numShares="1000" price="16.875" />

esp_playback is run below without modifying the playrate and the output is written by executing esp_subscribe connected to the cbret_006 window.

esp_playback -p remoteBox:19011/ws1/project1 -c sybase:sybase -C espxml:sum/input.xml
esp_subscribe  -p remoteBox:19011/ws1/project1 -c sybase:sybase  -s cbret_006:

The output is a single XML record with the correct share sum and average price of the three input records.

<cbret_006 ESP_OPS="i"  tradeId="3" symbol="EBAY" tradeDate="2000-05-04 12:00:04" numShares="1600" price="72.500000"/>

esp_playback is run below with the -R option changing the playrate to 1 record per 4000 milliseconds (4 seconds). The output is written to the standard output by executing esp_subscribe connected to the cbret_006 window.

esp_playback -p remoteBox:19011/ws1/project1 -c sybase:sybase  -C espxml:sum/input.xml -R 1:4000 -t1
esp_subscribe -p remoteBox:19011/ws1/project1 -c sybase:sybase  -s cbret_006:

The output of this execution of esp_playback is 3 XML records identical to the input file.

<cbret_006 ESP_OPS="i"  tradeId="1" symbol="EBAY" tradeDate="2000-05-04 12:00:01" numShares="500" price="150.000000"/>
<cbret_006 ESP_OPS="i"  tradeId="2" symbol="EBAY" tradeDate="2000-05-04 12:00:02" numShares="100" price="50.625000"/>
<cbret_006 ESP_OPS="i"  tradeId="3" symbol="EBAY" tradeDate="2000-05-04 12:00:04" numShares="1000" price="16.875000"/>

keep 1 seconds per (symbol)

Syntax

esp_playback -p [<host>:]<port>/workspace-name/project-name -c user[:password] [OPTION...]

Required Arguments

• -c user[:password] – (required) authenticates access to ESP project with a user ID and, optionally, a password. If you do not provide either the password or the -k or -G option, you are prompted for the password. If the Event Stream Processor successfully authenticates with these credentials, the connection is maintained, otherwise the Event Stream Processor immediately closes the connection.

Options

Some of these options require file paths. When entering file paths on Windows, use forward slashes instead of back slashes.

• -a – (optional) uses asynchronous publishing. In this mode, the publication does not wait for ESP project to acknowledge the received data. The default is synchronous publishing.
• -p [<host>:]<port>/workspace-name/project-name – (required) together, the host:<port>/<workspace name>/ <project name> arguments specify the URI of ESP project to connect to the host and port. For example, if you have started your ESP cluster manager in the port 19011, the host name is set as localhost, and you are running a project called prj1 in the default workspace, specify -p as: -p localhost:19011/default/prj1
• -D user[:password] – (dependent required) authenticates access to a database with a user ID and, optionally, a password. This option is typically required to authenticate with the source database. However, if the ODBC manufacturer provides for storing the username/password parameter in the DSN (and you have done so), this option is not required. You are prompted for a password if you do not provide one.
• -G – (optional) authenticates access to the Event Stream Processor with credentials held within a Kerberos authentication ticket. Environment variables determine where the system will look to find authentication tickets (See Administrators Guide > Security in the Event Stream Processor > Authentication > Setting Environment Variables for Kerberos for more information). You must use the -c option to specify a user name in order to locate the proper authentication ticket; if you do not use -c, the client prompts for a user name and password.
• -h – (optional) prints a list of possible options on the screen along with a brief explanation for each option.
• -i – (optional) turns on the shine-through flag. In this mode, if there are any missing columns in an update, they are filled in with the previous values for the updated row. The default behavior is to fill any missing columns with NULLS.
• -e – (optional) specifies that communications to ESP project should be encrypted. Use -e with servers running SSL. Note that Event Stream Processor must be started in encrypted mode for this to work.
• -k privateRsaKeyFile – (optional) authentication is performed using the RSA private key file mechanism instead of password authentication. The privateRsaKeyFile must specify the pathname of the private RSA key file.
  Note: Ensure that the ESP project has been started with the -k option specifying the directory in which to store the RSA keys.
• -L – (optional) resets the project clock to match the time when the data source was recorded. -L cannot function alone. -R columnName immediately follows -L to retrieve the timestamp value and set the specified time at which playback should occur (according to the timestamp column).
  Note: If -R is not defined, the -L option does not function and reverts to use the current project time for playback.
• -C connStr – (dependent required) specifies the connection string for the source. The connection string varies from source to source.

  Supported source formats include: ODBC, ESP XML, ESP DLM, Binary, Recorder, and KDB. When used with an ODBC source, the -D option is typically required to authenticate with the source database. However, if the ODBC manufacturer provides for storing the username/password parameter in the DSN (and you have done so), the -D option is not required.

  The connection string for these formats are:

  • ODBC:odbc:<dsnName>:<sql>|<file>:<fileName>|<query>
    • odbc – the ODBC source string identifier. An ODBC 3.0 compliant driver for the required source must be installed and configured on the machine running this utility to use this source option.
    • dsnName – the ODBC data source name. For an SAP Sybase IQ data source, the data source name resides in the .odbc.ini file (found in <homedirectory>/.odbc.ini). For other database sources, the data source names reside in the odbc.ini and odbcinst.ini files (Found in an ODBC folder such as /etc/odbc.ini and /etc/odbcinst.ini). The example below has a dsnName entry of "SAPSybaseIQ".

      [SAPSybaseIQ]
               Description  = SAP Sybase IQ database
               ServerName   = iqhost_iqdemo
               DatabaseName = iqdemo
               CommLinks    = tcpip{host=iqhost;port=2638}
               Driver       = /iqhome/IQ-15_1/lib64/libodbc.so
      

    • file|sql – specifies whether the following argument is a file name or a SQL query. One of these values is required.
    • fileName|query – the fileName is the name of the file containing a SQL statement. The query is the SQL query that needs to be executed to retrieve the data. One of these values is required.
  • EXML: espxml:<inputFile>
    • espxml – is an identifier for an ESP XML file source.
    • inputFile – is the full path and name of the file containing data in ESP XML format.

    For information on how a record in an XML file should be configured, see Publish and Subscribe Executables > esp_convert > Input Formats.

  • ESPDLM (ESP delimited file): espdlm:<inputFile>[:<delimiter>]
    • espdlm – an identifier for the ESPDLM (ESP delimited) file source.
    • inputFile – is the full path and name of the file containing data in ESP DLM format.
    • delimiter – this optional parameter is a single-character field delimiter; the default is a comma.

    For information on how a record in a DLM file should be configured, see Publish and Subscribe Executables > esp_convert > Input Formats.

  • Binary: binary:<inputFile>
    • binary – is an identifier for an ESP binary file source. The advantage to this format is that the loads are faster because there is no conversion required because the data is already in a format that the Event Stream Processor can absorb. The disadvantage to this format is that it is machine-architecture specific.

      One can use esp_convert to convert data from either XML or DML to binary format.

      Note: Binary files recorded in previous releases cannot be played back unless they are first converted to the new binary format using esp_convert. See Publish and Subscribe Executables > esp_convert for information on how to convert binary files.
    • inputFile – is the full path and name of the file containing data in binary format.
  • Recorder: recorder:<inputFile>
    • recorder – is an identifier for a file generated by the recorder. The recorder can be started using the ESP Studio or the through the recorder examples in $ESP_HOME/client/pubsub/ folder.
    • inputFile – is the full path and name of the file containing data generated by the recorder.
  • KDB: kdb:<host>:<port>:<sql>|<file>:<filename>|<query>:[<user>:<password>]
    • file|sql – specifies whether the following argument is a file name or a SQL query. One of these values is required.
    • fileName|query – the fileName is the name of the file containing a SQL statement. The query is the SQL query that needs to be executed to retrieve the data. One of these values is required.

• -R playRate – specifies how fast/slow the data must be played back. If this parameter is not supplied, the record plays as fast as possible. The following examples show the two ways that the playback rate can be supplied.

  records:milliseconds
            

  where:

  • records – is the number of records to publish in the given number of milliseconds. This value can be 0 only if the millisecond component is also 0. A value of 0 indicates play as fast as possible.
  • milliseconds – is the number of milliseconds to play back the given number of records.

    This property is not supported for the source type recorder.

  columnName

  where:

  • columnName – is a case-sensitive column name in the target stream that controls how fast the record plays back. The target stream ignores the -R option if columnName does not exist in the target stream. Valid datatypes for columnName are date, time, timestamp, bigdatetime, long, interval, and integer. However, when using long, interval or integer, values are interpreted as timestamp, which is the number of milliseconds since 1970-01-01 00:00:00 UTC (the Epoch).

    The columnName property is ignored for the source type Recorder and is currently not supported for binary file sources.

• -T timeScaleRate – specifies a multiplication factor for the delta between the times for two consecutive records. This is used with the playback rate, when the playback rate is controlled by a column in the source. It takes an integer value between -N to +M. A positive value greater than +1 speeds up the time and a negative value less than -1 slows down the rate. A value of +1 or -1 plays back at the rate specified in the column and a value of 0 specifies that the column values are ignored. Default value is 1.

• -r interval – specifies the minimum number of seconds to wait between each reporting of the published statistics. The default is 5 seconds. A value of 0 indicates no reporting. The time interval check is triggered when there is a record to be published. This means that even when the reporting interval has been exceeded the statistics are reported if there are no records to trigger the check.
• -B bufferSize – specifies the internal read and write buffer sizes. The default buffer size is 32K, which is also the maximum allowed value. Smaller buffer sizes use less memory but may run slower in some situations.
• -t size – specifies that transaction blocks must be used to publish data to ESP project, with each block size large. ESP project processes data faster if transaction blocks deliver the data. The performance boost is achieved in two ways: the network is used more efficiently because a larger number of records are packed into a single network packet; and because ESP project treats the records as a single block, which fails or passes in its entirety. This results in less processing overhead. Depending on the nature of the application, this option may not be suitable.
• -w size – specifies that envelopes must be used to publish data to the Event Stream Processor, with each envelope of size size. If neither the -t nor -w option is specified, the default value is -w64. The value of size must be between 1 and 1024. Using this option ensures network efficiency in delivering data to ESP project by modifying it to treat the records as individual records.
• -s streamName – (required) specifies the target stream name for this utility. This option is required for ODBC and KDB sources. All other sources have the target stream-name/stream-ID embedded in them.
  Note: XML and delimited sources may require configuration prior to use. See Server Executables > esp_playback > Input Formats for information on how to configure XML and delimited sources.

  The streamName variable is case-sensitive.

• -S maxStringSize – specifies the maximum string size that can be processed. This option is used in ODBC sources. The default value is 1024.

  This value is global for all strings in the source. Specifying a large value for this option may result in increased memory usage depending on the number of strings in the record and the value specified with the -B option.

• -m dateMask – the date mask to use for XML and delimited files. The default is ''%Y-%m-%dT%H:%M:%S'. The date mask is common for all date columns. This option is meaningful only for the ESP XML and delimited file sources.
• -v (print version) – prints out the version number, platform, and build date for ESP playback.

Examples

• Reading and playing back data – To read data from an ODBC source with a DSN name of 'myODBC_connection' into a stream called 'InputWindow1', using a command line SQL statement:

  esp_playback -c user:pass -D dbuser:dbpassword -p myserver:9786/default/odbc_playback_test -C ODBC:myODBC_connection:SQL:"select * from mytable" -s InputWindow1
      

  If you omit the password from the -c and -D options, you are prompted to provide them after issuing the command. For example, using the same ODBC source and stream as above without specifying a password or an ODBC password, use:

  esp_playback -c user -D dbuser -p myserver:9786/default/odbc_playback_test -C ODBC:myODBC_connection:SQL:"select * from mytable" -s InputWindow1

  You are then prompted to enter a password and an ODBC password.
  To read data from an ESP XML source trades.xml, play the data at a rate of 10,000 rows per second, and report the progress every 15 seconds:

  esp_playback -c user:pass -p localhost:19022/w1/p1 -C "espxml:/tmp/trades.xml" -R 10000:1000 -R 15
      

  To read data from an ESP DLM source trades.csv, play the data at a rate of 10,000 rows per second, and report the progress ever 15 seconds.

  $ESP_HOME/bin/esp_playback -c user:pass -p localhost:19011/w1/p1 -C "espdlm:qa/trades.csv:," -R 10000:1000 -R 15

• Using long value in a timestamp column – Use a long value as a timestamp value in the timestamp column to reset the project clock in esp_playback:

  $ESP_HOME/bin/esp_playback -c user:pass -p localhost:19011/w1/p1  -C "espdlm:qa/trades.csv:," -R time -L
  (When the CCL is : CREATE INPUT WINDOW VWAP SCHEMA (Sym STRING, VWAP Float, time long) PRIMARY KEY (Sym) keep 60 seconds;)