File CSV Input Adapter

Adapter type: dsv_in. The File CSV Input adapter reads a file in Event Stream Processor delimited format.

Use this adapter to poll new data appended to the data file. The file does not require a header. If the file includes a header, it specifies the field names.

Sample record formats for the data file:
1. hasHeader=true
delimiter=,
expectStreamNameOpcode=false

Ts,ItemID,Price,Quantity,WarehouseZipCode,DeliveryZipCode
2004/06/17 10:00:00.000000,SKU1276532,50.00,1,10012,94086
2004/06/17 10:00:05.000000,SKU6723143,23.00,2,10012,94043

2. expectStreamNameOpcode=true
delimiter=,

Trades_in,i,2004/06/17 10:00:00.000000,SKU1276532,50.00,1,10012,94086
Trades_in,i,2004/06/17 10:00:05.000000,SKU6723143,23.00,2,10012,94043

3. expectStreamNameOpcode=false
timestampFormat=%Y/%m/%d %H:%M:%S
delimiter=,

2004/06/17 10:00:00.000000,SKU1276532,50.00,1,10012,94086
2004/06/17 10:00:05.000000,SKU6723143,23.00,2,10012,94043

This adapter supports schema discovery. If you use the CCL ATTACH ADAPTER statement to attach an adapter, you must supply the adapter type. The type for this adapter is dsv_in.

Property Label Property ID Type Description
Directory dir directory

(Required) Specify the absolute path to the data files you want the adapter to read. For example, <username>/<folder name>.

No default value.
File (in Directory) file tables (Required) File to read. No default value.
Stream name, opcode expected expectStreamNameOpcode boolean (Optional) If true, the adapter interprets the first two fields as stream name and opcode respectively. Messages with unmatched stream names are discarded. Default value is false.
Field Count fieldCount uint (Optional) Count of fields in CSV file, if different from the value for the source stream. Default value is 0.
Repeat Count repeatCount uint (Optional) Number of times the input data is repeated. If set to -1, the input data is repeated indefinitely. Default value is 0.
Note: This parameter can be used for testing a continuous streaming source.
Repeat Field repeatField string (Optional) Determines which numeric field's values are bumped on each repeat. Default value is a hyphen (-).
Note:
  • If repeatCount has a nonzero value, specify the stream column name.
  • If the repeatColumn is a key column in the stream, ensure there are no duplicates when specifying multiple rows in the input file.
  • If the adapter is attached to a window, the repeatField must be a key column.
Delimiter delimiter string (Advanced) Symbol used to separate the column. Default value is a comma ( , ).
Has Header hasHeader boolean (Advanced) Determines whether the first line of the file contains the description of the fields. Default value is false.
Directory (runtime) runtimeDir runtimeDirectory (Advanced) Location of the data files at runtime, if the value is different from the location defined at discovery time. No default value.
File Pattern filePattern string (Advanced) Pattern used to look up files for discovery. Default value is *.csv.
Poll Period (seconds) pollperiod uint (Advanced) Period for polling for new contents, in seconds. If set to 0, the File CSV Input adapter will not poll the file for appended records. Default value is 0.
Convert to Safe Opcodes safeOps boolean (Advanced) Converts the opcodes INSERT and UPDATE to UPSERT, and converts DELETE to SAFEDELETE. Default value is false.
Skip Deletes skipDels boolean (Advanced) Skips the rows with opcodes DELETE or SAFEDELETE. Default value is false.
Date Format dateFormat string (Advanced) Format string for parsing date values. Default value is %Y-%m-%dT%H:%M:%S.
Timestamp Format timestampFormat string (Advanced) Format string for parsing timestamp values. Default value is %Y-%m-%dT%H:%M:%S.
Block Size blockSize int (Advanced) Number of records to block into one pseudotransaction. Default value is 1.
Field Mapping permutation permutation (Advanced) Mapping between the in-platform and external fields. No default value.

PropertySet

 propertyset string

(Advanced) Specifies the name of the property set (a group of properties and values) you want to use from the project configuration file. If you specify the same properties in the project configuration file and the ATTACH ADAPTER statement, the values in the property set override the values defined in the ATTACH ADAPTER statement. No default value.
The File CSV Input adapter has these limitations:
  • When polling, you can append to the file, but the file cannot be overwritten or replaced. The stream names in the file rows are ignored and all the data is sent to the same stream.
  • For discovery to work correctly, set the delimiter character and the header presence flag to match the actual data.
  • Do not mix files with different delimiters or files with and without headers in the same directory. Files with wrong delimiters or headers are incorrectly discovered.
Parent topic: Internal Adapters