Adapter type: dsv_in. The File CSV Input adapter reads a file in Event Stream Processor delimited format.
Use this adapter to poll new data appended to the data file. The file does not require a header. If the file includes a header, it specifies the field names.
1. hasHeader=true delimiter=, expectStreamNameOpcode=false Ts,ItemID,Price,Quantity,WarehouseZipCode,DeliveryZipCode 2004/06/17 10:00:00.000000,SKU1276532,50.00,1,10012,94086 2004/06/17 10:00:05.000000,SKU6723143,23.00,2,10012,94043 2. expectStreamNameOpcode=true delimiter=, Trades_in,i,2004/06/17 10:00:00.000000,SKU1276532,50.00,1,10012,94086 Trades_in,i,2004/06/17 10:00:05.000000,SKU6723143,23.00,2,10012,94043 3. expectStreamNameOpcode=false timestampFormat=%Y/%m/%d %H:%M:%S delimiter=, 2004/06/17 10:00:00.000000,SKU1276532,50.00,1,10012,94086 2004/06/17 10:00:05.000000,SKU6723143,23.00,2,10012,94043
This adapter supports schema discovery in normal mode only. If you use the CCL ATTACH ADAPTER statement to attach an adapter, you must supply the adapter type.
The File CSV Input adapter operates in two loading modes: dynamic and normal. In normal mode, the adapter reads records from a single source file until there are no more records or you stop the process. In dynamic mode, the adapter reads records from a source directory on a first-come, first-served basis. These files follow a format defined by a regular expression or the filePattern parameter. You can also specify a POSIX standard regular expression, such as [a-z]{4}\.csv, or a wildcard character sequence that is supported by the native operating system, such as *.csv. For example, if the expression is *.csv, then the adapter searches for .csv files only. After reading each file, the adapter checks the directory for new files and continues processing. If there is an error in opening or reading a file, the adapter skips it.
In dynamic mode, if a file is overwritten or updated after the adapter processes it and the reprocess parameter is set to false, the file it is not processed again. If the removeAfterProcess parameter is set to false, the adapter does not delete the file after it processes it. Both reprocess and removeAfterProcess can be set to false simultaneously, but only one can be set to true at a time.
Property Label | Description |
---|---|
Directory |
Property ID: dir Type: string (Required for adapter operation and schema discovery) Specify the absolute path to the data files you want the adapter to read. For example, <username>/<folder name>. Default value is "." (current directory in which the Server is running). Use a forward slash for both UNIX and Windows paths. |
File (in Directory) |
Property ID: file Type: tables (Dependent required) File to read. Set only in normal mode; in dynamic mode, the value is ignored. No default value.. |
Stream name, opcode expected |
Property ID: expectStreamNameOpcode Type: boolean (Required) If true, the adapter interprets the first two fields as stream name and opcode respectively. Messages with unmatched stream names are discarded. When you are using schema discovery on this adapter with this property enabled, two columns for the stream name and opcode are created in the schema. Manually remove these two columns from your schema. Default value is false. |
Field Count |
Property ID: fieldCount Type: uint (Optional) Count of fields in CSV file, if different from the value for the source stream. Default value is 0. |
Repeat Date Field Count |
Property ID: repeatCount Type: int (Optional) Number of times the input data is
repeated. If set to -1, the input data is repeated indefinitely.
Default value is 0.
Note: You can use this
parameter to test a continuous streaming source.
|
Repeat Data Field Name |
Property ID: repeatField Type: string (Optional) On each repeat, increment the value in
this field. You must specify the stream column if repeatCount has a nonzero value.
Default value is a hyphen (-).
|
Delimiter |
Property ID: delimiter Type: string (Advanced) Symbol used to separate the column. Default value is a comma ( , ). |
Has Header | Property ID: hasHeader Type: boolean (Advanced) Determines whether the first line of the file contains the description of the fields. Default value is false . |
Directory (runtime) |
Property ID: runtimeDir Type: runtimeDirectory (Advanced) Location of the data files at runtime, if the value is different from the location defined at discovery time. No default value. |
File Pattern | Property ID: filePattern Type: string (Advanced) In dynamic mode, the filePattern looks up files in a directory. In normal mode, it is the pattern used to look up files for discovery. If both filePattern and fileRegex are specified, filePattern is ignored and fileRegex is used. filePattern allows wildcard characters supported by the native operating system. Default value is *.csv. |
Regular Expression |
Property ID: fileRegex Type: string (Optional) In dynamic mode, the regular expression property allows the user to specify a POSIX regular expression standard to find matching file names in the directory. If both filePattern and fileRegex are specified, then filePattern is ignored and fileRegex is used. No default value. |
Poll Period (seconds) |
Property ID: pollperiod Type: uint (Advanced) In normal mode, specifies the period for polling a file to check for new records. pollperiod is ignored in dynamic mode. Default value is 0. |
Directory Poll Period |
Property ID:dirPollPeriod Type: uint (Advanced) In dynamic mode, continuously checks for new files in the directory. Default value is 60 seconds. |
Convert to Safe Opcodes |
Property ID: safeOps Type: boolean (Advanced) Converts the opcodes INSERT and UPDATE to UPSERT, and converts DELETE to SAFEDELETE. Default value is false. |
Skip Deletes |
Property ID: skipDels Type: boolean (Advanced) Skips the rows with opcodes DELETE or SAFEDELETE. Default value is false. |
Date Format |
Property ID: dateFormat Type: string (Advanced) Format string for parsing date values. Default value is %Y-%m-%dT%H:%M:%S. |
Timestamp Format |
Property ID: timestampFormat Type: string (Advanced) Format string for parsing timestamp values. Default value is %Y-%m-%dT%H:%M:%S. |
Block Size |
Property ID: blockSize Type: int (Advanced) Number of records to block into one pseudotransaction. Default value is 1. |
Use Envelopes |
Property ID: useEnvelopes Type: boolean (Advanced) Specify the block type the adapter uses to pass data to the engine. If you specify a blockSize property greater than zero, by default, the adapter packages rows into transaction blocks to send to the engine. To get the adapter to package rows into envelope blocks instead, set this property to true. Default value is false. |
Field Mapping |
Property ID: permutation Type: permutation Mapping between Event Stream Processor and external fields, for example: <esp_columnname>=<database_columnname>:<esp_columnname>=<database_columnname>. No default value. |
PropertySet |
Property ID: propertyset Type: string (Advanced) Specifies the name of the property set. Property sets are reusable sets of properties that are stored in the project configuration file. Using these sets allows you to move adapter configuration properties out of the CCL file and into the CCR file. If you specify the same properties in the project configuration file and the ATTACH ADAPTER statement, the values in the property set override the values defined in the ATTACH ADAPTER statement. No default value. |
Dynamic Loading |
Property ID: dynamicMode Type: boolean (Optional) Set to true to enable dynamic loading mode, which causes the adapter to read records from files as they arrive in the directory. Records are processed one after another on a first-come, first-serve basis. Default value is false. |
Remove File After Processing | Property ID: removeAfterProcess Type: boolean (Optional) In dynamic mode, if removeAfterProcess is set to true, the file is removed from the directory after the adapter processes it. You cannot set this property to true if reprocess is also set to true. Default value is false . |
Reprocess File |
Property ID: reprocess Type: boolean (Optional) In dynamic mode, if reprocess is set to true, the file is processed again after being updated or overwritten. You cannot set this property to true if removeAfterProcess is also set to true. Default value is false. |