File/Hadoop JSON Input Adapter Configuration

Type: string

(Optional) Specify a full path to the log4j.properties logging file you wish to use. The default value is $ESP_HOME/adapters/framework/config/log4j.properties.

(Required) Element containing all information for this module. It contains a type attribute for specifying the module type.

For example, transporter.

Type: string

(Required) Instance name of the specific module you want to use. For example, MyInputTransporter.

Type: string

(Required) Name of the module as defined in the modulesdefine.xml file. For example, <TransporterType>InputTransporter.

Type: string

(Required) Instance name of the module that follows this one.

Type: integer

(Advanced) Capacity of the buffer queue between this module and the next. The default value is 10240.

(Required) Element containing the FileInputTransporterParameters element.

(Required) Element containing elements for the File Input transporter.

Type: string

(Required) Specify the absolute path to the data files which you want the adapter to read. For example, <username>/<foldername>. No default value.

To use Hadoop system files, use an HDFS folder uri instead of a local file system folder. For example, hdfs://<hdfsserver>:9000/<foldername>/<subfoldername>/<leaffoldername>.

To use Hadoop, download the binaries for Hadoop version 1.2.1 from http://hadoop.apache.org. Copy the hadoop-core.jar file (for example, for version 1.2.1 hadoop-core-1.2.1.jar) to %ESP_HOME%\adapters\framework\libj. Ensure you use a stable version rather than a beta.

Use a forward slash for both UNIX and Windows paths.

Type: string

(Required) Specify the file you want the adapter to read or the regex pattern to filter the files on a given directory. See the DynamicMode element. No default value.

Type: string

Type: string

An example regex pattern is ".*\.txt", which selects only files that end with ".txt". In regex patterns, you must include an escape character, "\", before meta chars to include them in the pattern string.

Type: integer

(Advanced) Define the period, in seconds, to poll the specified file or directory. Set this element only if the value of the DynamicMode element is set to dynamicFile or dynamicPath.

The default value is 0, which, along with all other values less than 0, turns off polling.

Type: boolean

(Optional) If this property is set to true, the file is removed from the directory after the adapter processes it. This element takes effect if the value of the DynamicMode element is set to dynamicPath and ignored if it is set to dynamicFile instead.

The default value is false.

Type: integer

(Optional) Specify the depth of the schema discovery. The adapter reads the number of rows specified by this element value when discovering the input data schema.

The default value is three.

(Required) Element containing all information for this module. It contains a type attribute for specifying the module type.

For example, formatter.

Type: string

(Required) Instance name of the specific module you want to use. For example, MyInputTransporter.

Type: string

(Required) Name of the module as defined in the modulesdefine.xml file. For example, <TransporterType>InputTransporter.

Type: string

(Required) Instance name of the module that follows this one.

Type: integer

(Advanced) Capacity of the buffer queue between this module and the next. The default value is 10240.

(Required) Element containing the JsonStreamToJsonStringFormatterParameters element.

(Required) Element containing elements for the Streaming JSON to JSON String formatter.

Type: string

(Optional) Specify the name of a supported charset. The default value is US-ASCII.

(Required) Element containing all information for this module. It contains a type attribute for specifying the module type.

For example, formatter.

Type: string

(Required) Instance name of the specific module you want to use. For example, MyInputTransporter.

Type: string

(Required) Name of the module as defined in the modulesdefine.xml file. For example, <TransporterType>InputTransporter.

Type: string

(Required) Instance name of the module that follows this one.

Type: integer

(Advanced) Capacity of the buffer queue between this module and the next. The default value is 10240.

(Required) Element containing the JsonStringToEspFormatterParameters element.

(Required) Element containing the JSON String to ESP formatter elements.

(Required) Element containing the ColsMapping element.

Type: complexType

(Required) Element which contains the Column element. You can have multiple ColsMapping elements if you are using an ESPMultiStreamPublisher.

Type: string

(Required) Specify a JSONPath expression for the JSON data that you want to map to columns of an ESP stream. This expression is matched to the value specified in the rootpath attribute of the ColsMapping element, if applicable. You can have multiple Column elements.

{
"firstName": "John",       
"lastName": "Smith", 
"phoneNumbers": [
        {
            "type": "home",
            "number": "212 555-1234"
        },
        {
            "type": "fax",
            "number": "646 555-4567"
        }
],
"friends": [
["female1","female2","female3"],
["male1","male2","male3"]
]
}

If you want the numbers and types of phone numbers, specify phoneNumbers as the rootpath value and numbers and type as the JSONPath expressions in the Column element.

You can also specify * for Column which indicates you want all the data in an array (that does not have key value data within it). For example, if you wanted all the female friends data, specify friends[1] for rootpath and * for Column.

The first <Column/> element and its value are mapped to the first column of an ESP stream, the second <Column/> element and its value are mapped to the second column of an ESP stream, and so on.

Type: string

(Advanced) The format string for parsing date values. For example, yyyy-MM-dd'T'HH:mm:ss.

Type: string

(Advanced) Format string for parsing timestamp values. For example, yyyy-MM-dd'T'HH:mm:ss.SSS.

(Required) Element containing all information for this module. It contains a type attribute for specifying the module type.

For example, formatter.

Type: string

(Required) Instance name of the specific module you want to use. For example, MyInputTransporter.

Type: string

(Required) Name of the module as defined in the modulesdefine.xml file. For example, <TransporterType>InputTransporter.

Type: integer

(Advanced) Capacity of the buffer queue between this module and the next. The default value is 10240.

(Required) Element containing the EspPublisherParameters element.

(Required) Element containing elements for the ESP publisher.

Type: string

(Required if adapter is running in standalone mode; optional if it is running in managed mode) Name of the ESP project to which the adapter is connected. For example, EspProject2.

This is the same project tag that you specify later in the adapter configuration file in the Name element within the Event Stream Processor (EspProjects) element.

If you are starting the adapter with the ESP project to which it is attached (that is, running the adapter in managed mode), you need not set this element as the adapter automatically detects the project name.

Type: string

(Required if adapter is running in standalone mode; optional if it is running in managed mode) Name of the ESP stream to which the adapter publishes data.

If you are starting the adapter with the ESP project to which it is attached (that is, running the adapter in managed mode), you need not set this element as the adapter automatically detects the stream name.

Type: positive integer

(Optional) Maximum size of the record pool. Record pooling, also referred to as block or batch publishing, allows for faster publication since there is less overall resource cost in publishing multiple records together, compared to publishing records individually.

Record pooling is disabled if this value is 1. The default value is 256.

Type: positive integer

(Optional) Maximum period of time, in milliseconds, for which records are pooled before being published. If not set, pooling time is unlimited and the pooling strategy is governed by maxPubPoolSize. No default value.

Type: boolean

(Optional) If set to true, pooled messages are published to Event Stream Processor in transactions. If set to false, they are published in envelopes. The default value is false.

Type: boolean

(Advanced) Converts the opcodes INSERT and UPDATE to UPSERT, and converts DELETE to SAFEDELETE. The default value is false.

Type: boolean

(Advanced) Skips the rows with opcodes DELETE or SAFEDELETE. The default value is false.