Performance and Tuning Tips

Optimizing performance in Sybase Event Stream Processor requires tuning at the project level as well as at the infrastructure level (machine, OS, network configuration, and so on).

If you tune your projects to produce maximum throughput and minimum latency but do not configure your infrastructure to handle the throughput, you will see sub-optimal performance. Likewise, if you configure your infrastructure to handle maximum throughput but do not tune your projects, your performance suffers.

• Distributing Load through Parallelization
  To maximize performance of large ESP projects, separate the data into smaller chunks that are processed within their own partitions. Processing on multiple partitions in parallel greatly improves performance over processing in one large partition.
• Distributing Load through Modularization
  You can optimize performance by breaking projects into modules. This strategy spreads the load out to more cores, thereby increasing throughput.
• Data Flow in Event Stream Processor
  The throughput of an Event Stream Processor project depends on the throughput of the slowest component in the project.
• Log Store Considerations
  The size and location of your log stores can impact performance.
• Batch Processing
  When stream processing logic is relatively light, inter-stream communication can become a bottleneck. To avoid such bottlenecks, you can publish data to the ESP server in micro batches. Batching reduces the overhead of inter-stream communication and thus increases throughput at the expense of increased latency.
• Main Memory Usage
  There are no Sybase Event Stream Processor configuration settings that directly set up or control RAM usage on the machine. However, ESP reference counts records in the system, ensuring that at most one copy of a record is present in memory, although multiple references to that record may exist in different streams.
• Determining Stream Memory Usage
  When the server is running at log level 7 and it is shutdown cleanly, it reports the amount of memory consumed by every stream and any aggregation indices in the server log file.
• CPU Usage
  Sybase Event Stream Processor automatically distributes its processing load across all the available CPUs on the machine. If the processing of a data stream seems slow, monitor each stream's CPU utilization using either the esp_monitor utility from the command line or through Sybase Control Center. If the monitoring tool shows one stream in the project using the CPU more than other streams, refine the project to ensure that the CPU is used evenly across the streams.
• TCP Buffer and Window Sizes
  High throughput data transfers between clients and Sybase Event Stream Processor rely on the proper tuning of the underlying operating system's TCP networking system.