Example demonstrating how to determine the source of performance issues in an ESP project and increase throughput using automatic partitioning. Automatic partitioning is the creation of parallel instances of an element and splitting input data across these instances. Partitioning project elements this way can result in an overall project throughput increase as the workload is split across the parallel instances.
See Automatic Partitioning in the Programmers Guide for additional information.
While partitioning can increase the project throughput, it also introduces additional elements into the data flow pipeline which can impact the project latency. In production environments however, the overhead should be marginal.
The performance statistics presented in this example are for demostration purposes only and will vary from system to system depending on hardware configuration. The PerformanceTuningUsingPartitioning projects consists of four elements: an input window (Feed) and three output windows (AvgLong, AvgShort, and AvgCompare). The Feed input window receives data from the XML Input adapter, Adapter1.
- Identify the overall project performance and bottlenecks.
- Identify your available system resources.
- Partition the bottleneck elements.
- Repeat steps above until all elements in the project share equal load.
- Identify the source of the performance issues in the
PerformanceTuningUsingPartitioning project by first running it without any
partitioning enabled. Use the esp_monitor command, or
alternatively, the Performance Monitor in the
ESP Studio:
For additional details on the esp_monitor command, see esp_monitor in the Utilities Guide. For additional details on the Performance Monitor, see the Performance Monitor section in the Studio Users Guide.
- If using the esp_monitor command:
- Start Event Stream Processor.
- Using the esp_cluster_admin command, add the PerformanceTuningUsingPartitioning project from $ESP_HOME/examples/ccl.
- Use the esp_cluster_admin command to start the project.
- Use the _ESP_Streams_Monitor stream, which is a part of
esp_monitor output, to determine which
element has the highest CPU usage. For
example:
<_ESP_Streams_Monitor ESP_OPS="u" stream_name="Feed" target="Feed" cpu_pct="18.720100" trans_per_sec="13824.000000"… <_ESP_Streams_Monitor ESP_OPS="u" stream_name="AvgShort" target="AvgShort" cpu_pct="59.280400" trans_per_sec="13814.000000"… <_ESP_Streams_Monitor ESP_OPS="u" stream_name="AvgLong" target="AvgLong" cpu_pct="99.840600" trans_per_sec="13634.000000"… <_ESP_Streams_Monitor ESP_OPS="u" stream_name="AvgCompare" target="AvgCompare" cpu_pct="49.920300" trans_per_sec="27437.000000"…
Actual values may vary from system to system depending on your hardware configuration. By looking at the cpu_pct field, you can see that the AvgLong element has the highest CPU consumption (99.840600) and therefore, is the bottleneck in this project. The rate at which data flows through the project is determined by the number of transactions processed by the first element, the input window (Feed), and equals about14,000 transactions per second.
- If using the ESP Studio:
- Start Event Stream Processor.
- Start the Studio and load the PerformanceTuningUsingPartitioning example from $ESP_HOME/examples/ccl.
- Start the PerformanceTuningUsingPartitioning project.
- In the SAP Sybase ESP Run-Test perspective, select the Monitor view.
- Click Select Running Project (
).
- Click OK.
- Select Rows Processed and leave the default coloring. The adapter starts to load data and fill the windows.
- Hover your cursor over the elements in the project to determine which element has the highest CPU usage. Actual values may vary from system to system depending on your hardware configuration. In this case, AvgLong has 99.840 CPU usage and therefore, is the bottleneck in this project.
- If using the esp_monitor command: