Guidelines for Partitioning Modules

You can partition an entire module when loading it (using the LOAD MODULE statement) by specifying which stream you wish to partition and the partitioning method. You can only partition a module by its input streams or windows. Mention them only once, even if they are mapped to multiple internal module streams or windows.

A module can contain a partitioned stream or load other partitioned modules. However, be aware that the total amount of partitions that your ESP project has overall can quickly grow. Specifically, if you partition a module, all internal module partitions are partitioned again when loading the module. For example, if a module partitions a stream by 5 and the module is partitioned by 4, you would have 20 parallel instances of the stream that the module partitions.

If you create too many partitions, the ESP Server may not be able to start properly. It is recommended that the number of partitions within your ESP project stays below the number of processors on your machine.

It is not possible to partition a stream that has more than one downstream connection. As a result, ESP Server inserts a copy of the partitioned stream between the stream and the partitioner. This copy does the distribution to all the successor nodes. The ID of this copied node is suffixed by .clone.

The example below displays the principle of partitioning modules. You can find the complete and running example in $ESP_HOME/examples/ccl/SubmodulesPartitioned. Ensure you specify the PARTITION BY clause last in the LOAD MODULE clause.

import 'module1.ccl'; 

CREATE INPUT Window InStocks SCHEMA StocksSchema Primary Key (Ts)KEEP ALL;

LOAD MODULE Module1 AS Module1_instance_01 
   IN rawStockFeed = InStocks
   OUT infoByStockSymbol = CompStocks2 
   Parameters myparam = 1000
   STORES store1 = MyStore1
   PARTITION 
BY InStocks HASH (Ts)
   PARTITIONS 3;