Most RepAgent memory comes from allocated Adaptive Server procedure cache (shared memory). It is used for:
Overhead
Schema cache
Transaction cache
Text and image cache
Refer to the Adaptive Server Enterprise System Administration Guide Volume 2 for information about increasing server memory.