If this is an IIS based relay server, please check if the application pool setting is limiting memory usage of your relay server web server extensions. Increasing the maximum number of worker processes in the application pool on 32-bit IIS will help spread the load. Upgrading to a 64-bit relay server or adding relay server to the relay server farm can also be valid solutions. Offloading other applications running on the relay server computer to other computers may also help.