KBI 310341 Alerts Delayed By 5 to 10 Minutes
Version
Argent AT — All Versions
Date
7 Jan 2013
Summary
Customers may suffer a delay of up to 10 minutes in delivery of any Alert, say, an Email Alert
This can be seen by referring to the Event timestamp of the alert and the email delivery time showed on email client
Customers may also see numerous (30+) Argent monitoring processes at any given time, which may result in the overconsumption of non-interactive desktop heap
Technical Background
Assuming the email gateway is working correctly, then the next area to check is the Argent AT server load, this is best done by checking CPU and memory:
A delay in Relator Task execution is an key indicator of the Argent Engine being overloaded
For example, a Relator is configured to run every two minutes, but the “Task Status And History” screen below shows delays and irregular execution time
An even better indicator is a sustained high skipped rate found in the product service log:
Resolution
Argent AT provides two methods of executing tasks — this is configured on a per-Relator basis.
A new process can be spawned for each task, or a shared process can be used when executing the task.
In Argent AT, there are implications to consider when using either method:
Spawn New Monitor Engine Process
For each task, a new process is spawned. This allows tasks to execute independently, and in some cases concurrency e.g. (in parallel — such as vCenter, which only allows one connection per process)
However, this also increases the resource usage of creating new processes, and creating new initial connections to a remote server or application
Use Shared Monitor Engine Process Pool
For each task, a shared process pool is re-used. This re-uses open connections to the remote server which significantly reduces the load.
Less processes are spawned, but if a particular task fails and crashes the process, it affects other tasks as well.
Argent AT assigns three shared process pools by default. You can specify the exact process pool to use (Pool #1, Pool #2, Pool #3), or you can use the {Dynamic} option, which dynamically allocates the task to the next freed up Pool.
In other words, under the default settings, and when {Dynamic} is used with shared process pools – up to three simultaneous tasks can be executed at one time – the rest will be queued.
Changing to Use Shared Monitor Engine Process In Pool will always significantly save on resources.
If the default of three process pools is not enough, you can increase the total number of shared process pools by going to Administration, Supervising Engine, and changing the Shared Monitor Engine Process Pool Size.
Example: From 3 to 15.
This simply means that a maximum of 15 shared process pools will be spawned instead of the default of 3
After making this change, you can see that this has taken effect by going back to your Relator.
You’ll see the number of Pools have increased to what you specified:
If you have hundreds of Relators and would like an automated approach to changing all Relators to use the dynamic/shared process pool, please contact Argent Support