Version

Argent Advanced Technology all versions

Date

Wednesday, 25 March 2015

Summary

Argent Guardian Ultra may suddenly stop firing new Events or Auto Correcting Existing Events

Inspecting system health using Global System View shows no obvious issues

Relator trace log shows monitoring is being executed successfully

There is no file accumulation in PENDING_EVENTS folder

There is some file accumulation in POST_EVENTS folder, usually small amount less than 100 files

The only anomaly is that following lines show up frequently in the Engine Service Log:

Could not read status of event (#xxxxxx). Assume it is completed

In the meantime, Argent for Compliance or Argent for Security may have generated more than 3,000 Events and growing

Inspecting the generated Events, customer can recognize that latest fired Event is about Windows Event 4 or 5 hours ago

The flood of Events in Argent for Compliance or Argent for Security causes severe delay in processing of Events from all other products including Argent Guardian Ultra

The issue is caused by incorrect configuration of Argent AT

It can be addressed by throttling Argent Console, as well as identifying and correcting the ill-configured Event Log Rules

Technical Background

Argent Console Engine is shared among all Argent AT products

If one product floods the Argent Console Engine, all other products will be affected

The most common scenarios in Argent for Compliance or Argent for Security include:

1. The Event filter somehow let every Windows Security Log Event pass

   A typical Domain Controller can generate 10 million Event Log records per day

2. Rule is configured to fire Events for Windows Error Events only

   However, because of network or hardware issue, there is a sudden burst of error Events

   A typical Windows server can generate more than 10,000 error Events within 24 hours

Customer can easily identify the culprit Rule by reading Alert Email or checking the Events on Argent Console screen

If the Rule is incorrectly configured, this root cause must be corrected to prevent such a mishap in the future

Resolution

Argent Console Engine also has a very good throttling mechanism built in

Set the Maximum Pending Events To Process

Set value to 100 or up to 300

No Administrator reads more than 300 Events from the same source

If that happens, there must be some configuration issue that should be corrected

Click For Full Size

Argent Console Engine may have already queued more than 1,000 Events

Customer needs to wipe out the queued Events too by taking following steps:

1. Stop Argent Console Service
2. Delete file \ARGENT\ARGENTCONSOLE\DATA\ AAC_ALERTS.DAT
3. Start Argent Console service