KBI 311466 Memory Balloons On Server Requiring A Hard Reboot

Version

All Versions

Date

Wednesday, 19 October 2016

Summary

In certain cases Microsoft patches can break Argent and the production server

Recently a patch was made to update Internet Explorer from version 8 to 11

This bad Microsoft patch caused Argent Monitoring Engine processes that were spawned from Argent Sentry Ultra and Argent Guardian Ultra to not cleanly exit, resulting in virtual memory ballooning to the point remote access via Remote Desktop (RDP) was not possible and a hard reboot was required as a soft reboot failed to shut down (or restart)

Be extremely careful applying ANY Microsoft patches

Argent strongly recommends AGAINST automatic patching — this is always a weak and lazy engineer’s approach; NEVER CHANGE A WORKING PRODUCTION ARGENT SERVER WITHOUT FIRST TESTING THE PATCH

Technical Background

Analysis showed the system running process count had risen from 150 to 2,000 and approximately 1,800 where from the Argent Monitoring Engine processes and the process memory (private working set) had changed from initially 35MB (after monitor task had completed) to 500KB, with a ‘commit size’ therefore ~34MB which now is on the Paging File

A manual process kill via TASKKILL interestingly said ‘Successful’ but didn’t actually free the memory or remove the process from the Windows Process List (Processes) in Windows Task Manager TASKMGR

The resultant behavior shows the ‘System Commit Size’ (Task Manager – Performance – System) ballooned out to 100GB (initially 48GB (32GB physical memory)) and rising

Specific Case

A computer with Microsoft Windows 2008 R2 (x64) SP1 (6.1.7601) with no individual Windows Updates was updated from IE8 to IE11, but for this update several prerequisites were required, see list below:

KB 2888049

Update is available that improves the network performance of Internet Explorer 11 in Windows

– After you install this update, Windows sends ACK messages without a delay when it uses Transmission Control Protocol (TCP) protocol connections

KB 2882822

Update adds ITraceRelogger interface support to Windows Embedded Standard 7 SP1, Windows 7 SP1 and Windows Server 2008 R2 SP1

– The iTraceRelogger interface is a dependency for certain features to work (for example, the UI Responsiveness tool in Internet Explorer 11 F12 tools

KB 2786081

Internet Explorer 10 does not save credentials for a website after you log off or restart a computer that is running Windows 7 SP1 or Windows Server 2008 R2 SP1

KB 2670838

Platform update for Windows 7 SP1 and Windows Server 2008 R2 SP1

This update improves the range and performance of the following graphics and imaging components:

Direct2D, DirectWrite, Direct3D, Windows Imaging Component (WIC),

Windows Advanced Rasterization Platform (WARP),

Windows Animation Manager (WAM), XPS Document API,H.264 Video Decoder, JPEG XR codec

Issue resolved by removing all the installed Windows Updates

Conclusion

Be very careful and use a dedicated change control system to make changes to a production server.

Specifically for Windows, use a test (with Argent installed) server to perform the update, or update from one Service Pack to another only

Resolution

N/A