KBI 311466 Memory Balloons On Server Requiring A Hard Reboot
Version
All Versions
Date
Wednesday, 19 October 2016
Summary
In certain cases Microsoft patches can break Argent and the production server
Recently a patch was made to update Internet Explorer from version 8 to 11
This bad Microsoft patch caused Argent Monitoring Engine processes that were spawned from Argent Sentry Ultra and Argent Guardian Ultra to not cleanly exit, resulting in virtual memory ballooning to the point remote access via Remote Desktop (RDP) was not possible and a hard reboot was required as a soft reboot failed to shut down (or restart)
Be extremely careful applying ANY Microsoft patches
Argent strongly recommends AGAINST automatic patching — this is always a weak and lazy engineer’s approach; NEVER CHANGE A WORKING PRODUCTION ARGENT SERVER WITHOUT FIRST TESTING THE PATCH
Technical Background
Analysis showed the system running process count had risen from 150 to 2,000 and approximately 1,800 where from the Argent Monitoring Engine processes and the process memory (private working set) had changed from initially 35MB (after monitor task had completed) to 500KB, with a ‘commit size’ therefore ~34MB which now is on the Paging File
A manual process kill via TASKKILL interestingly said ‘Successful’ but didn’t actually free the memory or remove the process from the Windows Process List (Processes) in Windows Task Manager TASKMGR
The resultant behavior shows the ‘System Commit Size’ (Task Manager – Performance – System) ballooned out to 100GB (initially 48GB (32GB physical memory)) and rising
Specific Case
A computer with Microsoft Windows 2008 R2 (x64) SP1 (6.1.7601) with no individual Windows Updates was updated from IE8 to IE11, but for this update several prerequisites were required, see list below:
KB 2888049
Update is available that improves the network performance of Internet Explorer 11 in Windows
– After you install this update, Windows sends ACK messages without a delay when it uses Transmission Control Protocol (TCP) protocol connections
KB 2882822
Update adds ITraceRelogger interface support to Windows Embedded Standard 7 SP1, Windows 7 SP1 and Windows Server 2008 R2 SP1
– The iTraceRelogger interface is a dependency for certain features to work (for example, the UI Responsiveness tool in Internet Explorer 11 F12 tools
KB 2786081
Internet Explorer 10 does not save credentials for a website after you log off or restart a computer that is running Windows 7 SP1 or Windows Server 2008 R2 SP1
KB 2670838
Platform update for Windows 7 SP1 and Windows Server 2008 R2 SP1
This update improves the range and performance of the following graphics and imaging components:
Direct2D, DirectWrite, Direct3D, Windows Imaging Component (WIC),
Windows Advanced Rasterization Platform (WARP),
Windows Animation Manager (WAM), XPS Document API,H.264 Video Decoder, JPEG XR codec
Issue resolved by removing all the installed Windows Updates
Conclusion
Be very careful and use a dedicated change control system to make changes to a production server.
Specifically for Windows, use a test (with Argent installed) server to perform the update, or update from one Service Pack to another only
Resolution
N/A