The problem affects:
VMware vCenter Server 5.0 Update 1 and modules 5.0.0U1 (VMware-VIMSetup-all-5.0.0-639890.iso) VMware Tools 8.6.5 build-621624 vsepflt.sys 5.0.0.0 build-443031
The problem is fixed in:
VMware Tools 8.6.5 build-731933 vsepflt.sys 5.0.0.1 build-652273
VMware vShield Endpoint Thin Agent 1.0 Update 3 Release Notes
After installation of Thin Agent drivers (vShield Endpoint 1.0.0, vShield Endpoint 1.0.0 Update 1, or vShield Endpoint 1.0.0 Update 2) on Windows XP/2003 guests, overall memory usage of the guest VM may increase substantially with all processes consuming higher amounts of memory than usual. This behavior is not observed in later versions of Windows viz. Vista and later. It results from a combination of the way the Thin Agent reads a file during its scan and Windows internal behavior.