Legit Reviews tested the gaming performance of 2GB versus 4GB of Memory on Vista 64-bit. Here's a taster:

Inspired by the Corsair Performance Analysis of 4GB versus 2GB of memory we figured it would be fun to see for ourselves how gaming performance is impacted by adding more system memory. Corsair did a great job on memory usage and the difference between 32-bit and 64-bit operating systems, so instead of trying to re-invent the wheel, we suggest that you check out their report. The Corsair performance analysis only used three games so our goal was to use a wider selection of games and see how much, if at all, the average framerate improved. Our findings, based on nine game titles, have shown that 4GB of memory does offer framerate improvements over using 2GB of memory. The observed difference was so small it would be hard to notice while gaming. Synthetic testing was also a wash since 3DMark 2006 gained performance while 3DMark Vantage seemed to take a slight performance hit.

Our average framerate increase of 1.6% in nine games was slightly higher than the 1.1% shown by Corsair in the three games they tested, but our test system was a little newer and we were running Vista SP1. Something else that we noted while running the benchmarks is that game loading times were also significantly enhanced by installing 4GB of system memory. While that was not the focus of this article (framerates was) we noted that Crysis v1.21 loaded 54.1% faster (14.28s versus 22.00s) on the initial level load. After the level was loaded and then restarted (as if one died and started over) the load times were within a hundredth of a second. Obviously, having more memory will also help other areas and at the end of the day it helps performance. If you're going to be building a new gaming PC then by all means use 4GB as it does help performance, but don't expect a night and day difference in the gaming benchmarks.