In collaboration with zSpace, we developed high speed hand gesture recognition system and combined it with a 3D virtual holographic display system developed by zSpace. By integrating these two systems, an immersive virtual environment can be realized. Based on the Janken robot recognition system developed earlier, the high speed 3D hand gesture interface utilizes super high speed stereo cameras making it possible to recognize gesture and track 3D position (500 fps) with extremely small latency. With such small latency, very fast and tiny movements, such as shaking, can be detected easily. Not only fast, the system is also correctly aligned with the real world, making the interaction very intuitive. With this system, user can manipulate very high quality 3D image using only their hands.