We present a virtual reality display system simulator that accurately reproduces gaze-contingent distortions created by any viewing optic. The simulator hardware supports rapid prototyping by presenting stereoscopic distortions on a high-speed television paired with shutter glasses, eliminating the need to fabricate physical optics. We further introduce light field portals as an efficient and general-purpose representation for VR optics, enabling real-time emulation using our simulator. This platform is used to conduct the first user study of perceptual requirements for eye-tracked optical distortion correction. Because our hardware platform facilitates consistent head and eye movements, it enables direct comparison of these requirements across observers, optical designs, and scene content. We conclude by introducing a simple binocular distortion metric, built using light field portals, which agrees with key trends identified in the user study and lays a foundation for the design of perceptually-based distortion metrics and correction schemes.