In this blog post, we’ll examine the principles behind how virtual reality replicates sight, hearing, smell, taste, and touch, and explore the conditions that make a virtual space feel real, as well as the limitations of sensory reproduction technology.
- The Key to Virtual Reality Is the Alignment of Senses and Movement
- Visuals are achieved through a crisp display and precise viewpoint tracking
- To reduce motion sickness, the information received by the eyes and the body must align
- Hearing reproduces the location of sounds and the reverberation of space
- Smell and taste deliver stimuli tailored to the scene
- The sense of touch must distinguish between skin stimulation and the resistance of an object
- Virtual reality is changing the way we experience and share space
The Key to Virtual Reality Is the Alignment of Senses and Movement
Virtual reality (VR) is a technology that creates the sensation of being present in a space that does not actually exist by providing sensory information—such as visuals and sound—in response to the user’s movements. The key lies not in simply stimulating as many of the five senses as possible. Instead, various pieces of information must be consistently linked so that, for example, when you turn your head, you see the scenery in that direction, and sounds seem to come from objects right in front of you. While vision and hearing alone can create a significant sense of presence, we have not yet reached the point where we can freely reproduce all the sensory experiences of the real environment.
When perceiving their surroundings, humans rely not only on the commonly cited five senses but also on the vestibular sense—which detects body rotation and acceleration—and proprioception—which senses the position and movement of the limbs. Therefore, no matter how realistic the visuals in front of the user may be, if they conflict with the information the body is receiving, it can lead to a sense of awkwardness or discomfort. The sense of reality in virtual reality is determined by both the precision of individual senses and the harmony between them.
Virtual reality does not necessarily have to display scenes that are identical to reality. Even a space created with simple graphics can give users the feeling of being inside it if it responds naturally to their actions. Conversely, even if the visuals are as sophisticated as a photograph, the sense of spatial stability collapses if the screen lags behind when the user turns their head. What matters is not only how closely it resembles reality but also how it responds to the user’s actions.
Visuals are achieved through a crisp display and precise viewpoint tracking
The most representative piece of virtual reality equipment is the head-mounted display (HMD). An immersive HMD blocks the user’s external field of view and displays images of the virtual space to both eyes. By providing images with slightly different viewpoints to the left and right eyes, it creates cues that allow the user to perceive depth, just as when viewing real objects with both eyes.
Head tracking is the technology that reflects the user’s movements in the visuals. Sensors and cameras track the direction and position of the head, and the viewpoint within the virtual space is updated accordingly. When the user turns their head to the left, the scenery on the left should be visible, and when they approach an object, that object should appear larger. For the experience to feel natural, the entire virtual space should not move in tandem with the head; rather, objects within the space should appear to remain in place while the user’s viewpoint changes.
Resolution and the optical system are also important. If you magnify the display through a lens, the gaps between pixels may become visible in a grid-like pattern; this is known as the screen-door effect. Improvements in the display’s pixel density and lens design can reduce this phenomenon and render small text or the outlines of objects more sharply.
For example, the Meta Quest 3’s display resolution is 2064×2208 pixels per eye. However, resolution figures alone cannot fully explain the perceived sharpness. Even with the same number of pixels, the level of detail varies depending on how widely they are spread across the field of view, and the characteristics of the lenses and the sharpness at the edges of the screen also play a role. To assess the image quality of an HMD, one must consider not only resolution but also the field of view, optical system, brightness, and contrast.
Added to this is the issue of latency between movement and screen refresh. If there is a delay between the moment a user moves their head and the moment the corresponding image reaches their eyes, the virtual space may appear to shake or seem to lag behind. This latency occurs at various stages, including sensor measurement, motion calculation, image generation, and screen output.
The screen’s refresh rate and this latency are related but are not the same concept. While the refresh rate refers to how often the screen is updated, motion latency refers to the time it takes for actual movement to be reflected on the screen. Even if the screen is updated frequently, if other processing stages are slow, the motion and the image may become out of sync.
Generating continuous images for both eyes at high resolution requires significant computational power. Virtual reality devices utilize graphics processing units, dedicated processors, efficient image generation methods, and motion prediction to enhance both clarity and responsiveness. While increasing resolution does not necessarily mean sacrificing response speed, these two factors must be balanced within the device’s processing capabilities.
To reduce motion sickness, the information received by the eyes and the body must align
Nausea or dizziness experienced in virtual reality is commonly referred to as “cyber sickness.” One of the primary theories explaining this phenomenon is sensory mismatch. When the screen shows the body moving rapidly but the vestibular system fails to detect the corresponding acceleration, the eyes and the body convey conflicting information.
Delays between head movements and the screen can also contribute to this mismatch. However, it is difficult to attribute cyber sickness to a single cause. Factors such as the content’s movement patterns, the characteristics of visual motion, duration of use, and individual sensitivity also play a role; therefore, the degree of discomfort can vary even when using the same device and content.
Consequently, a high-resolution display and low latency alone do not fully resolve motion sickness. Even when the device is functioning normally, the content itself can create sensory mismatches—for example, in situations where the user remains stationary while the virtual viewpoint moves rapidly. Comfort in virtual reality can only be ensured by considering both hardware performance and the design of movement and interaction methods.
Hearing reproduces the location of sounds and the reverberation of space
Humans determine the direction of sound not only by its volume and pitch but also by differences in the time and intensity at which it reaches each ear, as well as changes in timbre caused by the shape of the ears and head. Spatial audio in virtual reality calculates these cues and transmits them through headphones or speakers.
A representative technology is the Head-Related Transfer Function (HRTF). This describes how sound coming from a specific direction changes as it travels to the ear, influenced by the head and the outer ear. Applying these characteristics to a sound source creates the sensation that the sound is coming from specific locations—such as in front, behind, above, or below—rather than simply from the left or right.
Combining this with head tracking allows for a more natural maintenance of the sound’s position. When a sound is coming from the right side of the virtual space and the user turns their head in that direction, the sound should gradually shift to come from directly in front. The more the position of the sound source in the video aligns with the direction perceived by the ears, the stronger the sense of being in a single space becomes.
To express the size and texture of a space, reflected sound and reverberation must also be considered. This is because even the same voice sounds different in a small room versus a large concert hall. However, since the shape of each person’s ears and head varies, the accuracy of directional perception may differ even when the same HRTF is applied. Spatial audio is not a technology that guarantees the exact same auditory experience for every user.
Smell and taste deliver stimuli tailored to the scene
A common method for implementing smell involves releasing actual scent substances near the nose in sync with a virtual scene. For example, when a flower garden appears on the screen, the scent of flowers is delivered. The video itself does not turn into a smell; rather, a device releases a physical scent in response to signals from the digital content.
A study published in 2023 presented a wireless, wearable olfactory device using a small scent generator. This research demonstrates that it is possible to miniaturize scent-emitting devices and control the timing and concentration of emission to integrate them with virtual reality experiences. However, delivering pre-prepared scents is different from freely generating arbitrary odors.
In olfaction, eliminating a scent is just as difficult as emitting one. If the scent from a previous scene lingers in the air, it can mix with the scent of the next scene, and the concentration reaching the nose varies depending on airflow and the user’s position. Since the start and end of scents must be precisely controlled to match scene transitions, a different design approach is required compared to video or sound.
Among the methods for simulating taste are those that apply electrical stimulation or temperature changes to the tongue. Researchers at the National University of Singapore have studied digital taste devices that induce or alter taste sensations through such stimuli. Attempts have also been made to adjust stimulation conditions to target specific sensations, such as research on temperature stimulation to induce a sweet taste.
However, the fact that a taste sensation was reported in a specific experiment does not mean that the same taste can be accurately conveyed to everyone. Responses can vary depending on the location and intensity of the stimulation, temperature changes, and individual sensory differences. Such devices should be distinguished from universal devices designed to replicate the taste of real food exactly as desired.
The “taste” of food as commonly understood in everyday life is a broader experience than just the taste perceived by the tongue. The aroma that reaches the nose when chewing, along with the food’s temperature and texture, all interact to create the overall flavor. Therefore, even if sweet or salty tastes are induced, the softness of bread or the aroma of fruit is not necessarily reproduced at the same time. To experience virtual food as if it were real, multiple senses must be synchronized simultaneously.
The sense of touch must distinguish between skin stimulation and the resistance of an object
The technology that transmits tactile information in virtual reality is called haptics. Vibration from a controller is the most familiar example. When vibrations occur the moment a user touches a virtual object or presses a button, the user can feel that their action has elicited a response. Research is also underway on methods that use gloves or devices attached to the skin to deliver stimulation to a wider variety of locations.
One method of delivering tactile feedback without wearing equipment on the hands is non-contact haptics using ultrasound. By adjusting the output of multiple ultrasonic transducers to concentrate energy on specific points on the hand, it is possible to create sensations on the skin that feel like gentle pressure or vibration. In 2013, researchers at the University of Bristol in the UK announced an “UltraHaptics” system that delivers tactile feedback to various points in mid-air using this method.
This technology can be used to indicate the location of a virtual button or the moment of contact. However, feeling a sensation on the skin is different from having a solid object block the hand. Even if one perceives surface cues in mid-air, it does not support hand movements like an actual desk or provide the same resistance as when lifting a heavy object.
The experience of touching a real object involves not only pressure and vibration on the skin but also the forces acting on the fingers and arms, temperature, and surface texture. Therefore, it is difficult to view non-contact haptics as a complete replacement for wearable devices in every respect. Depending on the type of tactile sensation to be conveyed, one must select or combine devices and methods that implement vibration, pressure, and force resistance.
Virtual reality is changing the way we experience and share space
Corporate investment in virtual reality and related technologies is linked to efforts to extend these experiences into everyday life. In 2014, Facebook announced an agreement to acquire Oculus VR for approximately $2 billion. Since then, devices such as Meta’s Quest lineup and Apple’s Vision Pro have provided a variety of interactions that leverage both immersive virtual spaces and the real world.
Here, it is necessary to distinguish between virtual reality and mixed reality. While virtual reality focuses on replacing the surrounding environment with a virtual space, mixed reality connects the physical space with digital objects. The color pass-through feature on the Quest 3 and Quest 3S displays the user’s surroundings, captured by cameras, on the device’s screen. This allows users to see their real-world room while placing virtual screens or objects within that space.
Mixed reality does not refer to virtual reality that perfectly replicates all five senses. Rather, it is a method of combining digital content while utilizing real-world space as a sensory foundation. Therefore, it is possible to create experiences—such as examining a virtual model on an actual desk—without synthesizing all the senses from the outset.
In education and collaboration, it enables people in different locations to view and manipulate the same virtual model together. After pilot programs with universities, Meta officially launched its educational service in February 2025. Apple also showcased use cases such as 3D design collaboration and employee training using Vision Pro. These applications go beyond simply watching the same video together; their significance lies in jointly examining objects within a space and exchanging actions. However, the mere adoption of such services does not guarantee better outcomes in every class or work setting.
There remains a significant technological gap between virtually touring the streets of New York or experiencing an audiovisual environment that makes you feel as if you’re in a theater, and reproducing even the scent, temperature, and texture of a hot dog eaten on those very streets. Actual usage scope is determined not only by the performance of sensory devices but also by price, comfort, battery life, safety, and the quality of the content.
Virtual reality feels real because sensory information is linked to the user’s actions. Clearer visuals allow users to see objects in greater detail, precise spatial audio indicates the location of sounds, and additional stimuli such as touch and scent enhance the concreteness of the scene. When this information points to the same space and the same event, the experience becomes even more convincing. The key criterion for evaluating the advancement of virtual reality lies not only in the number of senses it implements but also in how accurately and comfortably it combines the necessary senses into a single experience.