How did HMDs evolve from video viewing devices into virtual reality and mixed reality devices?

In this blog post, we’ll examine the principles behind how HMDs deliver video and the relationship between field of view and image quality, and explore how personal video viewing devices evolved into virtual reality and mixed reality devices capable of recognizing movement and the surrounding space.

 

From a Device for Experiencing Space on a Screen Right Before Your Eyes

The key driver behind the evolution of HMDs is the combination of display and optical technologies—which render small screens with high clarity—and tracking and processing technologies that reflect the user’s movements in the video. With the addition of cameras and sensors that recognize the surrounding environment, HMDs have evolved beyond mere video-viewing devices into tools for placing and manipulating digital content within the real world. However, improvements in resolution and functionality do not automatically guarantee a comfortable user experience. Weight, heat generation, visual fatigue, price, and content usability remain significant challenges.
HMD stands for Head-Mounted Display and refers to a display device worn on the head. Although it is sometimes used interchangeably with “virtual reality device,” not all HMDs deliver a virtual reality experience. Personal video viewing devices that display a movie screen right in front of the user’s eyes also fall under the category of HMDs. Virtual reality (VR) provides users with an experience that makes them feel as if they have entered a virtual environment, while mixed reality (MR) combines the real world with digital content. “HMD” describes the form of the device, whereas “VR” and “MR” describe the nature of the experience the device provides.
One example of this shift is the Gear VR, which Samsung Electronics unveiled in 2014 in collaboration with Oculus. The early Gear VR utilized a Galaxy Note 4, leveraging the smartphone’s display and processing power. By utilizing a high-resolution Super AMOLED screen for the VR experience, it served as an example of how advancements in smartphone technology had extended to head-mounted devices. However, the Gear VR was not the starting point for HMDs. Prior to that, personal video viewing devices and various research and industrial HMDs had already been developed.

 

How does a small display become a wide, three-dimensional screen?

The basic structure of VR headsets and personal video-viewing HMDs consists of a combination of a display and lenses. Instead of simply viewing the small screen placed directly in front of the eyes, the lenses make it appear as though a large screen is located at a certain distance ahead. Therefore, the actual distance between the display and the eyes differs from the perceived distance of the screen. This is why the placement of the lenses and the screen is so important.
To create a stereoscopic image, the left and right eyes are presented with images from different viewpoints. Just as a person’s two eyes view the same object from slightly different angles, a difference is introduced between the images sent to the left and right eyes to create a sense of depth. In this case, separate panels can be used for each eye, or a single panel can be divided horizontally to serve both eyes. The mere presence of two panels is not sufficient to distinguish between devices designed for video viewing and those designed for gaming.
To create a virtual space, the user’s movements must also be reflected. When the user turns their head to the right, the right-side scenery in the virtual space should be visible, and on devices that support position tracking, nearby objects should appear larger when the user moves forward. Only when these responses occur appropriately does the user move beyond simply looking at the screen to actually feel as though they are inside the space.
The process by which a computer generates the image to be displayed on the screen based on scene information is called rendering. In VR, images must be continuously generated to reflect the viewing angles of both eyes and the movement of the head. Even if the display is sharp, significant latency between motion detection and the display of new images can reduce visual stability. This is why it is difficult to evaluate an HMD’s performance based solely on screen resolution.

 

Does a wider field of view improve image quality?

The field of view is the range of angles that the image occupies in front of the eyes. A wider field of view allows the virtual scene to fill even the peripheral areas, which is advantageous for enhancing immersion. However, spreading the same number of pixels over a wider range reduces the number of pixels allocated to a given field of view. Therefore, to maintain fine detail while widening the field of view, both display resolution and optical performance must be improved.
This is also why, in early consumer VR devices, the gaps between pixels appeared grid-like, or small text appeared blurry. Pixel structures that are barely noticeable on a standard screen can become prominent when magnified through a lens. The perception of differing sharpness between the center and edges of the screen is also influenced by the characteristics of the lens and the position of the eyes.
Image processing to reduce lens distortion is also crucial. VR devices use distortion correction, which preemptively warps the image in the opposite direction so that the image passing through the lens appears as intended. Simply referring to this as “fisheye rendering” can easily lead to confusion between the visual representation of wide scenes and the processing used to offset lens distortion. Distortion correction is not so much a technology that creates a wide field of view in and of itself, but rather a process of correcting the image distorted by the optical system.
Such processing may be equally necessary for both gaming and video viewing. Distortion is not a problem exclusive to movies; it can also be an issue in games. While the design priorities differ between the experience of surveying a wide space and that of appreciating the details within a fixed screen, clear and stable video is crucial in both cases.

 

What was the difference between Sony’s HMD for video viewing and the Oculus?

The HMZ-T1, announced by Sony in 2011, was a personal video-viewing HMD that featured an OLED panel for each eye. Its successor, the HMZ-T3W, also used OLED panels with a resolution of 1280×720 for each eye and provided a field of view of approximately 45 degrees. The structure, which delivered separate left and right images, was advantageous for reducing the interference that occurs in 3D video, where the image from one eye blends with that of the other.
This product line focused on the experience of having a large, personal screen right in front of the user’s eyes. In contrast, the early Oculus Rift aimed to provide an experience where users could explore a virtual space by combining a wide field of view with head-tracking. The difference between the two product lines lay not simply in the type or number of panels, but in how they were designed to shape the user’s experience of the display.
Therefore, it is difficult to generalize the comparisons made at the time into a fixed framework such as “high resolution for video viewing, wide field of view for gaming.” Each product differed in resolution, optical system, price, and head-mounted design, and devices intended for developers had different sales objectives than those aimed at general consumers. This is why it is difficult to conclude which approach was more advantageous for mass adoption based solely on the price difference between the early Oculus Rift DK2 and the Sony HMZ-T3W.
The PlayStation VR2, which appeared later, highlights the limitations of this distinction. Released in 2023, this device uses an OLED display with a resolution of 2000×2040 per eye and supports a field of view of approximately 110 degrees along with refresh rates of 90Hz and 120Hz. It is an example of a gaming device that combines OLED-based image quality with a wide field of view. However, since the field of view figures announced by manufacturers are influenced by measurement standards and wearing conditions, one should not directly interpret the numerical differences between products as differences in perceived performance.
The PlayStation VR2 also supports foveated rendering using eye tracking. This method focuses video processing resources on the area the user is looking at while reducing the processing load on peripheral areas. While this is not a technology that increases the physical number of pixels on the panel, it helps ensure detailed rendering in the user’s field of view while efficiently utilizing limited computational resources.

 

How has mixed reality expanded the role of HMDs?

The consumer VR business that began with Oculus has expanded into Meta’s Quest series. The Quest 3 and Quest 3S are all-in-one headsets equipped with their own processing units and batteries, allowing them to run supported content without a wired connection to an external computer. Both devices provide a mixed reality experience by displaying the surrounding environment through color pass-through.
Pass-through is a feature that displays the real-world view captured by external cameras on the internal display. Users can check their surroundings without removing the headset, and the system can display digital content alongside the real-world view. When combined with spatial tracking, virtual screens or objects can be rendered as if they were placed in specific locations within the actual room. However, the view of reality filtered through the camera and display is not identical to what the naked eye sees directly and is subject to factors such as image quality and latency.
Even products offering similar features do not necessarily share the same optical design. The Quest 3 uses pancake lenses, while the Quest 3S uses Fresnel lenses. The Quest 3’s pancake optical system utilizes a structure that folds the light path, contributing to a thinner device design. These differences demonstrate that HMDs are evolving by striking different balances between price, size, and clarity, rather than converging on a single standardized approach.
Apple’s Vision Pro, announced in 2023, features two micro-OLED displays with a total of 23 million pixels and is designed to recognize eye and hand movements as well as the surrounding environment using cameras and sensors. The experience, which Apple calls “spatial computing,” focuses on placing on-screen content within the real world and manipulating it with eye movements and hand gestures. This demonstrates that the roles of video viewing, virtual reality, and a work display can overlap within a single HMD.
It is also important to distinguish between hand tracking and eye tracking. Hand tracking recognizes the position and movement of the hands, while eye tracking determines the direction in which the eyes are looking. Support for hand tracking does not necessarily imply support for eye tracking, and even among devices with the same capabilities, the actual scope of use varies depending on the software and content.

 

A User Experience That Can’t Be Solved by Performance Improvements Alone

Even as HMD technology advances, the physical reality of wearing the device remains. How the weight is distributed across the face and head, how much heat is generated, and how long the battery lasts all determine the user experience. For products that connect to external devices, cables can restrict movement, while for all-in-one models, striking a balance between computational performance, heat generation, and battery life becomes crucial.
Visual fatigue is not a problem that can be eliminated simply by increasing resolution. In typical fixed-focus stereoscopic HMDs, even though the degree to which the eyes converge varies depending on the distance of virtual objects, the optical distance at which the eyes must focus clearly may remain constant. “Convergence-accommodation mismatch”—where the two eye movements that would normally move together in reality become misaligned—can be a source of discomfort. Delays between head movements and screen responses, as well as discrepancies between perceived visual movement and actual physical movement, can also make comfortable use difficult.
Challenges related to handling real-world input devices also remain. When the user’s surroundings are obscured, it is difficult to locate a keyboard or gamepad. Pass-through and hand tracking can reduce these inconveniences, but they do not fully replace the familiar input devices used for all tasks. Selecting items on a virtual screen with your hands and accurately typing a long document require different modes of interaction.
The evolution of HMDs has expanded from technologies that make small screens appear larger to technologies that reflect the user’s movements and the surrounding space. As a result, personal movie screens, gaming devices for exploring virtual spaces, and devices that place work screens within the real world now coexist on the same technological foundation. The scope of future applications will not be determined by a single factor, such as pixel count or field of view alone. HMDs can become tools for continuous everyday use only when they display necessary content with sufficient clarity while allowing users to wear them comfortably and operate them accurately.

 

About the author

Cam Tien

I love things that are gentle and cute. I love dogs, cats, and flowers because they make me happy. I also enjoy eating and traveling to discover new things. Besides that, I like to lie back, take in the scenery, and relax to enjoy life.