Many philosophers debate the topic of reality, and the components that make up what we perceive to be “real”. Although an interesting mental exercise, we’re more interested in the practical applications of reality today, which is more accurately defined as “realism”.
Realism is defined as representing a person, thing, or situation accurately or in a way that is true to life. In this article, we seek to understand to what degree this definition of realism can be known in a completely simulated environment using virtual reality technology, and how this impacts our world.
To understand the importance of realism in virtual reality, we must first look to what the technology is trying to achieve— a simulation. A simulation is defined as an approximate imitation of the operation of a process or system. There’s an important distinction in that definition, which is to mimic the operation of something rather than just the thing itself. A simulation is a place of action; and that intent is what separates a simulation from a static learning tool such as text, video, or presentations. Ultimately, a simulation bridges the gap between knowledge and high performing actions.
Traditionally, a simulation would be run “live” in a simulated physical space. History tells us that live simulations have been used as a learning tool for centuries, dating back to clay and stone models being used to understand the anatomy of a person. In more modern applications, simulations have been used to train students and professionals to assist in learning—for example a medical student may practice in a room dressed up as an operating theatre, or they could be taken a step further and practice on-site in an actual operating theatre (also known as in-situ or in-situation) which was considered to have a higher degree of realism. The objects within that environment also benefit from being as real as possible, which could mean the difference between using mannequins or a paid actor to simulate a patient.
Simulations have more recently found a new environment in which to model themselves, known as “virtual”, where one would experience the imitation synthetically. Thanks to computer technology, virtual simulations have been able to imitate an operation without the need of costly physical environments and the challenging logistics of getting people to them. Virtual Reality (VR) specifically refers to synthetic imitations presented inside of a special headset, which enhances the immersion of a synthetic environment further by assuming the user’s entire field of vision rather than just on a two dimensional screen.
VR headsets have been used commercially by organizations such as NASA for decades already, and as the cost of production decreased the technology became more viable for consumer purchase. Additionally, the processing power of hardware being used to run virtual reality headsets has led to more realistic simulations. However, to what degree does realism impact the efficacy of a simulation, especially in training situations?
To understand the efficacy of realism, we must look at a key variable that defines the perceived realism—fidelity. Fidelity refers to how closely a simulation imitates or amplifies reality. The overall goal of high-fidelity simulations is to improve the performance of an action, or what a person actually does (rather than thinks). In 1990, Miller (1990) indicated that this action is built upon a person’s knowledge, competence, and performance.
Generally speaking, a low fidelity simulation is good for building upon knowledge – it won’t feel real, but it will outline the basics and theory well enough. Medium fidelity is used to build competence through greater interaction, then high fidelity is where high performance action occurs. This model of traditional Fidelity can be viewed as a pyramid. Without the base level of knowledge, you can’t reach the top level of action. As you begin to improve your knowledge, competence, and performance, the action level of fidelity in a simulated environment increases too, agnostic of how the environment is actually presented. For example, the fidelity of a commercial flight simulator will be different for someone with no knowledge of aviation in comparison to someone who has spent years studying the theory. How will the person with little knowledge be able to know if the simulation is high-fidelity or not, if they don’t understand what to look for? Therefore, to some extent the concept of fidelity is subjective.
This can be understood at a deeper level by examining the different components that make up fidelity, over and above the person experiencing it. At a conceptual level, the scenario at hand needs to make sense. For example, you would expect a flight simulator to have a high degree of conceptual fidelity if it flies through the air, rather than through Walmart. Physically, the simulation needs to reflect the actual physical properties of the real environment. The flight simulator wouldn’t have a high degree of physical fidelity if it used carrots for the levers. Emotionally, the simulation should try to capture the same feelings one would expect in the real environment. For example, a simulated emergency in the cockpit of a plane should feel naturally urgent to deal with, rather than casual and boring. New technology such as heart-rate monitors built into wearable devices have made it easier to test for emotional response, helping simulators find better ways to increase the degree of emotional fidelity.
However, this level of understanding goes one step deeper again when looked at in the context of virtual reality simulations. There is one important consideration when it comes to gauging fidelity in a virtual reality environment, especially with regards to training simulations. Stone (2011) found the misconception to be that technological advances lead to better training environments. Stone and his human interface technologies team found that this is far less important than the concept of human-centered design. When the goal is to maximize the transfer of simulated tasks, knowledge, skills and behaviors to real-world applications; psychological fidelity is key.
Psychological fidelity is the degree to which simulated tasks can reproduce behaviours required real-world applications. This, to some degree, combines the traditional ideas of conceptual, physical, and emotional fidelity as described; although typically one would look at fidelity in this context as a combination of physical and psychological fidelity.
It is through this definition that it also becomes obvious that pure technological advances aren’t a large part of that equation, but rather the technical advances make it more accessible and viable. The major drawback of new interfaces such as virtual reality is that of a human experience nature. Think back to the first few years of websites—every site was completely different with walls of text, hidden links, strange navigations, and unique layouts. It wasn’t until many years later that a standardized experience began to take shape that was more natural to the user, taking advantage of modern design principles, content structures, and human behaviour to deliver a seamless experience. Virtual reality is still in the early stages of exploring the best user interface, however it is far more complicated due to the additional dimension. It’s one thing to master the interface of a 2D screen with basic inputs, but quite another in a 3D environment.
Stone broke down the concept of fidelity for virtual reality simulations further, as follows:
- Context Fidelity should seek to represent an appropriate background of sensory and behavioral detail—think background extras on a movie set. They don’t get in the way, but they add to the context of the scene.
- Interactive Technology Fidelity is the degree to which the input controls (think headsets and hand controls) represent the real-life interfaces.
- Hypo-fidelity and Hyper-fidelity measures the significance of too little or too many sensory details, behavioural details, and interaction systems
Interestingly, many end-users already have a concept of fidelity in their head before entering a training simulation, a baseline of expectations based on what they’ve seen in other applications such as gaming and military applications. Now the complex web of what makes up a “high fidelity” training simulation begins to reveal itself. We know that at the highest level, the end-user needs a basic knowledge of the system and also a certain level of competence before reaching high-performance actions. However, simulations run in virtual reality require a balance of contextual sensory and behavioural details alongside realistic input characteristics; while catering to the potential expectation gap a user may have before entering the simulation. This is all impacted further by the challenges virtual reality face with regards to seamless interfaces that are not yet standardized or widely understood by users.
So, if a virtual reality simulation can have a high degree of psychological and physical fidelity, then it can be considered to be more realistic—assuming the user already has some knowledge and competence to be able to perceive it that way to begin with. However, this is all qualified by the ability for humans to actually use it, an idea often called human-centered design.
Now that we understand the wider makeup of realism in a simulation, we can use it to our advantage in virtual reality applications. For example, the PrecisionOS surgical training platform is designed to make use of modern graphic engines to mimic the physical environment, including sensory and behavioural details such as tools, lighting and machinery with a high degree of physical fidelity. The patient on the operating table is configured to show the symptoms that require a given procedure, ensuring that the representation is balanced to stave off hypo- or hyper-fidelity. The hand controllers are configured to more closely resemble the common surgical tools of a procedure, aiming for a higher interactive technology fidelity. Finally, the simulated procedure is guided in real-time with feedback and mentorship, both visually and via audio to simulate the psychological environment as if training with a senior surgeon.
Students or staff have already completed many years of medical school which includes traditional learning tools and simulations, so they have a high degree of knowledge and competence. Therefore, a realistic virtual environment can help them train for advanced skillsets and enhanced confidence without having to practice on real patients—which can be both costly to the trainee, the patient, their family, and the hospital; not to mention challenging to find appropriate training patients and matching them with trainees on a repeating basis. At a wider level, this ultimately means better surgeons and a higher success rate for patients.
The future of realism in virtual reality looks promising. Already, we’ve seen physical feedback in VR hand controllers which has been proven to enhance the perception of interactive technology fidelity (Hoffman, 2011), and this technology is expanding to full haptic-feedback suits. As other fields of technology such as biotech begin to crossover with virtual reality, we’ll see even more integration and immersion. For example, the user may be able to control their virtual environment using just their thoughts, or be able to connect sensors directly to nerve endings for realistic feedback.
As the technology improves, so too does the perceived realism of our virtual environments—and as we’ve seen, this will further engage our world’s future students and professionals to perform at a higher level. The result in many years’ time will be a safer, more productive world to live in, and we can’t wait to stake our hand in the efforts.