Defining Immersive Experiences

a field guide to 360º, AR and VR

There can be a lot of overlap and general confusion about where one technology ends and another begins when people talk about the various tools involved in taking part in an immersive experience. To help define the three frontiers that IMEX Lab continues exploring, the team put together this helpful guide to 360º video, Augmented Reality and Virtual Reality.

Immersive Experiences technology allows for faculty and students to engage with academic media and content in exciting, and sometimes intimidating, new ways. There is a universe of possibilities when it comes to this emerging technology and many ways that users can experience and interact with content made just for these platforms. But what are the different types of immersive experiences? How do they differ from each other and how might these different affordances shape your experience, whether you are consuming or creating the content? This resource breaks down the ways we currently are thinking about immersive experiences: as 360° Video, Virtual Reality (VR) and Augmented Reality (AR).

You may know some of these terms by other names – eXtended Reality (XR) is a term used by Educause and others to refer to immersive experiences. Mixed Reality is another term that is sometimes analogous to AR. As these technologies grow, evolve, and are accepted into the mainstream, the terminology we use to refer to them and explain their uses will evolve as well. Media Commons and Teaching and Learning with Technology will continue to unpack these terms and technology for the benefit of all of Penn State’s students and faculty.

Source: Educause

Plotting the connections between 360º video, AR and VR: 360º video captures the real world and can be supported by additional guidance,
progressively pushing content towards full VR simulation. AR bases itself on – and adds additional information to – live experiences in the real world.

What is Level of Guidance?

The Level of Guidance indicates how much the creator has imbued the video with a sense of at what the viewer should be looking.

For instance, a Level 1 video would mean that the camera has been left in place while life happens around it, allowing you, as the viewer, to simply observe. A Level 2 video might incorporate subtle visual, textual or audible hints or simple transitions. A Level 3 video would rely on a handful of hints and transitions. A Level 4 video would employ multiple hints and dialogue. And a Level 5 video would rely on all of the above as well as a narrative arc that pushes the viewer forward.

[See Examples]

Another way of looking at this would be how much production went into the video if you are considering making one of your own like it.

What is Level of Integration?

The Level of Integration indicates how much the developer has connected the AR app with the real world environment.

For instance, a Level 1 AR app would bring a pre-created object or projection into the viewer’s space. A Level 2 AR app might incorporate a connection between object and physical surroundings, such as anchoring to a particular plane or surface. A Level 3 AR app would rely on several connections to the surroundings, including functional awareness of conditions in the space. A Level 4 AR app would employ increased awareness of conditions and real-time mapping of content onto the physical environment. And a Level 5 AR app would employ all of the above and allow for sophisticated real-time integration of real world and generated content.

[See Examples]

What is Level of Interactivity?

The Level of Interactivity indicates how many interactive features an experience has.

For instance, a Level 1 would be a more passive experience where you will be able to look around freely, but you will not be triggering or interacting with things in the environment. A Level 2 will provide some level of interactivity but only very basically, such as being able to move between different scenes in the experience. A Level 3 will start to incorporate some more complex interactions using a remote or controller to trigger things within the scenes. A Level 4 would be a fully interactive experience but something you are still guided through. Finally, a Level 5 would be a fully interactive experience that gives you complete control of how to use it.

[See Examples]

360º Video

What is it?

360º video allows the viewer to momentarily enter another space in the real world. At its simplest, it is akin to standing in a spot the creator chose and watching a scene unfold in all directions. The author can direct the viewer’s experience by adding additional guidance like text prompts, cuts to new scenes, voiceover narration, audio cues and more. At its most sophisticated, a 360º video could employ question prompts and a logic tree to trigger a series of differing videos, mimicking a choose your own adventure story.

What does it look like?

What do I need to consume it?

360º video can be watched immediately in any web browser that supports the technology (Google Chrome is a good cross-platform option) and the viewer can change their vantage point by using a mouse or trackpad.

A smartphone can also play back 360º content with the added bonus of leveraging the device’s accelerometer and other sensors to allow for changing the vantage by simply moving the device in any direction.

There are several options that leverage the low cost Google cardboard headset platform to allow for increasing immersion by inserting a smartphone into a device that splits the image into a stereoscopic view, allowing head movement to trigger the accelerometer in order to change the vantage.

With a fully standalone headset, the phone is removed from the equation, instead allowing video to play back using a built-in stereoscopic display in much higher quality. Immersion is further increased with integrated speakers, adjustable head straps and handheld controllers to manipulate the interface.

What should I have to create it?

access to a location or event to be documented in 360º video
360º video camera, monopod or tripod and additional gear, as needed
editing software (Adobe Premier Pro, Final Cut)
cardboard or standalone headset for viewing during editing
an account on a 360º friendly publishing platform (YouTube, Kaltura)

Augmented Reality

What is it?

Augmented Reality (AR) grafts an added layer of information over the real world. AR experiences vary, but all share the premise that the viewer will use some sort of in-between device to overlay material on the environment around them. Typically, a mobile phone or tablet with rear facing camera is called into service as a window into an AR-enhanced space. Applications of AR can include placing furniture inside one’s home, trapping digital monsters in a game scenario, measuring items simply by pointing at their edges and the like.

What does it look like?

What do I need to consume it?

Many AR experiences are downloadable for mobile devices, so any smartphone and app created by a developer will allow a viewer to add various overlays to the world around them. Because the smartphone or other mobile device is acting as a window onto this additional information, at least one or both of the viewers hands are necessarily tied up as they interact.

AR glasses or goggles act as a mobile heads up display for the wearer, allowing for viewing additional information about their environment as they go about their tasks. The content presented via AR glasses covers only a portion of the field of vision, usually just in one eye.

A more sophisticated AR headset presents an encompassing overlay on the viewer’s environment, allowing for rich content to appear anywhere the wearer looks while moving about.

What should I have to create it?

a location, object or facet of the real world needing further context
game design software (and experience)
3D modeling tools (and experience)
mobile device or headset for testing while building
a distribution method for your finished AR app (App Store, Play Store, etc)

Virtual Reality

What is it?

Virtual Reality (VR) takes the viewer inside a wholly created world. This environment may draw directly from or be inspired by experiences in the real world, but does not, by definition, need to do so. A simple VR experience may only place the view inside a constructed space and let them observe. More advanced VR content allows the viewer to interact with and manipulate the environment around them, including some combination of moving objects, talking with simulated characters, building new items in 3D space and much more.

What does it look like?

What do I need to consume it?

Many VR experiences are designed in the same way as computer games and, as such can be loaded on a gaming PC or console. Generally, the viewer would manipulate the simulated space using keyboard and mouse/trackpad or a handheld controller. There would be a large amount of variability in audio and video quality based on the device and its accessories.

A standalone headset can also be used to view VR content. The vantage point would follow head movement and basic interaction would be navigated with a handheld controller. Owing to its self-contained nature, the headset would not provide forward/backward or side-to-side motion tracking, etc and A/V quality would be limited by the headset’s specifications.

Using a high end PC and advanced headset opens up the possibility for not only far higher quality audio and video but also, through included accessories, motion tracking within the simulated environment. The amount of physical space available would be a constraint, as well as the tactile/haptic feedback when interacting with objects using a handheld controller.

Attaching/augmenting a high end PC and advanced headset with accessories can dramatically increase the sensation of immersion for a viewer. Haptic feedback gloves and other clothing can allow for actually feeling objects and actions within the experience while a VR platform with moving floor or other dedicated spaces allow the viewer to roam freely in-world.

What should I have to create it?

a narrative, demonstration or simulation idea
game design software (and experience)
3D modeling tools (and experience)
standalone or advanced headset for testing while building
a distribution method for your finished VR app (Steam, etc)

Looking for great content to engage your students?

IMEX is hard at work developing a comprehensive Experience Catalog, organized by discipline, that makes it is easy for faculty to find content that will be a perfect fit for their courses. Take a look at what we’ve found so far – and submit a suggestion for other materials that you want to see.

Find Your Experience