360 Video

360 video can be viewed in a Virtual Reality headset such as Meta Quest 3 and, when uploaded to platforms such as YouTube, is also available on a desktop or mobile device. The viewer interacts with the content within a VR headset by moving their head or on a desktop by ‘clicking and dragging’ to change the point of view using an input device such as a mouse. It is also possible to achieve similar interactions using the gyro technology on a mobile phone or a screen with touch capability. YouTube can display interactive 360 video in VR, on desktop and mobile.

Video is recorded with a camera utilising a series of wide angle lenses designed to capture the surrounding environment. It is stitched together using compatible software which may be provided by the manufacturer such as Insta360 Studio or by a third party such as Mistika VR.

360 video is usually recorded in the same equirectangular format as 360 photography. Current cameras record video of at least 6 – 8K which results in Gigabytes of data per minute with the Insta360 Titan recording 11K (10K in 3D). The challenges posed in producing 360 video, such as hiding microphones, lights and other equipment, has led to a decline in use during recent years in favour of 3D VR180 video. However the format remains popular in real estate, tourism and journalism where a view of the entire environment is important.

VR180 Video

VR180 uses half the horizontal viewing angle of 360 video with just the front facing 180 degrees available. It is designed to be consumed within a VR headset, viewed on a screen with active glasses or converted to anaglyph for viewing with red / cyan glasses. Whilst there are only 180 degrees of recorded content, most VR headsets have a viewing angle of around 90 degrees which provides a realistic sense of immersion.

Content is typically recorded using 2 wide angle lenses covering a 180 degree viewing angle. Both of these lenses face the same direction with the centre of each lens placed at approximately the same distance as human eyes. When converted for use within a VR headset, the video provides realistic depth. HumanEyes Technologies released the Vuze XR in 2018 which had two 4K cameras which could be used in VR180 mode or 360 capture mode. A recent addition to the VR180 camera market is the CALF 3D VR180.

This format is used in vlogging and entertainment such as storytelling. However, as mentioned in the previous section, 360 video is still used when it is useful to see an environment in its entirety.

Virtual Reality (VR)

Virtual Reality experiences are designed to facilitate interaction where location, physicality and changes to the environment have meaningful consequences. They are usually viewed within a VR headset such as Vive XR Elite or Meta Quest 3 using controllers or hand tracking. However, platforms such as Spatial and Horizon Workrooms allow users access via a desktop environment as a ‘window’ to the virtual world. The user is able to shape the narrative and environment by their choices which may involve changing the state or position of physical objects within a space. Many VR applications are created with software such as Unity or Unreal Engine.

Examples of immersive VR applications range from as simple as the simulation of fairground games within Nvidia’s VR Funhouse, production of 3D art using Google Tilt Brush or involve the complexity associated with piloting an aircraft in Flight Simulator. Other examples may be found on Meta’s App Store.

The term ‘Virtual Reality‘ was first used by American academic Jaron Lanier in the 1980’s as a title for his research project. He is considered to be the ‘father of VR’ because of his groundbreaking work in the field.

Ambisonic Audio

Ambisonics is an audio technology that uses hardware and software capable of rendering spatial audio in Virtual Reality, Augmented Reality and Mixed Reality. As few as 4 audio channels can be used to represent sound within a virtual space. As the viewer’s head changes direction or objects emitting sound move within a space the audio is adjusted in a realistic manner to reflect the effect of these movements on the perceived sound. It is also possible to experience ambisonic audio in a limited manner when viewing 360 video on a desktop PC or mobile device by moving the point of view. The use of 4 audio channels to simulate spatial sound is referred to as First Order. However, it is possible to use more than 4 channels to enhance the effect in a similar way to improvements of 7.1 surround sound over 5.1.

For more information on ambisonics, see this excellent summary of ambisonic audio from Waves.com

Olfactory

Olfaction or olfactory sense is the sense of smell. There are devices capable of stimulating the olfactory sense as part of an immersive experience. One example is the Smell Engine described as “a system for artificial odour synthesis in virtual environments”

Gustatory

Gustatory perception refers to the taste sense. It is possible to trick the human brain to into thinking that food is being consumed using stimulation by computer controlled plates placed upon the tongue. In 2013 digital lollipop was created by researchers at the University of Singapore that stimulated sweet, sour, salty and bitter tastes.

Summary

Immersive digital media has the potential to elevate and enhance the process of storytelling, communicating research ideas, developing products and in the provision of training. The last 10 years has seen rapid growth of hardware and software technologies at both professional and consumer levels increasing the number of creators and immersive content. Despite these advances, many challenges remain including the size, weight, cost and uptake of VR headsets, the cost and quality issues associated with 360 and VR180 cameras and the technical complexities of generating spatial audio. There are also positive signs too – the release of the Apple Vision Pro, camera releases from manufacturers such as Insta360 and continued support for immersive content in Adobe’s Creative Cloud.

AI School

I’ve recently been exploring the use of Midjourney and Magnific.ai to create highly detailed 12K 360 images such as the example shown above (note this example has been enhanced with audio and lens flare using Pano2VR). After working out the most appropriate workflow through a series of experiments, a course was published on Eventbrite designed to impart this knowledge and experience to interested parties in the AI and virtual tour communities. In addition to Midjourney and Magnific, the process involves Photoshop and Pano2VR (or equivelent editing software tools) to edit the nadir and zenith (directly below and above the viewer) along with ensuring the vertical seam blends. This is to ensure objects and textures on the left side of the image fit seamlessly with the right when assembled in the final 360 form.

You can see a variety of images produced using this method by visiting the 360 image section of the Simcoemedia Shop or on my Facebook feed.

AI Workshop Details

The workshops have been designed to cater for 360 enthusiasts, immersive media artists and virtual tour producers looking to enhance their 360 workflow using AI tools. The 90 minute session includes the following activities:

• Creating equirectangular images in Midjourney by using appropriate commands, parameters and descriptive keywords
• Ensuring the image does not warp in 360 view by aligning the horizon correctly
• Resolving stitching problems and nadir issues using Photoshop and Pano2VR (or equivelent) to ensure the best interactive experience
• Using Magnfic to upscale and add detail to images whilst retaining a reasonable level of creative control
• Resolving metadata issues + testing images to ensure they are ready for publication
• Discussion of how these tools have developed and evolved in recent months

A reference page has been set up on this site at www.simcoe.co.uk/ai-school/ reminding participants of the key processes highlighted during demonstration and discussion. So far these sessions have been useful and enjoyable for those taking part with some returning for the second in the series which examines transfering image styles to existing 360 tour photography. For more information on upcoming events see Peter Simcoe’s Eventbrite page.

Related topics: Find out more about how Generative AI creates video and images by reading ‘What Is Generative AI and Is It Useful In Film Production?’