Google Tilt Brush | Simcoemedia

Immersive Digital Media Part 2 – Examples

Peter Simcoe — Wed, 20 Nov 2024 20:08:35 +0000

My previous post, Immersive Digital Media Part 1 – Definitions, defined the terminology associated with Extended Reality (XR) including VR, AR, MR and other technologies such as ambisonic audio. The second article in the series focuses on projects I have undertaken in the last 8 years including 360 video, VR180 and ambisonic audio.

AI Generated 360 Images

In November 2023 I initiated a series of experiments examining potential production methods for a short film. During experimentation Midjourneywas used to create equirectangular images which were edited in Photoshop (Beta). MagnificAI upscaling increased the resolution up to 12K. See them on the Simcoemedia website: www.simcoe.co.uk/product-category/360-images/

Also see:

12K examples on 360Cities.net
Applying styles to 360 photography experiment
Tutorials by Peter Simcoe

360 Video Handbook

Published in February 2018, the 360 Video Handbook is a coffee table book featuring a series of inspirational 360 video projects. The style and layout was designed to make it easy to access by featuring an explanatory diagram or photographs on one side of the spread throughout the publication. The book also answers frequently asked questions by beginners. See all Simcoemedia books at:

www.simcoe.co.uk/books/

Augmented Reality Music Video

An AR music video for the song ‘A Little On The Darker Side’ was created using Adobe’s Character Animator software. A series of character designs were developed. These were then animated and rendered in HD. The characters were exported as animated GIFs with transparency then imported into Adobe Aero to create an augmented reality experience. See the example video at:

https://youtu.be/ZfZQ5DZgq1w

Google Tilt Brush 3D 360 Video

3D 360 videos created with virtual reality art creation app Google Tilt Brush. These can be viewed in 3D on YouTube using red / cyan glasses:

Google Earth Studio

Google Earth Studio allows users to generate animated sequences using Google Maps including the 3D photogrammetry data available for many cities around the world.

Chester 4K 360 tour
London 4K 360 tour
London Interactive 4K with labels
Snowdonia 4K 360 tour

360 Video with Ambisonic Audio

In 2017 I became a content creator for HumanEyes Technologies. The company offered funding for 360 video projects using their camera technology including a series of travel videos in Chester, London and Barcelona, music videos, development of a second, upgraded light suit (the original low budget version can be seen in the One More Chance video). Some of these were featured in the 360 Video Handbook. HumanEyes Technologies supported the creation of music videos by supplying extended access to ambisonic audio software and funding the production of light suit v2.0.

Meta Workrooms Interview

Created a series of interviews with Emily Olman and Paul Tomlinson using Meta Workrooms. These interviews explore the connectivity and activities facilitated by Workrooms and explored the meaning of Spatial Computing:

Google Maps interactive 360 images

360 photographs taken at locations across the UK, Spain, Sweden and France have been uploaded to Google Maps to create a portfolio of immersive imagery capable of display on a desktop, mobile devices and within a VR headset.

See Google Maps images by Peter Simcoe

Simcoemedia Spatial Gallery

An interactive gallery created using Spatial IO. Can be viewed on mobile devices, desktop and VR headsets. See gallery.

Microsoft Flight Simulator

This VR video was recorded to demonstrate take-off and flight around Airbus (Broughton) plant in North Wales, UK using Microsoft Flight Simulator. It represents the level of realism the simulator currently offers.

Immersive Digital Media Part 1 – Definitions

Peter Simcoe — Mon, 28 Oct 2024 11:00:06 +0000

I recently had a conversation with a client regarding the use of immersive digital media in engineering and design. We discussed how it can enhance research, product development and training and increase the overall impact. Over the last 8 years I have conducted a variety of experiments exploring immersive media such as recording 360 video, drawing with Google Tilt Brush and mixing ambisonic audio. This article, the first of two posts exploring immersive media, provides an overview of terminology.

Traditional vs Immersive Media

The term traditional media usually refers to television, radio, newspapers and cinema. Content is presented to the audience in a passive manner, meaning there is little or no control over presentation or narrative. In contrast, immersive media interactivity and enhanced sensory experience using advanced hardware and software such as VR headsets or headphones designed to emulate spatial audio. Some technologies incorporate the simulation of touch and smell. Immersive experiences are designed to be consumed in a non-linear, participatory manner where choices and physical interaction affect narrative and environment.

Below is an example of a 360 video uploaded to YouTube with a resolution of 8K. The original video was recorded with a high resolution camera. It is important to note only a portion of the 7680 x 3840 pixels recorded by the 360 camera will be visible to the viewer at any given time (depending upon the Field Of View) which reduces displayed resolution to that approximating full HD (1920 x 1080). If viewed on a desktop PC in full screen you can use the mouse to direct the point of view by clicking and dragging in the desired direction

Immersive experiences are designed to increase the sense of realism and there are many different formats available, each with their own characteristics and advantages. Common media formats are:

360 Video

360 video can be viewed in a Virtual Reality headset such as Meta Quest 3 and, when uploaded to platforms such as YouTube, is also available on a desktop or mobile device. The viewer interacts with the content within a VR headset by moving their head or on a desktop by ‘clicking and dragging’ to change the point of view using an input device such as a mouse. It is also possible to achieve similar interactions using the gyro technology on a mobile phone or a screen with touch capability. YouTube can display interactive 360 video in VR, on desktop and mobile.

Video is recorded with a camera utilising a series of wide angle lenses designed to capture the surrounding environment. It is stitched together using compatible software which may be provided by the manufacturer such as Insta360 Studio or by a third party such as Mistika VR.

360 video is usually recorded in the same equirectangular format as 360 photography. Current cameras record video of at least 6 – 8K which results in Gigabytes of data per minute with the Insta360 Titan recording 11K (10K in 3D). The challenges posed in producing 360 video, such as hiding microphones, lights and other equipment, has led to a decline in use during recent years in favour of 3D VR180 video. However the format remains popular in real estate, tourism and journalism where a view of the entire environment is important.

VR180 Video

VR180 uses half the horizontal viewing angle of 360 video with just the front facing 180 degrees available. It is designed to be consumed within a VR headset, viewed on a screen with active glasses or converted to anaglyph for viewing with red / cyan glasses. Whilst there are only 180 degrees of recorded content, most VR headsets have a viewing angle of around 90 degrees which provides a realistic sense of immersion.

Content is typically recorded using 2 wide angle lenses covering a 180 degree viewing angle. Both of these lenses face the same direction with the centre of each lens placed at approximately the same distance as human eyes. When converted for use within a VR headset, the video provides realistic depth. HumanEyes Technologies released the Vuze XR in 2018 which had two 4K cameras which could be used in VR180 mode or 360 capture mode. A recent addition to the VR180 camera market is the CALF 3D VR180.

This format is used in vlogging and entertainment such as storytelling. However, as mentioned in the previous section, 360 video is still used when it is useful to see an environment in its entirety.

360 Video Examples

VR180 Video Examples

Virtual Reality (VR)

Virtual Reality experiences are designed to facilitate interaction where location, physicality and changes to the environment have meaningful consequences. They are usually viewed within a VR headset such as Vive XR Elite or Meta Quest 3 using controllers or hand tracking. However, platforms such as Spatial and Horizon Workrooms allow users access via a desktop environment as a ‘window’ to the virtual world. The user is able to shape the narrative and environment by their choices which may involve changing the state or position of physical objects within a space. Many VR applications are created with software such as Unity or Unreal Engine.

Examples of immersive VR applications range from as simple as the simulation of fairground games within Nvidia’s VR Funhouse, production of 3D art using Google Tilt Brush or involve the complexity associated with piloting an aircraft in Flight Simulator. Other examples may be found on Meta’s App Store.

The term ‘Virtual Reality‘ was first used by American academic Jaron Lanier in the 1980’s as a title for his research project. He is considered to be the ‘father of VR’ because of his groundbreaking work in the field.

Augmented Reality (AR)

Augmented Reality is the technology that overlays visuals, data or audio onto the real world, enhancing the user’s perception of the environment. One example of this is Google Maps Live View where the camera on a mobile phone is used to show a live view of the road ahead whilst superimposing directions and other visual guides. Another notable project is Glass, Google’s answer to Augmented Reality glasses. This project began in 2010 with the wearable tech available in 2014, It was later discontinued in 2015 due to safety and privacy concerns along with a lack of uptake in the healthcare sector – see this article for more information on the cancellation.

Mixed Reality (MR)

Mixed Reality is similar to Augmented Reality but allows the users to interact with the layers or objects superimposed upon the environment around the user. Meta Quest 3’s MR demo First Encounters is a great example of this. The surrounding environment is displayed on the headset in real time using front facing cameras whilst objects are overlaid onto the display to create game elements that can be interacted with.

Extended Reality (XR)

This term incorporates VR, MR and AR. XR refers to the technologies and experiences collectively.

Ambisonic Audio

Ambisonics is an audio technology that uses hardware and software capable of rendering spatial audio in Virtual Reality, Augmented Reality and Mixed Reality. As few as 4 audio channels can be used to represent sound within a virtual space. As the viewer’s head changes direction or objects emitting sound move within a space the audio is adjusted in a realistic manner to reflect the effect of these movements on the perceived sound. It is also possible to experience ambisonic audio in a limited manner when viewing 360 video on a desktop PC or mobile device by moving the point of view. The use of 4 audio channels to simulate spatial sound is referred to as First Order. However, it is possible to use more than 4 channels to enhance the effect in a similar way to improvements of 7.1 surround sound over 5.1.

For more information on ambisonics, see this excellent summary of ambisonic audio from Waves.com

Olfactory

Olfaction or olfactory sense is the sense of smell. There are devices capable of stimulating the olfactory sense as part of an immersive experience. One example is the Smell Engine described as “a system for artificial odour synthesis in virtual environments”

Gustatory

Gustatory perception refers to the taste sense. It is possible to trick the human brain to into thinking that food is being consumed using stimulation by computer controlled plates placed upon the tongue. In 2013 digital lollipop was created by researchers at the University of Singapore that stimulated sweet, sour, salty and bitter tastes.

Summary

Immersive digital media has the potential to elevate and enhance the process of storytelling, communicating research ideas, developing products and in the provision of training. The last 10 years has seen rapid growth of hardware and software technologies at both professional and consumer levels increasing the number of creators and immersive content. Despite these advances, many challenges remain including the size, weight, cost and uptake of VR headsets, the cost and quality issues associated with 360 and VR180 cameras and the technical complexities of generating spatial audio. There are also positive signs too – the release of the Apple Vision Pro, camera releases from manufacturers such as Insta360 and continued support for immersive content in Adobe’s Creative Cloud.

The post Immersive Digital Media Part 1 – Definitions first appeared on Simcoemedia.

Virtual Reality: Hopes for 2022 and Beyond

Peter Simcoe — Wed, 12 Jan 2022 14:29:59 +0000

2021 demonstrated that sales of Virtual Reality (VR) products for those wishing to explore this rapidly developing technology on a modest budget remains alive and kicking – with over 10 million Meta Quest 2 headsets sold throughout the year as one example. VR is set to make a splash in 2022 with the launch of the both the Playstation VR2 and the Meta Quest Pro some time later in the year. The Meta Quest 3 will likely arrive next year, whilst the lightweight ultra-portable Vive Flow glasses are currently attracting some interest with their recent release. From a designer’s perspective, Vive, Meta (formerly Facebook) have enabled designers and artists to develop an interest and skillset in VR by providing the following experiences:

Drawing in three dimensions using apps such as Google Tilt Brush. A variety of brushes and shapes can be used to build room sized 3D drawings animated using camera paths and audio reactive brushes. In 2021 Google released the source data for Tilt Brush. This has led to the development of multi-user drawing experiences and a selection of new brushes.
Painting on canvas with Painting VR. No mess or tidy up to deal with in the real world but a whole bundle of creative endeavours with paint and canvases in VR. This app is still in Beta.
Sculpting using Adobe Medium. All the fun of virtual clay and character modelling within an environment with a professional interface. This processing-power-hungry app is difficult to master but when used to its full potential the results are quite spectacular.
Designing with Gravity Sketch’s excellent 3D modelling tools. The ability to create objects including vehicles, household objects and even architectural models is enhanced by the intuitive way that Gravity Sketch facilitates the creation and manipulation of surfaces.

Hopes for the Future of VR

Here are my hopes as a designer for VR in the near future:

Augmented Reality (AR) will form a key component of any future headset from Meta (formerly Facebook). We have played games, designed and interacted with others within a completely virtual environment. The next step is to design, play and interact within our own environment using AR. As an example, as a designer I hope that the new generation of experiences will allow the user to view their own living or working spaces in high resolution whilst also being able to interact with objects or people within that environment. This approach has the potential to allow for collaborative design development on an office table, or even kicking a virtual football against the lounge wall at home. The industry standard Microsoft Hololens is currently leading the way in this type of Mixed Reality but it comes at a industry leading price of around £3000 ($3500) for a basic unit.
Lighter, more portable headsets to reduce the burden in size and weight of the product on your face. This is a real sticking point for some new users as the size and weight remains an annoyance. This issue can often detract from the overall VR experience, especially when the headset cannot be made to fit with at least a degree of comfort (the Meta Quest 2 original strap being a great example).
Increased resolution within the apps to create more realistic drawings and environments – the resolution of headsets is improving and the screen door effect has been reduced significantly in the past couple of years with improved OLED headset screen design. There are rumours of an even higher resolution uLED technology coming to products in 2022 – 2023.
Improved workflow when designing with surfaces or solids to allow greater efficiency in the workflow between applications, 3D printers or rendering programs such as 3D Studio as a few examples.

Relevant Videos

The post Virtual Reality: Hopes for 2022 and Beyond first appeared on Simcoemedia.

Atmospheric Tilt Brush Animation in 2D and 3D 360 “The Desert”

Peter Simcoe — Mon, 25 Jan 2021 13:00:21 +0000

Above is a 4K 2D video cinematic animation created entirely within VR using Google Tilt Brush along with a soundtrack created with a Digital Audio Workstation (DAW). The 3D model is expansive and additional fog / smoke / dust effects give the impression of depth and distance. There is also 3D 360 version exported from Tilt Brush – this can be seen below. To view this version in 3D, click on the 3D icon below the video, select #anaglyph then wear your red / cyan glasses. Move around the scene by clicking and dragging the mouse. Alternatively, if you have a Head Mounted Display (HMD) such as an Oculus Rift, HTC Vive, Oculus Quest2 or similar you can download the animated sequence in 3D 360 here – https://bit.ly/vr-desert.

If you have not tried Google Tilt Brush, you can purchase this software from the Oculus Store or Steam Store. Its a great tool for drawing within virtual reality and for me personally, its the software I used the most. See some of the other Tilt Brush projects I’ve been working on which include “The Canyon” and my experiments with Matterport .obj files.

2D 4K VERSION: https://youtu.be/806mkUoTanY
3D 360 VERSION: https://youtu.be/K8q9ZQCAIdU

The post Atmospheric Tilt Brush Animation in 2D and 3D 360 “The Desert” first appeared on Simcoemedia.

Google Tilt Brush Cinematic Animation Experiment

Peter Simcoe — Mon, 21 Dec 2020 11:44:56 +0000

I have recently been developing my exploration of cinematic visuals and camera movements created entirely within Google Tilt Brush. In this example I created scenery covering the entire workspace and included water, fire and cloud / mist effects using the smoke tool along with adjustments of the fog setting to increase the sense of depth. A soundtrack was added with synth, electric guitar and environment sound effects.

Creating VR artwork within an Oculus, HTC Vive or other HMD headset is both fun and educational in the sense that it develops skills in 3D modeling, spacial awareness and the principles of animating camera movements. There are a variety of brushes, special effects and meshes available to enable you to draw and construct within the immersive environment. It is available from most virtual reality app stores such as Steam and Oculus and you can learn more about this VR tool by watching the YouTube demonstration of Google Tilt Brush. An additional 3D 360 animated sequence (below) was exported and uploaded to YouTube – you can view this in 3D using red / cyan glasses and using the mouse or touchscreen to move around the scene as with other 360 YouTube videos.

Further experiments will be conducted within the VR tool within the next few months, so watch this space. See some of the other Google Tilt Brush Experiments I’ve been working on.

The post Google Tilt Brush Cinematic Animation Experiment first appeared on Simcoemedia.

Google Tilt Brush Experiments

Peter Simcoe — Sun, 08 Nov 2020 09:16:20 +0000

Google Tilt Brush is a virtual reality art and design application for Oculus Rift, Quest and HTC Vive created by Google. Over the past couple of weeks I’ve been experimenting with this tool to create a variety of scenes. Some of these have been built from scratch and others created by importing a Matterport .obj into the scene to push the boundaries of the software and explore the creation of fire, water, landscapes, weather conditions and other elements. To appreciate these models and scenes in ideal circumstances, they can be loaded back into Tilt Brush and explored in 3D.

These videos represent the output from camera movement paths and an atmospheric soundtrack added (created with Garage Band or Propellerheads Reason). For more information on how the Matterport models were created using the Matterport Pro2 camera with infrared scanning capability, see their website at www.matterport.com. The video above featuring some of the examples with my own narration can be seen on YouTube in addition to a brief demonstration of Google Tilt Brush. You can also see another example of Google Tilt Brush work in the example called “The Desert”.

The post Google Tilt Brush Experiments first appeared on Simcoemedia.