AI Generated 360 Images

In November 2023 I initiated a series of experiments examining potential production methods for a short film. During experimentation Midjourneywas used to create equirectangular images which were edited in Photoshop (Beta). MagnificAI upscaling increased the resolution up to 12K. See them on the Simcoemedia website: www.simcoe.co.uk/product-category/360-images/

Also see:

12K examples on 360Cities.net
Applying styles to 360 photography experiment
Tutorials by Peter Simcoe

360 Video Handbook

Published in February 2018, the 360 Video Handbook is a coffee table book featuring a series of inspirational 360 video projects. The style and layout was designed to make it easy to access by featuring an explanatory diagram or photographs on one side of the spread throughout the publication. The book also answers frequently asked questions by beginners. See all Simcoemedia books at:

www.simcoe.co.uk/books/

Augmented Reality Music Video

An AR music video for the song ‘A Little On The Darker Side’ was created using Adobe’s Character Animator software. A series of character designs were developed. These were then animated and rendered in HD. The characters were exported as animated GIFs with transparency then imported into Adobe Aero to create an augmented reality experience. See the example video at:

https://youtu.be/ZfZQ5DZgq1w

Google Tilt Brush 3D 360 Video

3D 360 videos created with virtual reality art creation app Google Tilt Brush. These can be viewed in 3D on YouTube using red / cyan glasses:

Google Earth Studio

Google Earth Studio allows users to generate animated sequences using Google Maps including the 3D photogrammetry data available for many cities around the world.

Chester 4K 360 tour
London 4K 360 tour
London Interactive 4K with labels
Snowdonia 4K 360 tour

360 Video with Ambisonic Audio

In 2017 I became a content creator for HumanEyes Technologies. The company offered funding for 360 video projects using their camera technology including a series of travel videos in Chester, London and Barcelona, music videos, development of a second, upgraded light suit (the original low budget version can be seen in the One More Chance video). Some of these were featured in the 360 Video Handbook. HumanEyes Technologies supported the creation of music videos by supplying extended access to ambisonic audio software and funding the production of light suit v2.0.

Meta Workrooms Interview

Created a series of interviews with Emily Olman and Paul Tomlinson using Meta Workrooms. These interviews explore the connectivity and activities facilitated by Workrooms and explored the meaning of Spatial Computing:

Google Maps interactive 360 images

360 photographs taken at locations across the UK, Spain, Sweden and France have been uploaded to Google Maps to create a portfolio of immersive imagery capable of display on a desktop, mobile devices and within a VR headset.

See Google Maps images by Peter Simcoe

Simcoemedia Spatial Gallery

An interactive gallery created using Spatial IO. Can be viewed on mobile devices, desktop and VR headsets. See gallery.

Microsoft Flight Simulator

This VR video was recorded to demonstrate take-off and flight around Airbus (Broughton) plant in North Wales, UK using Microsoft Flight Simulator. It represents the level of realism the simulator currently offers.

Other Links

www.simcoe.co.uk
www.youtube.com/simcoemedia
www.x.com/simcoemedia
www.linkedin.com/in/simcoemedia/
www.behance.net/simcoemedia

360 Video

360 video can be viewed in a Virtual Reality headset such as Meta Quest 3 and, when uploaded to platforms such as YouTube, is also available on a desktop or mobile device. The viewer interacts with the content within a VR headset by moving their head or on a desktop by ‘clicking and dragging’ to change the point of view using an input device such as a mouse. It is also possible to achieve similar interactions using the gyro technology on a mobile phone or a screen with touch capability. YouTube can display interactive 360 video in VR, on desktop and mobile.

Video is recorded with a camera utilising a series of wide angle lenses designed to capture the surrounding environment. It is stitched together using compatible software which may be provided by the manufacturer such as Insta360 Studio or by a third party such as Mistika VR.

360 video is usually recorded in the same equirectangular format as 360 photography. Current cameras record video of at least 6 – 8K which results in Gigabytes of data per minute with the Insta360 Titan recording 11K (10K in 3D). The challenges posed in producing 360 video, such as hiding microphones, lights and other equipment, has led to a decline in use during recent years in favour of 3D VR180 video. However the format remains popular in real estate, tourism and journalism where a view of the entire environment is important.

VR180 Video

VR180 uses half the horizontal viewing angle of 360 video with just the front facing 180 degrees available. It is designed to be consumed within a VR headset, viewed on a screen with active glasses or converted to anaglyph for viewing with red / cyan glasses. Whilst there are only 180 degrees of recorded content, most VR headsets have a viewing angle of around 90 degrees which provides a realistic sense of immersion.

Content is typically recorded using 2 wide angle lenses covering a 180 degree viewing angle. Both of these lenses face the same direction with the centre of each lens placed at approximately the same distance as human eyes. When converted for use within a VR headset, the video provides realistic depth. HumanEyes Technologies released the Vuze XR in 2018 which had two 4K cameras which could be used in VR180 mode or 360 capture mode. A recent addition to the VR180 camera market is the CALF 3D VR180.

This format is used in vlogging and entertainment such as storytelling. However, as mentioned in the previous section, 360 video is still used when it is useful to see an environment in its entirety.

Virtual Reality (VR)

Virtual Reality experiences are designed to facilitate interaction where location, physicality and changes to the environment have meaningful consequences. They are usually viewed within a VR headset such as Vive XR Elite or Meta Quest 3 using controllers or hand tracking. However, platforms such as Spatial and Horizon Workrooms allow users access via a desktop environment as a ‘window’ to the virtual world. The user is able to shape the narrative and environment by their choices which may involve changing the state or position of physical objects within a space. Many VR applications are created with software such as Unity or Unreal Engine.

Examples of immersive VR applications range from as simple as the simulation of fairground games within Nvidia’s VR Funhouse, production of 3D art using Google Tilt Brush or involve the complexity associated with piloting an aircraft in Flight Simulator. Other examples may be found on Meta’s App Store.

The term ‘Virtual Reality‘ was first used by American academic Jaron Lanier in the 1980’s as a title for his research project. He is considered to be the ‘father of VR’ because of his groundbreaking work in the field.

Ambisonic Audio

Ambisonics is an audio technology that uses hardware and software capable of rendering spatial audio in Virtual Reality, Augmented Reality and Mixed Reality. As few as 4 audio channels can be used to represent sound within a virtual space. As the viewer’s head changes direction or objects emitting sound move within a space the audio is adjusted in a realistic manner to reflect the effect of these movements on the perceived sound. It is also possible to experience ambisonic audio in a limited manner when viewing 360 video on a desktop PC or mobile device by moving the point of view. The use of 4 audio channels to simulate spatial sound is referred to as First Order. However, it is possible to use more than 4 channels to enhance the effect in a similar way to improvements of 7.1 surround sound over 5.1.

For more information on ambisonics, see this excellent summary of ambisonic audio from Waves.com

Olfactory

Olfaction or olfactory sense is the sense of smell. There are devices capable of stimulating the olfactory sense as part of an immersive experience. One example is the Smell Engine described as “a system for artificial odour synthesis in virtual environments”

Gustatory

Gustatory perception refers to the taste sense. It is possible to trick the human brain to into thinking that food is being consumed using stimulation by computer controlled plates placed upon the tongue. In 2013 digital lollipop was created by researchers at the University of Singapore that stimulated sweet, sour, salty and bitter tastes.

Summary

Immersive digital media has the potential to elevate and enhance the process of storytelling, communicating research ideas, developing products and in the provision of training. The last 10 years has seen rapid growth of hardware and software technologies at both professional and consumer levels increasing the number of creators and immersive content. Despite these advances, many challenges remain including the size, weight, cost and uptake of VR headsets, the cost and quality issues associated with 360 and VR180 cameras and the technical complexities of generating spatial audio. There are also positive signs too – the release of the Apple Vision Pro, camera releases from manufacturers such as Insta360 and continued support for immersive content in Adobe’s Creative Cloud.

AI School

I’ve recently been exploring the use of Midjourney and Magnific.ai to create highly detailed 12K 360 images such as the example shown above (note this example has been enhanced with audio and lens flare using Pano2VR). After working out the most appropriate workflow through a series of experiments, a course was published on Eventbrite designed to impart this knowledge and experience to interested parties in the AI and virtual tour communities. In addition to Midjourney and Magnific, the process involves Photoshop and Pano2VR (or equivelent editing software tools) to edit the nadir and zenith (directly below and above the viewer) along with ensuring the vertical seam blends. This is to ensure objects and textures on the left side of the image fit seamlessly with the right when assembled in the final 360 form.

You can see a variety of images produced using this method by visiting the 360 image section of the Simcoemedia Shop or on my Facebook feed.

AI Workshop Details

The workshops have been designed to cater for 360 enthusiasts, immersive media artists and virtual tour producers looking to enhance their 360 workflow using AI tools. The 90 minute session includes the following activities:

• Creating equirectangular images in Midjourney by using appropriate commands, parameters and descriptive keywords
• Ensuring the image does not warp in 360 view by aligning the horizon correctly
• Resolving stitching problems and nadir issues using Photoshop and Pano2VR (or equivelent) to ensure the best interactive experience
• Using Magnfic to upscale and add detail to images whilst retaining a reasonable level of creative control
• Resolving metadata issues + testing images to ensure they are ready for publication
• Discussion of how these tools have developed and evolved in recent months

A reference page has been set up on this site at www.simcoe.co.uk/ai-school/ reminding participants of the key processes highlighted during demonstration and discussion. So far these sessions have been useful and enjoyable for those taking part with some returning for the second in the series which examines transfering image styles to existing 360 tour photography. For more information on upcoming events see Peter Simcoe’s Eventbrite page.

Related topics: Find out more about how Generative AI creates video and images by reading ‘What Is Generative AI and Is It Useful In Film Production?’

Other Workrooms Articles

Exploring Workrooms with Emily Olman Episode
Virtual Reality hopes for 2022 and Beyond

About Spatial IO

Spatial is a desktop, mobile and virtual reality compatible platform designed to help brands, game designers and media producers build and share spaces containing interactive content. The type of media you may find in design galleries on Spatial include photography, 3D models, videos and graphic design. In addition, spaces can be converted to NFTs and then rented or sold to other hosts. The platform also provides the Creators Toolkit allowing users to integrate Unity 3D functionality and it is possible to link to your SketchFab account where models can be dropped into Spatial spaces with ease.

Spatial was founded by Anand Agarawala and Jinha Lee. They describe themselves and their employees as 3D design and metaverse experts based in New York and San Francisco. Check out their website and explore some of the spaces at www.spatial.io.

Simcoemedia Photograpy Gallery

The Simcoemedia Design Gallery was created within Spatial to demonstrate a variety of media including graphic design such as album cover artwork, video production, virtual reality such as Tilt Brush experiments, books (such as Freelancers Handbook) and photography from around the world. These are some of the key projects completed during the 21 years of creative industries freelancing.

One of the most useful features of Spatial is the ability to view content within VR (such as Meta Quest2), a mobile phone (via the Spatial app) and desktop via the website at www.spatial.io. This enables the gallery to be viewed by a wide variety of audiences, though you will need to sign up for a free account.

The second of the 2D 360 tours focuses on Snowdonia in North Wales. Fly down to the UK from space and take a trip across the mountain range for spectacular views generated by photogrammetry included with Google Earth. There are only a select few locations in each country with photogrammetry / 3D data – usually these are key cities such as London, New York, Los Angeles, Sydney and other major cities. There are also many national parks and points of interest for tourists that use 3D data. For other areas 2D photography is the only available option for the moment but it is possible to switch on a guide within Google Earth Studio to indicate where 3D data is available.

There are a variety of features not used in the experiments above that are available within Google Earth Studio including the ability to roll the camera, export of 3D camera data for purposes such as overlaying text in After Effects, control over the field of view and the influence of a focal point. I tried many of these options and the video below demonstrates use of the field of view (FOV) adjustment as well as leaving the labels for locations and businesses switched on. At the time of writing this article I could not get Googles cloud computing to render an 8 minute video at 60fps with labels (the labels did not even appear on shorter videos) – to enable the labels as seen in the 4K River Thames Tour below I had to enable a screen capture then stabilise the video in Premier Pro and this stabilisation process is why the Google Earth logo is a little jittery Its best to view this in 4K quality on a large 4K screen.

The final experiment involves recording motion over the city of Barcelona at the same angle of approx 60 degrees. The animation was export as a 4K video and then imported into Adobe Premier Pro. Gradient blur was added to the top and bottom of the image to simulate a camera lens with a small depth of field which in turn gives the impression that the subject matter is a small scale model. See below for the results of this experiment:

Many thanks to the Google Earth team for providing me with early access. You can find out more information about Google Earth Studio on their website and see other examples of experimental 360 video created by Peter Simcoe on the Google Tilt Brush blog post.