Immersive design overview
To make the user’s experience immersive I’ve tried incorporating the use of different visuals, lighting and sounds that make the viewer feel like they are actually in the space I created. I also made sure to render the image as stereoscopic to provide the depth of field that you get in real life.
Embodiment
With the vision I had for a 360 experience it would have been quite hard to implement embodiment as the viewer is encouraged to look around the full 360 degree range.
I did attempt this slightly in that I placed a stool in the center and suggested that when you watch its best while sitting down on your own stool that allows you to swivel around easily and take in the full 360 experience.
Chronological immersive events
To manipulate and influence the viewer’s attention I have used light, movement, colour and sound as detailed below.
1. Ambient sound
To start with, ambient birdsong tempts the viewer to look around their new surroundings, bright light illuminating the stone garden they are in. Having a HDRI dome and ambient sound helps sell that you are in the real world, with actual images of trees in the distance and bright blue sky.
2. Gate and bottle
As the viewer is looking around the gate on the outer wall slowly creaks open, pulling their view to a lone bottle that rolls towards them, creating sound as it rolls over the stone floor. It then floats right up to them, showing how you may have picked it up if in an interactive experience. The cork is taken off, making a popping sound and then hitting the ground, the viewer drinks from the bottle and the world starts to change.
The bottle is textured in the same way that the ‘other side’ is, implying it is not from our own world, creating a contrast between the two worlds. I tried to use bright colourful vines on the bottle in contrast to the dark grey surroundings to show the importance of it.
3. Swirling colour
To show the difference between the two worlds I tried to make use of colour, having brighter, higher saturation and brighter colours in the ‘wonderland’ section and darker, greyer colours in the real world.
Once the viewer drinks from the bottle the world starts to change colour, I used a blue area light and another Arnold skydome but coloured purple to change the tone of the whole scene.
4. Tile slam
The viewer fades into the next world which is filled with colourful mushrooms and 4 lanterns upon the wall. Shortly after they arrive, the tiles below them start to rise all around them along with binaural sound they always have something to look at. The tiles then slam down, blowing out the lanterns.
Putting the garden into darkness helps focus the viewer more using lighting to attract their attention.
5. Vines
You then hear the sound of something growing as vines sprout from the ground, growing on each side of the garden, funneling the viewer’s attention to the gate using movement and the light tip of the vines.
Having vines on both sides of the wall means that they will see the vines growing no matter where they look.
6. Light orb
At this point the viewer is most likely looking at the gate where a sound starts to play. If they are not already looking, the sound will draw their attention as a blue light orb appears. It floats there for a moment to allow anyone who missed it to see it, then starts moving towards and into the Rubik’s cube, making sound as it moves, which cuts off as it enters the cube.
7. Rubik’s cube
The cube slides out of the wall making a loud grinding sound, moving towards the viewer. It moves slowly, making sound as tiles lift beneath it. It starts to rotate a while after it starts moving giving you time to properly take in the view. Due to its large visual and audio presence it is hard to miss it and it slides back into the wall, solved.
8. Teapot
As the cube enters the wall, a sound is heard and a teapot teleports in front of it with a bright flash. This section is supposed to focus more on audio as the teapot teleports, rapidly moving around your head and around the room.
As it teleports, light appears on the wall and it hovers in front of each portal before quickly moving to the next.
Eventually it teleports from the gate and slowly glides back to the viewer, giving them time to see it once again. The hat then comes off, slowly increasing in size, rising above you, as it stops for a moment and slams shut over you.
Camera
Originally I planned to use depth of field on the camera so that the viewer’s attention wouldn’t go outside of the gate/garden and make them feel detached from the experience. From further research, however, I found that the VR camera in Arnold didn’t allow the use of DoF so I ended up blocking the gate and using a HDRI image to make the viewer feel like they were in a real space.
Sound
Using sound in immersive design can be used to tell a story, for example when transitioning between the real world and the ‘other side’ we made sure to make the audio sound more dream-like. Adding reverberation to the bottle as it is dropped, with the synth wave slowly coming in as the video transitions to the other side gives a more surreal feel to the world.
Binaural sound
It was important to me that the sound was finalised in a way that mirrors real life to make it feel more immersive, to do this I used binaural audio. Binaural recordings are optimised for headphones and can recreate the perception of distance – making the audio more immersive. As I was doing this digitally rather than recording the sounds binaurally I used a plugin called DearVR Micro that created binaural sound from the audio I was using.
DearVR Micro
Using DearVR Micro in Adobe Audition I was able to change the Azimuth (horizontal direction in degrees), Width (the size of the sound) and elevation (vertical direction in degrees) of sounds, taking the sound clips that Natalie produced and syncing them with a lower resolution video which was easier to work with than high resolution.
Exporting the entire multitrack session then provided a single binaural audio track that I put into Premiere Pro with the 8K footage and rendered it out.
I had a few issues with the audio, at first the sound wasn’t moving when rotating the video, still sounding binaural but not in sync with the video. I managed to fix this by following a few tutorials, using specific Premiere Pro presets for ambisonic over and under VR (Levine, 2017).
The next issue I had was that although the audio now moved when rotating the video, it was off by 90 degrees. I tried using a panner fx in Premiere but through several iterations I didn’t seem to be doing anything. I went back to the initial tutorial video and found that I hadn’t applied the specific binaural panner fx as the audio track sampling wasn’t set to adaptive.
Once I changed it I was able to use the panner to rotate the audio for export, syncing the video and audio up correctly, ready to upload to YouTube.
References
Levine J. (2017) Ambisonic audio with 360 video in Premiere Pro CC 2017 [Video]. Available online: https://www.youtube.com/watch?v=MqBDQLV7xZI. [Accessed 10/05/2021].