Adding 3D Assets To A 360 Virtual Reality (VR) Video Scene: Case Study

By Shane Folden, Gravity Jack Videographer & 3D Artist

Here at Gravity Jack, we recently completed a virtual reality (VR) project for Wind River (an Intel Company). The goal: Show users what it will be like to drive in an autonomous, self-driving car in the very near future!

This was, of course, a fun project for Gravity Jack — but not without its challenges. Notably, compositing 3D elements seamlessly into a 360º VR scene.

Interior Effects & Mapping Texture

When the experience begins, users find themselves sitting in the driver’s (or non-driver’s, if you will) seat of a futuristic, self-driving car. Throughout the experience, the windows and dash of the vehicle display a wide range of effects, including the user’s route, weather reports, music and entertainment features, and other useful information that people may like to see while being driven around town.

In addition to creating and attaching these effects to the interior of the car, we also needed to place additional effects on the road’s surface and other passing vehicles in the scene.

In order to solve the interior needs, Gravity Jack rendered all of the effects onto a flat plane. Once complete, we then mapped the texture to the 3D model. Easy! Done.

Exterior Scene Elements

The exterior effects for this project proved more difficult than originally anticipated.

For this challenge, we used Mettle’s 360 software within After Effects. Doing this allowed us to break the equirectangular image into 6 individual images.

Side note: “Equirectangular” is just a fancy word for the unwrapped version of a 360º video. Imagine a cardboard box that has been taken apart, each panel of the cube is a different image.

The purpose of this step is to effectively track the scene and place objects in the scene as though it were a normal, two-dimensional video. Once this is complete, we are able to take the six individual images and combine them, seamlessly, into one equirectangular image or video (for this case, it was video).

The Mission At Hand

We needed to place a green and red highlight on the road as the car traveled. Red, showing where the user could not travel due to a wreck or construction, and green to show an alternate route for the user.

To do this, we placed the plane onto the road surface and tracked it to the scene. This makes the plane “stick” to the road, ensuring it does not move around with the video. Ultimately, this effect gives the illusion that the green and red planes were part of the scene.

Up to this point — so far, so good. That was, until we start driving “over” the plane.

The Issues We Encountered

The first issue we encountered was that the 3D model of the vehicle limited the visibility of the road surface, directly in front of the car. Unfortunately, we only had about 5 seconds of video to work with for this particular scene. We needed to convey the fact that the plane on the ground was part of the scene and not just a brief flash of color.

Our second issue was that all of the windows on the car were transparent, allowing users to look around within the scene and be fully immersed. When we tracked the plane and placed it in the scene, it matched up and we were able to briefly drive over it; however, when we looked out the driver side window, rather than seeing a soft lit path, we saw a hard-edged green square that had been cut off by the other images.


This let us know that it had only been tracked within the front facing image.

Naturally, we went back and tried to track the plane in the other image views as well. The end result was a very sloppy look. Regardless of what we tried, getting the images to line up with each other (and seamlessly within the scene) proved entirely too difficult. So, that idea went…out the window. 😉

The Solution

Instead of trying to track the plane in a 360º space, we simply added it to the front windshield of the car and masked off the sections that would pull users out of the immersion. This made two important things possible:

A) Users could drive along the plane for as long as we wished, as it was now part of the car.

B) We could now allow the plane to pass by the car on the side windows, which masked out the section we didn’t want the user to see.

Additionally, we were now able to manually track the plane to fit naturally within the scene. Doing this ensured it was flat on the road, curved with the road, and didn’t jump onto the sidewalk or parked cars.

It was a perfect fit!

Our Takeaway

Our takeaway from all of this: Don’t make life more difficult by trying to overdo/overthink your effects. KISS (keep it simple stupid).

The solution was actually the fastest and most effective way to get our point across. Moving forward, we will continue to place 3D objects into a 360º scene, pushing the boundaries of our abilities every chance we get.

From time to time it’s important to take a step back and analyze what is truly needed and allow ourselves as creatives to step ‘outside of the box.’

To see the final experience for yourself, you can download the free Wind River Experience app for both iOS and Android.

If you’re a videographer, 3D artist or developer running into similar struggles, feel free to reach out to us with your question! We’re always happy to help answer questions!


  • The Revolutionary Impact of Virtual Reality in Tourism

    Step Beyond Boundaries: VR’s New Travel Dimension

    Embark on an exploration of how Virtual Reality (VR) is innovatively charting the course of the tourism industry. The VR landscape is bustling, poised to reach a staggering $62.1 billion market value by 2027, with travel experiences contributing a substantial slice to this tech pie. Pioneers like Google Earth VR, Ascape VR, and The Wild Within offer sweeping vistas and interactive treks, inviting users to tour globally from comfort zones.

    • Revolutionizing Exploration: VR transcends geographic limits, enabling virtual visits to the world’s marvels.
    • Tourism Titans: Leaders in tech exemplify VR’s vast potential to reshape how we uncover global wonders.
    • Gravity Jack: As adept navigators in VR and AR realms, we craft immersive voyages that enchant and inform.

    We stand at the cusp of a grand revolution, where virtual journeys ignite the spirit of travel. For an in-depth survey of VR’s trailblazing trajectory, dive into our full discourse on tech-fueled tourism!

  • Crafting Personalized Retail Expereinces With Augmented Reality

    Augmented Reality: Revolutionizing Retail Customization

    In the bustling realm of retail, Augmented Reality (AR) is ushering in an era where shopping becomes a personalized showcase. Imagine trying on clothes without stepping into a dressing room, or placing furniture in your living room with just a tap on your screen. With an anticipated market growth to USD 795.3 billion by 2025, it’s clear that consumers crave these cutting-edge experiences.

    • AR empowers consumers to customize products in real-time.
    • Big names like IKEA, Sephora, and Nike are pioneering this space.
    • Insightful data gleaned from AR interactions are redefining marketing strategies.

    At Gravity Jack, we’re not just observers but active innovators in AR development. We provide immersive solutions that resonate with consumers, paving the way for businesses to weave their narratives into the customer’s world. Dive into our blog to see how the tactile future of digital shopping is already at our fingertips.

  • Augmented Reality: A Guide to Understanding Its Impact on Business, Products, and Branding

    Dive into the expanding universe of Augmented Reality (AR) with Gravity Jack’s latest insights on creating captivating AR applications. As the industry projects a staggering growth, hitting a market value of $3664.5 million by 2025, we’re here delivering the golden rules and innovative solutions for AR success.

    • User-Centered Design: Just like Snap Inc. and Niantic have aced user feedback, we craft AR experiences with your audience as our north star.

    • Technical Mastery: Conquer challenges like real-time tracking and cross-platform compatibility with our expertise inspired by ARCore and ARKit.

    • Next-Gen Immersion: Get excited about optimizing AR for the freshest platforms like the Apple Vision Pro and Oculus Quest 3; we’re all about that realistic occlusion and silky-smooth refresh rate.

    Stay ahead in the AR game with a partner who marries professional finesse with a playful edge. For a sneak peek at the future tailored by hands-on creativity, peruse our full article on AR Application Development Best Practices. Discover more and let the magic of AR enhance your digital landscape.