Merging Entertainment and Tech: How to Build Immersive Experiences Together

  Disney / Lenovo’s Star Wars Jedi Challenges AR device invites players to engage in lightsaber duels with life-size Sith lords (Disney)

Disney / Lenovo’s Star Wars Jedi Challenges AR device invites players to engage in lightsaber duels with life-size Sith lords (Disney)

Hollywood and Silicon Valley, Bound Together

Technology and entertainment have always needed each other.

Technology enables new platforms for content, creation, interaction with experiences, and distribution. Yet consumers continue to be drawn to new technology primarily by the content and experiences it enables. Content, as the saying goes, is king.

When it comes to entertainment, audiences’ primary emotional connection is to the imagined world, character, feeling, or memory associated with content - not the device, file, or site they used to experience it. Although when designed in an integrated and inspired way, the quality and ease use of a technology platform can serve to further enhance and strengthen emotional connection to a story.

On the other hand, the entertainment industry frequently falls behind advances in technology and is slow to adopt emerging platforms. We have seen this over time from last century’s development of motion picture sound, to peer-to-peer file sharing, to today’s increasing dominance of streaming media services-tuned-studios and user generated content.

With a few notable exceptions (such as video gaming, which I will address in a subsequent post) advancements in the tech industry develop largely separately from the world of entertainment production and the content it depends upon.

TECHNOLOGY AND ENTERTAINMENT NEED EACH OTHER NOW MORE THAN EVER

Audiences have historically adopted the methods necessary to engage in their favorite entertainment experiences, such as going to a public theater or a movie theatre to catch a first run movie, recording a television program on a VCR or DVR device for later viewing, accessing a favorite show exclusively available via a Netflix subscription, contributing fanfic ideas or feedback on social media, or searching for a music video only available on YouTube. All it requires, of course, is a motivated interest in connecting with that story or experience, often times over and over again.

But currently Apple, Facebook’s Oculus, Amazon, Netflix, Sony, Samsung, and more are all in a race to deliver increasingly personalized content (media streaming services) and more compelling and immersive platforms (such as AR and VR) to increase engagement with users and attract new customers.

The battle for securing cutting-edge platforms/devices in consumer homes (with the development of smart speakers & displays, connected TVs, cameras, and head-mounted displays, just to name a few) is becoming more and more dependent on the content necessary to gain user interest and drive engagement. And content such as television, film, and music will need to be designed to leverage and showcase the new wave of devices’ tech-forward capabilities such as mixed reality, AI, 4k cameras, biometric sensors, facial recognition, and more.

 Clockwise, from top left: Netflix announces development of ‘choose your own adventure’ Black Mirror episode for 2019, @Lilmiquela is an AI Celebrity on Instagram, Spectator e-sports on the popular Twitch streaming platform, owned by Amazon.

Clockwise, from top left: Netflix announces development of ‘choose your own adventure’ Black Mirror episode for 2019, @Lilmiquela is an AI Celebrity on Instagram, Spectator e-sports on the popular Twitch streaming platform, owned by Amazon.

Thanks to the increasing impact of social media and data-curated streaming services, audiences have started to expect a more personal, direct interaction with their entertainment. Increasingly, people don’t just want to view a story and characters from a distance, they wish to experience it - even influence it - themselves. The evidence is rampant, from increased user-generated (or ‘prosumer’) content expanding story properties and marketing, the exploding popularity of cosplay and location-based entertainment such as immersive theater and escape rooms, continuing development and integration of virtual and augmented reality experiences where viewers are ‘part of’ a world or acting directly with a character, and more. Audiences are enthusiastically responding to new experiences which offer more personal interaction and a deeper connection to stories and characters, which of course data and technology can and have enabled on a large scale.

THE FUTURE IS IMMERSIVE, PERSONAL INTERACTIONS

 National Theater in London Offers Glasses With Live Subtitles (Courtesy New York Times)

National Theater in London Offers Glasses With Live Subtitles (Courtesy New York Times)

 Oculus Touch controllers precisely recreate users’ hands and gestures in VR media (Oculus Quest)

Oculus Touch controllers precisely recreate users’ hands and gestures in VR media (Oculus Quest)

Meanwhile, the entertainment and tech industries still struggle to develop products together with shared goals, motivations, and timelines. We’re still a long way off from creating truly seamless, cohesive technology experiences which prioritize and enhance content and emotional connection over layers of paywalls and complexity.

A few inspiring examples of product experiences demonstrating seamless integration of technology with the magic of story and character include:

  • AR head mounted displays for theater audiences which integrate subtitles in various languages (without obscuring the show itself)

  • Disney’s Star Wars Jedi Challenges AR kit for the home which enables a first-person lightsaber dual with a Sith lord

  • Oculus Touch controllers with haptic feedback recreating human hands and gestures within VR worlds - enabling users to feel the experience of petting a character directly

  • Amazon Echo smart speaker acting as gateway through which audiences can quickly and easily access a single story from over 400,000 audiobooks

These product experiences integrate technology and content in such a way that the devices act as the means, not the ends, to innovative entertainment while playing a critical role in access and immersion.

So how do we work better together to make truly compelling experiences for audiences that integrate the power of new technologies and interactions?

Developing and shipping meaningful entertainment experiences like those listed above will require a tighter than ever connection among entertainment and technology industries and practices. We need to ensure more fluent sharing of knowledge, processes expertise, and tools so that we can collaborate from the ground up and deliver to audiences (and users) the personalized, realistic, and overall seamless and delightful experiences they crave.


RECOGNIZING SIMILARITIES AND DIFFERENCES

Entertainment and tech inarguably share a goal to deliver audiences and (users) meaningful, compelling experiences that endure and build connection to brand.

And from one point of view, production practices across entertainment and tech can be viewed as quite similar, if given different titles:

Tech /Digital Production

Investor

Product Managers

Development & Content Authors

User Experience Design

Visual UI Design, Industrial Design

Cross-Platform Development

Elevator-pitch ideas to illustrate impact i.e “Uber for air transportation”

Interaction Design

User Experience Testing & Research

Data Analytics & Qualitative Research

Entertainment

Executive Producer

Show-runners / Producers

Storytelling & Writing

Direction

Artists & Art Department

360 Marketing and Brand Strategy

High concept pitches to predict audience appeal. i.e. “Jaws in space” (Alien)

Storyboarding

Pitching & Feedback

Box Office Numbers and Sales

…and more

 Pixar Story Artist Valerie Lapoint pitches story ideas for feedback and notes (Kahn Academy / Pixar in a Box)

Pixar Story Artist Valerie Lapoint pitches story ideas for feedback and notes (Kahn Academy / Pixar in a Box)

 The author user testing the Microsoft Surface mixed reality device to determine its intuitiveness and usability (Microsoft)

The author user testing the Microsoft Surface mixed reality device to determine its intuitiveness and usability (Microsoft)

However a few key differences should also be considered which heavily impact production goals and approaches.

  1. Overall tolerance for risk. Hollywood is the undisputed master of storytelling and emotional impact. Yet film, television, and music production yields such thin profit margins that Hollywood is equally notorious for strategies of risk avoidance and investment in established, predictable properties and experiences. Conversely, one can look at tech’s foundation of venture capital wealth for unbounded optimism and risk taking in technological innovation, demonstrated through the creation of computer animation & digital effects tools, VR, and more. Few tech ventures succeed, but when they do they are often widely impactful to users.

  2. Anticipated length of engagement. In entertainment, audiences fully submitting submit themselves to an experience for a limited time (such as 2 hours). This context allows creators to deliver a pre-scripted user experience that audiences abide by and can provide full attention to. In tech, the intent of building consumer technology platforms is for the user to be able to use, adopt, and live with the technology on an ongoing basis. This requires product experiences to be designed, developed, and tested to be primarily intuitive and easy to use - if they are to be successfully adopted. Technology teams constantly test with users, their version of audiences, to make sure the product is centered around their needs and expectations, not the developers’.

  3. The introduction of access and usability obstacles. Within the nature of developing and releasing new, innovative technology is the introduction of entirely new devices, uses, functions, and interactions. Unlike a typical song format or story spine, the goal is actually to break out of predictable formula. New technology requires that users willingly try something entirely new, learn and understand it, successfully interact with it, and adopt it. Think for example of the first time a user encounters a virtual reality head-mounted display or new game controllers; users are frequently not entirely successful with the experiences at first use, and the products are often unintuitive to use, requiring considerable guidance. And when we consider a wide range of audience or users with varied experience with technologies, the visible usability problems become more acute. The integration of approachability, usability, and easy access for all remain major challenges for technology development (and the tech industry as a whole) and can critically hinder access to entrainment content.

How to Move Forward Together

In practice, we’ll need to

  • Share concepts, pitches, ideas, and explorations more frequently - especially in areas where tech and creative ideas converge - so that we can drive experiences that leverage each others’ developments.

 Testing sketched content with an early cardboard prototype of a hardware device. This practice enables cheap, rapid iteration to ensure an intuitive ‘total experience’ across content and tech design. ( Stanford )

Testing sketched content with an early cardboard prototype of a hardware device. This practice enables cheap, rapid iteration to ensure an intuitive ‘total experience’ across content and tech design. (Stanford)

  • Develop and test concepts together. Combine early content ideas and technology prototypes to test the overall experience earlier and more frequently with audiences and users. For example, artist sketches and storyboards can be integrated into paper prototype testing and ‘Wizard of Oz’ practices (common in tech) to ensure that the combination of content and technology is intuitive, fluid, and delightful.

  • Trust each other's expertise. Technologists would benefit from giving artists and storytellers’ more of a platform to contribute early and often when developing interactive experiences, such as AR and VR. And artists must find ways to work more closely in partnership with technology design teams to test and iterate stories and characters quickly, so they can match the speed of Internet-based feedback and ensure a story interaction makes sense within the limitations of the technology.

  • Supplement each other’s risk tolerance. Tech should invest in more creative story talent that can help stretch the boundaries and impact of immersive experiences. Storytellers can imagine their world in the context of various platforms and levels of viewer interaction and influence.

  • Tech in particular will need to step up the human-centered design processes and work harder to remove barriers of usability, lack of familiarity, and user discomfort that still plagues most advanced technology products. This includes addressing critical barriers of trust with use of user data, and including inclusive design processes and accessibility. More than ever, storytellers will depend on platforms which enable them to reach as many audiences as possible.

  • Consider Audiences, Viewers, Consumers, Players, and Users as the same person at various moments of engagement. Once we adopt a shared perspective to consider all of these roles as various points of a single journey throughout a product experience, we have made major headway into carting a map for a more collaborative development process across entertainment and tech.

Look to the Legacy of Video Games For Tools

We must carve a path together sharing expertise, process, and techniques, and ideally build new tools together to create truly engaging, cohesive experiences.

Fortunately, one example we can look to is the development of video games over the past 50 years, which can show us some valuable clues how to achieve a productive synergy. In my next blog post, I’ll outline tools that the video game industry has developed to create cohesive and compelling experience across advances in content and technology.