Unreal Engine baseball fans on live TV are just the beginning

If I were watching Wilmer Flores hit his first Giants homer the way I normally watch baseball—half paying attention, half thinking about a different sport—I might not have noticed that the crowd was a little off. If I forgot, for a moment, that we're in the middle of a pandemic, I might have thought that the ball had actually dropped into rows of unimpressed Dodgers fans.

Of course, there were no fans in Dodger Stadium over the weekend. In certain shots, Fox Sports filled the bleachers with Unreal Engine puppets with the help of augmented reality production system Pixotope. Fox and other channels also piped in fake crowd noise taken from MLB The Show. 

Even if I were distracted, I would've noticed that some camera angles contained fans, while others showed empty seats and blue tarps, but on the whole the effect wasn't nearly as uncanny as I expected. Squint a bit, and Flores' homer looks like any other homer as it would appear on TV. When you're trying to pick out the ball, or watch to see if the outfielder makes a leaping catch, the crowd looks like a crowd so long as it's there. The mind, always guessing and simulating, sees what it expects to see.

“Our goal is to make sure that the view looks normal," said Fox Sports executive Brad Zager in an interview with Variety.

Should MLB games just be quiet, crowdless, and weird right now? I think so, because the weirdness will remind us that they probably shouldn't be playing . But this is what the bosses of televised reality have decided: Play the games, fake the crowds with videogame technology, make it feel "normal."

We can be certain that this won't be the end of it. Television may never have shown us objective reality (the existence of which is an ontological debate for elsewhere), but at least we could believe that the crowds we were seeing were human beings. As we get used to that not being the case, and as videogame graphics improve—check out the Unreal Engine 5 demo for a glimpse—network execs surely won't shy away from more realtime TV magic. 

According to the Variety article , another Fox Sports exec suggested that the fake crowds could be used as an ad format. I can't wait to see thousands of baseball fan sims simultaneously gorging themselves on virtual Taco Bell Crunchwrap Supremes. (I would actually like to see that, despite the queasy tone I'm striking.) 

Beyond baseball games, the mixing of computer generated stuff and live video is becoming ultra-convincing. The Unreal Engine video below demonstrates how it's possible to make a virtual environment a real one, at least in terms of what a camera sees. Want to shoot your motorcycle guy in a desert? He's in a desert. Forest? He's in a forest. 

Along with performance capture, machine learning video manipulation, AI voices that can cry, and all the other budding tech that's replicating and simulating our world, we're fast approaching a point where there won't be any noticeable difference between live human crowds and Unreal Engine crowds, living actors and dead actors, or environments that exist in the world and environments that exist in 3D rendering software. Within a decade or two, they won't have to rely on us being a little distracted: Fake crowds will just look real, if they want them to. When real fans return, a game that isn't sold out can become a sold out game, if they want it to.

Will they get away with it? Doesn't seem like it'll be hard. The comments on Fox's YouTube video aren't positive ("dumbest thing I've ever seen"), but I haven't run into any David Cronenberg movie-esque radicals on a mission to defend material reality. There's a lot going on. Subtly influencing millions of viewers by making live sports broadcasts more like videogames doesn't rate too high on the problems list. 

And maybe we don't mind being experimented on, just a little bit. After all, it is comforting to watch a baseball game with crowd noise right now. The ratings suggest as much, at least. And as videogames are praised for becoming more 'realistic,' it's only fair that reality should get to become more like a videogame. They'll converge faster that way, which is of course the ultimate dream of the entertainment industry: to become all that there is.