Now There’s a Way to Watch Your Friend’s (Simulated) Airplane Flying in the Real World (Sort of)

When Microsoft came out with its latest release of Flight Simulator last year, it was widely praised as the most realistic version yet, so convincing in its detail that you could almost feel shifts in the wind and forget you were flying over a fake landscape. Sales topped two million in just four months.

The software lets you choose from a few dozen airplanes of different types, each one faithful down to the last rivet. And although it comes with enough high-res graphics of airports and landscapes to satisfy most players, a free software developer kit (SDK) allows anyone to create and upload their own models of airports and landmarks, which other users can then download. Even the air traffic is realistic. When you’re flying online, whether it’s over your home town or half a world away, you can see other people’s sim airplanes, or at least the ones closest to you, flying around or taxiing on the runway.

All of this gave Christoph Leuze an idea. His brother Matthias is, in addition to being a real-world pilot, an avid player of Flight Simulator. Leuze is a research scientist at Stanford who specializes in mixed reality applications for medicine that allow doctors to see information-rich augmented reality displays over their patients. Given his background, Christof wondered if he could “see” Matthias’s simulated flights in augmented reality. First he used Microsoft’s SDK to grab the GPS positioning information for his brother’s flight. Then he transferred that information to a server, using his own computer for the simple demo. Then, using augmented reality development software like ARKit and ARCore, he mapped those coordinates to the real world seen through his smartphone.

The result? He can now watch Matthias, who lives in Austria, land his sim airplane on the Stanford campus, as if he were flying in the actual sky overhead.

He demonstrates in this video:

First Real World Flight of a Microsoft Flight Simulator Plane

Watch on

So far, Leuze hasn’t gotten much farther than proving that the concept works. “Right now it’s only a hobby,” he says, “but because there was such high interest, I’m looking right now at how we could expand it.”

For the demo he only tracked one airplane, but there’s no reason in principle that he couldn’t track more. Scaling up to a general phone app that could “see” all the nearby simulated airplanes flying overhead would be more involved, however. You’d need identifying information for each sim airplane and GPS coordinates for its flight path, which should be readily obtainable. But the more airplanes, the more computer capacity is required. Leuze figures it would be doable for maybe 100 sim airplanes, but beyond that it might get difficult. “The AR app itself wouldn’t change that much,” he says, “but the server infrastructure would be more involved and would cost more money. This is something we’ll have to figure out, because there was a lot of interest from people who wanted to try it out for themselves.”

He hasn’t approached Microsoft or any other commercial company to explore the possibilities, but his invention did win the “People’s Choice” award for best demo at the recent IEEE VR conference, so he knows the app could be popular.

Like practically everything else these days, it blurs the line between what’s real and what’s unreal, even for a scientist who works in the field. "Now every time I hear a plane overhead,” he jokes, “I wonder, Is that my brother?”

Tony Reichhardt | READ MORE

Tony Reichhardt is a senior editor at Air & Space.