Harnessing 5G, 6G, and AI to Accelerate Immersive Technologies

The realm of extended reality, or XR, spans virtual (VR), augmented (AR), and mixed reality (MR). VR, AR, and MR offer varying immersive experiences: VR allows users to step into another world entirely, AR imposes virtual features onto users’ real-world surroundings, and MR not only layers virtual objects into a real-world environment, but also allows users to interact realistically with them.

These technologies have given rise to aspirations of countless applications in virtual education, entertainment, gaming, and health care. But to realize a future of vivid 3D gaming, crisp holographic meetings, and smooth remote surgical operations, we’ll need to think bigger than the setups we have now, said electrical and computer engineering researcher Walid Saad.

Today’s immersive experiences are still typically confined to a few users in one room, hooked up to a laptop or desktop, he explained. To Saad, a boundless future for XR lies with a wireless infrastructure that harnesses cellular networks, including today’s 5G systems and tomorrow’s 6G systems. That kind of infrastructure could enable extended reality services to operate more seamlessly, long-distance, and in big spaces like stadiums, concert venues, and theme parks.

“Cellular is pervasive and reliable,” Saad said. “It exists everywhere, and it gives you this end-to-end perspective. If we ultimately want to have holographic societies with these very precise interactions between the virtual and physical world, this is where leveraging cellular networks comes in. We need higher rates and reliability, and very low latency.”

Saad, an expert in wireless networks and communications, leads the Network Science, Wireless, and Security Laboratory and is a member of Wireless@VT, a research group founded in the Bradley Department of Electrical and Computer Engineering in 2007. With support from a National Science Foundation grant that Saad’s team received last fall, he aims to lay the foundation for a large-scale, wireless communication infrastructure that can sustain the traffic of wireless XR applications, while supporting the unique needs of users in immersive experiences.

Saad sees cellular networks as the potential “backbone” for that wireless infrastructure. He envisions harnessing base stations already deployed worldwide, not only for communication, but for the extensive sensing and computing needs of XR. His model is a hybrid one, as it supplements cellular networks with wireless local area network (WLAN) technologies like WiFi.

Saad’s team would be the first to develop such a large-scale, wireless infrastructure with 6G capabilities — one that can support services in which the physical and virtual worlds can completely merge, he said. Researchers have tested the use of wireless local area networks for wireless XR, but no holistic solutions exist for end-to-end deployment (allowing for communication to go between distant endpoints) of wireless XR over cellular networks.

His team’s wireless architecture builds from previous work to deploy virtual reality over wireless networks, during which they discovered “knobs to turn” to optimize the user’s VR experience, Saad said. They are also factoring in their previous findings on the use of terahertz, or high frequencies, to fulfill the high-quality visual and perceptive requirements of XR experiences.

Saad’s research has two major phases: fundamental theory, then application of wireless XR.

His team has first worked to characterize the user experience within their wireless architecture. They’ve looked at how network attributes like link quality, network load, and available wireless and computing resources, as well as user preferences like desired quality of visual effects and responsiveness, affect users’ quality of experience while they’re immersed in wireless XR.

Network attributes like link quality — and variations in that quality — affect how VR, AR, or MR performs. One important performance indicator is latency — how long it takes for data to get to its destination from one point in the network to another. In immersive experiences, sustained low latency is critical to making the user feel like the experience is natural or realistic, Saad said.

He’s been studying how indicators like latency affect the user’s subjective experience of the virtual setting. Saad noted that 6G capabilities will be critical for the more demanding, futuristic holographic applications of XR, which require near-zero latency along with high rates and reliability. 5G alone will not cut it.

Saad’s team is now in the early stages of the second phase of their research: application. Once they’ve determined XR’s current capabilities over wireless networks and how it should perform, he hopes to find ways to manage the system’s communication and computing resources to maintain a high quality of experience. “That’s a big challenge,” Saad said. “When I understand my performance metrics, what are the knobs I can then turn to improve these metrics?”

That’s where artificial intelligence comes into play, Saad said. His team is working to develop learning algorithms that enable wireless XR to operate flexibly under a dynamic and complex network environment, continuously adapt network resources to user needs, and deliver content effectively in real time. AI will be part of the team’s approach to reaching near-zero latency, as they’ll harness its predictive abilities to lessen the time needed to run holographic applications.

In collaboration with the University of Colorado Colorado Springs, Saad’s team will then test their framework in real-world wireless XR environments. With it, they’ll collect feedback and XR user data under real-world constraints.

Saad believes wireless XR will eventually open society up to the highly-engaging immersive experiences we’ve been dreaming up, as well as those we haven’t.

“We are at the cusp of a new world, where virtual and physical worlds become one,” he said. “We are now imagining a future where we can send, using wireless networks, the five senses of a human, thus creating new, unimaginable holographic experiences. This research is the first step toward this long-term goal.”