Share this article

With KB5043178 to Release Preview Channel, Microsoft advises Windows 11 users to plug in when the battery is low

Copilot in Outlook will generate personalized themes for you to customize the app

Microsoft will raise the price of its 365 Suite to include AI capabilities

Death Stranding Director’s Cut is now Xbox X|S at a huge discount

Outlook will let users create custom account icons so they can tell their accounts apart easier

Microsoft’s latest patent generates hyperreal virtual avatars based on users' faces

The gaming industry can greatly benefit from this technology.

4 min. read

Published onApril 23, 2024

published onApril 23, 2024

Share this article

Read our disclosure page to find out how can you help Windows Report sustain the editorial teamRead more

Microsoft has a thing with virtual avatars: its recently-releasedMesh platformlets Teams users create avatars to engage in so-calledvirtual immersive spaces, which are contexts where employees can hang out together, virtually, even though they are miles away apart.

Microsoft Mesh is indeed a fun platform, and the Redmond-based tech giant says it’s working: many Microsoft employeesfeel a sense of bondingwhile hanging out there. But at the same time it feels off:the cartoon-like avatars, while cute and everything, seem displaced.

However, it seems that the Redmond-based tech giant might already be working on an update. The company recently published a patent that describes a technology capable of generating hyperreal virtual avatars based on users’ faces.

The technology is calledMulti-modal Three-Dimensional Face Modeling and Tracking for Generating Expressive Avatarsand describes a computer system that creates expressive avatars using 3D face modeling and tracking.

The computer system has a processor and a storage system of its own, which processes and keeps data needed to model the hyperreal virtual avatar. It also keeps instructions on how to render them.

The processor then receives initialization data, which is the starting look of a face model, and additional multi-modal data signals, which can be audio data (such as the voice of the user), that will be used to create a hyperrealist model of the face.

Multimodal 3D face modeling and tracking techniques can utilize multiple different sensor devices, each providing one or more input signals and/or measurements for a user’s face to detect, model, track, and/or animate a three-dimensional face model graphically as an avatar.

The initialization and multi-modal data are then taken by the system and are processed together in a fitting process. This process adjusts the data to fit the model. Based on the fitting process, the system then determines a set of parameters, which are used to describe the hyperreal virtual avatar.

3D face modeling and tracking techniques createe 3D vertices based on a user’s face and apply transformations to the vertices from a neutral face to depict expressions on a digital face model (e.g., an avatar representation of the user’s face).

The system uses deep learning to come up with a detailed virtual avatar that looks, acts, and is capable of facial expressions similar or identical to the user’s face. Deep learning is a form of AI technology that seeks to emulate the human brain as much as possible, and Microsoft has been investing in it over the last few years.

The framework utilizes deep learning techniques and forward modeling to perform a parametric fitting process that translate the multi-modal data signals into a set of parameteres or expression code that can be used to generate an expressive 3D facial model.

Since this is a computer system, the paper mentions the ability to integrate it on a plethora of devices, from VR/AR/MR headsets to mobile phones, laptops, gaming consoles, tablet computers, and many more, meaning users could have a device ready to transport their virtual avatars into a Meta-like or Mesh-like virtual space.

Microsoft might seek to implement this new hyperreal virtual avatar system in existing platforms, such as Microsoft Teams, Microsoft Mesh, or even Windows.

Even though virtual spaces already exist, they are not that popular, for now, due to their limiting capabilities, but a system like this one can encourage more people to join virtual spaces.

The gaming industry could also benefit from this technology, allowing users to customize their characters based on their looks, providing a highly personalized gaming experience, as well.

The full paper can be readhere.

More about the topics:microsoft,Microsoft Teams

Flavius Floare

Tech Journalist

Flavius is a writer and a media content producer with a particular interest in technology, gaming, media, film and storytelling.

He’s always curious and ready to take on everything new in the tech world, covering Microsoft’s products on a daily basis. The passion for gaming and hardware feeds his journalistic approach, making him a great researcher and news writer that’s always ready to bring you the bleeding edge!

User forum

0 messages

Sort by:LatestOldestMost Votes

Comment*

Name*

Email*

Commenting as.Not you?

Save information for future comments

Comment

Flavius Floare

Tech Journalist

Flavius is a writer and a media content producer with a particular interest in technology, gaming, media, film and storytelling.

With KB5043178 to Release Preview Channel, Microsoft advises Windows 11 users to plug in when the battery is low#

Copilot in Outlook will generate personalized themes for you to customize the app#

Microsoft will raise the price of its 365 Suite to include AI capabilities#

Death Stranding Director’s Cut is now Xbox X|S at a huge discount#

Outlook will let users create custom account icons so they can tell their accounts apart easier#

Microsoft’s latest patent generates hyperreal virtual avatars based on users' faces#