Map audio features to viseme weights for XR avatar mouth animations.
npx skills add https://github.com/google/xrblocks --skill lipsync-google
Please help me install this Agent Skill. Skill: lipsync Source: https://github.com/google/xrblocks/tree/main/src/addons/lipsync Command: npx skills add https://github.com/google/xrblocks --skill lipsync-google
This Skill provides real-time audio-driven mouth animations for XR avatars, allowing them to mouth along with spoken audio, creating a more immersive and expressive user experience.
Import the LipsyncMouth script and pass it a MediaStream and a target face. For example:
const face = xb.StylizedFace({showEyes: false});
puppetHead.add(face);
const driver = new LipsyncMouth(myMicStream, {target: face});
puppetHead.add(driver);
High-intent search queries and answers about installing and using this skill.
You can animate XR avatar mouths in real-time by mapping audio features to viseme weights. This Skill maps audio input to mouth shapes without using machine learning models, relying on a formant-based heuristic.
Avatar lipsync does not require machine learning models when using a formant-based heuristic. This Skill analyzes audio formants directly to determine viseme weights for driving the mouth shape of an XR face.
To sync avatar mouth movements across multiplayer XR sessions, this Skill integrates with the Netblocks platform. This ensures that characters' mouths move in sync with spoken words for all connected users.
Driving mouth animations with audio input requires an active MediaStream and a target face object that implements a setVisemes method. The Skill is designed for XR applications using the xrblocks framework.
Using formant analysis for viseme mapping limits you to heuristic-based audio feature detection rather than nuanced phoneme recognition. This approach is tailored for xrblocks faces and requires a consistent audio MediaStream input to function.
hiyenwongAnalyze synaptic weight distributions to infer plasticity geometry from neuroscience datasets.
diadia0000Predict and reconcile player movements with server data in multiplayer games.
huiferTrack vision records, intraocular pressure, and eye disease screening results.
GroundMountCompanyGenerate AI talking head videos with lip-sync using multiple model backends.