LatentSync
ByteDance's open-source lip sync — auto-adjusts character mouth shapes from audio
ByteDance's open-source lip synchronization that automatically adjusts character mouth shapes from audio.
AI video generation, digital humans, and video restoration.
Key features
- End-to-end audio-conditioned latent diffusion lip-sync
- No intermediate motion representation; Whisper audio embeddings via cross-attention
- Version 1.6 trained at 512x512 to reduce blurriness
- Temporal layer for consistency; improved performance on Chinese videos
- Gradio app and CLI inference; 8GB minimum VRAM on 1.5
Good for
- Syncing lip movements of talking-head videos to new audio
- Dubbing videos into other languages
Stack & tags
PythonPyTorchGradiolip-syncvideo-editingdiffusionopen-weightstalking-headwhispergradio
Quick start
python gradio_app.pyFrom the project README on GitHub.
Similar open-source projects
Hyperframes
★ 55,468Write HTML. Render video. Built for agents — HeyGen's open-source video framework
Hypit
★ 18,649Clone any viral video with AI agents — swap the face, the words, the B-roll, ship 100 variants