Kuaishou, Tsinghua Boost Video AI Consistency with Geometry-Aware Memory
Researchers from Kuaishou's Kling Team and Tsinghua University have developed GIM-World, a framework that uses geometry-aware implicit memory to enhance visual consistency in video world models. The system distills 3D scene structure into compact memory tokens, allowing for efficient, persistent environment simulation without high inference costs associated with explicit 3D reconstruction. This technical development aims to improve long-horizon consistency in generated video environments by integrating geometric understanding into the memory state.
Key Takeaways
- GIM-World compresses variable-length video history into fixed-size memory tokens, improving long-horizon consistency.
- A camera-queryable geometry head distills 3D scene structure into the memory during training, using a frozen foundation model as a teacher.
- The system prunes redundant historical frames to manage encoding costs, using an information-guided method.
- During inference, the geometry head and 3D teacher are discarded, resulting in a lightweight memory module that runs in less than 0.3% of the diffusion backbone's time.
- Experiments on MIND datasets demonstrated GIM-World's improved memory consistency, action control, and 3D geometric consistency over existing baselines.
Why It Matters
Enhancing long-horizon consistency in video world models addresses a key technical hurdle for advanced AI video generation. By encoding 3D scene geometry directly into implicit memory, Kuaishou and Tsinghua demonstrate a method to maintain visual accuracy without the computational burden of constant 3D reconstruction. This development sets a precedent for more stable and realistic AI-generated virtual environments, impacting applications from embodied AI agents to interactive simulations and gaming. Watch for increased adoption of geometry-aware memory techniques as large language and video models continue to advance towards more persistent and interactive content creation.
Read full article at arxiv.org