🔮 Method 🔮
MoVieS consists of a shared image encoder, an attention-based feature backbone, and three heads that simultaneously predict 🎨appearance, 🧱geometry and 💨motion. Image shortcut for splatter head and time-varying Gaussian attributes are omitted for brevity. The image encoder, feature backbone and depth head are initialized from a geometrically pretrained transformer, VGGT. Motion head is initialized from its point head. Remaining components, such as the splatter head and camera/time embeddings, are trained from scratch.
Given \(t_q\) target timesteps, the motion head is conditioned via adaptive layer normalization (AdaLN) and predicts 3D movements for each input pixel in a canonical space. After rasterization using the \(M\) corresponding query-time cameras, output images in shape \(M\times 3\times H\times W\) are rendered for supervision. Gaussian attribute deformation \(\Delta\mathbf{a}\) is omitted for brevity.
Given 3D point tracking datasets (e.g., PointOdyssey, DynamicReplica and Stereo4D), ground-truth motion \(\Delta\mathbf{x}\) is defined as the 3D displacement of each tracked point between any two frames in the world coordinate system. Two kinds of complementary losses are applied for motion supervision (1) a point-wise L1 loss and (2) a distribution-level loss: \[ \mathcal{L}_{\text{motion}} = \lambda_{\text{pt}}\mathcal{L}_{\text{pt}} + \lambda_{\text{dist}}\mathcal{L}_{\text{dist}}\\ = \frac{1}{P}\sum_{i\in\Omega}\lambda_{\text{pt}}\|\Delta\hat{\mathbf{x}}_i-\Delta\mathbf{x}_i\|_1 + \frac{1}{P^2}\sum_{(i,j)\in\Omega\times\Omega}\lambda_{\text{dist}}\|\Delta\hat{\mathbf{x}}_i\cdot\Delta\hat{\mathbf{x}}_j^\top - \Delta\mathbf{x}_i\cdot\Delta\mathbf{x}_j^\top\|_1, \] where \(\Omega\) denotes the set of all valid \(P\) tracked points, and \(\Omega \times \Omega\) is its Cartesian product.
