WorldSculpt splits crowded 3D scenes into hundreds of editable meshes

Why it matters

Editable meshes turn convincing 3D captures into assets that game engines, simulators and robots can manipulate. WorldSculpt advances that conversion, though its structured inputs keep it well short of one-tap phone scanning.

Muyao Niu, project lead Zhixiang Wang (@lightchaserx), and Alaya Lab director Kaipeng Zhang released WorldSculpt, a research system that rebuilds crowded multi-view scenes as collections of separate 3D meshes instead of one fused representation.

Wildmind AI on X

The distinction matters once someone wants to change the reconstructed world. WorldSculpt's output treats a chair, table, lamp or smaller object as its own mesh, allowing each asset to be selected, moved or edited independently while retaining its position in the larger scene.

Niu, the paper's first author, is a second-year PhD student in mechano-informatics at the University of Tokyo. His research spans generative 3D modeling, neural radiance fields, Gaussian splatting and computational photography. Wang, who identifies himself as a team leader building Alaya Lab with Zhang, previously worked as a research scientist at CyberAgent AI Lab after completing his University of Tokyo PhD in 2024. Zhang started leading Alaya Lab after research roles at Shanghai AI Lab and SenseTime, focusing the lab on world models and game-engineering systems.

That background helps explain WorldSculpt's emphasis on compositional output. Alaya Lab's site groups its research under game creation and world-model projects, including agents that build playable games and systems for playable worlds. A navigable 3D scene can look convincing as a video or Gaussian splat, but game engines, simulations and robotics systems often need discrete objects with geometry that software can address.

The phone-video claim needs a footnote

The post that brought WorldSculpt wider attention described it as turning cluttered phone video into editable objects. WorldSculpt's own materials describe a more structured pipeline.

The model receives RGB images along with camera poses, instance-level masks and coarse 3D bounding boxes. Those additional inputs tell WorldSculpt where the cameras were, which pixels belong to each object and roughly where each object sits in the scene. The method is therefore closer to reconstruction from grounded, posed multi-view imagery than one-tap processing of an arbitrary handheld clip.

That preparation does not erase the technical contribution. Occlusion is a central problem in crowded scenes: a camera may see a cabinet only behind a sofa, or capture a small object in a few partial glimpses. Traditional geometry pipelines can preserve the visible surfaces while leaving hidden sections incomplete. WorldSculpt instead uses a generative prior to propose a complete object mesh from the evidence available across views.

The hidden portions remain model-generated estimates. A complete back side inferred from partial observations should not be mistaken for measured geometry. The value is that WorldSculpt produces a usable, self-contained object rather than a fragment attached to one monolithic reconstruction.

How WorldSculpt assembles the parts

For each detected object, WorldSculpt chooses an anchor image where the object is most visible and creates a canonical 3D cube around it. Cropped and masked observations from the other camera views are mapped into that shared object space.

WorldSculpt builds on Pixal3D, a generative model designed for individual 3D objects. The researchers added a multi-view conditioning path that lifts DINOv3 image features into a voxel grid and combines evidence from multiple views. The adapted model generates a complete mesh in canonical coordinates, then applies a rigid transformation to place that object back into the scene's world coordinates.

Processing objects independently gives the method its compositional structure. The pipeline does not need to fuse all surfaces into one scene-level mesh after generation. It can also scale the same object model across scenes containing hundreds of items, even though the model was fine-tuned on individual objects rather than complete crowded environments.

The authors tested the method with between one and 16 views and synthetic occlusion levels reaching 75%. The WorldSculpt paper was submitted to arXiv on September 4, 2026, and revised on September 7. Its authors report that the system outperformed the ShapeR multi-view baseline across the metrics used for their most complex benchmark. Those results are the research team's evaluation and have not been independently reproduced in the materials reviewed here.

A benchmark built for clutter

The project also introduces UE-MeshyScene, a benchmark containing six photorealistic environments rendered in Unreal Engine 5.8 at 2560 x 1440 resolution. The scenes include as many as several hundred assets, along with camera poses, masks, 3D boxes, depth and reference object geometry.

WorldSculpt's learned view aggregator produced its largest reported gains in that dense setting. On UE-MeshyScene, the authors say it reduced one Chamfer-distance metric by 12% compared with averaging features across views, while raising the associated F-score from 0.944 to 0.951. The gap was smaller in scenes where objects were already well observed, suggesting the added fusion machinery earns its keep when views are uneven or heavily blocked.

Alaya Lab also demonstrated WorldSculpt on rendered views from World Labs' Marble. In that workflow, a generated Gaussian-splatting world becomes a collection of object meshes that can be pulled apart and edited. The example uses known camera poses and depth derived from the source world, so it does not establish the same performance on uncontrolled consumer footage.

Research code before consumer product

The official WorldSculpt repository includes inference scripts and directs users to download a WorldSculpt checkpoint from the project's Hugging Face page, along with UE-MeshyScene and Marble data. It does not establish the release of training code or every model weight used to build the system.

The repository's installation instructions begin with a TRELLIS.2 environment, add further dependencies, download WorldSculpt and Pixal3D checkpoints and run inference through shell commands. That process places the release squarely in research-pipeline territory rather than consumer phone scanning.

Its immediate audience is likely to be teams already able to produce camera calibration, masks and spatial boxes, whether from synthetic environments, instrumented capture systems or upstream computer-vision models.

World models are getting better at producing scenes that people can watch and navigate. Niu, Wang and Zhang are working on the conversion step that makes those scenes behave like assets inside software. If upstream tracking and segmentation become reliable enough, WorldSculpt's object-by-object approach could make captured or generated worlds considerably easier to reuse in games, simulation and robotics.