Overview

Can a generated 3D world become more than a visual environment? Can it be sensed, interpreted, and acted upon directly?

This exploration began with a series of experiments in Unity for querying surfaces inside Gaussian splat environments. Instead of converting the splats into conventional geometry, I used their position, scale, density, and opacity to estimate where a ray encounters something solid.

The first tests involved simulated LiDAR rays emitted from a drone moving through a generated environment. Once this started producing reliable spatial information, I began considering what else could use the same system.

The result was a procedural hexapod robot capable of walking directly through a World Labs-generated splat world. Each leg probes the underlying data to locate footholds, evaluate the terrain, adapt the robot’s body, and traverse uneven surfaces without relying on traditional collision geometry.

The project is less about the robot itself than the question behind it: can a generated world be perceived and acted on directly through its native representation?

From Raycasting to Locomotion

Much of my work involves volumetric capture, point clouds, and Gaussian splats—representations that retain some of the density, ambiguity, and imperfections of captured reality.

I had already been exploring splat raycasting as a way of querying these environments more directly. At some point, it became clear that the same information could be used for support detection, foothold placement, climbing, and navigation.

A walking machine was particularly suited to the experiment. Legs can continuously adapt to small variations in the terrain, while wheels depend on smoother and more predictable surfaces. It also gave me an opportunity to explore a long-standing fascination with hexapods and machines such as the Tachikoma.

Reading the Splat Field

The robot moves inside a Marble-generated Gaussian splat environment loaded into Unity. There is no navigation mesh, baked collision setup, or hidden geometric floor underneath the scene.

A custom system decodes the position, scale, and opacity of the splats and organizes them into spatial chunks. During each query, entire regions that a ray cannot intersect are skipped, reducing the number of individual splats that need to be evaluated.

The remaining splats are treated as spatial volumes that can be tested for likely intersections. Their density and opacity help determine whether a detected area is substantial enough to act as a surface rather than visual noise.

Gaussian splats do not provide a clean surface normal in the way that a mesh triangle does. To estimate the orientation of the terrain, the system fires several nearby rays and compares the pattern of returned points. From this local sample, it can approximate slope, direction, stability, and surface continuity.

This makes the system fast enough to perform multiple probes for every leg, on every frame.

Procedural Locomotion

Each leg searches the surrounding splat field for a usable foothold in the robot’s direction of travel.

A potential step is evaluated according to several conditions:

• Whether enough splat density exists to support the foot

• How steep or irregular the detected surface appears

• Whether the required change in height is manageable

• Whether the position fits the current gait and body configuration

This information drives the robot’s gait timing, stepping arcs, movement speed, body height, and orientation. Analytic inverse kinematics are then used to solve the final position of each leg.

The locomotion system supports different numbers and arrangements of legs rather than being limited to one specific hexapod configuration.

For climbing and wall traversal, the robot is not permanently aligned to world gravity. Instead, its local sense of “down” can be reconstructed from nearby surfaces, allowing its body and feet to progressively reorient themselves around walls and steep structures.

Why Work Directly with Splats?

Marble can export mesh-based colliders alongside its generated environments. For most production applications, those colliders provide a more practical foundation for physics and navigation.

For this experiment, however, I wanted to intentionally skip that conversion and work as closely as possible with the original splat representation.

The visual coherence of Gaussian splats makes an environment immediately readable to a person. A floor looks like a floor, and a wall appears solid. But once the same environment is sampled computationally, it reveals small gaps, density changes, uneven areas, and uncertain boundaries.

Rather than behaving like a cleanly designed game level, the terrain feels closer to a scanned physical location, containing noise and imperfections that the locomotion system must continuously interpret.

An early version used a wheeled robot, but the wheels struggled with these small irregularities. Procedural legs proved far more capable of adapting to the data, reinforcing the idea that the representation itself can influence the kind of machine best suited to navigating it.

Marble as a World-Generation Tool

Marble made it possible to produce believable, spatially coherent environments and immediately begin testing them as places that could be probed and traversed.

Because the generated worlds are represented as Gaussian splats rather than clean polygonal levels, they retain a dense, lifelike quality. The robot is not moving through a deliberately simplified game environment; it is attempting to interpret something closer to a captured place.

This ambiguity is precisely what made the environments interesting for the experiment. The robot must continuously read the world rather than assuming that every visible surface is geometrically perfect.

Autonomous Exploration

The current system also includes an autonomous navigation mode. I can select a destination and allow the robot to travel toward it, adapting to terrain, avoiding obstacles, and climbing surfaces when necessary.

The next stage is to develop this into a more believable rover or exploration system by introducing additional simulated sensing:

• LiDAR-style environmental scanning

• Depth and proximity sensing

• Sonar-like spatial feedback

• Environmental mapping

• Telemetry and diagnostic data

• Delayed or incomplete sensor responses

The intention is to make the machine feel less like an immediately controlled digital character and more like a remote robot gradually constructing an understanding of an unfamiliar place.

The World Labs environment then becomes more than a static scene. It becomes a world in which a rover has just arrived—and must learn how to move through it.

Tools and Technology

• Marble / World Labs — Generated Gaussian splat environments

• Unity — Simulation, rendering, and procedural control

• UnityGaussianSplatting — Customized version of Aras Pranckevičius’s renderer

• Custom splat decoder — Position, scale, and opacity extraction

• Spatial acceleration system — Chunked ray queries and region rejection

• Procedural locomotion — Generalized gait and leg-count system

• Analytic inverse kinematics — Leg solving and adaptive body positioning

• Custom navigation system — Destination-based traversal and obstacle response

Credits

Research and development: Ruben Frosali / RubenFro
Studio: TRIDOT
Generated environments: Marble by World Labs
Gaussian splat rendering base: UnityGaussianSplatting by Aras Pranckevičius

Features

• World Labs Community Showcase [Splat Raycasting for Robot Locomotion]

• 80 Level [Artist Shows a Walking Robot Navigating a Gaussian Splat Environment]