Improve frustum culling of skinned meshes through per-joint bounds

[greeble-dev]

Objective

Mostly fix #4971 by adding a new option for updating skinned mesh Aabb components from joint transforms.

skinned_bounds_pr.mp4

This fixes cases where vertex positions are only modified through skinning. It doesn't fix other cases like morph targets and vertex shaders.

The PR kind of upstreams bevy_mod_skinned_aabb, but with some changes to make it simpler and more reliable.

Why A Draft?

I'm fishing for design feedback and any other concerns before I finish up the PR. The current state is functional and mostly tested. The remaining work is documentation, error handling, final testing and release notes. There also needs to be a decision on whether the new option is enabled by default in the glTF importer.

Background

If a main world entity has a Mesh3d component then it's automatically assigned an Aabb component. This is done by bevy_camera or bevy_gltf. The Aabb is used by bevy_camera for frustum culling. It can also be used by bevy_picking as an optimization, and by third party crates.

But there's a problem - the Aabb can be wrong if something changes the mesh's vertex positions after the Aabb is calculated. The most common culprits are skinning or morph targets. Vertex positions can also be changed by mutating the Mesh asset (#4294), or by vertex shaders.

The most common solution has been to disable frustum culling via the NoFrustumCulling component. This is simple, and might even be the most efficient approach for apps where meshes tend to stay on-screen. But it's annoying to implement, bad for apps where meshes are often off-screen, and it only fixes frustum culling - it doesn't help other systems that use the Aabb.

Solution

This PR adds a reliable and reasonably efficient option for updating the Aabb of a skinned mesh from its animated joint transforms. See the "How does it work" section for more detail.

The glTF loader can add skinned bounds automatically if a new GltfSkinnedMeshBoundsPolicy setting is enabled in GltfPlugin or GltfLoaderSettings:

app.add_plugins(DefaultPlugins.set(GltfPlugin {
    skinned_mesh_bounds_policy: GltfSkinnedMeshBoundsPolicy::Dynamic,
    ..default()
}))

For non-glTF cases, the user can create a SkinnedMeshBoundsAsset from a Mesh and bind it to an entity with the SkinnedMeshBounds component.

let asset = assets.add(SkinnedMeshBoundsAsset::from_mesh(&mesh)?);
entity.insert(SkinnedMeshBounds(asset));

See the custom_skinned_mesh example for real code.

I'm unsure if the SkinnedMeshBounds name is right. It's not "the bounds of the skinned mesh" - that's the Aabb. It's the things that contribute to the skinned mesh bounds.

Bonus Features

GltfSkinnedMeshBoundsPolicy::NoFrustumCulling

This is a convenience for users who prefer the NoFrustumCulling workaround, but want to avoid the hassle of adding it after a glTF scene has been spawned.

app.add_plugins(DefaultPlugins.set(GltfPlugin {
    skinned_mesh_bounds_policy: GltfSkinnedMeshBoundsPolicy::NoFrustumCulling,
    ..default()
}))

Note that PR #21787 is doing something similar, or maybe adding a different option. I'm fine if that PR lands first and this PR is adjusted to fit.

Gizmos

bevy_gizmos::SkinnedMeshBoundsGizmoPlugin can draw the per-joint AABBs.

fn toggle_skinned_mesh_bounds(mut config: ResMut<GizmoConfigStore>) {
    config.config_mut::<SkinnedMeshBoundsGizmoConfigGroup>().1.draw_all ^= true;
}

Like with SkinnedMeshBounds, I'm unsure if the name is right. It's not technically drawing the bounds of the skinned mesh - it's drawing the per-joint bounds that contribute to the bounds of the skinned mesh.

Testing

# Press `B` to show mesh bounds, 'J' to show joint bounds.
cargo run --example scene_viewer --features "free_camera" -- "assets/models/animated/Fox.glb"
cargo run --example scene_viewer --features "free_camera" -- "assets/models/SimpleSkin/SimpleSkin.gltf"

# Press `B` to show mesh bounds.
cargo run --example custom_skinned_mesh

# More complicated mesh downloaded from https://github.com/KhronosGroup/glTF-Sample-Assets/tree/main/Models/RecursiveSkeletons
cargo run --example scene_viewer --features "free_camera" -- "RecursiveSkeletons.glb"

I also hacked custom_skinned_mesh to simulate awkward cases like rotated and off-screen entities.

How Does It Work?

Click to expand

Summary

When created from a Mesh, the SkinnedBoundsMeshAsset calculates an AABB for each joint - the AABB encloses all the vertices skinned to that joint. Then every frame, bevy_camera::update_skinned_mesh_bounds uses the current joint transforms to calculate an Aabb that encloses all the joint AABBs.

This approach is reliable, in that the final Aabb will always enclose the skinned vertices. But it can be larger than necessary. In practice it's tight enough to be useful, and rarely more than 50% bigger.

This approach works even with non-rigid transforms and soft skinning. If there's any doubt then I can add more detail.

Awkward Bits

There's a few issues that stop the solution being as simple and efficient as it could be.

Problem 1: Joint transforms are world-space, Aabb is entity-space.

• Ideally we'd use the world-space joint transforms to calculate a world-space Aabb, but that's not possible.
• The obvious solution is to transform the joints to entity-space, so the Aabb is directly calculated in entity-space.
  • But that means an extra matrix multiply per joint.
• This PR calculates the Aabb in world-space and then transforms it to entity-space.
  • That avoids a matrix multiply per-joint, but can increase the size of the Aabb.

Problem 2: Joint AABBs are in a surprising(?) space.

• When creating joint AABBs from a mesh, the intuitive solution would be to calculate them in joint-space.
  • Then the update just has to transform them by the world-space joint transform.
• But to calculate them in joint-space we need both the bind pose vertex positions and the bind pose joint transforms.
  • Those are in separate assets - Mesh and SkinnedMeshInverseBindposes - and those assets can be mixed and matched.
  • So we'd need to calculate a SkinnedMeshBoundsAsset for each combination of Mesh and SkinnedMeshInverseBindposes.
  • (bevy_mod_skinned_aabb uses this approach - it's slow and fragile.)
• This PR calculates joint AABBs in mesh-space (or more strictly speaking: bind pose space).
  • That can be done with just the Mesh asset, so for each Mesh there's a matching SkinnedMeshBoundsAsset.
• One downside is that the update needs an extra matrix multiply so we can go from mesh-space to world-space.
  • However, this might become a performance advantage if frustum culling changes - see the "Future Options" section.
• Another minor downside is that mesh-space AABBs (red in the screenshot below) tend to be bigger than joint-space AABBs (green), since joints with one long axis might be at an awkward angle in mesh-space.

Pseudo-Code

Here's a pseudo-code version of the update so we can compare against other options.

let mesh_entity = ...;
let inverse_bindposes = ...;
let skinned_mesh_bounds = ...;
let skinned_mesh = ...;

let accumulator = AabbAccumulator::new();

for (joint_index, modelspace_joint_aabb) in skinned_mesh_bounds {
    let joint_entity = skinned_mesh.joints[joint_index];
    let world_from_joint = entities.get(joint_entity).get::<Transform>();
    let joint_from_mesh = inverse_bindposes[joint_index];
    let world_from_mesh = world_from_joint * joint_from_mesh;
    let worldspace_joint_aabb == transform_aabb(modelspace_joint_aabb, world_from_mesh);
    accumulator.add(worldspace_joint_aabb);
}
    
let worldspace_mesh_aabb = accumulator.finish();
let entity_from_world = mesh_entity.get::<Transform>().inverse();
let entityspace_mesh_aabb = transform_aabb(worldspace_mesh_aabb, entity_from_world);

*mesh_entity.get_mut::<Aabb>() = entityspace_joint_aabb;

Future Options

For frustum culling there's a cheeky way to optimize and simplify skinned bounds - put frustum culling in the renderer and calculate a world-space AABB during extract_skins. It's a good fit because skin extraction has already grabbed the joint transform and calculated world_from_mesh. I estimate this would make skinned bounds 3x faster.

let skinned_mesh_bounds = ...;
let joint_transforms = ...;

let accumulator = AabbAccumulator::new();

for (joint_index, modelspace_joint_aabb) in skinned_mesh_bounds {
    let world_from_mesh = joint_transforms[joint_index];
    let worldspace_joint_aabb = transform_aabb(modelspace_joint_aabb, world_from_mesh);
    accumulator.add(worldspace_joint_aabb);
}

let worldspace_mesh_aabb = accumulator.finish();

(Note that putting frustum culling into the renderer doesn't mean removing the current main world visibility system - it just means the main world system would be separate opt-in system)

Another option is to change main world frustum culling to use a world-space AABB. So there would be a new GlobalAabb component that gets updated each frame from Aabb and the entity transform (which is basically the same as transform propagation and the relationship between Transform and GlobalTransform). This has some advantages and disadvantages but I won't get into them here - I think putting frustum culling into the renderer is a better option.

Performance

Click to expand

Initialization

Creating the skinned bounds asset for Fox.glb (576 verts, 22 skinned joints) takes 0.03ms. Loading the whole glTF takes 8.7ms, so this is a <1% increase. The relative increase could be higher for glTFs that need less processing (e.g. pre-compressed textures).

Per-Frame

The many_foxes example has 1000 skinned meshes, each with 22 skinned joints. Updating the skinned bounds take 0.086ms. This is a throughput of roughly 250,000 joints per millisecond, using two threads.

The whole animation update takes 3.67ms (where "animation update" = advancing players + graph evaluation + transform propagation). So we can kinda sorta claim that this PR increases the cost of skinned animation by roughly 3%. But that's very hand-wavey and situation dependent.

This was tested on an AMD Ryzen 7900 but with TaskPoolOptions::with_num_threads(6) to simulate a lower spec CPU. Comparing against a few other threading options:

• Non-parallel: 0.141ms.
• 6 threads (2 compute threads): 0.086ms.
• 24 threads (15 compute threads): 0.051ms.

So the parallel iterator is better but quickly hits diminishing returns as the number of threads increases.

Future Options

The "How Does It Work" section mentions moving skinned mesh bounds into the renderer's skin extraction. Based on some microbenchmarks, I estimate this would reduce non-parallel many_foxes on my CPU from 0.141ms to 0.049ms, so roughly 3x faster. Requiring AVX2 (to enable broadcast loads) or pre-splatting (to fake broadcast loads for SSE) would knock off another 25%. And fancier SIMD approaches could do better again.

There's also approaches that trade reliability for performance. For character rigs, an effective optimization is to fold face and finger joints into a single bound on the head and hand joints. This can reduce the number of joints required by 50-80%. Another option is to use bounding spheres instead of AABBs.

FAQ

Click to expand

Should the new option be enabled by default in the glTF importer?

I think so. Bugs caused by skinned mesh culling have been a regular pain for both new and experienced users. I'm guessing most people would pay the CPU cost to have skinned meshes Just Work(tm). Sophisticated users in search of performance can choose to disable the option and use a fixed AABB.

Why is the automatic option tied to the glTF importer? Why do custom meshes have to be done manually? Shouldn't this be automatic for any mesh?

bevy_mod_skinned_aabb took the automatic approach, and I don't think the outcome was good. It needs some surprisingly fiddly and fragile logic to decide when an entity has the right combination of assets in the right loaded state. And its broken by RenderAssetUsages::RENDER_WORLD.

So this PR takes a more modest and manual approach. I think there's plenty of scope to make things more general and automated in future, particularly as the asset pipeline matures. So if the glTF importer becomes a purer glTF -> BSN transform, then adding skinned bounds could be a general scene transform or asset transform that's shared with other importers and custom mesh generators.

Why is there a separate SkinnedMeshBoundsAsset? Couldn't that data go in the Mesh or SkinnedMeshInverseBindposes assets?

The data can't go in SkinnedMeshInverseBindposes - it has to match the Mesh it's used with, and a single SkinnedMeshInverseBindposes can be shared between multiple Mesh assets (see the "Awkward Bits" section above for more background).

The first version of this PR did put the data in Mesh. But a separate asset plus component has two advantages:

1. It works if the Mesh is RenderAssetUsages::RENDER_WORLD.
2. update_skinned_mesh_bounds can efficiently skip any skinned meshes without skinned bounds.

The situation could change. Maybe frustum culling moves to the render world and can safely use data from Mesh - although that could be a problem if some other system still wants skinned bounds in the main world. Another path is for RenderAssetUsages to be redesigned, so the renderer and main worlds can negotiate which parts they need.

Why SkinnedMeshBounds? Shouldn't it be SkinnedMeshAabbs?

Playing it safe in case things change, e.g. bounding spheres might become an an option.

Why is the bounds update in bevy_camera? Shouldn't it be in bevy_mesh?

This gets tricky because the behavior depends on who wants to use Aabb components. Maybe an app uses bevy_mesh but doesn't use bevy_camera, so updating the Aabb would be redundant. Or maybe the app doesn't use bevy_camera, but does use bevy_picking. Or maybe it uses bevy_picking but not the mesh picking parts. Etc etc. Plugins are hard!

Putting the update in bevy_camera seems like the path of least resistance for now. Maybe that changes if plugins can negotiate features.

What Do Other Engines Do?

Click to expand

• Unreal: Automatically uses collision shapes attached to joints, which is similar to this PR but fragile and inefficient. Also supports various fixed bounds options.
• Unity: Fixed bounds attached to the root bone. Automatically calculated from animation poses or specified manually (documentation).
• Godot: Unclear. There's an issue that suggests they only support fixed bounds (AABB broken on imported meshes with skeletons. godotengine/godot#85171), but also code in the engine related to per-joint AABBs (MeshStorage::mesh_get_aabb). I don't know what they're using for frustum culling.
• O3DE: Fixed bounds attached to root bone, plus option to approximate the AABB from joint origins and a fudge factor.

An approach that's been proposed several times for Bevy is copying Unity's "fixed AABB from animation poses". I think this is more complicated and less reliable than many people expect. More complicated because linking animations to meshes can often be difficult. Less reliable because it doesn't account for ragdolls and procedural animation. But it could still be viable for for simple cases like a single self-contained glTF with basic animation.

[janhohenheim]

I very much like the approach presented here and think it's a very reasonable default.

The policy enum should be expanded to include @pcwalton's approach: don't generate any AABB, but keep frustum culling, since all AABBs will be artist-authored and inserted manually

[pcwalton]

Oh nice, I just filed #21845 for the rest pose approach.

[greeble-dev]

The policy enum should be expanded to include @pcwalton's approach: don't generate any AABB, but keep frustum culling, since all AABBs will be artist-authored and inserted manually

I'd like to consider some different approaches:

enum GltfSkinnedMeshBoundsPolicy {
    ...
    /// Skinned meshes are assigned this `Aabb` component.
    Fixed { aabb: Aabb },
    /// Skinned meshes are assigned a `SkinnedMeshBounds` component, which will
    /// be used by the `bevy_camera` plugin to update the `Aabb` component.
    ///
    /// The `Aabb` component will be updated to enclose the given AABB relative to the named joint.
    /// If the joint name is `None`, the AABB is relative to the root joint.
    FixedJointRelative { aabb: Aabb3d, joint_name: Option<String> },
}

So Fixed is similar to Unreal's fixed options. FixedJointRelative is basically Unity's approach but more flexible (can be any joint, not just the root). Unfortunately this needs some changes to SkinnedMeshBounds to support bounds being in joint-space.

I'm not sure if I'll try adding these options in this PR as it's already a bit chonky.