Ignore backfaces during Node3D selection

[vaner-org]

Closes #12550.

Adds an editor setting 3D > Ignore Backfaces on Selection that is true by default.
Adds an optional parameter and an if statement to TriangleMesh's intersect_ray function.

[YeldhamDev]

This should also be implemented for remote selection.

[vaner-org]

This should also be implemented for remote selection.

✔️ Done.

[Calinou]

Tested locally on https://github.com/Calinou/godot-reflection, it works as expected. Code looks good to me.

In the example below, the camera is stuck inside geometry.

Before

ignore_backfaces_before.mp4

After

ignore_backfaces_after.mp4

[KoBeWi]

Does it need to be a setting? Why would someone want to turn it off?

[Calinou]

Does it need to be a setting? Why would someone want to turn it off?

I can't see a reason to turn it off, so I think the setting should be removed too. We can introduce it in a future PR if there's a proven need for it.

[vaner-org]

I've removed the editor setting and hardcoded our two calls to intersect_ray to ignore backfaces.

[kleonc]

Note that the cull mode the given mesh is rendered with is not being accounted for anyhow (it's the same before/after this PR though).

So while this PR fixes the behavior for the CULL_BACK/back case from the tables below, it breaks CULL_FRONT/back and CULL_DISABLED/back cases.
If CULL_BACK/back case is more common/annoying than CULL_FRONT/back and CULL_DISABLED/back cases together, then this PR could be considered as an improvement even in it's current state. Whether this is actually the case I have no idea.

Without this PR (v4.5.stable.official [876b290]):

Cull mode	mesh faces looked at from	visible	selectable	correct
CULL_FRONT	front	✗	✓	❌
CULL_FRONT	back	✓	✓	✅
CULL_BACK	front	✓	✓	✅
CULL_BACK	back	✗	✓	❌
CULL_DISABLED	front	✓	✓	✅
CULL_DISABLED	back	✓	✓	✅

With this PR (v4.5.beta.gh-110185 [4d10a65]):

Cull mode	mesh faces looked at from	visible	selectable	correct
CULL_FRONT	front	✗	✓	❌
CULL_FRONT	back	✓	✗	❌
CULL_BACK	front	✓	✓	✅
CULL_BACK	back	✗	✗	✅
CULL_DISABLED	front	✓	✓	✅
CULL_DISABLED	back	✓	✗	❌

E.g. selection for double sided Sprite3D (by default double_sided = true; it uses CULL_DISABLED mode internally) gets broken with this PR as it makes the sprite selectable only from the front side.

Before:

Godot_v4.5-stable_win64_NrlaBoyJEj.mp4

After:

godot.windows.editor.x86_64_eufXVIyjSr.mp4

[ratkingsminion]

In our case the PR would be an improvement, we have levels full of 3D models (each wall, floor, ceiling is a mesh on its own) and none of them have front face culling. I think that will be the case for the majority of users, at least I can't imagine otherwise. We use Sprite3Ds mostly as billboards, and I guess the PR does not account for that either (it's already a problem in the current way of selecting).

[vaner-org]

Addressing @kleonc's concerns, I've added an extra commit on which I'd appreciate some feedback.
Assuming the precedence used in MeshInstance3D's get_active_material(), for each case, from highest to lowest priority in how the cull mode is inferred:

MeshInstance3D	Sprite3D	CSGPrimitive3D
GeometryInstance3D material override	Geometry3DInstance material_override	Geometry3DInstance material_override
MeshInstance3D surface (0) override	SpriteBase3D double_sided	CSGShape's specific get_material
MeshInstance3D mesh material

This means that the new parameter accepted by intersect_ray is now an enum, one of CULL_BACK, CULL_FRONT and CULL_DISABLED.

All of this can be tested in the scene here: backface_test.zip

Here's the problem:
It seriously hurts editor performance, adding a noticeable delay when selecting nodes. If anyone here is aware of a more performant way of doing this, please let me know.

Alternatively, I can revert this commit, retaining the bandaid fix for Sprite3D, and then branch off the remaining functionality into another draft PR, where an optimised way of doing this can be discussed, and the fix for Sprite3D can be replaced with something that can represent all mesh planes.

[Repiteo]

@Calinou @kleonc What's the call then? I don't think we should integrate something that hurts editor performance, so this is more about if we should allow the standalone version to exist with known issues or if a fully optimized implementation is necessary to start

[Calinou]

What's the call then? I don't think we should integrate something that hurts editor performance, so this is more about if we should allow the standalone version to exist with known issues or if a fully optimized implementation is necessary to start

Indeed, this will introduce usability issues in large projects even if they didn't run into any issues previously. In this sense, it would be a regression.

I think we should hold off on merging this until we find a more optimized approach. Maybe reusing the depth buffer is something we should do eventually...