Skip to content

Incorrect handling of stencil reference values in RenderState #2538

@theMinka

Description

@theMinka

I found that the stencil buffer mechanics do not handle stencil reference values correctly.

Most stencil buffer-related parameters are stored in the additional render state of the material. Among them are a pair of stencil reference parameters - one for front faces and one for back faces. These reference values serve two purposes:

  1. They are used as the comparison value in many stencil test functions.
  2. They are written directly into the stencil buffer when the StencilOperation.Replace operation is used.

In addition, there is a pair of stencil mask parameters, which provide simple bitmasks. These masks are applied to both the current value in the stencil buffer and the reference value before each stencil test.

The interaction between reference and mask values is documented here:
https://registry.khronos.org/OpenGL-Refpages/gl4/html/glStencilFuncSeparate.xhtml

In jME, there is currently a bug in the RenderState.copyMergedTo(...) method. The stencil reference value gets incorrectly overridden by the stencil mask value, causing the reference value to be ignored entirely.

Example:
In the TestStencilOutline example, changing the stencil reference value on the outline object from 1 to 0 should invert the colored area. The inverted area should only cover the interior of the boxes, ignoring the overlapping regions that normally create the outline. However, due to the aforementioned bug, changing the reference value has no effect—the outline is rendered exactly as before.

public OutlineControl(AssetManager assetManager, ColorRGBA colorRGBA) {
    ...
    outlineMaterial.getAdditionalRenderState().setFrontStencilReference(0);
    outlineMaterial.getAdditionalRenderState().setBackStencilReference(0);
    outlineMaterial.getAdditionalRenderState().setFrontStencilMask(0xFF);
    outlineMaterial.getAdditionalRenderState().setBackStencilMask(0xFF);
}

This bug was introduced by the changes made in #2325. I can provide a pull request to fix this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething that is supposed to work, but doesn't. More severe than a "defect".

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions