Bug: In redisvl FT.CREATE fails when using INDEXMISSING, SORTABLE, and UNF together

[nkanu17]

Summary

TextField with index_missing=True, sortable=True, and unf=True fails at index creation due to incorrect field option ordering
When a RedisVL TextField is configured with all three of:

• index_missing=True
• sortable=True
• unf=True

the generated FT.CREATE field modifiers are ordered incorrectly:

• Generated: TEXT ... SORTABLE UNF INDEXMISSING → Redis rejects this.
• Required: TEXT ... INDEXMISSING SORTABLE UNF → Redis accepts this.

RediSearch’s parser currently requires INDEXEMPTY and INDEXMISSING to appear before SORTABLE/UNF in the field definition (see redis/redis#5177). RedisVL’s TextField.as_redis_field() does not enforce that ordering, causing FT.CREATE to fail with:

• ResponseError: Field 'INDEXMISSING' does not have a type.

A similar issue may affect other field types that support index_missing / index_empty and sortable / unf (e.g., NumericField, possibly TagField / GeoField).

---

Current Behavior

• TextField.as_redis_field() (in redisvl/schema/fields.py) assembles kwargs for redis-py’s TextField, including:
  • sortable (bool)
  • index_missing (bool)
  • index_empty (bool)
  • no_index (bool)
• After constructing the redis-py TextField, it then mutates field.args_suffix to insert UNF relative to SORTABLE.
• With index_missing=True, sortable=True, unf=True, the resulting suffix is:
  • ["SORTABLE", "UNF", "INDEXMISSING"].
• This produces an FT.CREATE field definition equivalent to:
  • ... TEXT SORTABLE UNF INDEXMISSING,
    which RediSearch parses as if INDEXMISSING were a field name, hence:
  • ResponseError: Field 'INDEXMISSING' does not have a type.
• There is a small script in the repo root (test_unf_sortable.py) that demonstrates this; its final test case fails with precisely this error and prints the problematic suffix ordering.

---

Expected Behavior

• A TextField with index_missing=True, sortable=True, and unf=True should generate field modifiers in an order acceptable to RediSearch, e.g.:
  • TEXT ... INDEXMISSING SORTABLE UNF.
• More generally, the canonical ordering should be:
  • [INDEXEMPTY] [INDEXMISSING] [SORTABLE [UNF]] [NOINDEX].

So that:

• INDEXEMPTY and INDEXMISSING always precede SORTABLE / UNF.
• UNF is only present if the field is sortable.
• NOINDEX appears at the end.

---

Reproduction Steps

1. Configure a TextField in RedisVL with:

   • sortable=True
   • unf=True
   • index_missing=True
     either directly or via an IndexSchema definition, for example (conceptually):
   • Text field: work_experience_summary
   • Attributes: { "sortable": True, "unf": True, "index_missing": True }.

2. Inspect the generated redis-py field arguments:

   • Call field = TextField(...).as_redis_field().
   • Inspect field.redis_args() or field.args_suffix.

3. Observed output (from test_unf_sortable.py):

   • Base args: ["TEXT", "WEIGHT", 1].
   • Suffix: ["SORTABLE", "UNF", "INDEXMISSING"].

4. Use this field in an index and call SearchIndex.create() against a Redis instance with RediSearch ≥ 2.10 (e.g., Redis Stack 7.4.0-v1).

5. Actual result:

   • FT.CREATE fails with:
     • ResponseError: Field 'INDEXMISSING' does not have a type.

6. Manually modifying the command to reorder the modifiers as:

   • TEXT INDEXMISSING SORTABLE UNF
     produces a successful FT.CREATE, confirming that the problem is purely the option ordering.

---

Actual Behavior

• RedisVL currently lets redis-py assemble args_suffix such that:
  • SORTABLE and UNF come before INDEXMISSING.
• RediSearch’s parser expects INDEXEMPTY / INDEXMISSING before SORTABLE and treats INDEXMISSING at the end as if it were a field name rather than an option.
• Index creation fails whenever index_missing=True, sortable=True, and unf=True are combined on the same TextField.

---

Root Cause

• RediSearch parser limitation / bug (see AOF file corrupted after a power loss aof-load-truncated yes fails too redis#5177):
  • INDEXEMPTY and INDEXMISSING must appear before SORTABLE in the FT.CREATE field definition.
  • Certain permutations of options that are “semantically OK” are rejected because of this ordering constraint.
• RedisVL’s TextField.as_redis_field():
  • Enables index_missing / index_empty via kwargs.
  • Then injects UNF by inserting it relative to SORTABLE, without re-normalizing the full list of modifiers to satisfy RediSearch’s required order.
• Similar logic exists in NumericField.as_redis_field() and may require the same normalization.

---

Proposed Solution

1. Normalize args_suffix ordering in as_redis_field()

   • For TextField:
     • After constructing the underlying redis-py TextField, collect any of:
       • INDEXEMPTY, INDEXMISSING, SORTABLE, UNF, NOINDEX.
     • Remove them from field.args_suffix, remember their presence, and then rebuild the suffix in a canonical order:
       • Other tokens (if any, not in this set), followed by:
       • INDEXEMPTY (if present),
       • INDEXMISSING (if present),
       • SORTABLE (if present),
       • UNF (if sortable and requested),
       • NOINDEX (if present).
   • Apply similar logic to NumericField.as_redis_field() (and any other field types that can combine these options).

2. Respect UNF constraints

   • Ensure UNF is only emitted if sortable=True.
   • If unf=True but sortable=False, drop UNF or raise a validation error.

3. Testing

   • Unit tests for TextField (and NumericField where applicable):
     • Validate that for combinations like:
       • index_missing=True, sortable=True, unf=True.
       • index_empty=True, sortable=True, unf=True.
       • Both index_empty=True and index_missing=True with sortable=True, unf=True.
         the resulting redis_args() put INDEXEMPTY/INDEXMISSING before SORTABLE / UNF.
   • Integration tests with Redis:
     • Create indices using these combinations via RedisVL.
     • Assert that SearchIndex.create() completes successfully without FT.CREATE parse errors.

4. Optional: centralize ordering rules

   • To avoid duplication, consider a small helper function for normalizing field modifier order that can be reused across field types (TextField, NumericField, etc.).