[FEATURE] FT.INFO returned information are not aligned with returned result of RedisSearch

[hwware]

Some of our clients could call API from Mem0 library (https://github.com/mem0ai). It provides some interfaces with RedisSearch (https://github.com/mem0ai/mem0/blob/main/mem0/vector_stores/redis.py). When I play some commands of the ValkeySearch, I noticed that the information returned by FT.INFO is quite different between RedisSearch and ValkeySearch.
Following, I will list four cases to indicate the difference.

1. Some fields are missing from the returned information by FT.INFO from ValkeySearch
   Test script：
   FT.CREATE idx1 ON HASH PREFIX 1 blog:post: SCHEMA sku AS sku_tag TAG

Following is part of the returned information from RedisSearch

You can see, the field indexes_all only appear in the returned result of RedisSearch
PR #306 fix this issue

2. Some fields only appear in the returned information by FT.INFO from ValkeySearch
   Test script：
   FT.CREATE idx1 ON HASH PREFIX 1 blog:post: SCHEMA sku AS sku_tag TAG

Following is part of the returned information from ValkeySearch

You can see, the size, one of the subfields of attributes, and backfill_in_progress etc only appear in the ValkeySearch returned information. This is not a bug, they are extra meta data for ValkeySearch

3. Some fields appear in the ValkeySearch and RedisSearch, but the format is not matched
   Test script：
   FT.CREATE idx3 SCHEMA my_hash_field_key VECTOR HNSW 10 TYPE FLOAT32 DIM 20 DISTANCE_METRIC COSINE M 4 EF_CONSTRUCTION 100

Following is part of the returned information from RedisSearch

Following is part of the returned information from ValkeySearch

You can notice, for the attribute part, the format of ValkeySearch is different from RedisSearch

PR #312 fix this issue

4. Some fields appear in the ValkeySearch and RedisSearch, but the value or type is not matched
   Test script：
   JSON.SET doc:1 $ '[{"arr": [1, 2, 3]}, {"val": "hello"}, {"val": "world"}]'
   FT.CREATE idx2 ON JSON PREFIX 1 doc: SCHEMA $..arr AS arr NUMERIC

Following is part of the returned information from RedisSearch

Following is part of the returned information from ValkeySearch

You can see, the fields: num_docs, num_terms, and num_records are different type or value

PR #319 to fix case 4 issue

I think we should at least align with the returned information by FT.INFO from RedisSearch. Later, I will focus on this part and create PRs if there is no argue with this. @PingXie @madolson @yairgott

[madolson]

I think it's fine to have extra metadata, but it is odd that we have a lot of the same information just in a different format. I think it would make sense to align it as much as possible. It is a breaking change though, so maybe something to release as a 2.0 along with Valkey 9.0 ?

[allenss-amazon]

Looking at the code.

1. num_terms is hard-coded at zero -- yes a string 0 not a number 0. Since we don't support text yet, this makes sense.
2. num_records and num_docs are correctly collected, but reported as a string instead of a number.

IMO, we should treat it as a bug and change the data types in a patch release.

[allenss-amazon]

We have enough serious bugs to justify a patch release. So, I'd vote to just fix this in the next patch release.

[hwware]

I think it's fine to have extra metadata, but it is odd that we have a lot of the same information just in a different format. I think it would make sense to align it as much as possible. It is a breaking change though, so maybe something to release as a 2.0 along with Valkey 9.0 ?

IMO, we should append the extra metadata in the end after aligning with RedisSearch metadata, because client could parse it by its index or string.

[madolson]

@yairgott Are you okay with the proposed breaking changes? I'm a little bit more worried for you guys since you likely have more production usage.