Missing entries in searching with the new indexing

[etj]

Some records are not properly returned when using the new indexing procedure.

English: searching for

• "ken" -> 5 results
• "keny" -> 0 results
• "kenya" -> 5 results

Italian, Spanish: searching for

• "ken" -> 5 results
• "keny" -> 5 results
• "kenya" -> 5 results

French: searching for

• "ken" -> 5 results
• "keny" -> 5 results
• "kenya" -> 0 results

The GeoNode instance does not have any multilang field.

Settings:

>>> settings.LANGUAGES
(('de-de', 'Deutsch'), ('en-us', 'English'), ('es-es', 'Espanol'), ('fr-fr', 'Francais'), ('it-it', 'Italiano'))
>>> settings.LANGUAGE_CODE
'en'

Being 3762 the resource id of one of the returned resource, in the DB we have:

# select * from indexing_resourceindex where resource_id = 3762;
  id  | lang |      name      |                        vector                         | resource_id
------+------+----------------+-------------------------------------------------------+-------------
 6244 |      | title          | 'kenya':2 'kenya-map':1 'map':3                       |        3762
 6245 |      | title_abstract | 'kenya':2 'kenya-map':1 'map':3                       |        3762
 6246 |      | all            | 'inform':5 'kenya':2 'kenya-map':1 'map':3 'provid':6 |        3762
(3 rows)

[etj]

The index is created with the default language:

geonode/geonode/indexing/manager.py

Lines 55 to 56 in f5df740

	# this index is not localized
	pg_lang = multi.get_pg_language(multi.get_default_language())

but the query is created using the user currently selected language:

geonode/geonode/indexing/api/filters.py

Lines 48 to 59 in f5df740

	search_lang = request.query_params.get("search_lang", None)
	if search_lang:
	search_lang = validate_lang(search_lang)
	if not search_lang:
	search_lang = get_language_from_request(request)[:2]
	search_lang = validate_lang(search_lang)
	lang_filter = Q(lang__isnull=True)
	if search_lang:
	lang_filter = lang_filter | Q(lang=search_lang)
	pg_lang = get_pg_language(search_lang)

The tokenizer may not match.

We may want to use the same language tokenizer for both storing and querying.
The local default language may work, as the simple tokenizer.
Let's got for the local language.

[etj]

select to_tsquery('english'::regconfig, 'kenya:*')
--> 'kenya:*'

select to_tsquery('english'::regconfig, 'keny:*')
--> 'keni:*'

select to_tsquery('english'::regconfig, 'ken:*')
--> 'ken:*'

ChatGPT says:

---

This looks strange at first — why turn keny → keni?

But it’s not random:

✔️ Explanation

• With a wildcard (:*), PostgreSQL still applies stemming unless the word exactly matches the dictionary as-is.
• The English stemmer tries to find the morphological base.
• “keny” is not a valid English word, so the stemmer tries to reduce it.
• The Porter stemmer sometimes rewrites stems to normalize them — in this case, it produces "keni" as the stem.
  (You can see similar effects with words like “pony” → “poni”, “enemy” → “enemi”, etc.)

---

[etj]

Ok, the simple configuration is a better shot.
Let's keep the indexing language agnostic to avoid unexpected stemming.