Is replacing LLM-generated variations with a fixed mock list in map_query_semantic_space a mistake?

[zboyr]

Hi team,

I was reviewing the code for map_query_semantic_space in adaptive_crawler.py (lines 668–737), and I noticed that the original logic for generating query variations using an LLM (with perform_completion_with_backoff) has been commented out. Instead, a hardcoded set of mock queries is now being used:

variations = {'queries': [
    'what are the best vegetables to use in fried rice?',
    'how do I make vegetable fried rice from scratch?',
    ...
]}

Is this change intentional, or could it be an error? Using a static list of queries seems to defeat the purpose of generating diverse semantic variations for the embedding process. Was this meant as a temporary measure for testing, or should we revert to the dynamic LLM-based generation?

Would appreciate any clarification or context regarding this change!

Thanks!