Incorrect Context on Models with Vector Stores Enabled

[nathanwong91]

Unsure if this was answered or discussed before but I couldn't find any when I tried to search for it.

We have a vector store setup going to AWS Bedrock Knowledge Base and have turned it on similar to the example in the docs

model_list:
  - model_name: claude-3-5-sonnet-with-vector-store
    litellm_params:
      model: anthropic/claude-3-5-sonnet
      vector_store_ids: ["T37J8R4WTM"]

If we use this model for example, debugging code or asking it about the weather it loses its mind and starts talking about the files in vector store. Is this intended? I know I can just add a model without the vector store always on.