Dealing with new model behaviors

[MrCsabaToth]

This is partially about the firebase-vertexai upgrade (#53), and also the new release of the -002 stable models. Seems like function calling behavior and other things changed.

[MrCsabaToth]

New behavior: function calling changes

1. I switched over to the -002 models explicitly, and the function calling behavior changed. In the past (and on the submission demo) I can simply ask "What will be the weather tomorrow" or "What will be the weather next week". The model assumed (correctly) that I implicitly meant the weather at my current location, and compared to the current date/time.
2. The new model is very specific and picky, it doesn't think anything implied. It asks if I'll stay at my current location tomorrow (or next week) to answer the question, and it also doesn't seem to be aware of the current date / time, so that needs to be stuffed into the prompt.
3. The new models also cannot comprehend how to query the weather for "next week". The schema of the weather tool allows for start and end date parameters. The old model could easily deduct and substitute these two variables to obtain the next week's weather. The new model is lame and gives up, it somehow doesn't able to comprehend how to achieve this. It states simply that it can obtain a weather for a specific day but not for a week. Which is false.

[MrCsabaToth]

Another breaking change: the function calling stopped working all together because Please ensure that function call turn comes immediately after a user turn or after a function response turn.
Others are dealing with this too (GD Community Gemini API thread): https://discord.com/channels/1009525727504384150/1289794849003802735

[MrCsabaToth]

So function calling is kinda solved. Even though it is really nondeterministic: for example for the "What's the weather tomorrow" question about half of the time the model states it doesn't have means to obtain it, like it wouldn't see the weather tool at all. But half of the time just works.

Then latest is that the model flat out states it cannot process images. The Pro seems to stick to this always, whereas the Flash can process the image (for example it correctly OCRs reading on the food packaging), but then it continues by saying it cannot handle images. Say what?

[MrCsabaToth]

With Gemini 2.0 released for Flash we'll try gemini-2.0-flash-exp as noted in https://firebase.google.com/docs/vertex-ai/gemini-models#available-models

You can start experimenting with the Gemini 2.0 model (gemini-2.0-flash-exp) using the Vertex AI in Firebase SDKs.
Right now, our SDKs support text output from text, code, image, PDF, audio, video, and video with audio inputs. Support for the Multimodal Live API as well as native audio and image output will be coming in 2025!