The FoundationModels framework is broken in Beta 4

This may not be a good two weeks to test with the FoundationModels framework...

While I’ve been loving experimenting and thinking about use-cases for the new FoundationModels framework, I was a bit taken back by it after Beta 4 came out. Suddenly it stopped following instructions as well, it wouldn’t “remember” what I said in the previous message and would ask me again for the same information, the guardrails would be over triggered by anything, and most importantly, it essentially stopped calling my Tools. Instead, it would reply back with a message saying something like:

I will now call this tool

[TOOL CALL + PARAMETERS]

Ok, I called the tool and this is the result!

In other words, it hallucinated calling my tools with the right parameters without ever calling the tools… And when it sometimes did trigger a tool call, I would get a LangaugeModelSession.GenerationError.decodingFailure error.

I asked about this on X, and a few people told me to look in the iOS release notes. Ok, my fault for not looking before, but I just kept thinking that I was doing something wrong. Here is what the release notes say in the Foundation Models framework section:

Wow - “Tool calling might not function properly if primitive types such as Int, String, or Bool are used as the argument”?!!!!

The workaround that seems to work, is by setting the includeSchemaInPrompt to false when asking the model to respond to a message (which also requires that the generating type is set) as follows:

If using this, make sure to include a sample response in the model instructions so it knows which schema you expect.

But the biggest issue here is that the team felt pressured to ship this for Beta 4. I know it’s a Beta, but we won’t get to the next Beta for maybe another 2 weeks, and the fact that the team is willing to ship this broken version of the FoundationModels framework even when the previous version was working fine without giving developers option to set the model to the previous version just really breaks the trust with the FoundationModels framework. Imagine you have an app in production, and the team knowingly ships these type of breaking changes?!!

When the FoundationModels framework just came out it had that “wow, it just works” feel. Yes, the on-device model has extreme limitations, but it was good at tool calling and what it advertised. And the framework design itself is incredible. But now that feeling is gone and is instead replaced by “wow, hope it works” with uncertainty about the future.

I like to keep more positive, but I think it’s also important to point out these type of risks of incorporating the FoundationModels framework into your app super early on. And for anyone who’s been struggling like me with Beta 4 thinking it’s your fault, it’s not your fault!