Natural Language Search with Apple's FoundationModels framework?!

Can Apple's FoundationModels framework be used for searching through house listings?

Earlier today I saw an interesting tweet from Rudrank Riyam as he’s attending Apple’s #WWDC25 recap event in Bangalore:

What surprised me here was Apple’s example of using the FoundationModels framework for “Natural language search” - to search through house listings for example…

First of all, the FoundationModels framework has an extremely limited context window of only 4096 tokens - that is both for INPUT AND OUTPUT combined. So in the context of text search, you wouldn’t be able to pass in that many objects to the model for the user to search.

But let’s say you do have a more limited number of options and want to use the FoundationModels framework to do the search. Would it be able to give you relevant results given how small the model is? I went ahead and tested it out using Apple’s example of house listings to see!

The House Listings

First - I created a Generable HouseListing object so that I create sample listings and easily pass them into the model.

Imagine you receive the house listings from your API and load them into an array of HouseListing structs.

The results is five of the following house listings:

The House Search Results

Next, I created a Generable HouseSearchResults objects that will be generated by the FoundationModels framework that match the user’s search query:

Searching for the Perfect House

Using the new #Playgrounds macro, I can quickly test out if the model will return back the correct results based on the user query:

The result for “a house with a pool” was generated as follows:

WOW!!! IMPRESSIVE!!! IT WORKS… except… let’s take a look at the description for house number 5 again…

THERE IS NO MENTION OF A POOL! From running different versions of this search prompt, my assessment is that it’s confusing the “kitchen" island with waterfall edges as a pool.

In fact, if you look closely at the reasoning of that the house “features a resort style pool with spa”, you can see that this specifically is mentioned for house #3:

Maybe it confused the 5 numberOfBedrooms as house number 5?!!

Also note that house #2, which does have a pool was NOT included in the results:

Now, sometimes the model does work… When I searched for a house near hiking trails, it correctly showed house #5:

However, I would argue that house #1, a farmhouse on 2.5 acres of land, should also show up in the results as it can easily have a trail on the property… This just shows that the model cannot deduce information.

You can see the deduction flaws as well. When I searched for “a house with an ocean view", it gave the following reasoning for the Cape Cod Cottage (house #4):

The answer is technically correct - in the description it says that “the second floor offers additional bedrooms with dormer windows and ocean glimpses.” But the model never mentioned the “ocean glimpses” as the reason for why it chose house number 4. Instead siting the proximity to the beach, which doesn’t necessarily mean ocean views directly from the house.

Conclusion

The example of searching using natural language with the FoundationModels framework just exposes the extreme limitations of this small on-device model. Make sure to test thoroughly for all use-cases, as the word relationships of this model are not as sophisticated (e.g. kitchen island = pool) as the super big LLMs we’re used to like ChatGPT and Claude.