Natural Language Search with Apple's FoundationsModels framework Part 2: Using Tools

Learn how to use Apple's FoundationModels framework to parse user input into powerful filters for real-estate listings.

Previously, I wrote Natural Language Search with Apple's FoundationsModels framework?!, which explores Apple’s FoundationModels ability to find the correct answer when given a mix of natural language options. For example, if the model is given 5 real-estate listings, can it figure out which of the 5 matches the user’s query (e.g. pet-friendly)? TLDR - not surprisingly, this DID NOT work well and truly showed the limitations of this model.

Today, I’m going to explore one alternative that does work well for this use-case. The key is to stop thinking of the FoundationModel as “intelligent” and think of it more as a natural language parser. Then build sophisticated tools that can take in parameters parsed by the model.

So let’s get started!

Having a Comprehensive DataSet

The most difficult part of the this example is making sure you have a comprehensive dataset that can anticipate most of what the user is looking for. In the case of real-estate, these datasets already exist, but they can further be enhanced by running Python scripts in combination with much bigger and better models such as Google Gemini Pro to further take out extra non-standard parameters.

I wrote a few blog posts about pre-processing your data here:

• The Case for Preprocessing: Using LLMs Before Your Users Do

• How to Work with OpenAI's Batch API (not in Swift!)

• Building a Sanskrit Dictionary App: An AI-Powered Retrospective

Let’s assume I have a comprehensive real-estate backend, which will translate to the following HouseListing object in Swift:

import Foundation
import FoundationModels

@Generable
struct HouseListing {
    
    // MARK: - Basic Property Details
    let id: String
    let title: String
    let description: String
    
    let address: String
    let priceInUSD: Double
    let squareFootage: Int
    let lotSizeInAcres: Double
    let yearBuilt: Int
    
    // MARK: - Bedrooms & Bathrooms
    let numberOfBedrooms: Int
    let numberOfBathrooms: Double
    let hasMasterSuite: Bool
    let hasGuestBedroom: Bool
    
    // MARK: - Parking & Garage
    let hasGarage: Bool
    let numberOfGarageSpaces: Int
    let hasDriveway: Bool
    let hasCarport: Bool
    let streetParking: Bool
    
    // MARK: - Outdoor Features
    let hasPool: Bool
    let hasHotTub: Bool
    let hasBackyard: Bool
    let hasFrontYard: Bool
    let hasGarden: Bool
    let hasDeck: Bool
    let hasPatio: Bool
    let hasBalcony: Bool
    let hasFirepit: Bool
    let hasOutdoorKitchen: Bool
    
    // MARK: - Indoor Features
    let hasFireplace: Bool
    let hasBasement: Bool
    let hasAttic: Bool
    let hasWalkInCloset: Bool
    let hasLaundryRoom: Bool
    let hasHomeOffice: Bool
    let hasLibrary: Bool
    let hasWinecellar: Bool
    let hasPantry: Bool
    let hasGameRoom: Bool
    let hasHomeGym: Bool
    let hasHomeTheater: Bool
    
    // MARK: - Kitchen Features
    let hasUpdatedKitchen: Bool
    let hasKitchenIsland: Bool
    let hasGraniteCountertops: Bool
    let hasStainlessAppliances: Bool
    let hasGasStove: Bool
    let hasDoubleOven: Bool
    let hasWalkInPantry: Bool
    
    // MARK: - Flooring & Interior
    let hasHardwoodFloors: Bool
    let hasCarpeting: Bool
    let hasTileFlooring: Bool
    let hasMarbleFlooring: Bool
    let hasVaultedCeilings: Bool
    let hasOpenFloorPlan: Bool
    let hasCrownMolding: Bool
    
    // MARK: - Utilities & Systems
    let hasCentralAC: Bool
    let hasCentralHeating: Bool
    let hasWasherDryer: Bool
    let hasSecuritySystem: Bool
    let hasSmartHome: Bool
    let hasSolarPanels: Bool
    let hasBackupGenerator: Bool
    
    // MARK: - Accessibility
    let isWheelchairAccessible: Bool
    let hasSingleStoryLiving: Bool
    let hasElevator: Bool
    let hasWideDoorways: Bool
    let hasAccessibleBathroom: Bool
    
    // MARK: - Pet-Friendly Features
    let isPetFriendly: Bool
    let hasDogRun: Bool
    let hasFencedYard: Bool
    let allowsCats: Bool
    let allowsDogs: Bool
    let allowsLargePets: Bool
    let hasBuiltInPetFeatures: Bool
    
    // MARK: - Neighborhood & Location
    let nearPublicTransit: Bool
    let nearSchools: Bool
    let nearShopping: Bool
    let nearRestaurants: Bool
    let nearHospitals: Bool
    let nearHikingTrails: Bool
    let nearBeach: Bool
    let nearPark: Bool
    let nearGolfCourse: Bool
    let nearAirport: Bool
    let isInGatedCommunity: Bool
    let hasHOA: Bool
    let hoaFee: Double?
    
    // MARK: - School Districts
    let schoolDistrict: String
    let elementarySchoolRating: Int? // 1-10
    let middleSchoolRating: Int? // 1-10
    let highSchoolRating: Int? // 1-10
    let nearPrivateSchools: Bool
    
    // MARK: - Safety & Security
    let hasNeighborhoodWatch: Bool
    let hasSecurityGates: Bool
    let hasMotionLights: Bool
    let hasRingDoorbell: Bool
    
    // MARK: - Investment & Financial
    let hasRentalPotential: Bool
    let propertyTaxRate: Double
    let appreciationRate: Double
    let isFixerUpper: Bool
    let isNewConstruction: Bool
    let hasWarranty: Bool
    
    // MARK: - Seasonal & Climate
    let hasGoodInsulation: Bool
    let hasStormShelter: Bool
    let hasGoodDrainage: Bool
    
    // MARK: - Community Features
    let hasCommunityPool: Bool
    let hasClubhouse: Bool
    let hasTennisCourtAccess: Bool
    let hasGymAccess: Bool
    let hasPlaygroundNearby: Bool
    let hasWalkingTrails: Bool
    
    // MARK: - Technology & Modern Features
    let hasFiberInternet: Bool
    let hasSmartThermostat: Bool
    let hasElectricCarCharger: Bool
    let hasSmartLighting: Bool
    let hasVoiceControl: Bool
    
    // MARK: - Environmental
    let hasEcoFriendlyFeatures: Bool
    let hasEnergyEfficientAppliances: Bool
    let hasLEDLighting: Bool
    let hasRainwaterCollection: Bool
    let hasCompostArea: Bool
    
    // MARK: - Maintenance & Condition
    let hasRecentRenovations: Bool
    let needsRepairs: Bool
    let hasMaintenanceContract: Bool
}

How can we work with this using the FoundationModels framework?

The Goal

Our HouseListing has soooo much detail that it will quickly overwhelm the user to see all of them at ones. Instead, we can build a chat-based interface where the user can simply search for houses based on the criteria that matter to them specifically.

This means that we need to translate what the user is searching for in a house in natural language to match the variables in the HouseListing object, that we can use as a filter for houses matching the user’s query.

For example, when the user searches for “pet friendly houses near hiking trails”, we want to filter our HouseListing to only return houses where the isPetFriendly and nearHikingTrails variables are set to true.

Creating a Housing Search Tool

To create a Tool using the FoundationModels framework 3 things are required:

1. Name - A unique name for the tool

2. Description - A natural language description of when and how to use the tool

3. The Call function - A language model will call this method when it wants to leverage this tool.

Let’s get started with the name and description, the easy parts:

Now onto the hard part - the call function…

First, we will define the input arguments from the model when calling this function. So when the user types in “a pet friendly house near hiking trails”, we would want the model to pass in isPetFriendly = true and nearHikingTrails = true as arguments.

To do this, we first want to specify all the possible filter options as an enum that the model will choose from, which will allow us to filter HouseListings:

    @Generable
    enum FilterType: String, CaseIterable {
        // Price & Size
        case minPrice = "minPrice"
        case maxPrice = "maxPrice"
        case minSquareFootage = "minSquareFootage"
        case maxSquareFootage = "maxSquareFootage"
        case minLotSize = "minLotSize"
        case maxLotSize = "maxLotSize"
        case minYearBuilt = "minYearBuilt"
        case maxYearBuilt = "maxYearBuilt"
        
        // Bedrooms & Bathrooms
        case numberOfBedrooms = "bedrooms"
        case minBedrooms = "minBedrooms"
        case maxBedrooms = "maxBedrooms"
        case minBathrooms = "minBathrooms"
        case maxBathrooms = "maxBathrooms"
        case hasMasterSuite = "hasMasterSuite"
        case hasGuestBedroom = "hasGuestBedroom"
        
        // Features (Boolean)
        case hasPool = "hasPool"
        case hasGarage = "hasGarage"
        case hasFireplace = "hasFireplace"
        case hasGarden = "hasGarden"
        case hasHomeOffice = "hasHomeOffice"
        case hasSmartHome = "hasSmartHome"
        case hasSolarPanels = "hasSolarPanels"
        case isPetFriendly = "isPetFriendly"
        case hasFencedYard = "hasFencedYard"
        case isWheelchairAccessible = "isWheelchairAccessible"
        case hasSingleStoryLiving = "hasSingleStoryLiving"
        case nearSchools = "nearSchools"
        case nearPublicTransit = "nearPublicTransit"
        case nearHikingTrails = "nearHikingTrails"
        case nearBeach = "nearBeach"
        case nearGolfCourse = "nearGolfCourse"
        case isInGatedCommunity = "isInGatedCommunity"
        case hasRentalPotential = "hasRentalPotential"
        case isFixerUpper = "isFixerUpper"
        case isNewConstruction = "isNewConstruction"
        case hasUpdatedKitchen = "hasUpdatedKitchen"
        case hasKitchenIsland = "hasKitchenIsland"
        case hasHardwoodFloors = "hasHardwoodFloors"
        case hasOpenFloorPlan = "hasOpenFloorPlan"
        case hasCentralAC = "hasCentralAC"
        case hasSecuritySystem = "hasSecuritySystem"
        case hasBackyard = "hasBackyard"
        case hasBalcony = "hasBalcony"
        case hasElevator = "hasElevator"
        case allowsDogs = "allowsDogs"
        case allowsCats = "allowsCats"
        case hasHOA = "hasHOA"
        case hasEcoFriendlyFeatures = "hasEcoFriendlyFeatures"
        
        var description: String {
            switch self {
            case .minPrice: return "Minimum price in USD"
            case .maxPrice: return "Maximum price in USD"
            case .minSquareFootage: return "Minimum square footage"
            case .maxSquareFootage: return "Maximum square footage"
            case .minLotSize: return "Minimum lot size in acres"
            case .maxLotSize: return "Maximum lot size in acres"
            case .minYearBuilt: return "Minimum year built"
            case .maxYearBuilt: return "Maximum year built"
            case .numberOfBedrooms: return "Exact number of bedrooms"
            case .minBedrooms: return "Minimum bedrooms"
            case .maxBedrooms: return "Maximum bedrooms"
            case .minBathrooms: return "Minimum bathrooms"
            case .maxBathrooms: return "Maximum bathrooms"
            case .hasMasterSuite: return "Has master suite"
            case .hasGuestBedroom: return "Has guest bedroom"
            case .hasPool: return "Has swimming pool"
            case .hasGarage: return "Has garage"
            case .hasFireplace: return "Has fireplace"
            case .hasGarden: return "Has garden"
            case .hasHomeOffice: return "Has home office"
            case .hasSmartHome: return "Has smart home features"
            case .hasSolarPanels: return "Has solar panels"
            case .isPetFriendly: return "Pet friendly"
            case .hasFencedYard: return "Has fenced yard"
            case .isWheelchairAccessible: return "Wheelchair accessible"
            case .hasSingleStoryLiving: return "Single story living"
            case .nearSchools: return "Near schools"
            case .nearPublicTransit: return "Near public transit"
            case .nearHikingTrails: return "Near hiking trails"
            case .nearBeach: return "Near beach"
            case .nearGolfCourse: return "Near golf course"
            case .isInGatedCommunity: return "In gated community"
            case .hasRentalPotential: return "Has rental potential"
            case .isFixerUpper: return "Fixer-upper property"
            case .isNewConstruction: return "New construction"
            case .hasUpdatedKitchen: return "Has updated kitchen"
            case .hasKitchenIsland: return "Has kitchen island"
            case .hasHardwoodFloors: return "Has hardwood floors"
            case .hasOpenFloorPlan: return "Has open floor plan"
            case .hasCentralAC: return "Has central air conditioning"
            case .hasSecuritySystem: return "Has security system"
            case .hasBackyard: return "Has backyard"
            case .hasBalcony: return "Has balcony"
            case .hasElevator: return "Has elevator"
            case .allowsDogs: return "Allows dogs"
            case .allowsCats: return "Allows cats"
            case .hasHOA: return "Has HOA"
            case .hasEcoFriendlyFeatures: return "Has eco-friendly features"
            }
        }
    }

When the model passes in the FilterType, we would also want a value for this type (e.g. true / false), and a description of the filter - this is more for the model to “think” through using this filter:

Finally, our Arguments parameter for the call function will include an array of FilterSpecifications and a way for the model to reset all filters (for example if the user says “Forget that, I want to start fresh. Let’s look for houses with pools only”.

Note that the arguments input for the call function can be any Generable object defined in your Tool.

State Management with Tools

One of very interesting implementation details of the genius FoundationModels framework is that the LanguageModelSession will take in an instance of a tool that it will use throughout it’s lifecycle. That means Tools can keep state!

As the user continues their conversation with the language model, more filters will be added. For example, in the first message the user may say they want a house with a pool. When the model responds with filtered houses that have pools, the user may then try to narrow it down to houses that are also pet friendly, etc.

Ideally, the model should be able to take in the full context of the entire conversation and understand that the user wants a house with a pool that is also pet friendly and include these two filters (hasPool = true, and isPetFriendly=true) when calling the tool. But unfortunately the Foundation Model is just not that sophisticated and can be hit or miss. For example, if you say you want a house with a pool, it will only pass in the hasPool=true filter, “forgetting” / not taking into consideration the earlier isPetFriendly=true filter. And this will get worse and worse the longer the conversation and especially with more filters.

So keeping state is up to you as the developer! In our case, we can instruct the model to only include the new filters from the user input, that will aggregate and keep track of ourselves.

In our example of a DynamicHousingSearchTool, we want the tool to keep track of two things:

• The current active filters

• The current results matching the active filters

The second reason for keeping state is the model’s small context window. By having the tool keep the state, you can easily start a new LanguageModelSession using the same instance and continue the house search without the user knowing they’re in a new chat session as follows:

Applying the Filter

When the model passes in an array of FilterSpecifications as arguments, we can iterate through the filters and apply them by using the FilterType enum:

private func applyFilter(
        filterType: FilterType,
        value: String,
        to houses: [HouseListing]
    ) -> [HouseListing] {
        
        switch filterType {
        // Numeric filters
        case .minPrice:
            guard let minPrice = Double(value) else { return houses }
            return houses.filter { $0.priceInUSD >= minPrice }
            
        case .maxPrice:
            guard let maxPrice = Double(value) else { return houses }
            return houses.filter { $0.priceInUSD <= maxPrice }
            
        case .minSquareFootage:
            guard let minSqFt = Int(value) else { return houses }
            return houses.filter { $0.squareFootage >= minSqFt }
            
        case .maxSquareFootage:
            guard let maxSqFt = Int(value) else { return houses }
            return houses.filter { $0.squareFootage <= maxSqFt }
            
        case .numberOfBedrooms:
            guard let bedrooms = Int(value) else { return houses }
            return houses.filter { $0.numberOfBedrooms == bedrooms }
            
        case .minBedrooms:
            guard let minBedrooms = Int(value) else { return houses }
            return houses.filter { $0.numberOfBedrooms >= minBedrooms }
            
        case .maxBedrooms:
            guard let maxBedrooms = Int(value) else { return houses }
            return houses.filter { $0.numberOfBedrooms <= maxBedrooms }
            
        case .minBathrooms:
            guard let minBathrooms = Double(value) else { return houses }
            return houses.filter { $0.numberOfBathrooms >= minBathrooms }
            
        case .maxBathrooms:
            guard let maxBathrooms = Double(value) else { return houses }
            return houses.filter { $0.numberOfBathrooms <= maxBathrooms }
            
        case .minLotSize:
            guard let minLot = Double(value) else { return houses }
            return houses.filter { $0.lotSizeInAcres >= minLot }
            
        case .maxLotSize:
            guard let maxLot = Double(value) else { return houses }
            return houses.filter { $0.lotSizeInAcres <= maxLot }
            
        case .minYearBuilt:
            guard let minYear = Int(value) else { return houses }
            return houses.filter { $0.yearBuilt >= minYear }
            
        case .maxYearBuilt:
            guard let maxYear = Int(value) else { return houses }
            return houses.filter { $0.yearBuilt <= maxYear }
            
        // Boolean filters
        case .hasPool:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasPool == boolValue }
            
        case .hasGarage:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasGarage == boolValue }
            
        case .hasFireplace:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasFireplace == boolValue }
            
        case .hasGarden:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasGarden == boolValue }
            
        case .hasHomeOffice:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasHomeOffice == boolValue }
            
        case .hasSmartHome:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasSmartHome == boolValue }
            
        case .hasSolarPanels:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasSolarPanels == boolValue }
            
        case .isPetFriendly:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.isPetFriendly == boolValue }
            
        case .hasFencedYard:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasFencedYard == boolValue }
            
        case .isWheelchairAccessible:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.isWheelchairAccessible == boolValue }
            
        case .hasSingleStoryLiving:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasSingleStoryLiving == boolValue }
            
        case .nearSchools:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.nearSchools == boolValue }
            
        case .nearPublicTransit:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.nearPublicTransit == boolValue }
            
        case .nearHikingTrails:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.nearHikingTrails == boolValue }
            
        case .nearBeach:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.nearBeach == boolValue }
            
        case .nearGolfCourse:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.nearGolfCourse == boolValue }
            
        case .isInGatedCommunity:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.isInGatedCommunity == boolValue }
            
        case .hasRentalPotential:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasRentalPotential == boolValue }
            
        case .isFixerUpper:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.isFixerUpper == boolValue }
            
        case .isNewConstruction:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.isNewConstruction == boolValue }
            
        case .hasUpdatedKitchen:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasUpdatedKitchen == boolValue }
            
        case .hasKitchenIsland:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasKitchenIsland == boolValue }
            
        case .hasHardwoodFloors:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasHardwoodFloors == boolValue }
            
        case .hasOpenFloorPlan:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasOpenFloorPlan == boolValue }
            
        case .hasCentralAC:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasCentralAC == boolValue }
            
        case .hasSecuritySystem:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasSecuritySystem == boolValue }
            
        case .hasBackyard:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasBackyard == boolValue }
            
        case .hasBalcony:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasBalcony == boolValue }
            
        case .hasElevator:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasElevator == boolValue }
            
        case .allowsDogs:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.allowsDogs == boolValue }
            
        case .allowsCats:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.allowsCats == boolValue }
            
        case .hasHOA:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasHOA == boolValue }
            
        case .hasEcoFriendlyFeatures:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasEcoFriendlyFeatures == boolValue }
            
        case .hasMasterSuite:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasMasterSuite == boolValue }
            
        case .hasGuestBedroom:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasGuestBedroom == boolValue }
        }
    }

Implementing the Call Function

The call functions will take in the Arguments that we defined earlier:

And output a ToolOutput object, which can be initiated with either a String or a GeneratedContent type:

First, if the model passes in true for resetFilters, we reset the state variables:

Next, we will iterate through each filter and add the new filters to the activeFilters array and filter the currentResults:

We want to give the new filtered house results back to the Foundation Model to display to the user, but the original HouseListing object is too big and overwhelming both for the model (in tokens) and the user. So we want to create a smaller version of the HouseListing that just includes the most important information. We do this by creating a new Generable HouseResult object:

We also want to give a custom message about the results for the model:

We then include all the information in a GeneratedContent final output for the model:

That’s it! The model will not have the list of filtered houses to present to the user based on their natural language criteria.

Advanced

Since you’re controlling the end-to-end UI of this application, instead of passing the house results to the language model to display to the user, you can use the Tool call to trigger a UI update to directly DISPLAY the results to the user while sending the response to the model as a message simply saying you applied the requested filters.

Check out this project for inspiration!

The Prompt

Finally, it’s time to run the model! The model instructions in this case is a bit more complex to get it working just right:

Giving examples to the model is super important. The examples included above are:

Finally, we just pass the tool and the instructions to the model and it runs well!

Conclusion

The FoundationModels framework is best used for parsing natural language. When you use it only for that - just parsing user’s input into specific function arguments that you have full control over, you truly get to experience the power of the model. Set up your tools and code accordingly!