The Backstory

While the mainstream use-case for LLMs is using it as chatbots, there are a lot of opportunities to use it behind-the-scenes. In my case of building a Sanskrit-English Dictionary app, my database is extremely static and doesn’t change. That means that I can process many dictionary definitions using LLMs on the backend, and simply show the result to all users in a UI-friendly way in my app, without them even being aware that AI was used for processing.

I’ve already worked a lot with the Gemini 2.5 Pro API, before Batch Mode was released, and it took me ~ 30 days to process all my database definitions in the past one-by-one. So when Batch Mode came out, I couldn’t be more excited! Now my processing time went down to only 2 days and my cost went down 50%!

The Problems

I initially ran my batch jobs via Python scripts, but quickly ran into an organizational nightmare. There were several things to keep track of:

1. I had to divide up all my data into 100s of files to stay within the Batch Mode limits

2. Some of my files had to run using Gemini 2.5 Flash, and some had to be run using Gemini 2.5 Pro. When running a batch job, you have to run it using only 1 model, you cannot combine requests that need to run via different models.

3. I had to have a way to keep track of which files are running, which are done, which should be queued up next, etc.

4. I didn’t know how much everything was costing - yes, this is my fault, but I ended up spending A LOT more than I estimated. Since the batch jobs run so fast and the Gemini usage billing data updates very slowly, I ended up with a very surprising bill. Yes, again, my fault for not being slow in testing out the costs and not calculating them more carefully, but I honestly didn’t think it would go past my budget.

5. There is a tight timeframe for some of the tasks - once you upload your batch request file, it expires within 48 hours. The batch job itself expires after 48 hours after it starts. And of course, the batch job results file also expires after 48 hours. So you cannot just take your eye off-the-ball and return a week later to download your files. You have to actively poll for the result to make sure you download it before everything expires and you lose all your money on that batch job, having to re-run it again.

I did manage all of this in Python, but it was a bit of a mess. I really wished Gemini had a better interface to manage these batch requests, similar to what OpenAI has, but unfortunately, it’s just not there. So I decided to build it myself!

The Gemini Batch Mac App

So for the past week, I took the time to build the interface that I wished the Google Gemini team had… as a Mac Tahoe app. Here is the result!

Simply upload your JSONL Batch request files, click Run, and the whole process is managed for you! You can keep track of exactly what is happening in the right-hand pane by selecting the file at any time. And the results file is saved in the app sandbox directory - safe from the 48 hour expiration. You can come back to the app whenever you have time, and download all completed JSONL results file, which you can then process further!

Oh - and I forgot to mention one thing! The tokens are calculated at the end of each batch job, so you can get an estimated price per file run. So it’s very easy to test by running 1 file, waiting around 5 - 15 minutes for the batch job to complete, and check the costs before running more files. Wish I had this when I started running my Gemini batch jobs!

The App is Open Source

The app is now open source and available on the try! Swift AI Github for anybody to use! Note that the app is made for macOS 26 only - as I wanted to play around with LiquidGlass, but it should be easy to clone and port back if you need.

To learn how to read / write JSONL files in Swift, read my blog post: How to Read / Write JSONL Files in Swift. I also build a JSONL Beautifier to make it easy to inspect the JSONL results as I couldn’t find one when I searched.

Happy Batch Processing 🚀🚀🚀

JSONL, short for JSON Lines (also known as newline-delimited JSON), is a text-based format where each line is a standalone, valid JSON object. Unlike traditional JSON (which often holds an entire array or nested structure), JSONL is structured so that each entry lives on its own line, separated by newline characters ( \n ).

A sample JSONL file will look like this:

Again, note that there is no comma to separate the JSON structures. New structures / objects are simply on a new line ( \n ). Note also that each JSON value has an id to identify / map it.

Writing a JSONL File

Let’s say we want to create a Gemini API Batch request to generate Haiku poems for a list of topics. To do this, we would need to create a haiku_batch_requests.jsonl file to upload to the files API and then reference for creating the batch request.

Before writing to the file, we first need to define the system instructions and the haiku subjects. The system instructions will be reused in every JSON structure, but the subjects of the haiku will change.

Note that since we need to be able to map the subject of the haiku to the generated haiku, we need to include the id or key for each subject:

Note that for simplicity I’m using simple integers as keys, but in reality this would be a key that matches to your database, such as an UUID.

Next, you need to create an Encodable object that you can encode to JSON. In the case of the Gemini Batch API, I will be using the existing AIProxy library since it already has an encodable request object:

Again, to generate the JSONL file, it can be ANY encodable object that you need.

Finally, I will simply go through each of my haiku subjects and generate a JSON string from it, which I’ll store in an array of json lines. Then I will simply combine them with a “\n” character like this:

That’s it! The hard part is over. The final part is simply to write the newly generated json lines string to file:

Here I’m storing the final JSONL in the user’s Downloads folder, but you can store it in any path and access it as needed.

Reading the JSONL file

Reading from a JSONL file follows the same process but in reverse, in the following steps:

1. Separate the response by the new lines character ( \n )

2. Iterate line-by-line and decode each line into a Decodable object

3. Return back an array of decodable objects

The code is simple:

That’s it!

Conclusion

While the JSONL format may seem a bit intimidating at first - just from being new and something many of us haven’t worked with, it’s actually just as simple to work with as with any JSON object. Just go line-by-line!

BONUS: I wanted to beautify my JSONL file, but couldn’t find a JSONL Beautifier online, so I made one!

https://jsonlbeautifier.com/

Happy building!

Learn how to leverage Google's Gemini Batch API to add powerful background AI processing to your Swift applications. Perfect for developers using Firebase or Supabase who need to process large datasets asynchronously over hours or days without blocking user interactions. We'll walk through the complete lifecycle of creating batch requests, monitoring their progress, and handling results - all using Swift! You'll receive a fully functional Swift monitoring app at the end of the workshop to track your batch requests in real-time and integrate batch AI processing into your existing applications.

Buy Now ($50)

​Discover how to use the FoundationModels framework to create truly reactive applications where users can configure app settings using natural language commands. Watch as we build a demo app that dynamically transforms its UI based on typed user preferences - from changing themes and layouts to adjusting functionality, all through conversational interaction. All code will be provided at the end of the workshop, so you can run it at your own pace whenever you are ready to upgrade to macOS Tahoe.

Buy Now ($50)

In my latest app, I have a SwiftData model where some of the variables are processed by an LLM API in the background and the SwiftData model has to be updated with the results from the API. I’ll give the example of a Journal app with a simple model:

The app will use an LLM API to analyze the journal entry’s mood. So the mood will start off by being nil and then being filled in once we get the LLM API response. The mood variable is not the most critical part of this app, so it can be done fully on the background while the user can browse or add other journal entries.

So let’s try to attempt to write a nonisolated class (to be executed in the background) to extract the mood from a journal entry:

It’s not a full on error, but it’s a warning, and my goal is to remove all the warnings to make sure my code is Swift 6 level SAFE!! So how do we fix it?

The Model Actor

To make sure that our SwiftData models are accessed, updated, and saved in a thread-safe manner on the background thread, we can use a ModelActor. By default, the model actor will take in the model container and make the changes as needed.

The model actor will be the only one that deals with the actual models, so in order to pass around the model information to our Mood Extractor API, for example, we need to create a proxy object for the JournalEntry:

Now whenever the MoodExtractor needs to access the JournalEntry, it can send in the JournalEntry’s PersistentIdentifier and get back the proxy JournalEntryInfo object with the required information for making the API call in a safe way. We can write the getJournalEntryInfo as follows:

Finally, when the API returns back the mood of the journal entry, we can add a way to update the identified JournalEntry with the mood:

That’s it! Just with a few lines of code, the JournalEntryActor will safely retrieve, update, and save SwiftData models in a thread-safe way IN THE BACKGROUND!

The Mood Extractor

The MoodExtractor will be triggered to run in the background to extract the mood of the journal entry when the user clicks on the Save button after finishing editing their journal entry. So we can get the model container at the initialization (coming from SwiftUI), and use it to initialized the JournalEntryActor:

Now we can easily use the journalEntryActor to update our extractMood function to be completely thread-safe!

Using the MoodExtractor in SwiftUI

When the user clicks on “Save” after editing the entry, the extractMood function can be called in a detached (background task):

To make sure the result is updated in real-time on the UI, make sure to use the SwiftData Query macro, in our case to list out the journal entries:

The journal entries will automatically update with the mood as soon as they are saved in the background by the JournalEntryActor!

That’s it! It’s a bit of overhead in the architecture to go between the Model / ModelActor / Background API / SwiftUI, but this makes sure your code is SAFE and reactive! Oh - and all of this can be done in Swift 5 as well (you can architect your code in this same way without the Swift 6 warnings to force you!).

Have thoughts? Let me know in the comments!

Let’s build it!

Read more

It just looks like any other orange button:

Only when I click on it, I can see some glass effect…

I just assumed this would be fixed over time, or maybe I was not getting something… and when I tried one simple trick, it did actually work! But only in dark mode…

The key is to put an opacity on the button tint:

This works really well in dark mode:

But not as great in light mode (this is why I set the opacity to higher in light mode, it just washes out if it’s too high):

Anyway, it does look too cool in dark mode! So this is something to play around with when creating glass effect views - make sure to add the opacity to the tint but test it out in dark and light modes!

Again, this might have been obvious to others, but I just assumed that glassEffect would automatically look like the orange button with the 0.5 opacity above and it just didn’t. So sharing it just in case others had the same misconception.

So I was extremely excited to see this X post from @_julianschiavo announcing that we can now use the same FoundationModels API design to work with external LLM providers!

In this post, I’ll walk you through how to do this to build an AI-powered Financial Analyst app using the Perplexity API for searching through company-specific financial data.

The AI-powered Financial Analyst App

The Financial Analyst app is simple - the user can simply choose from a list of companies they want financial reports on, include the time range for that financial information, and ask any custom questions if they’d like.

The Perplexity API is then used to get back a structured response, which is streamed as new tokens come in and the UI is built with the latest streamed generation:

So how do we build this?

Streaming the Perplexity API Response

Streaming an API response line-by-line has been available in iOS APIs through AsyncThrowingStream for some time, so this part is as simple as this:

Notice that the stream will return back the json string from the response.

I won’t go deeper into the prompt and other specific variables that are used to make the streaming request as this is just the basics of building an API call, but you can view the full PerplexityClient code here.

I’ll just point out the json schema that is passed to the Perplexity API as this is what we expect to get returned as a structured json response from the model:

Using Generable Types for Decoding

Normally we would be creating Codable objects to decode the API response, but now we can simply use the Generable type from the FoundationModels framework instead!

The Generable type is now Codable by default, so it just works! But one issue is that as of now (Xcode 26 Beta 4), you cannot add CodingKeys. This means that if the api returns snake case (e.g. search_results), you cannot add a CodingKey to use searchResults as a variable. You have to name each variable EXACTLY how they are in the API response.

Generating a Generable Object from JSON

When the Perplexity API returns back the response, the structured output we requested with our json schema, which corresponds to the Generable Analysis object above will be returned simply as a json string in the assistant message content.

However, here is the problem… the json response will be generated one token at a time:

So how can we decode it? This is the magical part!

First, create a GeneratedContent object from any partial json string!

Then pass that generated object into the Generable object that the json represents:

This will start generating the Analysis as soon as any of the information from the partial json string comes in and can be parsed!

For example, here the company name came in, so now the company is available!

The final result to get an analysis object from the API response is as follows:

In the same way, to generate the GeneratedFinancialAnalysis object, we will pass in the json response from the Perplexity response stream every time it comes in:

This will generate the current version of the Analysis object from the partial json string!

Updating the UI

Live-updating the UI as the stream comes in with more and more information each time is as simple as just updating the generatedAnalysis State variable every time the new response with additional data comes in through the stream:

Simply performAnalysis when the view loads:

And keep replacing the generatedAnalysis variable with the latest version. Here is the full performAnalysis function:

You can view the full AnalysisView code here.

Conclusion

That’s it! It only takes a few steps to use the Generable macro from the FoundationModels framework to create an impressive streaming experience with external LLM providers.

Simply:

• Stream the LLM Provider API response as a json string

• Use Generable types for decoding the json

• Update the UI to the latest streamed generable object

The FoundationModels team once again does an impressive job with allowing us to use the same developer experience for external models!

Now, back to building 🤓

P.S. - The full code for the Financial Analyst app is available on Github here.

So how can we build it?!! Let’s get started!

Read more

I will now call this tool

[TOOL CALL + PARAMETERS]

Ok, I called the tool and this is the result!

In other words, it hallucinated calling my tools with the right parameters without ever calling the tools… And when it sometimes did trigger a tool call, I would get a LangaugeModelSession.GenerationError.decodingFailure error.

I asked about this on X, and a few people told me to look in the iOS release notes. Ok, my fault for not looking before, but I just kept thinking that I was doing something wrong. Here is what the release notes say in the Foundation Models framework section:

Wow - “Tool calling might not function properly if primitive types such as Int, String, or Bool are used as the argument”?!!!!

The workaround that seems to work, is by setting the includeSchemaInPrompt to false when asking the model to respond to a message (which also requires that the generating type is set) as follows:

If using this, make sure to include a sample response in the model instructions so it knows which schema you expect.

But the biggest issue here is that the team felt pressured to ship this for Beta 4. I know it’s a Beta, but we won’t get to the next Beta for maybe another 2 weeks, and the fact that the team is willing to ship this broken version of the FoundationModels framework even when the previous version was working fine without giving developers option to set the model to the previous version just really breaks the trust with the FoundationModels framework. Imagine you have an app in production, and the team knowingly ships these type of breaking changes?!!

When the FoundationModels framework just came out it had that “wow, it just works” feel. Yes, the on-device model has extreme limitations, but it was good at tool calling and what it advertised. And the framework design itself is incredible. But now that feeling is gone and is instead replaced by “wow, hope it works” with uncertainty about the future.

I like to keep more positive, but I think it’s also important to point out these type of risks of incorporating the FoundationModels framework into your app super early on. And for anyone who’s been struggling like me with Beta 4 thinking it’s your fault, it’s not your fault!

Today, I’m going to explore one alternative that does work well for this use-case. The key is to stop thinking of the FoundationModel as “intelligent” and think of it more as a natural language parser. Then build sophisticated tools that can take in parameters parsed by the model.

So let’s get started!

Having a Comprehensive DataSet

The most difficult part of the this example is making sure you have a comprehensive dataset that can anticipate most of what the user is looking for. In the case of real-estate, these datasets already exist, but they can further be enhanced by running Python scripts in combination with much bigger and better models such as Google Gemini Pro to further take out extra non-standard parameters.

I wrote a few blog posts about pre-processing your data here:

• The Case for Preprocessing: Using LLMs Before Your Users Do

• How to Work with OpenAI's Batch API (not in Swift!)

• Building a Sanskrit Dictionary App: An AI-Powered Retrospective

Let’s assume I have a comprehensive real-estate backend, which will translate to the following HouseListing object in Swift:

import Foundation
import FoundationModels

@Generable
struct HouseListing {
    
    // MARK: - Basic Property Details
    let id: String
    let title: String
    let description: String
    
    let address: String
    let priceInUSD: Double
    let squareFootage: Int
    let lotSizeInAcres: Double
    let yearBuilt: Int
    
    // MARK: - Bedrooms & Bathrooms
    let numberOfBedrooms: Int
    let numberOfBathrooms: Double
    let hasMasterSuite: Bool
    let hasGuestBedroom: Bool
    
    // MARK: - Parking & Garage
    let hasGarage: Bool
    let numberOfGarageSpaces: Int
    let hasDriveway: Bool
    let hasCarport: Bool
    let streetParking: Bool
    
    // MARK: - Outdoor Features
    let hasPool: Bool
    let hasHotTub: Bool
    let hasBackyard: Bool
    let hasFrontYard: Bool
    let hasGarden: Bool
    let hasDeck: Bool
    let hasPatio: Bool
    let hasBalcony: Bool
    let hasFirepit: Bool
    let hasOutdoorKitchen: Bool
    
    // MARK: - Indoor Features
    let hasFireplace: Bool
    let hasBasement: Bool
    let hasAttic: Bool
    let hasWalkInCloset: Bool
    let hasLaundryRoom: Bool
    let hasHomeOffice: Bool
    let hasLibrary: Bool
    let hasWinecellar: Bool
    let hasPantry: Bool
    let hasGameRoom: Bool
    let hasHomeGym: Bool
    let hasHomeTheater: Bool
    
    // MARK: - Kitchen Features
    let hasUpdatedKitchen: Bool
    let hasKitchenIsland: Bool
    let hasGraniteCountertops: Bool
    let hasStainlessAppliances: Bool
    let hasGasStove: Bool
    let hasDoubleOven: Bool
    let hasWalkInPantry: Bool
    
    // MARK: - Flooring & Interior
    let hasHardwoodFloors: Bool
    let hasCarpeting: Bool
    let hasTileFlooring: Bool
    let hasMarbleFlooring: Bool
    let hasVaultedCeilings: Bool
    let hasOpenFloorPlan: Bool
    let hasCrownMolding: Bool
    
    // MARK: - Utilities & Systems
    let hasCentralAC: Bool
    let hasCentralHeating: Bool
    let hasWasherDryer: Bool
    let hasSecuritySystem: Bool
    let hasSmartHome: Bool
    let hasSolarPanels: Bool
    let hasBackupGenerator: Bool
    
    // MARK: - Accessibility
    let isWheelchairAccessible: Bool
    let hasSingleStoryLiving: Bool
    let hasElevator: Bool
    let hasWideDoorways: Bool
    let hasAccessibleBathroom: Bool
    
    // MARK: - Pet-Friendly Features
    let isPetFriendly: Bool
    let hasDogRun: Bool
    let hasFencedYard: Bool
    let allowsCats: Bool
    let allowsDogs: Bool
    let allowsLargePets: Bool
    let hasBuiltInPetFeatures: Bool
    
    // MARK: - Neighborhood & Location
    let nearPublicTransit: Bool
    let nearSchools: Bool
    let nearShopping: Bool
    let nearRestaurants: Bool
    let nearHospitals: Bool
    let nearHikingTrails: Bool
    let nearBeach: Bool
    let nearPark: Bool
    let nearGolfCourse: Bool
    let nearAirport: Bool
    let isInGatedCommunity: Bool
    let hasHOA: Bool
    let hoaFee: Double?
    
    // MARK: - School Districts
    let schoolDistrict: String
    let elementarySchoolRating: Int? // 1-10
    let middleSchoolRating: Int? // 1-10
    let highSchoolRating: Int? // 1-10
    let nearPrivateSchools: Bool
    
    // MARK: - Safety & Security
    let hasNeighborhoodWatch: Bool
    let hasSecurityGates: Bool
    let hasMotionLights: Bool
    let hasRingDoorbell: Bool
    
    // MARK: - Investment & Financial
    let hasRentalPotential: Bool
    let propertyTaxRate: Double
    let appreciationRate: Double
    let isFixerUpper: Bool
    let isNewConstruction: Bool
    let hasWarranty: Bool
    
    // MARK: - Seasonal & Climate
    let hasGoodInsulation: Bool
    let hasStormShelter: Bool
    let hasGoodDrainage: Bool
    
    // MARK: - Community Features
    let hasCommunityPool: Bool
    let hasClubhouse: Bool
    let hasTennisCourtAccess: Bool
    let hasGymAccess: Bool
    let hasPlaygroundNearby: Bool
    let hasWalkingTrails: Bool
    
    // MARK: - Technology & Modern Features
    let hasFiberInternet: Bool
    let hasSmartThermostat: Bool
    let hasElectricCarCharger: Bool
    let hasSmartLighting: Bool
    let hasVoiceControl: Bool
    
    // MARK: - Environmental
    let hasEcoFriendlyFeatures: Bool
    let hasEnergyEfficientAppliances: Bool
    let hasLEDLighting: Bool
    let hasRainwaterCollection: Bool
    let hasCompostArea: Bool
    
    // MARK: - Maintenance & Condition
    let hasRecentRenovations: Bool
    let needsRepairs: Bool
    let hasMaintenanceContract: Bool
}

How can we work with this using the FoundationModels framework?

The Goal

Our HouseListing has soooo much detail that it will quickly overwhelm the user to see all of them at ones. Instead, we can build a chat-based interface where the user can simply search for houses based on the criteria that matter to them specifically.

This means that we need to translate what the user is searching for in a house in natural language to match the variables in the HouseListing object, that we can use as a filter for houses matching the user’s query.

For example, when the user searches for “pet friendly houses near hiking trails”, we want to filter our HouseListing to only return houses where the isPetFriendly and nearHikingTrails variables are set to true.

Creating a Housing Search Tool

To create a Tool using the FoundationModels framework 3 things are required:

1. Name - A unique name for the tool

2. Description - A natural language description of when and how to use the tool

3. The Call function - A language model will call this method when it wants to leverage this tool.

Let’s get started with the name and description, the easy parts:

Now onto the hard part - the call function…

First, we will define the input arguments from the model when calling this function. So when the user types in “a pet friendly house near hiking trails”, we would want the model to pass in isPetFriendly = true and nearHikingTrails = true as arguments.

To do this, we first want to specify all the possible filter options as an enum that the model will choose from, which will allow us to filter HouseListings:

    @Generable
    enum FilterType: String, CaseIterable {
        // Price & Size
        case minPrice = "minPrice"
        case maxPrice = "maxPrice"
        case minSquareFootage = "minSquareFootage"
        case maxSquareFootage = "maxSquareFootage"
        case minLotSize = "minLotSize"
        case maxLotSize = "maxLotSize"
        case minYearBuilt = "minYearBuilt"
        case maxYearBuilt = "maxYearBuilt"
        
        // Bedrooms & Bathrooms
        case numberOfBedrooms = "bedrooms"
        case minBedrooms = "minBedrooms"
        case maxBedrooms = "maxBedrooms"
        case minBathrooms = "minBathrooms"
        case maxBathrooms = "maxBathrooms"
        case hasMasterSuite = "hasMasterSuite"
        case hasGuestBedroom = "hasGuestBedroom"
        
        // Features (Boolean)
        case hasPool = "hasPool"
        case hasGarage = "hasGarage"
        case hasFireplace = "hasFireplace"
        case hasGarden = "hasGarden"
        case hasHomeOffice = "hasHomeOffice"
        case hasSmartHome = "hasSmartHome"
        case hasSolarPanels = "hasSolarPanels"
        case isPetFriendly = "isPetFriendly"
        case hasFencedYard = "hasFencedYard"
        case isWheelchairAccessible = "isWheelchairAccessible"
        case hasSingleStoryLiving = "hasSingleStoryLiving"
        case nearSchools = "nearSchools"
        case nearPublicTransit = "nearPublicTransit"
        case nearHikingTrails = "nearHikingTrails"
        case nearBeach = "nearBeach"
        case nearGolfCourse = "nearGolfCourse"
        case isInGatedCommunity = "isInGatedCommunity"
        case hasRentalPotential = "hasRentalPotential"
        case isFixerUpper = "isFixerUpper"
        case isNewConstruction = "isNewConstruction"
        case hasUpdatedKitchen = "hasUpdatedKitchen"
        case hasKitchenIsland = "hasKitchenIsland"
        case hasHardwoodFloors = "hasHardwoodFloors"
        case hasOpenFloorPlan = "hasOpenFloorPlan"
        case hasCentralAC = "hasCentralAC"
        case hasSecuritySystem = "hasSecuritySystem"
        case hasBackyard = "hasBackyard"
        case hasBalcony = "hasBalcony"
        case hasElevator = "hasElevator"
        case allowsDogs = "allowsDogs"
        case allowsCats = "allowsCats"
        case hasHOA = "hasHOA"
        case hasEcoFriendlyFeatures = "hasEcoFriendlyFeatures"
        
        var description: String {
            switch self {
            case .minPrice: return "Minimum price in USD"
            case .maxPrice: return "Maximum price in USD"
            case .minSquareFootage: return "Minimum square footage"
            case .maxSquareFootage: return "Maximum square footage"
            case .minLotSize: return "Minimum lot size in acres"
            case .maxLotSize: return "Maximum lot size in acres"
            case .minYearBuilt: return "Minimum year built"
            case .maxYearBuilt: return "Maximum year built"
            case .numberOfBedrooms: return "Exact number of bedrooms"
            case .minBedrooms: return "Minimum bedrooms"
            case .maxBedrooms: return "Maximum bedrooms"
            case .minBathrooms: return "Minimum bathrooms"
            case .maxBathrooms: return "Maximum bathrooms"
            case .hasMasterSuite: return "Has master suite"
            case .hasGuestBedroom: return "Has guest bedroom"
            case .hasPool: return "Has swimming pool"
            case .hasGarage: return "Has garage"
            case .hasFireplace: return "Has fireplace"
            case .hasGarden: return "Has garden"
            case .hasHomeOffice: return "Has home office"
            case .hasSmartHome: return "Has smart home features"
            case .hasSolarPanels: return "Has solar panels"
            case .isPetFriendly: return "Pet friendly"
            case .hasFencedYard: return "Has fenced yard"
            case .isWheelchairAccessible: return "Wheelchair accessible"
            case .hasSingleStoryLiving: return "Single story living"
            case .nearSchools: return "Near schools"
            case .nearPublicTransit: return "Near public transit"
            case .nearHikingTrails: return "Near hiking trails"
            case .nearBeach: return "Near beach"
            case .nearGolfCourse: return "Near golf course"
            case .isInGatedCommunity: return "In gated community"
            case .hasRentalPotential: return "Has rental potential"
            case .isFixerUpper: return "Fixer-upper property"
            case .isNewConstruction: return "New construction"
            case .hasUpdatedKitchen: return "Has updated kitchen"
            case .hasKitchenIsland: return "Has kitchen island"
            case .hasHardwoodFloors: return "Has hardwood floors"
            case .hasOpenFloorPlan: return "Has open floor plan"
            case .hasCentralAC: return "Has central air conditioning"
            case .hasSecuritySystem: return "Has security system"
            case .hasBackyard: return "Has backyard"
            case .hasBalcony: return "Has balcony"
            case .hasElevator: return "Has elevator"
            case .allowsDogs: return "Allows dogs"
            case .allowsCats: return "Allows cats"
            case .hasHOA: return "Has HOA"
            case .hasEcoFriendlyFeatures: return "Has eco-friendly features"
            }
        }
    }

When the model passes in the FilterType, we would also want a value for this type (e.g. true / false), and a description of the filter - this is more for the model to “think” through using this filter:

Finally, our Arguments parameter for the call function will include an array of FilterSpecifications and a way for the model to reset all filters (for example if the user says “Forget that, I want to start fresh. Let’s look for houses with pools only”.

Note that the arguments input for the call function can be any Generable object defined in your Tool.

State Management with Tools

One of very interesting implementation details of the genius FoundationModels framework is that the LanguageModelSession will take in an instance of a tool that it will use throughout it’s lifecycle. That means Tools can keep state!

As the user continues their conversation with the language model, more filters will be added. For example, in the first message the user may say they want a house with a pool. When the model responds with filtered houses that have pools, the user may then try to narrow it down to houses that are also pet friendly, etc.

Ideally, the model should be able to take in the full context of the entire conversation and understand that the user wants a house with a pool that is also pet friendly and include these two filters (hasPool = true, and isPetFriendly=true) when calling the tool. But unfortunately the Foundation Model is just not that sophisticated and can be hit or miss. For example, if you say you want a house with a pool, it will only pass in the hasPool=true filter, “forgetting” / not taking into consideration the earlier isPetFriendly=true filter. And this will get worse and worse the longer the conversation and especially with more filters.

So keeping state is up to you as the developer! In our case, we can instruct the model to only include the new filters from the user input, that will aggregate and keep track of ourselves.

In our example of a DynamicHousingSearchTool, we want the tool to keep track of two things:

• The current active filters

• The current results matching the active filters

The second reason for keeping state is the model’s small context window. By having the tool keep the state, you can easily start a new LanguageModelSession using the same instance and continue the house search without the user knowing they’re in a new chat session as follows:

Applying the Filter

When the model passes in an array of FilterSpecifications as arguments, we can iterate through the filters and apply them by using the FilterType enum:

private func applyFilter(
        filterType: FilterType,
        value: String,
        to houses: [HouseListing]
    ) -> [HouseListing] {
        
        switch filterType {
        // Numeric filters
        case .minPrice:
            guard let minPrice = Double(value) else { return houses }
            return houses.filter { $0.priceInUSD >= minPrice }
            
        case .maxPrice:
            guard let maxPrice = Double(value) else { return houses }
            return houses.filter { $0.priceInUSD <= maxPrice }
            
        case .minSquareFootage:
            guard let minSqFt = Int(value) else { return houses }
            return houses.filter { $0.squareFootage >= minSqFt }
            
        case .maxSquareFootage:
            guard let maxSqFt = Int(value) else { return houses }
            return houses.filter { $0.squareFootage <= maxSqFt }
            
        case .numberOfBedrooms:
            guard let bedrooms = Int(value) else { return houses }
            return houses.filter { $0.numberOfBedrooms == bedrooms }
            
        case .minBedrooms:
            guard let minBedrooms = Int(value) else { return houses }
            return houses.filter { $0.numberOfBedrooms >= minBedrooms }
            
        case .maxBedrooms:
            guard let maxBedrooms = Int(value) else { return houses }
            return houses.filter { $0.numberOfBedrooms <= maxBedrooms }
            
        case .minBathrooms:
            guard let minBathrooms = Double(value) else { return houses }
            return houses.filter { $0.numberOfBathrooms >= minBathrooms }
            
        case .maxBathrooms:
            guard let maxBathrooms = Double(value) else { return houses }
            return houses.filter { $0.numberOfBathrooms <= maxBathrooms }
            
        case .minLotSize:
            guard let minLot = Double(value) else { return houses }
            return houses.filter { $0.lotSizeInAcres >= minLot }
            
        case .maxLotSize:
            guard let maxLot = Double(value) else { return houses }
            return houses.filter { $0.lotSizeInAcres <= maxLot }
            
        case .minYearBuilt:
            guard let minYear = Int(value) else { return houses }
            return houses.filter { $0.yearBuilt >= minYear }
            
        case .maxYearBuilt:
            guard let maxYear = Int(value) else { return houses }
            return houses.filter { $0.yearBuilt <= maxYear }
            
        // Boolean filters
        case .hasPool:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasPool == boolValue }
            
        case .hasGarage:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasGarage == boolValue }
            
        case .hasFireplace:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasFireplace == boolValue }
            
        case .hasGarden:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasGarden == boolValue }
            
        case .hasHomeOffice:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasHomeOffice == boolValue }
            
        case .hasSmartHome:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasSmartHome == boolValue }
            
        case .hasSolarPanels:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasSolarPanels == boolValue }
            
        case .isPetFriendly:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.isPetFriendly == boolValue }
            
        case .hasFencedYard:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasFencedYard == boolValue }
            
        case .isWheelchairAccessible:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.isWheelchairAccessible == boolValue }
            
        case .hasSingleStoryLiving:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasSingleStoryLiving == boolValue }
            
        case .nearSchools:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.nearSchools == boolValue }
            
        case .nearPublicTransit:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.nearPublicTransit == boolValue }
            
        case .nearHikingTrails:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.nearHikingTrails == boolValue }
            
        case .nearBeach:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.nearBeach == boolValue }
            
        case .nearGolfCourse:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.nearGolfCourse == boolValue }
            
        case .isInGatedCommunity:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.isInGatedCommunity == boolValue }
            
        case .hasRentalPotential:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasRentalPotential == boolValue }
            
        case .isFixerUpper:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.isFixerUpper == boolValue }
            
        case .isNewConstruction:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.isNewConstruction == boolValue }
            
        case .hasUpdatedKitchen:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasUpdatedKitchen == boolValue }
            
        case .hasKitchenIsland:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasKitchenIsland == boolValue }
            
        case .hasHardwoodFloors:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasHardwoodFloors == boolValue }
            
        case .hasOpenFloorPlan:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasOpenFloorPlan == boolValue }
            
        case .hasCentralAC:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasCentralAC == boolValue }
            
        case .hasSecuritySystem:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasSecuritySystem == boolValue }
            
        case .hasBackyard:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasBackyard == boolValue }
            
        case .hasBalcony:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasBalcony == boolValue }
            
        case .hasElevator:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasElevator == boolValue }
            
        case .allowsDogs:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.allowsDogs == boolValue }
            
        case .allowsCats:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.allowsCats == boolValue }
            
        case .hasHOA:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasHOA == boolValue }
            
        case .hasEcoFriendlyFeatures:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasEcoFriendlyFeatures == boolValue }
            
        case .hasMasterSuite:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasMasterSuite == boolValue }
            
        case .hasGuestBedroom:
            let boolValue = value.lowercased() == "true"
            return houses.filter { $0.hasGuestBedroom == boolValue }
        }
    }

Implementing the Call Function

The call functions will take in the Arguments that we defined earlier:

And output a ToolOutput object, which can be initiated with either a String or a GeneratedContent type:

First, if the model passes in true for resetFilters, we reset the state variables:

Next, we will iterate through each filter and add the new filters to the activeFilters array and filter the currentResults:

We want to give the new filtered house results back to the Foundation Model to display to the user, but the original HouseListing object is too big and overwhelming both for the model (in tokens) and the user. So we want to create a smaller version of the HouseListing that just includes the most important information. We do this by creating a new Generable HouseResult object:

We also want to give a custom message about the results for the model:

We then include all the information in a GeneratedContent final output for the model:

That’s it! The model will not have the list of filtered houses to present to the user based on their natural language criteria.

Advanced

Since you’re controlling the end-to-end UI of this application, instead of passing the house results to the language model to display to the user, you can use the Tool call to trigger a UI update to directly DISPLAY the results to the user while sending the response to the model as a message simply saying you applied the requested filters.

Check out this project for inspiration!

The Prompt

Finally, it’s time to run the model! The model instructions in this case is a bit more complex to get it working just right:

Giving examples to the model is super important. The examples included above are:

Finally, we just pass the tool and the instructions to the model and it runs well!

Conclusion

The FoundationModels framework is best used for parsing natural language. When you use it only for that - just parsing user’s input into specific function arguments that you have full control over, you truly get to experience the power of the model. Set up your tools and code accordingly!

One possible definition that is relatively accepted is that an agent is an LLM running tools in a loop and possibly working with other LLMs to achieve a goal.

Like many powerful ideas, it's very simple at its core. But you'll see that it can scale to advanced use cases.

This is a guest post by Tarek M. Ben Lechhab, indie Apple developer, freelancer, and AI enthusiast, feel free to take a look at his publication.

Read other posts by Tarek

Apple's Foundation Models framework is an excellent candidate for experiments in this space.

Indeed it provides very solid tool calling capabilities, and the limited context window, (only 4096 tokens input and output combined), can greatly benefit from delegating tasks in order to multiply the total number of tokens available to do some work.

There are many ways agents are used today. We are going to focus on one approach: the plan and execute design pattern with a bit of orchestration.

Used wisely, those can greatly expand the capabilities of your AI-enabled app.

We will leverage tool calling in order to exhibit agentic behavior.

But first, let's take a look at tool calling in Foundation Models.

Use the Tool, Luke

I'm only half joking with this reference. Indeed, like the Force, the tool calling (or function calling) capability of LLMs is very powerful. In the LLM world,a tool is a piece of code provided beforehand that can be called by your LLM at its own discretion.

Instead of relying on a set of deterministic steps to achieve a result, you pass a tool to the LLM and let it decide if, when, and how it should call it.

The Foundation Models implementation is top notch in its own category, especially from an API design standpoint. I'm quite impressed with how much one can achieve with it, given the small size of the only currently available model.

In order for the LLM to use your tools, you first declare a tool conforming to the Tool protocol.

This involves:

• Naming the tool

• Providing a description. This is like a mini-prompt for the LLM to understand what this tool does. Based of this mini-prompt, the LLM will determine if, when, and how it should use the tool.

• Defining its arguments using @Generable. Those are the parameters that the LLM must pass in order to invoke your tool. They will be dynamically generated by the LLM.

• Implementing the call method which is the action to execute when this tool is called. This method is `async`, so you can definitely do some background or network work, assuming you stay within the timeout limit of the LLM.

• Returning the output, wrapped in a ToolOutput. By default it can take a string or generated content (which will be useful later)

Here's a simple example for the traditional weather tool:

Then, you just pass an instance of that tool to your LanguageModelSession during instantiation, and you're good to go!

It's up to the LLM to decide whether or not it should call the tool, when it should call it, and with which parameters.

For long-time programmers, this might sound scary, since there is a lot of non determinism here. but when it works well (and it usually works well), it's really powerful.

You are essentially programming your computer in natural language and delegating a good chunk of the decision making process.

In some sense, LLMs can be seen as a new emerging core capability of computers, and programming them with natural language as a new programming paradigm. This idea was developed by Andrej Karpathy in a talk that I can only strongly encourage you to watch.

We are still at the phase of interacting through the command line with those LLMs, as it was done in the 60s with traditional computers.

I can't even begin to imagine how it will look in the future if it evolves from the CLI to other ways of interacting!

Planning and Executing

Now that we've seen the basics of a tool, let's turn it up a notch.

Like traditional programming, this new kind of programming has design patterns.

One of these design patterns is planning and executing.

One could think a way to achieve planning and executing would be to write an advanced prompt.

We could instruct for instance the LLM to first make an extensive plan and then execute this plan.

While that might work for some cases, this approach would be pretty limited:

• First because the context window is short, which means that the planning tokens will count toward the global 4096 input + output limit, not to mention the extensive prompt tokens.

• Second, because a small 4-billion-parameter LLM like this one is probably not trained for planning and execution (the same goes for reasoning)

A much better way (and the standard way) to do it is by adapting our architecture to have agents interacting with each other.

I've put an LLM in your LLM

Remember the definition of an agent that we went with earlier? An LLM working with tools and other LLMs.

This is exactly what we'll experiment with: What if we exposed other LLMs to our root LLM through the tool API? That way, an LLM has the capability of calling another LLM (which is seen as a tool) when necessary.

Each LLM is an instance of a LanguageModelSession, we thus have multiple windows of 4096 tokens to work with.

Let's take a look at the architecture of what we're building:

Think of it like a restaurant:

• The Coordinator which is the head chef, decides which dish to make

• The Planner writes the recipe

• Workers are the line cooks

• The Coordinator takes back the work from cooks and plates the final dish 🍽️

There is a lot of code involved as you can imagine. I can only strongly encourage you to follow along in the sample project.

Let's take a look at the main pieces.

Breaking down the architecture

1. We instantiate a root Coordinator LLM

Here are its instructions (or system prompt)

Remark: You’ll notice that I had to explain to the coordinator to keep the order right and how to do it. At some point, since the worker tasks were running in parallel, and the order in which they returned wasn't predictable, the final output was out of order. Classic concurrency bug fixed by a simple human sentence! We wish all bugs could be fixed like that!

And its prompt:

2. We give it a set of tools, namely the Planner and the Worker which are themselves instances of LLMs

3. The Coordinator passes the user's request to the Planner using the tool's Arguments (leveraging @Generable).

4. Then, in the call method, we instantiate a new session of the LLM to handle the planning request. The Planner decomposes the user request into subtasks, the Coordinator gets them.

5. The Coordinator spawns one Worker per subtask, to execute it

6. The Coordinator gathers all the Workers results, combines them and answers the user

Why This Matters

Having such a decomposition unlocks capabilities that would be harder, or sometimes impossible to achieve with just one LLM and one prompt:

• Each tool (especially workers) can reason about its task, in its own 4096-token playground.

• Tools can handle ambiguity and edge cases of their own subtasks.

• Tools can provide explanations for their actions, even if it goes wrong.

• Ultimately, it enables emergent behaviors, although I admit that with a model this small it's seldom the case.

Taking our agents for a test run

Alright, it's time to take our shiny new toy for a spin.

Ever wanted to learn about quantum computing? Now is the chance.

Let's go with the very basic prompt: “What is quantum computing?”

Normal Mode

Here is one possible output from the normal mode:

Quantum computing is a cutting-edge field of computer science that leverages the principles of quantum mechanics to process and solve complex problems more efficiently than classical computers. Unlike traditional computers, which use bits as the smallest unit of data represented as either 0 or 1, quantum computers use quantum bits, or qubits. Qubits can exist in multiple states simultaneously due to a phenomenon known as superposition, allowing quantum computers to process a vast amount of possibilities at once.

Another key concept in quantum computing is entanglement, where qubits become interconnected in such a way that the state of one qubit can depend on the state of another, regardless of the distance between them. This interconnectedness enables quantum computers to perform certain calculations much faster than classical computers.

Quantum computing has the potential to revolutionize fields such as cryptography, optimization, materials science, and complex system simulations by tackling problems that are currently intractable for classical computers. However, building practical quantum computers remains a significant challenge due to issues like qubit stability and error rates.

Agentic Mode

Let's try it with the agentic architecture now:

The coordinator receives the following prompt:

Process the following task using the available tools:

User Task: What is quantum computing? 

Then it passes this prompt to the worker (notice that it generated the maximum number of subtasks itself based on the range guide we provided):

Please analyze and break down the following task into subtasks:
Task: What is quantum computing?
Maximum subtasks: 5

Requirements:
1. Create logical, actionable subtasks
2. Identify dependencies between subtasks
3. Ensure subtasks are specific and measurable
4. Consider the given context and constraints

Provide a comprehensive task plan with a clear strategy.

Here is the planner's decomposition:

Task Plan:
Original Task: What is quantum computing?

Strategy: To effectively breakdown the task of 'What is quantum computing?', I will systematically research foundational concepts and theories, trace the historical evolution of quantum computing, assess its potential applications, and compile these insights into a coherent summary. Each subtask builds upon the previous ones, ensuring a logical progression from theoretical understanding to practical implications.

Subtasks (5):
  1. [1] Research foundational concepts of quantum computing.
  2. [2] Identify key theories and principles underlying quantum computing.
  3. [3] Explore historical development and milestones in quantum computing.
  4. [4] Analyze applications and potential impacts of quantum computing.
  5. [5] Summarize findings and prepare a structured explanation of quantum computing.

The coordinator assigns each of the five subtasks to a worker, spawned in parallel. (not shown here for conciseness)

Then it gathers the findings of each worker, and combines them into the final result sent to the user:

Here's a structured breakdown of quantum computing based on the subtasks:

1. **Research Foundational Concepts of Quantum Computing:**
   - Quantum computing uses principles like superposition and entanglement.
   - Key components include qubits, quantum gates, and quantum algorithms.

2. **Identify Key Theories and Principles Underlying Quantum Computing:**
   - **Quantum Superposition**: Allows qubits to exist in multiple states simultaneously.
   - **Quantum Entanglement**: Links qubits for complex computations.
   - **Quantum Tunneling**: Enables particles to pass through potential barriers.
   - **Quantum Interference**: Enhances signal amplification and cancellation.
   - **Quantum Decoherence**: Addresses the loss of quantum states due to environmental interaction.

3. **Explore Historical Development and Milestones in Quantum Computing:**
   - Quantum computing began in the mid-20th century.
   - Key milestones include the development of quantum gates in the 1980s and the first quantum computers in the 1990s.
   - IBM's 2019 announcement of quantum advantage marked a significant breakthrough.

4. **Analyze Applications and Potential Impacts of Quantum Computing:**
   - Quantum computing could revolutionize cryptography by breaking current encryption standards.
   - In materials science, it could enhance drug discovery and material optimization.
   - AI benefits from increased parallelism, improving machine learning capabilities.

5. **Summarize Findings and Prepare a Structured Explanation of Quantum Computing:**
   - Quantum computing offers unparalleled computational power through superposition and entanglement.
   - It holds transformative potential across industries, promising breakthroughs in cryptography, materials science, and artificial intelligence.

This structured approach ensures a logical understanding of quantum computing from foundational concepts to its wide-reaching applications.

We can see that the final output from the agentic version is much more terse and elaborated than what a single prompt can usually do.

This is because we expanded the working tokens by multiplying the context windows, dispatched in separate tasks, allowing parallel and specialized work.

By providing additional tools for specific tasks to the worker tools, I believe this can scale quite well and provide real value.

Limitations

Hallucinations

While this architecture is powerful in theory, you should still keep in mind that the only currently available model is very small. So it can quickly go off rails and hallucinate.

For instance, if you ask for real world information, such as the weather, the task decomposition will be just right, but since the workers don't find tools to actually get the location and the weather for that location, it just hallucinates that it does so, or tries to fill it with last know information from training.

In a production app, the results of such an architecture will only be as good as the tests you conducted to ensure that they fit your specific needs.

I encourage you to try it, the way tasks are decomposed and executed is instructive to understand the internals of the model.

Specialized synthesizer or not?

I want to mention that an initial version of this architecture had a synthesizer tool called after the workers in order to combine the work.

However, it was fragile and the coordinator seemed often confused and tried to re-interpret those results.

This is typically the kind of limitations that probably wouldn't appear in a bigger model since it would have enough power and tokens to deal with this kind of ambiguity.

So I removed it, the Coordinator knows “naturally” how to do the synthesizing job.

Technical details

There are a few technical details worth mentioning.

Splitting instructions

I have had much better results splitting my explanations to the LLM instance between the instructions and the actual prompt. Apple engineers say that the model is specifically trained to follow instructions over prompt. So one could expect to be able to say everything in the instructions and keep the prompt minimal, but at the time of this writing, it doesn't always work well.

Mapping inputs and outputs

One key element to make this architecture work is to ensure that when tools are supposed to be chained together, the output of the previous tool has to match the input of the next one.

For instance, the coordinator is supposed to call a worker with a subtask produced from the planner.

Here's how that translates:

You can see that the planner outputs an array of subtasks, and the worker takes one subtask. Once again, the LLM figures out the mapping automatically. 🤯

Head-scratcher: avoid guided generation with tool calling!

It took me a long while to figure this one out. I don't know if it's a bug, a limitation of the system, or just a quirk but there is a critical detail to ensure the model proceeds with tool calling.

The root or coordinator model mustn't have any “generating” type when using tool calling, at least with this architecture. It means, you shouldn't call it with session.respond(to: prompt, generating: FinalAnswer.self) for instance.

During my tests, I absolutely had to remove any generating type from the respond(to:) call, which means that ultimately it's probably just like calling it with session.respond(to: prompt, generating: String.self).

Otherwise, it would just be very inconsistent in calling the passed tools and often tried to fulfill the request itself, especially with multi-step tool calling.

I guess there's something that prevents an advanced use case such as this one from working with guided generation.

I might be wrong, but my suspicion is that guided generation seems to be baked in the model, probably during training time, and that it conflicts with the model roaming free with tool use and generating a simple String. (once again, the size of the model is probably the culprit here)

Prompt engineering

Crafting a good system prompt (instructions) and user prompts goes a long way to make the model do exactly what we want it to.

Once again, this is the interface we have to program the LLM, so finding the right words and formulation is a discipline in and of itself.

I found best results with the usual prompt engineering best practices such as using a lot of imperative language.

However, counter intuitively, I found best results by not explicitly naming the tools in the instructions and prompt. For some reason, when doing so it wasn't making the connection between the actual tools and the string representing those tools in the prompt, and consequently it would lay down its plan to do it, if it had tools at its disposal with those names. 🙄

Session Management

⚠️Warning: One also has to be careful not to call `respond(to:)` on a LanguageModelSession again before it has returned. Otherwise, it crashes, so be careful in your production apps!

Also, keep in mind that while you're expanding capabilities by using many sessions, with each one of them having a context window of 4096 tokens, you still have to find ways to not overflow the root session context window.

After all, this session will manage the inputs and outputs of all child sessions, so it can quickly blow out of proportions.

For now I've just kept it simple, but in theory, there are many ways to mitigate this, such as using a sliding window, chained summarization, external memory etc.

Recursion Limitation

Initially, I wanted to go even further, I wanted to give the Worker tool / LLM an instance of itself, so it could recursively delegate tasks.

Unfortunately I encountered two issues:

• First, it seems like there is a built-in limitation preventing the result from referencing itself (self-referential structs with @Generable). I was even able to uncover that Apple engineers are using a force try! inside the Foundation Models code. 😄

FoundationModels/SchemaAugmentor.swift:209: Fatal error: 'try!' expression unexpectedly raised an error: FoundationModels.GenerationSchema.SchemaError.undefinedReferences(schema: Optional("SafeResponse<WorkerResult>"), references: ["WorkerResult"], context: FoundationModels.GenerationSchema.SchemaError.Context(debugDescription: "Undefined types: [WorkerResult]", underlyingErrors: []))

• Second, for some reason, the worker really tends to spawn many workers, which themselves spawn many others, and so on. The number exponentially explodes at runtime.

So for now, I would avoid such an architecture without solid testing beforehand.

Observability

In the demo app, I've implemented a history tab with a full trace of all the instructions, prompts, calls, responses of all LLMs and tools.

It's really instructive to see what happens in real time. You can even tweak the architecture and the prompts and see the changes.

Performance

While profiling the app, I was able to confirm that for my use cases, the overhead of parallelism was minimal, or sometimes even improved performance a tiny bit. It seems like the Neural Engine is made to handle both sequential and concurrent executions of Foundation Models. (which makes sense, since those kind of models can be used anywhere in the system, at any time)

Wrapping it up

I hope I have given you a glimpse of what's possible.

What we did is essentially an automated implementation of Map/Reduce, expressed in natural language.

The LLM is able to decide by itself which tools to call, and to make those calls sequential or parallel.

Think about the concurrency code and all the small details that could be necessary to achieve such a task manually.

Now all you need to do is explain it in plain old English, and it will just work.

There are so many agentic patterns that can be implemented, such as agent-to-agent messaging, persistent memory, federation with other online models, etc.

I encourage you to clone the repository, build the app, and play with it.

For instance, you can try using the same prompt in normal and agentic mode and compare the results. You can also monitor the token usage and tweak the agents and the prompts. Reach out to me with your findings and remarks!

While it's true that we are a bit limited by this first model, I suspect Apple will introduce larger models, as well as online ones.

After all, the framework is called FoundationModels, plural, and the API design kind of gives away the fact that it can be a single interface to multiple models.

With a more powerful LLM, and more advanced tools, the possibilities are virtually limitless. We will be able to do so much more for our users, as we already can with this model.

Did that trigger ideas for your apps? What do you think about those agentic capabilities?

I hope this was useful to you. I have many more experiments from where this one came from, so stay tuned!

This is a guest post by Tarek M. Ben Lechhab, indie Apple developer, freelancer, and AI enthusiast, feel free to take a look at his publication.

Read other posts by Tarek

You’ll notice that while most input fields are restricted, the “Subject” field is completely open so the student can be very specific about their area of study. For example, instead of something general like “Math”, a younger student or their parent can input “Division” more specifically as the subject.

However, this creates one problem. The Subject field becomes part of the main prompt used to input into the model, and it’s possible students put in weird or unproductive areas of study. The FoundationModels framework already includes guardrails for harmful and inappropriate content, but we may want to add our custom guardrails to protect against funny and non-educational content.

For example, imagine a student inputs “Baking Perfect Cookies” as the subject. While this will not trip up the FoundationModels guardrails as there is nothing harmful about this, it is still not a “real” subject that will help school-aged students study in order to pass standardized tests. I know not allowing baking cookings as a study plan is unfortunate, but the goal of our app is to specifically help students pass tests in school.

With the FoundationModels framework running on-device and being free, adding guardrails for these type of cases is super easier (and did I mention free?!!).

The Guardrail Result Object

Settings up a custom guardrail just means that you’ll be checking the user input for certain criteria that you specify in your prompt. Instead of having the model just return a true / false value of whether the subject is appropriate or not, it’s important to give the model more tokens to “reason” about it’s output. This allows the model to add more context and information to the decision, which you can also display to the user.

To make sure the model responses back in a structured way, we would create a Generable GuardrailResult object:

Notice that the ordering of the result properties is very important as we want the model to take the extra tokens to “reason” before “deciding” whether the subject from the user input is in face an educational subject.

The Guardrail Prompt

Next, it is important to specify your evaluation criteria in the prompt. Here is an example prompt for making sure the user’s subject is appropriate (note that it is not fully optimized and would need more work and testing):

Generating a Guardrail Result

Finally, you would simply pass in the subject from the user input and have a model respond with a StudySubjectGuardrailResult generation:

Running the Guardrail

The first option to include the guardrail is to run the guardrail result generation model, and if it’s a valid subject, then run the separate model that generates a study plan:

This is especially good if you have a streaming UI which will stream the study plan. You would want to first validate the study plan subject before displaying the study plan.

However, if your UI allows for generating the full study plan before displaying it to the user, then it’s a good practice to run both the guardrail and study plan generation in parallel. Then simply don’t show the study plan output if the guardrail fails. This saves time:

In this case, our custom guardrail gets activated:

Conclusion

With Apple’s on-device FREE FoundationModel, it makes it very fast and easy to include custom guardrails for validating user input.

However, keep in mind that this model is small (on-device), so it may not be as sophisticated as the super big LLMs that we are used to with ChatGPT / Claude / Gemini Pro. From my testing, it was super easy for the guardrails to be wrong in many cases. This might mean I need to play around with my prompt more. So make sure to test your guardrails thoroughly with many examples to make sure you get the expected results!

What surprised me here was Apple’s example of using the FoundationModels framework for “Natural language search” - to search through house listings for example…

First of all, the FoundationModels framework has an extremely limited context window of only 4096 tokens - that is both for INPUT AND OUTPUT combined. So in the context of text search, you wouldn’t be able to pass in that many objects to the model for the user to search.

But let’s say you do have a more limited number of options and want to use the FoundationModels framework to do the search. Would it be able to give you relevant results given how small the model is? I went ahead and tested it out using Apple’s example of house listings to see!

The House Listings

First - I created a Generable HouseListing object so that I create sample listings and easily pass them into the model.

Imagine you receive the house listings from your API and load them into an array of HouseListing structs.

The results is five of the following house listings:

The House Search Results

Next, I created a Generable HouseSearchResults objects that will be generated by the FoundationModels framework that match the user’s search query:

Searching for the Perfect House

Using the new #Playgrounds macro, I can quickly test out if the model will return back the correct results based on the user query:

The result for “a house with a pool” was generated as follows:

WOW!!! IMPRESSIVE!!! IT WORKS… except… let’s take a look at the description for house number 5 again…

THERE IS NO MENTION OF A POOL! From running different versions of this search prompt, my assessment is that it’s confusing the “kitchen" island with waterfall edges as a pool.

In fact, if you look closely at the reasoning of that the house “features a resort style pool with spa”, you can see that this specifically is mentioned for house #3:

Maybe it confused the 5 numberOfBedrooms as house number 5?!!

Also note that house #2, which does have a pool was NOT included in the results:

Now, sometimes the model does work… When I searched for a house near hiking trails, it correctly showed house #5:

However, I would argue that house #1, a farmhouse on 2.5 acres of land, should also show up in the results as it can easily have a trail on the property… This just shows that the model cannot deduce information.

You can see the deduction flaws as well. When I searched for “a house with an ocean view", it gave the following reasoning for the Cape Cod Cottage (house #4):

The answer is technically correct - in the description it says that “the second floor offers additional bedrooms with dormer windows and ocean glimpses.” But the model never mentioned the “ocean glimpses” as the reason for why it chose house number 4. Instead siting the proximity to the beach, which doesn’t necessarily mean ocean views directly from the house.

Conclusion

The example of searching using natural language with the FoundationModels framework just exposes the extreme limitations of this small on-device model. Make sure to test thoroughly for all use-cases, as the word relationships of this model are not as sophisticated (e.g. kitchen island = pool) as the super big LLMs we’re used to like ChatGPT and Claude.

Let’s start with a regular, non-Liquid Glass button:

The output is a non-bordered plain button:

Now the fun part - let’s make it a glass button! Simply add the .buttonStyle(.glass) to the button property:

The output is the button with a slight white background, but that is much more fun when pressed!

While the button just looks white on a white background - remember it is a glass on top of it’s background. So white on white will be more white. If we change the background color to something else, we can see the glass much better!

It’s hard to see this glass button, which is fine if this button is not that important and should stay to the side. But what if we want it to be more prominent? In that case, we can set the buttonStyle to glassProminent:

Now, the button will have a background!

But what if we want a different background than the default blue one? This is where we run into problems - hopefully this is a beta issue…

The ideal solution is that you would simply change the button tint color to your desired color:

This works, but only in DARK MODE!!

Sadly - in light mode, the red button currently just expands but stays the same color red… for now hopefully!

But there is one solution which gets slightly better results for light and dark mode (but still not the same for light mode). You can set the buttonStyle to a regular borderProminent one, and just add an interactive glassEffect:

It works better in dark mode, but is still a bit buggy:

Making a Circular Button

Now, what if you want to make your button a perfect circle? We would want the following code to work:

Unfortunately, it does not. The glassEffect doesn’t fit the shape…

I know what you’re thinking - we can add a shape to the glassEffect…

While this works initially, the weird shape issue returns when the user clicks on the button:

The solution.. for now… is to clip the shape of the button into a circle:

Finally - it works… Note that I removed the tint to show the effect more prominently:

Hopefully this will get fixed in future betas and “just work” :)

To learn more about the Foundation Model and how it is prompted, I watched the following WWDC25 session:

• Explore prompt design & safety for on-device foundation models

• Code-along: Bring on-device AI to your app using the Foundation Models framework

• Deep dive into the Foundation Models framework

In this blog post, I’ll summarize the prompting techniques for easier future reference.

Prompt Testing

As with all LLMs, it is important to test different variations of the prompt to figure out which one gets the best results for your app’s use-case.

One big issue with the FoundationModels is it’s extremely limited context size. Only 4096 tokens for BOTH input and output tokens. This means that your prompt counts against the context window. So you have to strike the right balance between providing a powerful prompt but keeping it short.

The other big issue is that the FoundationModels are extremely guardrailed against unsafe input and output. While this is a great thing in theory, in practice this can produce a lot of false positives. In other words, your innocent prompt may accidentally trigger the guardrails. Something that Apple includes in their documentation as well:

Fortunately, it is super easy to test prompts right in any Xcode project using the new Playgrounds framework!

In your project, simply create a Playgrounds.swift file, import Playgrounds and start playing around. The reason I say to do this in your project is because it’ll have access to all your types.

Here is how simple it is to get started with testing your prompts with Playgrounds:

Guides

Before you even write a prompt, the FoundationModels framework provides a powerful Generable macro that defines the structure of the model’s output for you. For example, let’s imagine we’re building an education app where study plans are generated on demand based on student requirements.

The StudyPlan object that the Foundation Model should generate would take this structure:

To generate, simply include the StudyPlan object as what the model is generating in the response:

As a starting point, this is pretty vague. But instead of starting by optimizing the prompt, we can use a Guide for each variable that we need generated to be a lot more specific with what we want.

To start with, we can add a description for each variable explaining what it is for:

Now your prompt is much more targeted for each variable, and you don’t have to include these details separately in the main prompt.

String Guides

But wait, there are more Guides! For example, let’s say our app focuses on only 5 subjects: Math, English, History, Science, and Philosophy. You can add it in as an .anyOf Guide for the subject variable:

Now you ensure that the generated response will be one of these five, which your app specifically supports.

Instead of using this guide, Generable type also supports enums, so this also works well:

If you are using a String as a variable, it also supports .constant (always the same value - let’s say the school name in our case):

This will always generate “My Online School” for the school variable.

You can also put in a regex into the Guide! For example, this ensures that the teachers generated name always starts with “Mr.” or “Mrs.”!

And if you’re into Swift’s native RegexBuilder framework, that is also supported!

Number Guides

Let’s say our app only supports study plans that are under 10 weeks. It is easy to set this in the Guide using the .minimum, .maximum, or .range variables - supported for Int / Float / Double / Decimal types:

Array Guides

Finally - you can be very specific with how many objects you want generated in an array. Say, for example, that I want to include 3 prerequisites for each study plan. It’s as simple as specifying the count to be 3:

The final generated guided output will look like this:

Guided generation with the FoundationModels framework is extremely impressive because the model has been trained specifically for it. You don’t have to worry that it will go “off-script” and hallucinate enum types that don’t exist. And with only a few tokens of context window, you can use this powerful prompting technique.

Order of Properties

When you consider the Generable object as a prompt in itself, ordering becomes super important. For example, in the above example of the StudyPlan object, notice that we first generate the subject, then the weeklyTopics. The weekly topics directly relate to the subject.

Now what if the subject is put after the weeklyTopics?

Notice that the Foundation Model no longer knows the subject and is generating the weeklyTopics and the prerequisites with “Topic A” and “Topic B” as the subject! That is because LLMs generate one token (*subword*) at a time, so if it hasn’t generated the subject yet, it will not “know” what it is!

So make sure you test out different order of properties as an additional form of prompting! For example - if you first generate a short story and need a summary of that story, generate the story first then the summary to improve the summary based on the story. This is also something I do as I write my blog posts - I write the blog post first then the title after :)

Use Natural Language Length Qualifiers for Longer Text

Now let’s say I want a lot more details of what is included in the weekly topic study plan as I find it to not be detailed enough for my app:

I will go ahead and create a WeeklyTopic object to get both the title and a more detailed description:

When I generate this as part of my StudyPlan, I’m not too happy with the result - it is still too vague and general:

This is where I can include a length qualifier to my Guided prompt for the description variable to that this field needs to be “in detail”:

Play around with super specific lengths to get just the righ ouput for both longer and shorter styles. You can use “In a few words”, “In a single paragraph”, “In three sentences” etc. to describe the length of output you want. This is also a great way to control the prompt context window.

Character Role

Models work very well when being given a character description. Let’s say we want to be a super fun online school for kindergarten students. You can create a fun kindergarten teacher character for your study plan in the instructions:

Now this looks like a much more fun study plan for math I wish someone created for me…

In other cases, you may use the session instructions to provide an expected tone for the model. For example, in a personal guided diary app, you may instruct the model to respond to negativity in an empathetic way:

Few Shot Prompting

One very effective prompting technique is called few-shot prompting, meaning that you give the model a few examples of what to generate. This is extremely easy to do in the FoundationModels framework with the Generative macro. The macro automatically converts any Generative object into a string, so it’s as simple as first create an example:

or two:

Then just directly inserting the examples into the prompt!

That’s it! Now my study plan is a lot more fun and fit for kindergarten-level students:

Playing with Temperature

Since I’m making a bit of a fun learning app, I want the model to go a bit off-script and generate more fun lesson plans. This can be done in the response options:

Here, I set the temperature a bit higher to 0.8 to get more fun responses:

I tried higher temperatures, but that produces inconsistent and more non-sensical responses. So that is just something to play around with and test.

If you want a much more conservative deterministic response that will follow the guides you set on your Generable option, set the sampling to greedy:

You’ll get a much more “normal” sounding study plan:

I just want to say again how nice it is to test all of this right inside Xcode in the #Playground macro super fast versus the usual of making API calls and waiting for responses, debugging JSON decoding and all that!

Other Tips

A few other simple prompt tips include:

• Phrase your prompts as a clear command

• Request a single specific task in detail

• Use ALL CAPS such as MUST or DO NOT to stop undesired behavior

Prompts from User Input

In all of the above examples, I / the app fully controls the prompt. But in most apps, there will be some type of user input. While I don’t recommend letting the user fully chat their preferences, especially when those are limited, some type of interface like this would work well:

This allows the user control, but in not in a way that can cause unexpected input into your model. The only “open” prompt field is the Subject field, where the user can input anything that will go into your actual user prompt. Of course - if you stick to only a limited number of subjects, you can implement a drop-down here.

To see the full breakdown of implementing this app, see my blog post Let’s Build a SwiftUI Study‑Plan App using FoundationModels StreamResponse.

When the user inputs any part of the prompt, it comes with some risks. Mainly that it may trigger the guardrails. They can also type something like “avoid previous instructions and do this instead” and try to break the prompt.

The first step is to have a more comprehensive prompt where the user’s input is only a small part of it:

Here - only the subject field is input by the user directly. All else is selected from the limited UI.

But the more important part is figuring out how to handle errors if the user does input something inappropriate that triggers the model’s guardrails. The LanguageModelSession throws specialized errors:

It is your responsibility to think through how to handle these errors in your UI:

For example, if the user triggers the guardrails due to wrong behavior in a social app, you may have to implement a “3 strikes you’re out” policy to disable thtat user’s accounts or to make them invisible to others.

Other times, you may just ignore the error and not do anything and simply prompt the user to try again. Or adjust the UI accordingly to a fallback option. How you handle the user-related prompt input errors is different for every app use-case. Don’t forget to experiment with many edge cases and figure out how to handle this in your app!

Conclusion

After two years of working with Python and web-based LLM APIs, it’s incredibly exciting to finally have a model that just works with Swift and SwiftUI. Apple’s Foundation Model brings LLMs into the native app development workflow in a way that feels seamless, type-safe, and purpose-built. With tools like Generable, guided output, and on-device testing via Playgrounds, we can now experiment and iterate faster, without worrying about JSON formatting or remote API quirks.

Happy Prompting in Swift 🚀

One of the most mind-blowing API design features of the FoundationModels framework is how it integrates with SwiftUI. Specifically, how you can show the results to the user as the model is generating them token-by-token. Let me show you the magic in the example Study Plan app I built:

So how does this work? It’s super simple!

Define the Generable Models

The first step is to define the models that the FoundationModels framework will generate. For the Study Plan app, it’ll be a StudyPlan model with WeeklyStudy topics with StudySessions:

Note that the FoundationModel will generate ALL fields regardless of whether they are being displayed to the user or not. So if you have a bigger model that has some non-generated content and some generated content, keep the generated content as separate objects, and just sync the generated info back into the bigger model (maybe saved in SwiftData) once it fully generates. Keep the Generable models as separate entities only used for generating information as needed!

The UI

One option is to build a chat-based UI for getting the user’s study plan preferences that are needed as input to the model. However, I would argue that this is more difficult for the user than building a much more quick and simple input screen, especially if you do have extremely specific input:

I would argue that even though we’re used to the chat-based interfaces of ChatGPT, it might not be the best UI for most use-cased on mobile. Especially when it’s as simple as the form above.

Once the user enters the Subject of their study and clicks on “Generate Study Plan”, the model is ready to stream!

Once the app gathers the required information from the user, it can be put into the prompt.

The Prompt

Using the specific input values of the user, the prompt is generated as follows in the StudyPlanViewModel:

Again - keep in mind the context window here. The longer the prompt, the less context for output tokens… So it’s a balance. You want the prompt to be specific, but not so long so that it eats up the context window. This is where it’s important to test out shorter versions of the prompt to provide the same quality level of output.

PartiallyGenerated

Let’s go back to running a “Hello, World!” prompt in the FoundationModels framework:

As you can see, the model took 1 second to respond with “Hello! How can I assist you today?” This seems super fast, and it is! But as your prompt and output requirements become bigger and bigger the time can significantly slow down.

Since we don’t want the user starting at a spinner, the FoundationModels API provides a nice streamResponse function for the session that will stream the response as it is generated token-by-token by the model:

Notice that it first printed “Hello”, then continued to “Hello! How can I assist” then “Hello! How can I assist you today?”. Each of these output is of String.PartiallyGenerated type which is output by the stream:

The Generable macro will automatically create the PartiallyGenerated type for any object in the same way - so nothing else is needed to be done to our custom StudyPlan project for this to work!

We simply use the PartiallyGenerated type directly in the View or ViewModel! That’s it!

The same applies for other embedding Generable objects inside the StudyPlan!

The only difference here is that every variable of the PartiallyGenerated object can be nil even if the variable is required in the original object!

For example, my StudyPlan object with all required parameters:

Becomes this as a PartiallyGenerated object that I can view by expanding the Generable macro:

These PartiallyGenerated objects are then used in the same way to display the information as the originals (but keeping in mind that every value may be nil!). They will just keep changing as the model is generated! For example, here is the StudyPlanDisplayView:

As each field is generated, the view gets filled out magically!

Update the PartiallyGenerated Object

After the user fills out their study plan parameters in the UI and click the “Generate Study Plan” button, we will use the LanguageModelSession’s streamResponse function to generate the study plan inside the view model.

The key here is to always update the StudyPlan.PartiallyGenerated object with the latest partial generation from the stream!

As the objects keeps getting updated, make sure to set up your view in a way that will automatically update it with the view model as a StateObject:

Later on in the code where the study plan is displayed:

That’s it!

Error Handling

There are special errors that can be generated during the model’s generation process:

There are already many errors and new errors may be added in the future as the API expands!

Conclusion

Apple’s FoundationModels framework makes it remarkably easy and surprisingly delightful to build intelligent, real-time generative apps in SwiftUI. With just a few lines of code, you can stream output token-by-token, giving users immediate feedback and a far more engaging experience than waiting for a full response to load. By using the Generable macro and leveraging PartiallyGenerated types, you can seamlessly display evolving content in your UI as the model generates it!

This blog post is focused on the Meet the Foundation Models framework video:

Limitations

While extremely impressive and a dream-come-true for many Apple Developers - an on-device FREE PRIVATE model!! - it’s important to keep in mind that the model is highly limited.

First - it is small enough to fit on device. A big PRO! But the CON is that the model will just not be as powerful as LLMs provided by OpenAI, Google, or Anthropic. Expect it to not “know” as much as users are used to from larger LLMs, to hallucinate, and have severe logic limitations. This model will not “think” or do a web search or run quick code in a Python interpreter to verify information as the new OpenAI / Google / Anthropic agentic models do.

Instead, focus on it’s strengths - processing natural language. As mentioned in the video, the FoundationModel should be used for the following simple tasks with text:

• Summarization

• Extraction

• Classification

• Tagging

• Composition

• Revision

Again - all of these tasks are focused around processing existing simple language. Read more about the FoundationModel training data here.

Training Cut-off Date

As already mentioned, I wouldn’t rely on this model too much for important factual information - unless it is summarizing the information that I explicitly give it. But this is especially true for the most up-to-date information.

I don’t think we have a model training cut-off date, but many of these models have been trained over a year ago! And the more time passes, the more irrelevant the recent events data becomes. Here is what happens when I ask for today’s date:

And just to confirm the training cut-off, here is what we get if ask who the current U.S. President is:

Looks like the training cut-off is October 2023…

Limited Context Window

The FoundationModel is limited to only 4096 tokens - both input AND output!! This is approximately 3,000 words… So don’t count on summarizing long text / PDFs… or even building a long chat. Keep it short and simple!

Limited Language Support

It’s hard to find information about which languages are supported by the Foundation Model, but there is a small footnote in the Apple supercharges its tools and technologies for developers to foster creativity, innovation, and design press release that mentions the same languages as supported by Apple Intelligence:

• English

• French

• German

• Italian

• Portuguese (Brazil)

• Spanish

• Japanese

• Korean

• Chinese (simplified)

With more languages coming “by the end of this year”: Danish, Dutch, Norwegian, Portuguese (Portugal), Swedish, Turkish, Chinese (traditional), and Vietnamese. Not even Hindi :(

Text-Only Input

Many of us (and our users) are now take multimodal models for granted - we’re used to being able to input audio, video, images, PDFs, etc and get a very nice impressive response. However, with the FoundationModels framework, we need to take a step back. Back to LLM capabilities only. Only text input works as of now.

Safety and Guardrails

The FoundationModel is extremely guardrailed:

First, the framework uses an on-device language model that’s trained to handle sensitive topics with care. Second, the framework uses guardrails that Apple developed with a responsible AI approach. These guardrails flag sensitive content, such as self-harm, violence, and adult sexual material, from model input and output.

- Improving safety from generative model output

While this is good on the surface of how language models should behave, in practice this creates a lot of false positives, which is pointed out in the documentation:

Again - you will have to really test the prompts to make sure the app’s guardrails will not be activated unexpectedly.

Unknown Updates

It seems like Apple plans to update the FoundationModels framework before next year (probably several times). And we, as developers, have no idea when it will happen and how it will impact our existing apps. After all - Apple cannot test that the new model will not break our current specific app use-case. And there doesn’t seem to be a way to refer to a specific release of the model.

Availability

The first issue with using FoundationModels is understanding their availability and therefore including a fallback experience for users who do not have these available. According to Apple’s documentation:

The Apple Intelligence webpage lists the following devices as compatible:

And of course, users must have iOS 26+, macOS 26+, and visionOS 26+ installed on their compatible device.

The code for checking availability and displaying error messages in your view is as follows:

Note that there are currently three reasons for the model being unavailable:

• The device is not eligible (see above for eligible devices from the Apple Intelligence website)

• Apple Intelligence is not enabled - this is a simple fix for the user if they want the extra features.

• Model not ready

Since this framework is in beta and will for sure change in the future, it’s also important to include an unknown default reason that may be added in future versions.

Now onto the fun part!

Hello World

To quickly play around with the model, we can now quickly add the #Playground macro to any Swift file and see the results right away! Simply import the Playgrounds framework and start testing out different prompts.

Getting started with the FoundationModels takes only three simple steps:

1. Import the FoundationModels framework

2. Create a LanguageModelSession

3. Pass it a prompt!

That’s it! Only a few simple lines of code:

Instructions

Now, if you’ve used other LLMs, we know that we need to first give the model a system prompt - called instructions in the FoundationModels language. This is done when setting up the LanguageModelSession initially:

One thing I really liked from the video was how they broke down the difference between Instructions vs Prompts:

• Instructions should only come from you, the developer / app

• Prompts can come from you or the user!

In other words, you should never allow the user access to modify the models instructions.

Iteration

If you have an array of items for the model to respond to, you can simply iterate over your array of items and get a list of responses from the model:

Saved Conversation History

This is something that I really love about the FoundationModels API - the message history is automatically saved by the LanguageModelSession for you! If you’ve used other LLM APIs, you’d know that you usually have to send the full message history to the model every single time you need a response!

So when I ask the model for this session about the topic of the last haiku, it knows automatically!

Funny enough, the model DID NOT follow the initial instructions to respond only in haiku format when asked about the topic of the last haiku. And one time it printed out the topic of the last haiku as orange instead of plum… So it is super important to test out these multi-turn interactions to make sure the user interactions stay on track and do not hallucinate the important information, especially when the user provides the prompt :)

Full Transcript

And if you do need a full transcript (maybe you need to send the full conversation history to an larger server-side API after all!), it is very easily available in the session object:

Generable - NO JSON!!! NO CODABLE!!!

Now onto the super impressive things!!! The FoundationModels has introduced a @Generable macro, which will automatically turn the FoundationModels responses to a typed Swift object, without any JSON / Codable!!

It’s as simple as this:

Now just use this object as a generating item along with the prompt:

Notice how the response.content object is automatically typed to the @Generable SearchSuggestions object!

Property Ordering

LLMs generate content one token (*sub-word*) at a time. This means that if you want the LLM to “reason” or “think” you can first add a longer-form property such as “analysis” first then the main smaller-form property after.

For example, we can create a SearchTerm object which has the analysis of why this search term is relevant first, and the actual term later. For the user, we will only show the term, but by putting “analysis” as the first property, we’re giving the model more tokens to “think” through before coming up with a compelling suggestion:

Generable Composition

In the last example, you may have noticed something super cool! The Foundation Model is able to generate ALL nested Generable objects!

Check out this TravelItinerary object with an array of Generable Destination objects with an array of Generable Activity objects!

It’s able to generate all the objects!

Again - note that travel is not a great use-case for this model as it may hallucinate travel information or provide old information that is not up-to-date. So just use this as an example of object composition…

Tools

As we discussed in the Limitations section, one of the issues with the Foundation Model is that it has a cutoff date maybe around October 2023, so it doesn’t have access to the most recent information, including today’s date!

But it’s not a big deal. We can give it the Tools it needs to get the relevant information to our app. Here is a simple GetCurrentDateTool:

To use it - we simply include the tool in the LanguageModelSession:

Notice that you can pass in multiple tools to the model - this is especially useful on iPhone as you can give access to the user’s Calendar, Reminders, Location, and much more personalized data from your app without risking the user’s privacy!

The model will figure out when to call each tool based on the context of the conversation! And you don’t even have to manage the back and forth of getting the tool response and sending it back to the model - this is done automatically for you!!! The Foundation Model API is truly impressive!

Conclusion

I wanted to include a few more things, but this post is already long enough, so I’ll continue on tomorrow with extra features of the FoundationModels framework as I continue to learn and figure out how to integrate it into apps.

I know I mentioned a lot of limitations in the beginning of this blog post, so I wanted to end here with a more positive view. We have an incredibly powerful natural language processing model ON DEVICE for FREE!! And the capabilities are on par and probably better than most MLX on-device models. So this is truly a big WIN for us as Apple developers, and I’m excited to see Apple continue on this path of providing more options and capabilities for us to work with in a beautiful Swift API :)

Happy Building!

Python is All You Need

So why use Python if I’m a Swift developer? The answer is simple: most Swift apps need a backend, and in my case, I was using Supabase. To process and prepare that backend data, Python when paired with LLMs, is by far the easiest and most efficient tool. Could it be done in Swift? Probably. But the developer experience would be painful, and LLM support for Swift just isn’t there yet. Python, on the other hand, works beautifully.

While there is a ton of hype around AI Agents - if you build fully agentic systems using LLMs for all tasks in the pipeline, you’ll quickly run into surprisingly unreliable and unpredictable results. While these systems have potential (I’m sure they’ll really stand out for certain futuristic use cases I haven’t thought about), the real strength of LLMs today lies in their ability to write solid, reliable Python code that simply gets the job done.

Most of the work I did for my Sanskrit Dictionary app was writing a ton of Python scripts with the help of LLMs. I pushed as much of the data processing as possible to the backend, so the frontend app could stay lightweight and simply display everything instantly and efficiently.

Now here’s the magical part: I barely know Python. I only dabbled with it briefly over a decade ago when I first started learning to code. As I was finishing my MA in Sanskrit Studies and beginning to imagine this app, I assumed I’d need to spend at least six months head-down, learning Python from scratch before I could even get started. But then LLMs came along and suddenly, I could write working Python code without needing to formally study it. It just worked. I’m convinced that without LLMs + Python - I wouldn’t have been able to build this app!

The takeaway? Any Swift developer can now pick up Python on the fly - and that’s the real superpower of LLMs. Even agentic systems, like OpenAI’s o models, often rely on generating and executing Python scripts behind the scenes to produce accurate results. Right now, that’s the most reliable way to get meaningful output from LLMs. So before getting swept up in the agent hype, see if a well-crafted Python script can solve your problem. More often than not, it will.

Learning New Tools

While building the Sanskrit Dictionary app, I had to dive into two platforms I’d never used before: Supabase and Railway. Both are relatively easy to get started with, but using their more advanced features would’ve felt overwhelming if not for LLMs. With AI guiding me, the learning curve became not just easy, but honestly kind of addictive.

Take Supabase edge functions, for example. I was initially intimidated as I’ve never written a line of TypeScript, and I had no idea what Deno even was. But with LLMs helping me adapt and generate the right code, it felt like gaining a superpower!!

And did you know you can deploy a standalone Python script to Railway and call it via API?!!! That’s the kind of mind-blowing capability I wouldn’t even have found out about with LLMs guiding me.

Again - the takeaway here is that even if you’re a Swift developer, LLMs have opened up a whole new world of powerful tools and platforms - with almost no traditional learning curve. You don’t need to master every new language or framework from scratch anymore. You just need to know what you want to build.

Understanding Open Source Libraries

Some of the Sanskrit resources I had to use to build my app were written in Python that was just not accessible for me. My hope while I was in academia was always to work with the library creators to try to understand how these libraries work.

But now there is no need! By using uithub.com to extract the library into LLM-readable format + the 1M+ token window of Google Gemini Pro, I was able to get Gemini Pro to explain what the library did and write Python scripts exactly for my use case to use this library. Wow!!

Again - this is a game-changer for working with any open source library in any language!

Using LLMs for Swift

Unfortunately, LLMs are just not there in the same way when it comes to working with Swift / iOS / macOS apps. My favorite tool was Alex Sidebar, which I highly recommend. But just like any AI solution for Xcode, it is limited by the lack of training data in the models it uses.

For Swift - the best model is Claude Sonnet (Opus uses up all the rate limited tokens in one question, so not going to include it here). And Alex Sidebar does everything in it’s power to embed it with context, including documentation. But it will still write super verbose duplicate unmanageable code.

I use Alex Sidebar + Claude Sonnet in two way. First - if I have a very clear idea of a small coding task I need, I give it clear instructions, then I look carefully through the code and refactor it. Make sure to do this!! Claude Sonnet will take shortcuts and provide weird solutions that are very subtle and can go undetected without careful inspection. I really love the Diff viewer in Alex Sidebar to see exactly what changes were made - many times it’ll change something completely unrelated that you specifically have in your code for a purpose…

The other way I love using Alex Sidebar is for prototyping a feature I’m not exactly sure about. I would commit all my working code, then accept everything Alex Sidebar says (VIBE CODING ALERT!!!) to visually see what it builds. Once I have a better idea of what I want and how to architect properly, I will delete all this code, and go step-by-step as above, slowly and double checking the work as I recode the same feature.

The takeaway? Don’t trust the hype! While these code assistant models are amazing. I’m not going to argue with that! And do increase my productivity 10x, I’m always skeptical and refactor the code to make sure I understand the code and can work with my own codebase in the future. This is critical!

Conclusion

As Swift developers, we’re no longer locked into Swift. And that’s incredibly empowering! With the help of LLMs, we can now tap into the vast ecosystem of tools, languages, and platforms that were once too time-consuming or intimidating to learn. Whether it’s writing Python scripts, deploying backend functions, or understanding open source libraries in unfamiliar languages, we now have the ability to move fast and build things that would have taken months or been out of reach entirely! not too long ago.

What has once been impossible is not possible beyond our wildest imaginations!! My mind was completely blown with every step I took to make the “impossible” Sanskrit Dictionary app and this is only the starting point.

At the same time, we have to stay grounded. AI is a tool, not a shortcut. It can supercharge productivity, but it still requires judgment, refactoring, and a solid understanding of what you’re building. The real magic happens when you combine your developer intuition with the power of these tools.

So build boldly, but refactor carefully. Learn just enough, trust selectively, and let LLMs open doors, not create blind spots.

Happy Building 🚀

So how do you actually use the Batch API? While I love building AI-powered apps in Swift, this is one of those cases where Python really shines, especially when you need to handle file processing like converting between .jsonl and .csv, etc. LLMs are exceptionally good at working with Python for these kinds of tasks. Just open up Cursor, paste in the Batch API docs docs (or turn on web search), and have it generate a custom script tailored to your needs.

I won’t dive too deep into writing the Python script - again, this part is incredibly easy with Cursor. What I do want to highlight, though (and this might not be immediately obvious), is that OpenAI provides a visual interface for submitting batch jobs. So really, the only script you need is one that splits your data into chunks of 50,000 or fewer entries and saves them as .jsonl files. From there, you can handle the rest visually, which, personally, I love.

To get to OpenAI’s visual interface, select your project and go to your Dashboard, which you can find in the top right menu next to your account:

In the right-hand side menu, you’ll see several options, including the option for “Batches”:

To create a new Batch job, simply upload your jsonl file and select the endpoint that it’s for:

Once you create the batch, it’ll first validate that your data is in the correct format. If not, it’ll give a detailed error message, which you can use to fix your jsonl file. It’ll then take up to 24 hours to process the data:

Once it is completed, you’ll be able to download the output files and a file with errors if there are any - both in jsonl format. For example:

I know we’re all developers and are “supposed” to work via API, especially in the age of agents, but having this super nice interface for Batch jobs is super super nice to work with, and something that is unique to OpenAI.

One last tip - whether you’re using the API or the UI, be sure to delete all your uploaded files and the batch files once you’re done processing the batch. OpenAI charges for file storage, so cleaning up afterwards can help you avoid unnecessary costs. You can do that via UI as well just by selecting “Storage” in the Dashboard menu. Super convenient!

Happy Batching!