· 3 min read
On-Device AI: Right Tool, Wrong Job?
Solo Scores is offline first, so its AI playlist builder started on the tablet. It moved to the cloud because the feature's value is the model's knowledge of the repertoire, and a small model doesn't have it.
The feature
Ensembles spend a surprising amount of time building set lists. A wedding wants an hour of light music for the drinks reception; a church service wants something quieter. The music is all there in the library. The work is picking it and putting it in an order that flows.
So Solo Scores has a box where you describe the set ("An hour for a summer wedding drinks reception, upbeat but nothing too loud") and it builds a playlist from your collection.
Building it showed me that what the model is for decides where it runs, even in an app built to work offline.
Starting on the tablet
The app is offline first. Players open it in church basements and wedding venues where there's no signal, so my instinct was to keep the AI offline too. The first version ran a small, 3-billion-parameter model on the tablet itself.
It felt like the right fit. No connection needed, nothing leaving the device, and the feature would work in exactly the places the rest of the app does.
What it cost
It worked, in the sense that it produced playlists. But the costs added up quickly.
The download was about 2 GB. That's a lot to ask of someone who just wants to try a feature once.
It also ran on the tablet's own hardware, alongside an app that's already busy rendering PDFs and syncing.
Both of those I could have lived with. The third problem I couldn't.
What it didn't know
A small model doesn't know much about repertoire. It can match "jazz" to pieces tagged jazz, but that's something a filter could do. The useful part is knowing that Pachelbel's Canon suits a ceremony and Libertango doesn't.
That was the whole value of the feature: the model's knowledge of the music. And it's exactly what a small model doesn't have.
So the question that settled it was simple. Is the model there to reformat text, to classify, to follow a rule you could have written yourself? Or is it there for what it knows? For those mechanical jobs, a small model on the device is a great fit: no connection, nothing leaving the tablet. But if you want a model for its breadth of knowledge, the size of the model is the feature, and that settles where it lives.
Moving it to the server
So the model moved to the server, and that's the version that shipped. The app sends what's specific to the request (what the user asked for, the pieces available and how long the set should be) and gets back a title and an ordered list.
The rest of the work was the usual: the prompt was refined a number of times, mostly by describing a good set in the terms a musician would use, and the app checks every answer before showing it. None of that would have helped the on-device version. You can't prompt your way to knowledge a model doesn't have.
Takeaways
- Decide what the model is for first. On-device models are great at mechanical work; knowledge needs a model big enough to have it.
- Don't let a good principle pick the model for you. Offline first is right for the app, but it was the wrong reason to choose a model that didn't know the music.