Solo Scores: How Do I Know the AI Got Better?

Trying it isn't measuring it

Solo Scores has two AI features: Bulk Import, where a vision model reads each imported PDF so the app can sort it into scores, and a playlist builder that picks a set from your library.

The obvious way to check either is to try it. Import a few PDFs, ask for a playlist or two, and see if it looks right. But a handful of tries says little about a change. Better on the three I looked at can mean worse on the ten I didn't.

So both features have an eval: a fixed set of examples with a scorecard that says, in numbers, how well the current version does. That's what tells me whether a change actually helped.

A set with known answers

For import, the set is a couple of dozen real scores and their parts, with what's printed on each written down by hand: title, composer, key and the name each part should get.

Some are made hard on purpose. There are tilted scans, faint photocopies and pages with the title cut off, because that's what musicians actually import.

The scorecard runs every PDF through the same steps the app uses and counts how many scores were grouped correctly and how many titles, composers, keys and part names came out right. Each run is saved alongside the version of the app it tested, so I can compare any two.

Judging what can't be counted

Playlists don't have one right answer, so their eval works in two layers. Simple checks catch the clear failures: is the set close to the length asked for, does it suit the occasion, and does a request the library can't really fill stay short rather than padding itself with unrelated pieces?

Then a separate model acts as judge, so the builder isn't marking its own work. It scores each playlist on a few qualities, such as fit and flow, and names the pieces behind any problems. Those named problems are often more useful than the scores.

What the numbers decided

The scorecards have settled several questions I'd otherwise have argued with myself about:

A prompt change. Asking the playlist builder to use what it knows of each piece and shape the set's energy raised the judge's flow score, and fixed a carol service that had filled with secular Christmas songs.
An idea that didn't pay. Giving the playlist builder more detail about each piece made every request much bigger and added nothing the scores could show, so it stayed out.
Model size. A larger vision model read keys a little better but got fewer composers right, and a small model that could run on the tablet grouped only a handful of the scores correctly.

The scorecards also showed how much results wobble. Running the same version twice gives slightly different numbers, so one run on its own isn't enough to trust a change.

Takeaways

Write down the right answers first. A fixed set of real, awkward examples turns "it seems better" into a count.
Check what you can, judge what you can't. Plain rules catch the obvious failures; a separate judge handles taste.
Run it more than once. In Solo Scores, the same version scores a little differently each time, so one run isn't enough to judge a change.