Hi Trevor, thanks for your answer. Yes, the field is a string, and it was marked as Searchable, as well as Tokenized.
The issue appears to be that the Kentico search engine sorts by the tokenized fields, so some blocks of results were sorted by the first term, some by the last terms, and some didn't seem to have a discernible pattern. Sorting worked if I set the the field to not tokenized, and switched the index to Keyword, but then it's not very searchable because only exact matches returned results.
My solution was to create a duplicate of the column I wanted to sort by on the custom table to be used for sorting. The values are all the same as the first column, but while the first column is set to Tokenized (on) the duplicate is set to Tokenized (off)
Then I switched the index to Subset and set the code to sort by the second (untokenized) column.