Back at this again, and now I'm noticing something interesting..
When I leave the Sort field blank on the Smart Search Results control, the relevance scores work for individual content items. But the second I add in a sort condition (documentmodifiedwhen desc), it breaks the relevancy and everything now appears as 0%.
What I want is the results sorting by documentmodifiedwhen desc by default AND have the relevancy score working. Can I not have both?