I'm not aware of a way to find this information. As you say, there is nothing in the table OM_ContactGroupMember
that could record this. Going forward, you could create a custom event to track this?
From a GDPR perspective, do you need to see when or just whether they are or not?