What kind of environment are you in? On-premise? Azure? Amazon?
I'd check the actual smart search index files on the drive. You can find them under /App_Data/CMSModules/SmartSearch/<index code name>
Delete the files in that directory and attempt to rebuild the index. Most likely a corrupt file in there.