I have seen issues like that with Smart Search. I can't say I know of an exact fix, however, these are the tings I normally try. I know you have done some of these but there may be some you haven't tried.
- Confirm that you did not include any SmartSearch index files in the deployment package
- Reboot the web and worker roles
- Reimage the web and worker roles
- RDP to the worker roles. Delete any files in the C:\Resources[Some GUID]\CMSTemp and CMSCache folders. (I may have the exact path) on the WEb and Worker roles.
- Delete the physical files from Azure Storage
- Create a new index and try and create it
- Update the web.config with bigger timeout values for Smart Search Lock and Timeout values (https://docs.kentico.com/display/K8/Reference+-+Web.config+application+keys#Reference-Web.configapplicationkeys-Smartsearchsettings)
I often will try one of the above and then reboot / reimage the servers. I find that re-imaging reset them and cleans up a lot of stuff, unless your deployment package has the index files in them. If that is the case, you will have to rebuild / redploy without any SS index files included.