Running Smart search on Azure
I decided to write this article because I wanted to share two things with you. The first one is how smart search works when running on Azure. The second one is how to troubleshoot the Smart search if it seems that it’s not working. This article is aimed at developers that want to know a little more about Kentico Smart search when using Microsoft Azure.
Storing search indexes in Azure blob storage
First thing that comes into play when running on Azure is the Azure blob storage. Using the blob storage has some specifics that cause Smart search to work a little differently. The difference is connected with the fact that the blob storage is shared between all instances of Kentico and that Smart search doesn’t work well in environments where multiple instances manipulate (update, rebuild, optimize) the same index at the same time.
So, to prevent multiple instances from manipulating the same index at the same time, we introduced the concept of locking the index for exclusive write. Every time the Smart search wants to manipulate some search index, it creates a special file with .lock extension. This file indicates that some search indexer is working with the index. When the work is done, the file is deleted from the storage. This works as a traffic light for other Smart search instances. If the lock file is present (red light, not safe to work) the instance is not going to start manipulating the index. If it is not present (green light, safe to work) the instance is going to proceed.
You can find two different lock files in your blob storage. First one is named searchtaskindexer.lock and is located in ~/App_Data/CMSModules/SmartSearch directory. Second one is named write.lock and is located in the directory of some particular search index. They serve a little different purpose but they are both connected to the shared storage.
In some rare ocasions, this lock file may not get deleted. This may happen when the process that works on the search index ends unexpectedly. When this happenes, Smart search is going to be stuck because no Smart search instance in the future is going to touch the search indexes.
The fix for this situation is quite obvious. You need to delete the lock files. The easiest way is to connect to your blob storage and manually delete them.
Using Smart search worker role
The Smart search worker role is part of our Kentico Azure project that is designed for use on Azure Cloud Services. If you are not running on the Cloud Services or not using the worker role, you can skip this section right away.
The worker role just processes the search tasks in the loop. The code for the worker role is not a secret and you can look for it in a file named WorkerRole.cs. This will give you a general idea about what the worker role is doing when it runs.
/// <summary>
/// Main method of worker role.
/// </summary>
public override void Run()
{
if (!SmartSearchEnabled())
{
while (!SmartSearchEnabled())
{
Thread.Sleep(AzureHelper.SEARCH_INIT_SLEEP);
}
while (true)
{
AzureHelper.RestartAzureInstance();
}
}
CMSApplication.WaitForDatabaseAvailable.Value = true;
CMSApplication.Init();
WebSyncHelper.InstanceIsHiddenWebFarmServer = true;
// Infinite loop for processing tasks
while (true)
{
try
{
// Clear cache and all the hashtables
CacheHelper.ClearCache(null, logTask: false);
ModuleManager.ClearHashtables(false);
SearchTaskInfoProvider.ProcessTasks(false, true);
Thread.Sleep(AzureHelper.SEARCH_PROCESS_SLEEP);
}
catch (Exception ex)
{
// Catch all exceptions and log them
LoggingStrategy.CurrentEventlogStrategy.LogEvent(EventType.ERROR, "SmartSearchWorker", "Run", exception: ex);
}
}
}
Sometimes you get into a situation when you are fully convinced that every setting and every configuration is correct and the worker role runs just fine but the worker role is still not processing the search tasks. This can be annoying, especially when there is no error in Kentico Event log. So, how to find out what is wrong?
There is one practice that proved to be useful when trying to troubleshoot this scenario. Just follow these steps:
- Make sure that remote desktop is configured for your worker role. You can do that in Azure portal.
- Connect to the worker role through the remote desktop access.
- Look into Windows event log using Event viewer. Try looking for errors from around the time the worker role was starting.
Next steps can differ a lot based on the error you find. But the error should provide enough information to get you going.
Sometimes you are not able to find any error. What to do next? The last resort is to modify the code of the worker role to see which lines of code are being executed and which are not. This can be done by creating custom logging to the file system and logging some information before every line of code.
After that you can redeploy the project and see your logs. The logs should give you some hints about what line of code is not being executed and therefore where the problem might be.
Conclusion
I hope that the article shed some light on Kentico Smart search. You have learned a little more about the inner workings of Smart search and hopefuly can now troubleshoot some of the issues you may run into when using Kentico Smart search on Azure.
Any feedback related to this topic is always welcome :)