Constantly growing local search index tasks

Hi Sean,

You can see these tasks constantly growing because of the web farms. Especially, if you are hosting in Azure App Services there could be situations when the app gets re-created therefore the name of the web farm server changes. Cross-check the names of web farm servers and in search tasks. Then feel free to delete the irrelevant search tasks. We do it directly from CMS_SearchTask table.

The site is being hosted on a dedicated server, so there is only the CMS and APP instance in the local search tasks. With that in mind I was looking to disable Web Farm, however the Kentico documentation recommends to have the Web Farm enabled for MVC sites to keep the CMS and APP in sync.

We have cleared the backlog of local search index tasks and we will continue to monitor.

Monitoring the local tasks, we were seeing a huge number of tasks being created for a single product page from 11.48 AM onwards. We have stopped the thread and cleared the local tasks.

With web farm enabled Kentico is seeing 2 web farm servers, WIN-U7MCMMVDD0X_admin, which only had 5 local tasks to be processed, and WIN-U7MCMMVDD0X_AutoExternalWeb, which had over 900 (and growing) local tasks to be processed. All the tasks were for a single product page.

@Sean - this looks like the web farms are not working. Are the servers marked as healthy? What are your storage settings? Are you using shared storage in some way? Also, please check the web farm tasks and Event log for any errors.
@Dmitry - have you tried using CMSWebFarmServerName keys in the config files so when the app gets re-created in the cloud, it will use the same server name? I believe that this should help.

Are the servers marked as healthy?

Yes.

We are running the Kentico 12 SP MVC site on a single dedicated server.

The site license, Enterprise marketing solution, only allows 1 web farm server. In Smart Search > Local tasks, I can see 2 web farms servers, WIN-U7MCMMVDD0G_admin and WIN-U7MCMMVDD0G_AutoExternalWeb. Is this correct?

What are your storage settings? Are you using shared storage in some way?

Any particular storage settings your are referring to. For example, Files is set to File system.

Also, please check the web farm tasks and Event log for any errors.

No Web Farm tasks are showing at the moment.

For the Event Log we are entries similar to the following

Message: Cannot find table 0.

Exception type: System.IndexOutOfRangeException
Stack trace:
at System.Data.DataTableCollection.get_Item(Int32 index)
at CMS.WebFarmSync.WebFarmTaskInfoProvider.DeleteOrphanedTasks()
at CMS.WebFarmSync.WebFarmTaskInfoProvider.DeleteServerMemoryTasksInternal(Int32 serverId, DateTime taskCreated)
at CMS.WebFarmSync.WebFarmTaskInfoProvider.DeleteServerMemoryTasks(Int32 serverId, Nullable`1 taskCreated)
at CMS.WebFarmSync.WebFarmTaskCleaner.DeleteOldMemorySynchronizationTasksInternal(WindowsIdentity windowsIdentity)
at CMS.Base.CMSThread.RunThread()
at CMS.Base.CMSThread.Run()

We occasionally see NOT RESPONDING message from the WebFarmMonitor.

Part of the issue could be the license key. You have license key which allows you to have one Kentico admin app + one MVC app. But you wrote "In our local development environment, we have several developers working on the site using a shared database. " - you need to have valid license key for given number of connected apps. Otherwise, the web farm sync will not work and it will pile up the tasks for the connected servers which are not licensed. Please, contact sales@kentico.com and ask for a development (key) with more web farm servers enabled.
The error you are seeing is partly caused by the fact that there is big amount of orphaned tasks in the queue and partly Kentico bug which caused that these tasks were not correctly cleared once the not responding servers were removed. What is the hotfix you are using? The bug was fixed in hotfix no. 36 - if you are not using newer hotfix, please apply the latest one on the Kentico app and also update the NuGet packages in your MVC apps to match the hotfix (this is also crucial to happen so the MVC app us also using the new code)

For our local environment, we have added the following to the CMS and APP web.config respectively

<add key="CMSWebFarmServerName" value="avoiracmssite" />
...
<add key="CMSWebFarmServerName" value="avoiramvcsite" />

The staging environment, dedicated server, only has a CMS site and APP site so should be compatible with the relevant license?

Looking at the Kentico Installation Manager and NuGet packages, we are currently running 12.0.43.

but your developers are connecting to the same DB, right? This generates new web farm servers. Once they disconnect, the servers may be listed for a while and then the tasks for them are piling up.

Also, make sure that the SQL query in the CMS_Query table, called DeleteOrphanedTasks (QueryName column)looks like this:

-- Delete server tasks that are violating foreign key policy because of nolocks in inserting servertaks
DELETE FROM [CMS_WebFarmServerTask] WHERE [ServerID] NOT IN (SELECT [ServerID] FROM [CMS_WebFarmServer])
-- Delete old tasks that were already processed
DELETE TOP(@deleteTaskCount) FROM [CMS_WebFarmTask] WHERE [TaskIsAnonymous] = 0 AND [TaskID] NOT IN (SELECT [TaskID] FROM [CMS_WebFarmServerTask]) AND [TaskCreated] < deleteOlderThan
-- Return number of deleted tasks
SELECT @@ROWCOUNT

If not, you can update it using this command (and then restart Kentico and MVC apps):

UPDATE CMS_Query SET QueryText = '-- Delete server tasks that are violating foreign key policy because of nolocks in inserting servertaks

DELETE FROM [CMS_WebFarmServerTask] WHERE [ServerID] NOT IN (SELECT [ServerID] FROM [CMS_WebFarmServer])

--Delete old tasks that were already processed

DELETE TOP(@deleteTaskCount) FROM [CMS_WebFarmTask] WHERE [TaskIsAnonymous] = 0 AND [TaskID] NOT IN (SELECT [TaskID] FROM [CMS_WebFarmServerTask]) AND [TaskCreated] < @deleteOlderThan

--Return number of deleted tasks

SELECT @@ROWCOUNT' 

WHERE QueryName = 'DeleteOrphanedTasks'

If the SQL query is the same as above, then maybe the hotfix was not applied correctly and the CMS.WebFarmSync.dll file was not updated with the new code (what is the file version in your project?). In this case the best practice is rolling back the hotfix and applying it again, making sure all the files from the hotfix are copied over and old files are overwritten. You can also try copying just that DLL file from the hotfix files, then clean the solution in Visual studio, clear the .Net cache (c:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files) and rebuild the solution.

but your developers are connecting to the same DB, right? This generates new web farm servers. Once they disconnect, the servers may be listed for a while and then the tasks for them are piling up.

Yes for our development environment we have several developers with their own Visual Studio instances sharing the same database. We've set the Web Farm to manual and added CMSWebFarmServerName appSettings in the CMS and APP web.config. This should solve our localhost/development server issues on our end because no new localhost servers are being added to the Web Farm in manual mode.

For our staging environment, we have just the database, the CMS site and the APP site running. No developers connect to this environment.

We will look at the other suggestions and feedback as soon as we can.

Our local development environment, we have upgraded to 12.0.93 and I can confirm that the DeleteOrphanedTasks in the CMS_Query table is as follows

-- Delete server tasks that are violating foreign key policy because of nolocks in inserting servertaks
DELETE FROM [CMS_WebFarmServerTask] WHERE [ServerID] NOT IN (SELECT [ServerID] FROM [CMS_WebFarmServer])

-- Delete old tasks that were already processed
DELETE TOP(@deleteTaskCount) FROM [CMS_WebFarmTask] WHERE [TaskIsAnonymous] = 0 AND [TaskID] NOT IN (SELECT [TaskID] FROM [CMS_WebFarmServerTask]) AND [TaskCreated] < @deleteOlderThan

-- Return number of deleted tasks
SELECT @@ROWCOUNT

Currently monitoring the Local tasks and we are seeing the following for the respective web farms

cmssite | 7 records (for 2 nodes/pages)

mvcsite | 3,256 records (and growing for 2 nodes/pages)

This indicates that the web farm sync is not working correctly and the MVC app is not picking up the tasks. What is your IIS and network setup? What are the domain names set for the presentation and admin URLs? What license keys are in the system right now for how many servers? (I assume that right now you have only Kentico app and MVC app connecting to the DB). Are there any other errors in the Event log or logged within the web farm tasks?

Below is information related to our development environment. Hoping that fixing the development environment would also fix the staging environment.

| What is your IIS and network setup?

IIS Express

Shared SQL database

| What are the domain names set for the presentation and admin URLs?

• http://localhost:51873/admin
• http://localhost:55224/

| What license keys are in the system right now for how many servers?

• License domain localhost
• License expiration 1/21/2022
• License edition Enterprise marketing solution
• Web farm servers 50

We only have a Kentico app and MVC app connecting to the database.

There could be up to 5 developers working in the development environment. With this in mind, the web farm mode has been set to manual and the following CMSWebFarmServerName appSetting has been set.

• CMS | cmssite
• APP | mvcsite

| Are there any other errors in the Event log or logged within the web farm tasks?

Filtering the Event logs before source containing WebFarm, and we are mainly seeing cmssite is healthy and mvcsite is healthy. We are also seeing the following logs

WebFarmTaskProcessor | System.Data.SqlClient.SqlException_.Net SqlClient Data Provider | A network-related or instance-specific error occurred while establishing a connection to SQL Server.

This is likely to be when we have switched git branches or our VPN connection has stopped.

| Update

Coming back to the project today, a task processor is running, indicating that tasks are being finished successfully. However there are still local tasks being created for the same few pages, 1647 pages (25 per page) and growing.

Are there any other errors logged? Even if they sound unrelated. What tasks are being created? Could you please post some sample? I would try clearing the the CMS_WebFarm task table and restart the apps by doing fake change in the web.config. And I would also try setting up the sites in the IIS. I guess you are running the apps from Visual Studio - maybe the built in server is doing some magic.

Are there any other errors logged?

Not that we can see.

Even if they sound unrelated. What tasks are being created? Could you please post some sample?

04/21/2021 1:26:54 PM | Search Index | UPDATEOBJ | Update Search index 'Custom - Pages'

04/21/2021 1:25:19 PM | Smart search | PROCESSSEARCHTASK | Thread was being aborted.

I would try clearing the the CMS_WebFarm task table and restart the apps by doing fake change in the web.config. And I would also try setting up the sites in the IIS. I guess you are running the apps from Visual Studio - maybe the built in server is doing some magic.

We have set the Web Farm to manual and added CMSWebFarmServerName appSettings cmssite and mvcsite, respectively, and set CMSProcessSearchTasksByScheduler to true.

Both Web Farms are showing as healthy.

Cleared the CMS_SearchTask table.

Re-build local index Custom - Pages.

Run the scheduled task Execute local search tasks.

Navigate to Web Farm > Tasks.

Briefly saw two tasks, one for cmssite and one for mvcsite.

Navigate to Smart Search > Local tasks.

All tasks for Web Farm server cmssite have been processed.

No tasks have been processed for Web Farm server mvcsite.

This is very strange. I would try using full IIS setup and not Express or VS Server. Just to see if it makes any difference. If not, please submit a ticket to support, refer to this thread - we would need to have the projects and DB backups to take a closer look what is going on.