We found Database lock wait

Thawatchai Kongsuk asked on March 25, 2022 04:19

Today we found Database lock wait for this queries. How to solve this critical issue? Current version is Kentico 12.0.98 (Portal Engine)


Query1:

DELETE FROM CMS_ScheduledTask WHERE (([TaskObjectType] = @TaskObjectType AND [TaskObjectID] IN ( SELECT [StateID] FROM CMS_AutomationState WHERE ([StateObjectType] = @StateObjectType AND [StateObjectID] = 192875297) )) AND [TaskObjectID] IS NOT NULL)


Query2:

WITH q AS ( SELECT TOP(@BatchSize) [TaskID], [TaskName], [TaskDisplayName], [TaskAssemblyName], [TaskClass], [TaskInterval], [TaskData], [TaskLastRunTime], [TaskNextRunTime], [TaskEnabled], [TaskIsRunning], [TaskSiteID], [TaskDeleteAfterLastRun], [TaskServerName], [TaskGUID], [TaskExecutions], [TaskResourceID], [TaskRunInSeparateThread], [TaskUseExternalService], [TaskAllowExternalService], [TaskLastExecutionReset], [TaskCondition], [TaskRunIndividually], [TaskUserID], [TaskType], [TaskObjectType], [TaskObjectID], [TaskExecutingServerName] FROM CMS_ScheduledTask WHERE TaskID > @LastProcessedId AND TaskNextRunTime IS NOT NULL AND TaskNextRunTime <= @DateTime AND TaskEnabled = 1 AND TaskIsRunning = 0 AND (TaskSiteID = @TaskSiteID OR TaskSiteID IS NULL) AND (TaskServerName IS NULL OR TaskServerName = '' OR TaskServerName = @TaskServerName) ORDER BY TaskID ) UPDATE q SET TaskIsRunning = 1, TaskExecutingServerName = @TaskServerName OUTPUT INSERTED.[TaskID], INSERTED.[TaskName], INSERTED.[TaskDisplayName], INSERTED.[TaskAssemblyName], INSERTED.[TaskClass], INSERTED.[TaskInterval], INSERTED.[TaskData], INSERTED.[TaskLastRunTime], INSERTED.[TaskNextRunTime], INSERTED.[TaskEnabled], INSERTED.[TaskIsRunning], INSERTED.[TaskSiteID], INSERTED.[TaskDeleteAfterLastRun], INSERTED.[TaskServerName], INSERTED.[TaskGUID], INSERTED.[TaskExecutions], INSERTED.[TaskResourceID], INSERTED.[TaskRunInSeparateThread], INSERTED.[TaskUseExternalService], INSERTED.[TaskAllowExternalService], INSERTED.[TaskLastExecutionReset], INSERTED.[TaskCondition], INSERTED.[TaskRunIndividually], INSERTED.[TaskUserID], INSERTED.[TaskType], INSERTED.[TaskObjectType], INSERTED.[TaskObjectID], INSERTED.[TaskExecutingServerName]


Query3:

SELECT TOP 1 * FROM CMS_ScheduledTask WHERE ([TaskName] = @TaskName AND [TaskSiteID] = @TaskSiteID)

Recent Answers


Juraj Ondrus answered on March 25, 2022 05:44

How many scheduled tasks are in the system? What is the load and configuration? This is really hard to tell based on some SQL queries. It is also possible that these are just the victims and the issue is somewhere else. What steps are leading to this issue?

0 votesVote for this answer Mark as a Correct answer

Thawatchai Kongsuk answered on March 25, 2022 10:21

Hi Juraj,

This are tasks

  • Process analytics log [CMS.WebAnalytics] {CMS.WebAnalytics.HitLogProcessor}
  • Delete old file system cache files [CMS.OutputFilter] {CMS.OutputFilter.FileSystemCacheCleaner}
  • Contact group 'X1, but not submit form (...)' rebuild [CMS.ContactManagement] {CMS.ContactManagement.ContactGroupEvaluator}
  • Contact group 'X2, but not submit form (...)' rebuild [CMS.ContactManagement] {CMS.ContactManagement.ContactGroupEvaluator}
  • Contact group 'X3, but not submit form (...)' rebuild [CMS.ContactManagement] {CMS.ContactManagement.ContactGroupEvaluator}
  • Contact group 'X4, but not submit form (...)' rebuild [CMS.ContactManagement] {CMS.ContactManagement.ContactGroupEvaluator}
  • Contact group 'X4, but not submit form (...)' rebuild [CMS.ContactManagement] {CMS.ContactManagement.ContactGroupEvaluator}
  • Contact group 'X5, but not submit form (...)' rebuild [CMS.ContactManagement] {CMS.ContactManagement.ContactGroupEvaluator}
  • Contact group 'X6' rebuild [CMS.ContactManagement] {CMS.ContactManagement.ContactGroupEvaluator}
  • Contact group 'X7' rebuild [CMS.ContactManagement] {CMS.ContactManagement.ContactGroupEvaluator}
  • Contact group 'X8' rebuild [CMS.ContactManagement] {CMS.ContactManagement.ContactGroupEvaluator}
  • Delete old temporary attachments [CMS.DocumentEngine] {CMS.DocumentEngine.TemporaryAttachmentsCleaner}
  • Delete old temporary upload files [CMS.Base.Web.UI] {CMS.Base.Web.UI.TemporaryUploadsCleaner}
  • Content publishing [CMS.DocumentEngine] {CMS.DocumentEngine.ContentPublisher}
  • Delete image editor history [CMS.Scheduler] {CMS.Scheduler.DeleteImageEditorHistory}
  • Delete old temporary Page builder widgets [CMS.DocumentEngine] {CMS.DocumentEngine.PageBuilder.TempPageBuilderWidgetsCleaner}
  • Clear content of old shopping carts [CMS.Ecommerce] {CMS.Ecommerce.ShoppingCartCleaner}
  • Clean email queue [CMS.Scheduler] {CMS.Scheduler.QueueCleaner}
  • Send queued emails [CMS.Scheduler] {CMS.Scheduler.QueueSender}
  • Process forum thread views [CMS.Forums] {CMS.Forums.ThreadViewsProcessor}
  • Process external integration tasks [CMS.SynchronizationEngine] {CMS.Synchronization.IntegrationExternalTasksProcessor}
  • Membership reminder [CMS.Membership] {CMS.Membership.MembershipReminder}
  • Send module usage data [CMS.ModuleUsageTracking] {CMS.ModuleUsageTracking.Dispatcher}
  • Check bounced e-mails [CMS.Newsletters] {CMS.Newsletters.BounceChecker}
  • Send marketing emails [CMS.Newsletters] {CMS.Newsletters.NewsletterSender}
  • Persona's contact history recalculation [CMS.Personas] {CMS.Personas.Internal.PersonaContactHistoryTask}
  • Recalculate campaign reports [CMS.WebAnalytics] {CMS.WebAnalytics.CalculateCampaignConversionReportTask}
  • Report subscription sender [CMS.Reporting.Web.UI] {CMS.Reporting.Web.UI.ReportSubscriptionSender}
  • Execute local search tasks [CMS.Scheduler] {CMS.Scheduler.SearchTaskExecutor}
  • Remove expired sessions [CMS.Membership] {CMS.Membership.RemoveExpiredSessions}
  • Update database session [CMS.Membership] {CMS.Membership.UpdateDatabaseSession}
  • Recalculate time zone [CMS.Scheduler] {CMS.Scheduler.TimeZoneRecalculate}
  • Users delete non activated user [CMS.Membership] {CMS.Membership.DeleteNonActivatedUser}

What steps are leading to this issue?

After we completed update hotfix 12.0.98 on 23/03/2022, we just found this error today.

0 votesVote for this answer Mark as a Correct answer

Juraj Ondrus answered on March 25, 2022 12:04

Thanks, those are default tasks plus some tasks related to contact group re-calculation - these could be related in case you have a lot of contacts to recalculate. Anyway, by steps leading to the issue I meant what actions are done on the web site or in admin? Are you able to reliably reproduce the issue? If yes, how? Maybe it was one time issue only or, you should take a look at your online marketing data and maybe clean up inactive contacts and so on. Really hard to tell without knowing what leads to the issue.

0 votesVote for this answer Mark as a Correct answer

Bowling Christopher answered on April 6, 2022 10:57

Enter MySQL. mysql -u your_user -p. Let's see the list of locked tables. mysql> show open tables where in_use>0; Let's see the list of the current processes, one of them is locking your table(s) mysql> show processlist; Kill one of these processes.

0 votesVote for this answer Mark as a Correct answer

Thawatchai Kongsuk answered on April 7, 2022 05:40

Our Customer have actived OM_Contact about 35M records within 180 days and OM_Activity about 30M records, that they need to use this data for Online Marketing functions.

How to solve this problem?

0 votesVote for this answer Mark as a Correct answer

Juraj Ondrus answered on April 7, 2022 06:37

Is the DB server able to handle the load? Does it have computing power? Does the customer really needs 30M contact records? I would say there will be many inactive contacts and so on. What is your setting for deleting inactive contacts?

0 votesVote for this answer Mark as a Correct answer

Thawatchai Kongsuk answered on April 7, 2022 06:58

Our customer don't need to collect all of 30M records, but we setup on Kentico to Delete Inactive Contact last 180 days and the Kentico's result is 30M records.

0 votesVote for this answer Mark as a Correct answer

Thawatchai Kongsuk answered on April 7, 2022 07:07 (last edited on April 7, 2022 07:08)

Database Server Specification is below.

CPU = 32 cores RAM = 128 GB OS = Windows 2008R2 Service pack 1

0 votesVote for this answer Mark as a Correct answer

Juraj Ondrus answered on April 7, 2022 07:14

I do not know your project - what is the traffic, what other features are making DB calls, how are you using caching. There are like dozens of other variables affecting this. By seeing hard numbers - it does not mean anything., You should be using SQL profiler and other debugging tools to see what is going on, what are the most expensive SQL queries, etc.

0 votesVote for this answer Mark as a Correct answer

Thawatchai Kongsuk answered on April 7, 2022 10:34

All task that occured issues are Kentico's tasks.

All queries that occured issues are Kentico's queries.

How to solve this problem?

0 votesVote for this answer Mark as a Correct answer

   Please, sign in to be able to submit a new answer.