SQL Connection Error

It seems the issue resolved itself. Not sure how but there must have been a connection issue with the database server or a cached referenced that took some time to update the digested change. I'm chalking it up to the "it just works" theorem.

Hi,

We're getting the same error on one of our live sites (v9.0.5) in Azure.

Message: [ObjectGenerator.RegisterDefaultFactory]: You cannot change the default type of the generated objects after some objects were created by the generator. 1 objects were already created by this generator. To fix this problem, run the registration sooner in the application life cycle or create new objects later.

Exception type: System.Exception Stack Trace: at CMS.Base.ObjectGenerator`1.RegisterDefaultFactory(IObjectFactory factory, Boolean topPriority) at CMS.Ecommerce.ECommerceModule.OnInit() at CMS.Core.ModuleEntry.Init() at CMS.Core.ModuleEntryManager.Init() at CMS.Core.AppCore.Init() at CMS.DataEngine.CMSApplication.Init()

It happens infrequently and (so far) unpredictably. It's not linked to deployments or traffic levels from what I can tell. The only unusal activity I see in the Event Log is a high rate (dozens per minute) of Application_Start events at certain periods. However the crash does not always occur at these times either.

Any thoughts?

I've seen the same thing happen on occasion, and now twice in the last few times I restarted the app. v9.0.50 as well.

This has occurred after deploying .dll changes, as well as just clicking the "Restart application" button in the System application.

What makes matters worse is Kentico appears to cache the sql connection error state, so the whole app goes down for a couple minutes, unless someone intervenes by restarting the app pool/IIS first.

We're also receiving the same error on a v9.0.50 site, the site was previously running in v8 and only seems to have become a problem since the v9 upgrade.

The exact error is as follows: [ObjectGenerator.RegisterDefaultFactory]: You cannot change the default type of the generated objects after some objects were created by the generator. 2 objects were already created by this generator. To fix this problem, run the registration sooner in the application life cycle or create new objects later.

And the full stack trace:

Exception type: System.Exception
Stack Trace: 
at CMS.Base.ObjectGenerator`1.RegisterDefaultFactory(IObjectFactory factory, Boolean topPriority)
at CMS.Ecommerce.ECommerceModule.OnInit()
at CMS.Core.ModuleEntry.Init()
at CMS.Core.ModuleEntryManager.Init()
at CMS.Core.AppCore.Init()
at CMS.DataEngine.CMSApplication.Init()

Message: [ObjectGenerator.RegisterDefaultFactory]: You cannot change the default type of the generated objects after some objects were created by the generator. 4 objects were already created by this generator. To fix this problem, run the registration sooner in the application life cycle or create new objects later.

Exception type: System.Exception
Stack Trace: 
at CMS.Base.ObjectGenerator`1.RegisterDefaultFactory(IObjectFactory factory, Boolean topPriority)
at CMS.Ecommerce.ECommerceModule.OnInit()
at CMS.Core.ModuleEntry.Init()
at CMS.Core.ModuleEntryManager.Init()
at CMS.Core.AppCore.Init()
at CMS.DataEngine.CMSApplication.Init()
at CMS.Base.AbstractHandler.CallEventHandler[TArgs](EventHandler`1 h, TArgs e)
at CMS.Base.AbstractHandler.Raise[TArgs](String partName, List`1 list, TArgs e, Boolean important)
at CMS.Base.SimpleHandler`2.RaiseExecute(TArgs e)
at CMS.Base.SimpleHandler`2.RaiseExecute(TArgs e)
at CMS.Base.SimpleHandler`2.StartEvent(TArgs e)
at CMS.Base.ApplicationModule.BeginRequest(Object sender, EventArgs e)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

As Kurt Schindler said, this problem persists until someone manually recycles the IIS app pool so the live site can be down for some time...

Any help here would be appropriated.

Kurt and Steven,

Have you properly registered any custom modules or updated the code from v8 to v9? I'd check this out if you have any custom modules or code you've implemented.

Hi Brenden,

We do have a few custom modules installed on the site, all of our custom modules are registered as such:

[assembly: RegisterModule(typeof (ApiModule))]

namespace NetC
{
    public class ApiModule : Module
    {
        /// <summary>
        ///     Create new instance of ApiModule
        /// </summary>
        public ApiModule() : base("ApiModule")
        {
        }

        /// <summary>
        ///     Initialise the object
        /// </summary>
        protected override void OnInit()
        {
            base.OnInit();

            ...
        }
    }
}

Looking at the stack trace however it seems the problem in both Killian's case and mine lies in the CMS.Ecommerce.ECommerceModule.OnInit() method. We haven't registered any custom objects to be part of the CMS.Ecommerce module.

Thanks, Steven.

My stack trace also points to CMS.Ecommerce.ECommerceModule.OnInit() and its the same situation, we haven't registered any custom objects to be part of the CMS.Ecommerce module either.

I believe we don't actively use this module but it is attached to all our environments, including production.

Logic would dictate that disabling this module from our site, I would assume this error will subside. I may try disabling it from one of our staging sites and monitor site connectivity for the next couple of weeks in hopes that this error goes away.

An update:

The error continues to persist even after disabling the Ecommerce module.

I raised a support ticket with Kentico and sent them some of our custom code. They identified a potential issue with our WCF services and suggested the following...

It seems that problem is with WCF services (files with .svc.cs extension). The service code is calling Kentico API but application initialization is not ensured here. Kentico is initialized automatically for all .NET requests but WCF lifecycle is not attached here.

Therefore it is necessary to call CMS.DataEngine.CMSApplication.Init(); (as described here: https://docs.kentico.com/k10/integrating-3rd-party-systems/using-the-kentico-api-externally#UsingtheKenticoAPIexternally-InitializingtheKenticoapplication) prior any usage of Kentico API within WCF service.

I can't say for sure yet if this has resolved the issue but I will continue to keep an eye on it.

This doesn't seem to have fixed our issue. The site still drops unpredicatably and the same initialisation errors appear.

Hi Brendan, we're also experiencing this issue and it's becoming a real issue. We've tried the suggestions about but as with the other guys experience, it made no difference. Please can you advise ASAP?

Thanks Alix.

I will follow up on this thread. We were working on this with Alix in support ticket and there was a bug discovered related to a way in which application was initialized (this applies for v9 and partially v10).

With RunMenagedModulesForAllRequests set to true, managed code was used also for other requests as well. This along with our interface based objectFactory allowed object instantiation prior to application initialization.

We had released a fix for this in 10.0.41 and this should not happen in v11(new pattern for module registration was created).

Hi Michal,

Thanks for the update. Could this issue have been causing the crashes experienced by myself and others in this thread? Is there a fix available for v9 or what is the recommended action to prevent it from recurring?

Thanks, Killian

Hi Killian,

It could be caused by a bug, but it is hard to tell whether you ran into exact same initialization issues(in older versions exception falls to ). Workaround for this case was to set runAllManagedModulesForAllRequests to false and this could help in your case.

In case this does not work, you can try checking app start for module initialization order. There you can see order in which modules were initialized and first one is most likely causing trouble. Also you can use following code in global asax:

using System;
using CMS.Base;
using CMS.Core;
using CMS.DataEngine;

/// <summary>
/// Application methods.
/// </summary>
public class Global : CMSHttpApplication
{
    #region "Methods"

    static Global()
    {
#if DEBUG
        // Set debug mode based on current web project build configuration
        SystemContext.IsWebProjectDebug = true;
#endif

        // Ensure initialization of dynamic modules after application pre-initialization. Must be called before the StartApplication method.
        ApplicationEvents.PreInitialized.Execute += EnsureDynamicModules;

        CMS.DocumentEngine.DocumentGenerator.RegisterDefaultFactory(new LogFactory());

        // Initialize CMS application. This method should not be called from custom code.
        InitApplication();
    }


    /// <summary>
    /// Ensures that modules from the App code assembly are registered.
    /// </summary>
    private static void EnsureDynamicModules(object sender, EventArgs e)
    {
        // Ensures CMSModuleLoader module, which has IsDiscoverable set to false
        ModuleEntryManager.EnsureModule<CMSModuleLoader>();

        // Ensures other possible modules within app code assembly
        ModuleEntryManager.EnsureAppCodeModules(typeof(CMSModuleLoader).Assembly);
    }

    #endregion
}

public class LogFactory : CMS.DocumentEngine.DocumentFactory<CMS.DocumentEngine.TreeNode>
{
    public LogFactory() : base(typeof(CMS.DocumentEngine.TreeNode)) { }

    public override object CreateNewObject()
    {
        LogIfCalledPriorAppInit();

        return base.CreateNewObject();
    }

    private static void LogIfCalledPriorAppInit()
    {
        try
        {
            if (!CMSApplication.ApplicationInitialized.GetValueOrDefault(false))
            {
                // Log message = Time, URL, StackTrace
                System.Text.StringBuilder sb = new System.Text.StringBuilder();
                sb.AppendLine(DateTime.Now.ToString());
                if (System.Web.HttpContext.Current != null && System.Web.HttpContext.Current.Request != null)
                {
                    sb.AppendLine("URL: " + System.Web.HttpContext.Current.Request.Url);
                }
                sb.AppendLine(Environment.StackTrace);

                // Log to the Event Log (DB), might be lost because of failed initialization
                CoreServices.EventLog.LogEvent("E", "GENERATOR", "CreateNewObject", sb.ToString());

                // Log to the file system '/App_Data/generator.log'
                string filePath = System.Web.Hosting.HostingEnvironment.MapPath("~/App_Data/generator.log");
                using (System.IO.StreamWriter sw = System.IO.File.AppendText(filePath))
                {
                    sw.Write(sb.ToString());
                }
            }
        }
        catch
        {
            // Site should not fail because of issue with logging code
        }
    }
}

This code should create additional logging and log stack trace from this faulty initialization pointing you to code from which this originates.

Hi Michal,

One of your colleagues gave me the same code in a support ticket and it has been in place of our site since then. The site has not crashed recently but the generator.log file shows the following (examples from today), so far I have only seen it log for asset files e.g images or css.

10/31/2017 1:23:24 PM
URL: https://[domain]/App_Themes/[site theme]/css/gulp_default.min.css
   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at LogFactory.LogIfCalledPriorAppInit()
   at LogFactory.CreateNewObject()
   at CMS.Base.ObjectGenerator`1.TryFactory(IObjectFactory factory, Object parameter)
   at CMS.Base.ObjectGenerator`1.CreateNewObject(Object parameter)
   at CMS.DocumentEngine.DocumentGenerator.NewInstance[NodeType](String className)
   at CMS.DocumentEngine.TreeNode.NewInstance[NodeType](String className)
   at CMS.DocumentEngine.TreeNode.New[NodeType](String className, DataRow dataRow, TreeProvider treeProvider)
   at CMS.DocumentEngine.TreeNodeProvider.CreateInfo(DataRow dr, Boolean useGenerator)
   at CMS.DataEngine.AbstractInfoProvider`3.get_InfoObject()
   at CMS.DataEngine.AbstractInfoProvider`3..ctor(Boolean initialize)
   at CMS.DocumentEngine.TreeNodeProvider..ctor()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at CMS.Core.ObjectFactory`1.CreateInstance(ConstructorInfo ctor)
   at CMS.Core.ObjectFactory`1.CreateInstance()
   at CMS.Core.ObjectFactory`1.CreateNewObject()
   at CMS.Core.ObjectFactory`1.New()
   at CMS.DataEngine.AbstractInfoProvider`1.get_ProviderObject()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
   at CMS.DataEngine.AbstractProvider.GetProviderInternal(String objectType, Boolean exceptionIfNotFound)
   at CMS.DataEngine.AbstractProvider.GetProvider(String objectType, Boolean exceptionIfNotFound)
   at CMS.DocumentEngine.TreeNodeProvider.GetProviderObject(String className)
   at CMS.DocumentEngine.TreeNodeProvider.GetDocuments()
   at CMS.DocumentEngine.TreeProvider.SelectNodes()
   at CMS.URLRewritingEngine.CMSDocumentRouteHelper.RegisterDocumentRoutes(String where, String siteName, List`1 routes)
   at CMS.URLRewritingEngine.CMSDocumentRouteHelper.RegisterRoutes(String siteName)
   at CMS.URLRewritingEngine.CMSDocumentRouteHelper.EnsureRoutes(String siteName)
   at CMS.URLRewritingEngine.URLRewritingHandlers.AuthorizeRequest(Object sender, EventArgs e)
   at CMS.Base.AbstractHandler.CallEventHandler[TArgs](EventHandler`1 h, TArgs e)
   at CMS.Base.AbstractHandler.Raise[TArgs](String partName, List`1 list, TArgs e, Boolean important)
   at CMS.Base.SimpleHandler`2.RaiseExecute(TArgs e)
   at CMS.Base.SimpleHandler`2.RaiseExecute(TArgs e)
   at CMS.Base.SimpleHandler`2.StartEvent(TArgs e)
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
10/31/2017 1:23:26 PM
URL: https://[domain]/[media library folder]/media/Site-Media/tmp/Generic_SV_STicky.jpg?ext=.jpg
   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at LogFactory.LogIfCalledPriorAppInit()
   at LogFactory.CreateNewObject()
   at CMS.Base.ObjectGenerator`1.TryFactory(IObjectFactory factory, Object parameter)
   at CMS.Base.ObjectGenerator`1.CreateNewObject(Object parameter)
   at CMS.DocumentEngine.DocumentGenerator.NewInstance[NodeType](String className)
   at CMS.DocumentEngine.TreeNode.NewInstance[NodeType](String className)
   at CMS.DocumentEngine.TreeNode.New[NodeType](String className, DataRow dataRow, TreeProvider treeProvider)
   at CMS.DocumentEngine.TreeNodeProvider.CreateInfo(DataRow dr, Boolean useGenerator)
   at CMS.DataEngine.AbstractInfoProvider`3.get_InfoObject()
   at CMS.DataEngine.AbstractInfoProvider`3..ctor(Boolean initialize)
   at CMS.DocumentEngine.TreeNodeProvider..ctor()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at CMS.Core.ObjectFactory`1.CreateInstance(ConstructorInfo ctor)
   at CMS.Core.ObjectFactory`1.CreateInstance()
   at CMS.Core.ObjectFactory`1.CreateNewObject()
   at CMS.Core.ObjectFactory`1.New()
   at CMS.DataEngine.AbstractInfoProvider`1.get_ProviderObject()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
   at CMS.DataEngine.AbstractProvider.GetProviderInternal(String objectType, Boolean exceptionIfNotFound)
   at CMS.DataEngine.AbstractProvider.GetProvider(String objectType, Boolean exceptionIfNotFound)
   at CMS.DocumentEngine.TreeNodeProvider.GetProviderObject(String className)
   at CMS.DocumentEngine.TreeNodeProvider.GetDocuments()
   at CMS.DocumentEngine.TreeProvider.SelectNodes()
   at CMS.URLRewritingEngine.CMSDocumentRouteHelper.RegisterDocumentRoutes(String where, String siteName, List`1 routes)
   at CMS.URLRewritingEngine.CMSDocumentRouteHelper.RegisterRoutes(String siteName)
   at CMS.URLRewritingEngine.CMSDocumentRouteHelper.EnsureRoutes(String siteName)
   at CMS.URLRewritingEngine.URLRewritingHandlers.AuthorizeRequest(Object sender, EventArgs e)
   at CMS.Base.AbstractHandler.CallEventHandler[TArgs](EventHandler`1 h, TArgs e)
   at CMS.Base.AbstractHandler.Raise[TArgs](String partName, List`1 list, TArgs e, Boolean important)
   at CMS.Base.SimpleHandler`2.RaiseExecute(TArgs e)
   at CMS.Base.SimpleHandler`2.RaiseExecute(TArgs e)
   at CMS.Base.SimpleHandler`2.StartEvent(TArgs e)
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

Does that seem like the same issue you were referring to?

Hi Killian,

Yes it seems like you ran into same problem and rewriting and document api is called before application is fully initialized.

In case it is possible you should consider an upgrade as there is hotfix for v10 and v11 is IOC instead of object factories so it should not be a problem for next versions.

If you need to stick with v9, simplest workaround would be to set RunMenagedModulesForAllRequests to false. Is this possible for you?