Web farm configurations may be causing you trouble--here’s a solution

Example 1: Cache inconsistencies

A frequent issue that arises in web farm configurations is puzzling inconsistencies in cache invalidation. In dev and test environments, cache invalidation may be working perfectly. However, in production, authors may be reporting sporadic problems. They make an update, but it isn’t seen in the live environment because old content is still cached. It may be even more baffling, because when you look you see the updates. The problem seems random and is difficult to troubleshoot.

This is likely caused because a web farm cache invalidation task is not processed by all application instances, due to web farm configuration problems. Whether you, authors, or end users see the problem, depends on which web farm member is serving the content.

Example 2: Media synchronization issues

Another common issue is the incomplete synchronization of media files across app instances when they are staged from a lower environment. When comparing local media storage, you may discover that some files are missing on one server, and other files are missing on another. How does this happen? It is likely caused because a web farm media task is not processed by all applications instances, again due to web farm configuration problems. Again, whether someone sees an image or is missing an image, depends on which web farm member is serving pages to them.

Kentico Xperience web farm race conditions

These example problems are caused when multiple web application instances within a Kentico Xperience web farm share the same server name. It sets the stage for a race condition that can cause app instances to only receive a subset of important web farm synchronization tasks, like media updates & cache invalidation. To illustrate, think of the app instances as dogs trained to come for treats when their names are called. Imagine the effect of a dog trainer giving them all the same name, Rover, and providing a single serving of treats—a web farm queue of tasks in this analogy. When she yells their name, the first “Rover” will get the treats and the other’s will not. Next time, one of the other “Rovers” will win when their shared name is called. To solve the problem, the trainer needs to give each dog its own name, call each one individually, and give each its own reward of treats. When Xperience app instances are sharing the same web farm names, it causes the symptoms to disappear and reappear in each app instance in a baffling way. The fundamental problem becomes illusive. We’ve seen people troubleshooting network infrastructure, verifying if each app instance has the correct code, and adding extra code to invalidate caches. All attempts to solve symptoms caused by this web farm configuration problem. In Xperience, if each app instance is given a unique web farm server name, Xperience will create a separate queue of tasks for each app instance to ensure each one receives all the synchronization tasks needed to keep it up to date with the entire farm.

Kentico Xperience web farm legacy terms and confusion

Confusion is caused by the legacy terminology used in Kentico's documentation and object naming. When the Xperience web farm feature was introduced, the Portal Engine development model was king, and generally featured a one-to-one relationship between an app instance and a server. At the time, “server” and “app instance” were practically synonymous in the Kentico Xperience architecture. However, the shift towards Kentico Xperience's MVC development model necessitated multiple app instances per environment. The first and immediate reason was to segregate CMS admin functionalities from MVC websites. Kentico solved this by automatically adding “_AutoExternalWeb” to the web farm server name of the MVC app instances. The MVC development model also caused creating separate MVC app instances for each Xperience website to be very common. Additionally, the adoption of modern hosting strategies like Azure autoscaling and app service deployment slots further multiplied the use of separate app instances. This evolution means that each 'server' in modern configurations likely hosts multiple app instances, making the term "web farm app instance" a more accurate reflection of today's Xperience infrastructure realities. So, to avoid confusion when configuring Kentico Xperience web farms, think "web farm app instance" every time you read "web farm server".

The out-of-the-box solution

Kentico Xperience provides an automatic mode for naming web farm members that works as long as there is never more than one MVC app and one CMS admin app on each machine or Azure App Service plan. It works by using the machine name, plus “_AutoExternalWeb” for MVC apps, to create each web farm server name.

However, if more than these two app instances are needed on each machine—scenarios common when hosting multiple websites, using Azure autoscaling, or using Azure app service deployment slots—the only out-of-the-box solution is to use the manual web farm mode, and configure each app instance with a custom name. However, this has two down sides: 1) It is really easy to forget to configure a custom name, and 2) it doesn’t handle Azure autoscaling.

The just-works solution: Enhanced Web Farm Name Provider

To solve this challenge, members of the Kentico community at BlueModus created a solution to provide automatic web farm names that are unique to each app instance. Now, whether your environment is hosted in Azure or on-prem with IIS, or whether your solution is built with .NET Core or with .NET Framework 4.8, there is an easy solution. Simply add the NuGet package, XperienceCommunity.EnhancedWebFarmNameProvider to your Kentico Xperience 13 projects, including both the CMS and MVC projects. It will ensure unique web farm server names are created for each app instance.

Closing

Adding the Kentico Xperience Enhanced Web Farm Name Provider can save hours of troubleshooting and significantly improve the stability and consistency of your Kentico Xperience environments. Adding this solution helps ensure that all instances are always up-to-date, providing a seamless experience and preventing frustration for content editors and end-users alike.

Acknowledgements

Sincere thanks to Brandon Henricks and David Rector at BlueModus, for helping research this solution and testing it across multiple hosting environments and target frameworks. David’s dog bowl/queue/race-condition analogy was brilliant.