Contact groups improvements in detail
As you know, in Kentico 8.1, the Online marketing team focused on Contact groups, Personas, and Scoring performance (see
Vita's post).
This technical post explains in depth the improvements we made to Contact groups to help you better understand how the changes affect the performance of your site. It assumes you’ve used Contact groups already and doesn’t go into how they work in general.
So what was improved in Kentico 8.1 for Contact groups? They got faster. A
lot faster. In every way; from complete recalculations of all contacts, to smaller recalculations that now occur almost immediately when visitors perform activities on your website.
Let’s jump straight into the details.
The major improvements came primarily (but not solely) as a result of the following:
·
All macro rules were converted to SQL queries
·
All operations were made asynchronous
·
All recalculations are now done in batches
·
A lot of unnecessary recalculations are now filtered
Let’s talk about each of those a little.
Macro rules to SQL queries
Contact group macros are evaluated on 2 events. The first is
once a day, unless you turn it off. Why every day? Just to make sure each contact group stays up to date day by day. But in most cases, the contact group gets recalculated much more frequently —
every 10 seconds, in fact. In this second event, just the contacts that actually performed activities (or those that changed in some way) during the last 10 seconds are re-evaluated.
In Kentico 8.1, we improved the recalculation time on both of these events. Let's see a pseudo-code of how it works:
foreach(var contactGroup in allContactGroups)
{
// Do all the macros used in the Contact Group provide translators to SQL?
if(contactGroup.IsTranslatableIntoSQL())
{
// Yay! All of the macro rules can be translated. Process the contacts in one query.
evaluateContacts(contactGroup, contactsToRecalculate);
}
else
{
// Fallback to old recalculation.
foreach(var contact in contactsToRecalculate)
{
// This can cause a lot of DB queries.
evaluateMacros(contact, contactGroup);
}
}
}
You always want the
contactGroup.IsTranslatableIntoSQL method, (invented for explanation purposes), to return true. What exactly does that mean? If you’re only using our predefined macro rules, you don't have to do anything. All our macro rules can already be translated into SQL queries. But what if you have custom macro rules? Then, you have to provide the implementation yourself. In C# it is done
like this — you actually bind your custom macro rule to the ObjectQuery<ContactInfo> that represents the SQL query. This makes things easier, since you don't have to worry about database separation or the database schema.
If you can’t do the implementation yourself but still use custom macros, the event log will tell you. And the UI in contact groups will tell you. Even though it is still usable, you would lose a lot of performance gain by ignoring that.
Async everything
All recalculations are asynchronous. Logging activities to a database is asynchronous. Even contact changes are now recalculated asynchronously. What this means is that you get contact groups
evaluated every 10 seconds, but response time isn’t affected. On high-traffic websites, you might want to amend that 10-second interval — more about that in the next section.
Batch everything!
In Kentico 8.1, we do all the recalculations in batches as long as you provide the SQL translator for your custom macro rules. We group together contacts that were active during the last 10 seconds and
bulk insert their activities into the database. We then do one DB query to recalculate these contacts for each contact group that may be affected — more about that in Filtering. You can group those batches into different time frames through the
CMSProcessContactActionsInterval web.config key. It's always a tradeoff between performance and how fast you want to have your contact groups / scoring / personas up to date for your current visitors.
Filtering
When a contact performs, for example, a page-visit activity, only contact groups that actually contain this page-visit activity need to be and are recalculated. You can set
which activities affect your custom macro rule, so it doesn't get recalculated when it doesn't need to be. You can set that up in your MacroRuleMetadata. For more, see
this documentation section.
In-memory queue for activities
In Kentico 8.0, when a contact performed an activity, that activity got saved into a file located in the App_Data/Modules/OnlineMarketing folder. Then, a scheduled task read that file and processed every single line of it, triggering the recalculations. In Kentico 8.1, all activities and contact changes are put into an in-memory queue that gets processed by an always-running worker thread, processing them every 10 seconds. These activities are immediately inserted into the database when an application pool needs to be recycled, so you don’t lose any data. This doesn't apply to unexpected server crashes, though.
Does any of this affect custom activities?
All of the above applies to custom activities — asynchronous logging, batch recalculations, and SQL translations (if translators are provided). That is, as long as you log them via the ActivityLogger.LogActivity method.
Upgrade from Kentico 8.0 to Kentico 8.1
If you are upgrading from Kentico 8.0 to Kentico 8.1, you don't have to worry about anything other than optimizing your existing contact groups so that they use only predefined macro rules or your
custom macro rules with translators. Those recalculation scheduled tasks that are left from previous versions get executed one last time after the upgrade and then get deleted. Go ahead and upgrade! :)
I tried to keep this post concise so that you wouldn't get overwhelmed with details and only picked the most important ones. Thanks for reading!