The sections below describe the general steps that need to be taken in order to successfully set up a multivariate test on your website.

 

Enabling multivariate testing

 

Before you can start creating MVT tests for pages, it is first necessary to enable the module by going to Site Manager -> Settings -> On-line marketing and checking the Enable multivariate testing field.

 

Since web analytics are used to log conversion statistics for individual versions of tested pages, the Enable web analytics and Track conversions fields must additionally be enabled in the On‑line marketing -> Web analytics sub‑category of the settings. All other requirements for the correct functioning of the web analytics module also need to be fulfilled.

 

Defining an MVT test

 

To start optimizing a page on your website via multivariate testing, create and configure an MVT test object for it. This can be done in the Content -> Edit mode of CMS Desk by selecting the appropriate document from the content tree and switching to its Analytics -> MVT tests tab. Here you can find a list of all MVT tests assigned to the current document and manage them as required. The New MVT test link may be used to add an MVT test.

 

When creating or editing MVT tests, the properties described in the table below are available:

 

 

 

 

Creating testing variants on the page

 

Once a page contains an MVT test, you can start creating the content options that you wish to evaluate. This is done by defining variants for the elements that make up the content of the page. It is possible to use the following objects as variables:

 

•Web parts - these variants are separate instances of the original web part. Each variant's properties may be configured differently and an alternative Web part layout can be specified.

•Web part zones - in this case, variants are added as entire web part zones. A zone variant may contain any type or number of child web parts as required. The basic properties of the zone may also be set differently. When a new variant is added to a zone, the content of the original is automatically copied into it, so you do not have to rebuild the zone from scratch if you only need to make small modifications. Please note that creating variants for individual web parts inside zone variants is not supported.

•Editor widgets - like with web parts, each variant is a widget of the same type as the original that provides the option to set different values for its properties.

 

If you are not familiar with the basics of page development in Kentico CMS, it is recommended to read through the Development -> Web parts and Widgets chapters in the Kentico CMS Developer's Guide before you continue.

 

To add an MVT variant to a web part or zone, open the page on the Design tab, expand the context menu of the given object by right clicking its header (or through the icon), hover over the MVT variants option and select Add <object type> variant from the second level of the menu.

 

 

Then, fill in the following properties in the displayed dialog:

 

•Display name - the name used in lists of MVT variants in the administration interface.

•Code name - sets a code name that serves as an identifier of the variant.

•Description - can be used to enter a text description of the variant.

•Enabled - indicates if the variant should be considered as a possible testing option. If you disable an MVT variant, all testing combinations that include this variant will also be disabled.

 

After you enter and confirm these basic values for the new variant, a configuration dialog will be opened, just like when creating a standard web part or widget. Here, you can set up the available properties (according to the type of the given object) so that the variant generates the content that you wish to test. By default, the values set for the original will be loaded into the properties, so you only have to change the parts of the configuration that are unique for the given variant.

 

Each object for which testing variants are created will have a slider available in its header as shown below:

 

 

This slider can be used to switch between individual variants as needed (including the original). The content specified by the current selection on the page's variant sliders will be displayed in CMS Desk on the Design and Page tabs, and in Preview mode. You can also switch between different versions of the page's content by using the combination panel, which is described below in the next section. If you wish to view the content of variants while cycling through the slider on the Design tab, make sure that the Display web part content checkbox on the right side of the Edit mode header is checked.

 

The buttons on the right of the slider allow you to add new variants (), remove the currently selected one () or open a list () of all variants that the given object currently has. At any time, you can Configure () the properties of the variant currently chosen on the slider.

 

 

When editing a variant like this, you can access the properties that affect the content on the General tab. The specific settings related to the MVT testing functionality of the specific variant (the same options that were offered when creating a new variant) can be changed on the additional Variant properties tab.

 

In the case of editor widgets, multivariate testing variants are handled using a very similar approach. The only difference is that editing is done on the Page tab of CMS Desk and the slider and actions for variant management are located on the pop‑up menu of individual widgets.

 

 

You can access a list of all multivariate testing variants defined on a given document (page), by selecting it from the content tree in CMS Desk -> Content -> Edit and going to Analytics -> MVT variants. The variants of all three object types are included here, and they can be removed or edited as necessary.

 

There are some general rules that apply to all MVT variants:

 

•If you delete an object from the page, any variants that it might have will be removed along with it.

•Variants are not linked to a specific MVT test. Instead, they are either stored on the page template used by the given document (in the case of web parts and zones) or bound to the document itself (editor widget variants). This means that existing variants may be used by other MVT tests performed on the given page at another time. Also keep this in mind if you wish to Export a multivariate testing scenario to another website. Variants will be transferred along with documents or page templates, not MVT test objects.

 

 

Setting up testing combinations

 

Individual testing scenarios are represented by combinations of the MVT variants created on the page. By default, all possible combinations of the page's content will be included in a multivariate test, but you may perform some additional configuration to fine‑tune your test as required.

 

Combinations can be managed for documents that have an MVT test defined via a panel located at the top of the page editing interface in CMS Desk. This panel is available on both the Page and Design tabs, and in Preview mode.

 

 

You can choose any of the available combinations through the Combination selector. Doing so will cause the page to display the content defined by the variants that make up the given combination. The selection made through the combination panel is linked with the positions set on the MVT sliders of variable objects on the page. If you switch to a different variant through an object's slider, the current combination will also change accordingly and vice versa.

 

Pages where there are several variants for multiple objects will have a large amount of possible combinations. For this reason, it is recommended to carefully choose which combinations should be included in the page's MVT test. You can include or exclude a combination by toggling the Enabled checkbox located on the panel next to the selector. There is a close relationship between the status of a combination and its variants. If an individual variant is disabled through its properties, all combinations that include it will also be disabled. If a disabled combination is enabled at a later point, all of its variants will be enabled automatically.

 

A good way to evaluate potential combinations is to check them in the Preview mode of CMS Desk, and disable those where the variants clash visually, have conflicting logic or are otherwise incompatible. We recommend leaving the Default page combination enabled, so you can compare any improvements with the baseline statistics of the original page.

 

Another best practice is to set a descriptive custom name for each active combination, so that you can easily differentiate between combinations in the editing interface and identify them in the test's result reports. To do this, simply select a combination through the panel, enter the appropriate text into the Custom name field and click the Change button. The default names assigned to new combinations are composed of a number indicating their order, followed by a list of the display names of all variants that are included in the given combination.

 

Once the page's variants and combinations are all configured as needed, you can enable your MVT test on the Analytics -> MVT tests tab of the document. Making further modifications on the page while an MVT test is running is not recommended, since this may affect the accuracy and relevance of the measured statistics. A warning will be displayed below the combination panel on pages where a test is running.

 

The Set as result action should only be used after the conclusion of the test, once you have analyzed the logged conversion data and identified the combination that provides the best results. The action allows you to easily set the winning combination as the permanent content of the page. Simply select the given combination via the combination panel, then click the button and confirm the action. This will replace the original web parts, zones and widgets with the variants included in the currently selected combination and remove all other MVT variants from the page.

 

Configuring conversions

 

Conversions allow you to track the behaviour and actions of the website's visitors, so they must be used in order to get MVT testing results. They work on a site‑wide level and do not need to be assigned to individual MVT tests in any way. All that needs to be done is define conversion objects and link them with the actions that you wish to log for the test. There are many ways to specify that an action should be logged as a conversion. You can find information about conversion management in the Conversions chapter of this guide.

 

When an action designated as a conversion is performed anywhere on the website, the system checks if the given user has passed through a page with a running MVT test (according to the presence of a cookie). If this is the case, the conversion hit will be logged for the content combination that was assigned to the user.

 

Once you enable the MVT test and it starts running on the live site, the conversion statistics for individual combinations can be monitored using pre‑defined reports, as described in the Analyzing MVT test results topic.