Kentico CMS 7.0 Developer's Guide

Creating an A/B test

Creating an A/B test

Previous topic Next topic Mail us feedback on this topic!  

Creating an A/B test

Previous topic Next topic JavaScript is required for the print function Mail us feedback on this topic!  

The sections below describe the tasks that must be performed to successfully set up an A/B test on your website.

 

Enabling A/B testing

 

Before you can start creating A/B tests for pages, it is first necessary to enable the module by going to Site Manager -> Settings -> On-line marketing and checking the Enable A/B testing field.

 

devguide_clip0649

 

Since web analytics are used to log conversion statistics for individual page variants, the Enable web analytics and Track conversions fields must also be enabled in the On‑line marketing -> Web analytics sub‑category of the settings. All other requirements for the correct functioning of the Web analytics module also need to be fulfilled, so please refer to Modules -> Web analytics if you encounter any problems with conversion tracking while performing an A/B test.

 

Defining an A/B test for a page

 

To start optimizing a page on your website via A/B testing, create and configure an A/B test object for it. To do this, open CMS Desk, select the appropriate document from the content tree and switch to its Analytics -> A/B tests tab. Here you can find a list of all A/B tests assigned to the currently selected document and manage them as required.

 

Click NewABTest New test and fill in the A/B test's properties according to the information in the table below:

 

Display name

The name of the A/B test displayed in the administration interface.

Code name

Sets a code name that serves as an identifier for the test. It is also used in the name of the browser cookie used to store which of the test's page variants was assigned to a visitor.

Test description

Can be used to enter a text description for the test in order to give information about its purpose, goals, etc.

Test culture

Used to selects which culture version(s) of the document should be included in the test.

Target number of conversions

Sets the number of conversion hits that must be logged to complete the test. Once this number is reached, the A/B test will automatically stop working and switch to the Finished status.

 

If the total option is selected, then the number will be compared with the total sum of the conversion hits logged for all test page variants. If any variant is chosen, then the test will be concluded when the specified number is reached by one variant (the one with the most conversion hits).

 

Leaving this property empty or setting it to 0 means that there will be no conversion hit limit for the test.

Test from/to

Sets the time interval during which the test should occur. The Calendar button (Calendar) can be used to select the exact date and time.

Test enabled

This property may be used to manually start or stop the test.

Status

Displays the current status of the A/B test. The following statuses are possible:

 

Disabled - indicates that the test is not active. The original tested page will always be displayed to visitors and conversions will not be logged for the test's variants.

Not running - indicates that the test is not active yet. Used when the test is enabled, but the Test from date is still in the future.

Running - indicates that the test is currently active.

Finished - this status is automatically assigned after the Test to date passes or when the Current number of conversions reaches the Target number of conversions. Tests with this status are no longer active.

 

Each page may have only one A/B test running at a given time. It is however possible to have multiple finished or inactive (disabled) tests assigned to a page, which can be used to archive data from previous tests or when preparing future tests. Different culture versions of a page may each have a different test running at the same time.

 

devguide_clip1226

 

Click OK when everything is configured. The test will now be assigned to the document.

 

 

InfoBox_Note

 

Managing A/B tests through the Web analytics interface

 

If you wish to manage all tests assigned to different pages from a single location, go to CMS Desk -> Tools (or On-line marketing) -> Web analytics. Then expand the Optimization -> A/B tests category, select any of the contained items (reports) and switch to the A/B tests tab.

 

Here, all A/B tests defined on the current website and their page variants may be managed in the same way as described for individual documents. The only difference is that the Original page property is additionally available when editing a test, which can be used to assign tests to specific pages.

 

Adding page variants to an A/B test

 

Test variants are created the same way as any other pages, since each variant is represented by a separate document in the website's content tree and can be managed in CMS Desk -> Content. Because of this, you can utilize all page configuration and design options provided by Kentico CMS. Please refer to the Development -> Web development overview chapter if you require basic information about page development.

 

Usually, the page variants of an A/B test will all use the same basic concept, but have one key difference that sets them apart from the original page, such as:

 

The position of an important page element.

A different color scheme intended to highlight a part of the page.

Alternative text content of headings or descriptions that could potentially be more interesting for visitors.

 

These are only basic examples of ideas that can be tested by individual variants. You can implement any other type of modifications according to your specific requirements.

 

You may also leverage existing functionality to help present all variants defined for an A/B test as a single page. The following two properties are available on the Properties -> Navigation tab when editing any document:

 

Show in navigation - indicates if the page variant should be displayed by navigation controls and web parts (i.e. in the website's menus).

Show in site map - indicates if the page variant should be included in the website's Google Sitemap and displayed by the Site map web part.

 

devguide_clip1708

 

In most cases, it is recommended to have these properties disabled for page variants in order to prevent users from encountering multiple pages with nearly identical content. The appropriate variant will automatically be displayed to users when they view the test's original page, so standard navigation is not necessary.

 

Additionally, the Exclude from search flag can be enabled for page variants under the Search & SEO section. This will ensure that the given variant will not be returned in the results of searches performed on the website.

 

Once the modified versions of the original document are added to the content tree, they must be registered as variants of the given A/B test. To do this, select the original document, go to Analytics -> A/B tests, edit (Edit) the test and switch to its Variants tab. Click NewABVariant New variant and enter the following properties:

 

Variant display name - the name of the variant displayed in the administration interface.

Variant code name - sets a code name that serves as an identifier for the page variant. It is also used as the value of the A/B testing cookie stored for visitors assigned to this variant.

Test page - must contain the path of the associated page (document). When users assigned to the variant access the original page of the given A/B test, they will see the page specified here instead. The Select button may be used to choose an existing page from the website's content tree.

 

devguide_clip1233

 

Repeat this process for all documents that should be included as possible page options for the A/B test. The original document is automatically added as a variant when the A/B test is created, and highlighted in the list by a yellow background.

 

devguide_clip0642

 

 

 

Important!

 

When editing a running A/B test, it is necessary to always keep the variant Test page paths up‑to‑date and ensure that the target documents actually exist in the content tree. Otherwise visitors may encounter a Page not found error if the path of their variant is not valid.

 

To preserve conversion statistics, variants remain in the system even if their associated document is deleted.

 

It is possible to save time when defining page variants by using an action that combines the two steps described above. Simply select the A/B test's original page in CMS Desk -> Content and click the New button in the menu above the content tree. Instead of selecting a document type, choose the A/B test page variant option below the list.

 

devguide_clip0651

 

This can also be done by right‑clicking the document in the content tree and selecting the New -> A/B test page variant item from the context menu. A dialog with the following options will be opened:

 

Document name - sets the name of the new document that will be associated with the page variant.

Save under location - sets the path of the document under which the page variant will be placed. The Select button may be used to choose a parent page directly.

Assign to A/B test - selects the A/B test to which the variant will be added.

Show in navigation, Show in site map, Exclude from search - these options may be used to directly set the corresponding properties of the new document. The default values will hide the page variant from all standard website navigation.

 

Fill in these properties accordingly and click Save Save to confirm the entered data. This will create a copy of the selected document and automatically add it as a page variant of the specified A/B test. Now you can implement the required modifications that will distinguish the variant from the original page. Please keep in mind that this copy will use the same page template as the original document, so the template must be Cloned as ad-hoc on the document's Properties -> Template tab if you wish to make any changes to the design or layout of the page.

 

Configuring conversions

 

Conversions allow you to track the behaviour and actions of the website's visitors, so they must be used in order to get A/B testing results. They work on a site‑wide level and are not assigned to A/B tests in any way. All that needs to be done is define conversion objects and link them with the actions that you wish to log for the test.

 

There are many ways to specify that an action should be logged as a conversion. For example, to have the system log a conversion hit whenever a specific page is viewed by a user, select the given document from the content tree in CMS Desk and switch to its Analytics -> Settings tab. Use the Track conversion name property to either create a conversion via the New button, or Select an existing one.

 

devguide_clip0692

 

Additionally, many web parts and widgets that allow users to perform important actions (e.g. registration) also have the Track conversion name property, which can be used to specify a conversion in the same way. Further details about conversion management may be found in the Modules -> Web analytics -> Conversions chapter.

 

When an action designated as a conversion is performed anywhere on the website, the system checks if the given user has passed through a page with an A/B test (according to the presence of a cookie). If this is the case, the conversion hit will be logged for the page variant assigned to the user.

 

Once the A/B test starts running on the live website, you can monitor the conversion statistics for individual page variants using pre‑defined reports, as described in the Analyzing A/B test results topic.