|
|
Managing A/B tests |
|
|
|
|
|
Managing A/B tests |
|
|
|
|
|
|
Managing A/B tests |
|
|
|
|
|
Managing A/B tests |
|
|
|
|
|
|
||
The sections below describe the general steps that need to be taken in order to successfully set up an A/B test on your website.
Before you can start creating A/B tests for pages, it is first necessary to enable the module by going to Site Manager -> Settings -> On-line marketing and checking the Enable A/B testing field.
Since web analytics are used to log conversion statistics for individual page variants, the Enable web analytics and Track conversions fields must additionally be enabled in the On‑line marketing -> Web analytics sub‑category of the settings. All other requirements for the correct functioning of the web analytics module also need to be fulfilled.
To start optimizing a page via A/B testing, create and configure an A/B test object for it. This can be done in the Content -> Edit mode of CMS Desk by selecting the appropriate document from the content tree and switching to its Analytics -> A/B tests tab. Here you can find a list of all A/B tests assigned to the current document and manage them as required. The 
New test link may be used to add an A/B test.
When creating or editing A/B tests, the properties described in the table below are available:
Display name |
The name of the A/B test displayed in the administration interface. |
Code name |
Sets a code name that serves as an identifier for the test. It is also used in the name of the browser cookie used to store which of the test's page variants was assigned to a visitor. |
Test description |
Can be used to enter a text description for the test in order to give information about its purpose, goals etc. |
Test culture |
Used to selects which culture version(s) of the document should be included in the test. |
Target number of conversions |
Sets the number of conversion hits that must be logged to complete the test. Once this number is reached, the A/B test will automatically stop working and switch to the Finished status.
If the total option is selected, then the number will be compared with the total sum of the conversion hits logged for all test page variants. If any variant is chosen, then the test will be concluded when the specified number is reached by one variant (the one with the most conversion hits).
Leaving this property empty or setting it to 0 means that there will be no conversion hit limit for the test. |
Test from/to |
Sets the time interval during which the test should occur. The Calendar button ( |
Test enabled |
This property may be used to manually start or stop the test. |
Status |
Displays the current status of the A/B test. The following statuses are possible:
•Disabled - indicates that the test is not active. The original tested page will always be displayed to visitors and conversions will not be logged for the test's variants. •Not running - indicates that the test is not active yet. Used when the test is enabled, but the Test from date is still in the future. •Running - indicates that the test is currently active. •Finished - this status is automatically assigned after the Test to date passes or when the current number of conversions reaches the Target number of conversions. Tests with this status are no longer active.
Each page may have only one A/B test running at a given time. It is however possible to have multiple finished or inactive (disabled) tests assigned to a page, which can be used to archive data from previous tests or when preparing future tests. Different culture versions of a page may each have a different test running at the same time. |
|
Managing A/B tests through the Web analytics interface
If you wish to manage all tests assigned to different pages from a single location, go to CMS Desk -> On‑line marketing (or Tools) -> Web analytics. Then expand the Optimization -> A/B tests category, select any of the contained items (reports) and switch to the A/B tests tab.
Here, all A/B tests defined on the current website and their page variants may be managed in the same way as described for individual documents. The only difference is that the Original page property is additionally available when editing a test, which can be used to assign tests to specific pages. |
Test variants are created the same way as any other pages, since each variant is represented by a separate document in the website's content tree and can be managed in CMS Desk -> Content. Because of this, you can utilize all page configuration and design options provided by Kentico CMS. Please refer to the Development -> Web development overview chapter of the Developer's Guide if you require basic information about page development.
Usually, the page variants of an A/B test will all use the same basic concept, but have one key difference that sets them apart from the original page, such as:
•The position of an important page element.
•A different color scheme intended to highlight a part of the page.
•Alternative text content of headings or descriptions that could potentially be more interesting for visitors.
These are only basic examples of ideas that can be tested by individual variants. You can implement any other type of modifications according to your specific requirements.
You may also leverage existing functionality to help present all variants defined for an A/B test as a single page. The following two properties are available on the Properties -> Menu tab when editing any document:
•Show in navigation - indicates if the page variant should be displayed by navigation controls and web parts (i.e. in the website's menus).
•Show in site map - indicates if the page variant should be displayed by the Site map web part and included in the website's Google Sitemap.
In most cases, it is recommended to have these properties disabled for page variants in order to prevent users from encountering multiple pages with nearly identical content. The appropriate variant will automatically be displayed to users when they view the test's original page, so standard navigation is not necessary.
Additionally, the Exclude from search flag can be enabled for page variants on the Properties -> General tab. This will ensure that the given variant will not be returned in the results of searches performed on the website.
Once the modified versions of the original document are added to the content tree, they must be registered as variants of the given A/B test. To do this, select the original document, go to Analytics -> A/B tests, edit (
) the test and switch to its Variants tab. Click the 
New variant link and enter the following properties:
•Variant display name - the name of the variant displayed in the administration interface.
•Variant code name - sets a code name that serves as an identifier for the page variant. It is also used as the value of the A/B testing cookie stored for visitors assigned to this variant.
•Test page - must contain the path of the associated page (document). When users assigned to the variant access the original page of the given A/B test, they will see the page specified here instead. The Select button may be used to choose an existing page from the website's content tree.
Repeat this process for all documents that should be included as possible page options for the A/B test. The original document is automatically added as a variant when the A/B test is created, and highlighted in the list by a yellow background.
|
Important!
When editing a running A/B test, it is necessary to always keep the variant Test page paths up‑to‑date and ensure that the target documents actually exist in the content tree. Otherwise visitors may encounter a Page not found error if the path of their variant is not valid.
To preserve conversion statistics, variants remain in the system even if their associated document is deleted. |
It is possible to save time when defining page variants by using an action that combines the two steps described above. Simply select the A/B test's original page in CMS Desk -> Content and click the New button in the menu above the content tree. Instead of selecting a document type, choose the A/B test page variant option below the list.
This can also be done by right‑clicking the document in the content tree and selecting the New -> A/B test page variant item from the context menu. A dialog with the following options will be opened:
•Document name - sets the name of the new document that will be associated with the page variant.
•Save under location - sets the path of the document under which the page variant will be placed. The Select button may be used to choose a parent page directly.
•Assign to A/B test - selects the A/B test to which the variant will be added.
•Show in navigation, Show in site map, Exclude from search - these options may be used to directly set the corresponding properties of the new document. The default values will hide the page variant from all standard website navigation.
This will create a copy of the selected document and automatically add it as a page variant of the specified A/B test. Now you can implement the required modifications that will distinguish the variant from the original page. Please keep in mind that this copy will use the same page template as the original document, so the template must be Cloned as ad-hoc on the document's Properties -> Template tab if you wish to make any changes to the design or layout of the page.
Conversions allow you to track the behaviour and actions of the website's visitors, so they must be used in order to get A/B testing results. They work on a site‑wide level and do not need to be assigned to individual A/B tests in any way. All that needs to be done is define conversion objects and link them with the actions that you wish to log for the test. There are many ways to specify that an action should be logged as a conversion. You can find information about conversion management in the Conversions chapter of this guide.
When an action designated as a conversion is performed anywhere on the website, the system checks if the given user has passed through a page with an A/B test (according to the presence of a cookie). If this is the case, the conversion hit will be logged for the page variant assigned to the user.
Once the A/B test starts running on the live website, you can monitor the conversion statistics for individual page variants using pre‑defined reports, as described in the Analyzing A/B test results topic.
) can be used to select the exact date and time.
