Kentico MVC and Google Analytics: Custom reports, part 2
We all ask ourselves: Where are the bad hyperlinks in our website? Are there removed pages that haven't been treated with a proper 30x redirect? Are there file download links that no longer live? In this article, we'll take on the Invalid pages report in Kentico Web Analytics and create a better alternative in Google Analytics. We'll demo the steps with our standard Dancing Goat sample site that you used in the previous articles in this series.
Benefits
The original Invalid pages report in Web Analytics (WA) displays a chart with a cumulative number of failed HTTP requests in a given time scope. It also contains a table with pages (URLs) that fail at most.
In the new Invalid URLs report in Google Analytics (GA), you'll also be able to see where the failed HTTP requests originated. This will bring two major advantages to the table:
- You'll be able to distinguish between broken local links and other failed requests.
- You'll quickly learn where exactly the broken links are on your website.
You'll be able to filter out all the failed attempts to hack into /wp-login.php (the login page of WordPress), /.env (a file with environment variables on Unix-based systems), and other kinds of irrelevant requests. That way, you'll be able to just focus on cleaning broken links in your site or sites that you can influence.
In the end, you'll have a report like this:
/ga-saved-reports-invalid-urls.png "ga-saved-reports-invalid-urls.png")
Prerequisites
Hands down, the implementation will take a few more steps. GA per se does not have any knowledge of your website's error handling measures. You need to tell it what exactly it can consider a failed request. Then, you'll also need to supply GA with the values of the referer HTTP request header. Yes, you read it right. GA does not collect that information by default. We'll again demonstrate the implementation in the standard Dancing Goat (DG) MVC sample site that you can deploy to your IIS with the Kentico installer. To make the report work, you should have your DG site set up according to the third article of this series.
How a failed request looks in the Dancing Goat site
The very last MVC route in the RouteConfig.cs file points to the HttpErrorsController class. If all previous routes fail to serve the request, this controller eventually picks it up in its NotFound action method, just thanks to the routing conventions in MVC. If the request ends in any preceding route and if the corresponding controller action throws an unhandled exception, the Application_Error method in the Global.asax.cs file takes care of it. This method, together with the Handle404Error helper method, also covers exceptions from all other ASP.NET assemblies. Both the NotFound and the Handle404Error methods set the 404 status:
Response.StatusCode = 404;
As a last resort, there is also the httpErrors setting under configuration > system.webServer in the web.config file. Thanks to this directive, IIS intercepts all responses with the 404 status code and executes the /en-US/HttpErrors/NotFound URL that ends up in the same NotFound action method as before. Not only it catches all problems that arose in all other action methods and other ASP.NET assemblies, but it also handles all 404 requests that were short-cut at the bare IIS level.
Thanks to this analysis, you now know that indeed all failed requests end up solely in the NotFound action method.
Setting up an event in GTM
Let's bind a custom GA event to the view of the NotFound action method through Google Tag Manager (GTM). But before you set off for clicking around in GA and GTM, you'll need to create a small JavaScript file that works together with the GTM snippet you added in the third part of this series. The JS file would invoke the event every time the NotFound action method runs. Under DancingGoat > Scripts, add a new gtmInvalidUrlReporter.js file with the following contents:
window.dataLayer = window.dataLayer || [];
window.dataLayer.push({ "event": "invalidUrl" });
This code utilizes the data layer of GTM to pass the event to GA.
Then, add a reference to the file into the Views > HttpErrors > NotFound.cshtml file:
@{
ViewBag.Title = ResHelper.GetString("DancingGoatMvc.NotFound.Title");
}
<div class="row">
<div class="not-found">
<div class="col-md-5 col-lg-4 col-lg-offset-1 sm-hidden xs-hidden">
<img class="img-responsive" alt="Page not found" src="@Url.Content("~/Content/Images/404.jpg")">
</div>
<div class="col-md-7 col-lg-6">
<div class="not-found-content">
<h2 class="">
@ResHelper.GetString("DancingGoatMvc.NotFound.Title")
</h2>
<p>@ResHelper.GetString("DancingGoatMvc.NotFound.Message")</p>
<p>
@Html.Raw(ResHelper.GetStringFormat("DancingGoatMvc.NotFound.Instructions",
Html.ActionLink(ResHelper.GetString("DancingGoatMvc.NotFound.Homepage"), "Index", "Home"),
Html.ActionLink(ResHelper.GetString("DancingGoatMvc.NotFound.ContactUs"), "Index", "Contacts")))
</p>
<a href="@Url.Action("Index", "Home")" class="btn">
@ResHelper.GetString("DancingGoatMvc.NotFound.GoHome")
</a>
</div>
</div>
</div>
</div>
@Scripts.Render("~/Scripts/gtmInvalidUrlReporter.js")
With the event being prepared at the MVC app side, test your event in GTM. Open up the home page of the DG live site in your browser. Make sure you have agreed to using cookies. Then open the GTM dashboard in another tab. It is vital to open the dashboard in the same browser as the DG site, as the preview mode wouldn't work otherwise. Click the Preview button.
/gtm-preview.png "gtm-preview.png")
Switch back to the tab with your DG site and refresh the home page. You should see a GTM preview bar at the bottom of the page. If you don't, we recommend that you check out the troubleshooting guide on analyticsmania.com. In the GTM preview bar, you'll see three records in its left pane:
- Window Loaded
- DOM Ready
- Page View
Head back to the GTM dashboard > Triggers. Click New. Select Custom Event as the trigger type and invalidUrl as the event name. Save this trigger with a friendly name (e.g. Invalid URL).
Return to the DG live site and navigate manually to /en-US/HttpErrors/NotFound. Now you should see the GTM preview bar having the invalidUrl record in its left pane:
- Window Loaded
- DOM Ready
- invalidUrl
- Page View
This indicates that your DG site communicates with GTM correctly.
The next thing you need to do in GTM is setting up a new tag. So, in the navigation of the GTM dashboard, click Tags > New. Give the tag a friendly name (e.g. Invalid URL) and configure it according to the following screenshot.
The naming of the fields in the form might be a bit confusing but according to Google's conventions, the most specific information (the page URL) should be passed through the Label field. Note that once you add more GA events to your site, you can give them various weights in the Value field. When you're done in the Tag Configuration section, don't forget to bind your new tag with the trigger you've just set up.
Publish the GTM container.
Passing the HTTP referer value
The last major task that precedes the creation of the GA report is extracting the value of the referer HTTP header from the response provided by the NotFound action method. This will tell GA where the request came from.
In GA, navigate to Admin > Property > Custom Definitions > Custom Dimensions. Click NEW CUSTOM DIMENSION, give the dimension a friendly name, make sure it is scoped to hits and active.
Once you save the form, note down the Index value of your newly created dimension.
In the GTM dashboard, navigate to Variables > User-Defined Variables > Google Analytics Settings. In the Variable Configuration section, expand More Settings > Custom Dimensions. Add a dimension with the Index value you noted down previously and the Dimension Value being set to {{Referrer}}.
/gtm-variables-google-analytics-settings-more-settings-custom-dimensions.png "gtm-variables-google-analytics-settings-more-settings-custom-dimensions.png")
Save the configuration and publish the container again.
Mocking up failed requests
Before you can configure the Invalid URLs report in GA, you should allow GTM to gather some data for you. You can quickly mock some requests by navigating to the Articles or Products section, viewing a specific article and then trying to load the page with a slightly malformed GUID in its URL.
For instance, instead of the original URL of the Coffee processing techniques article ...
http://your-fqdn/your-dg-site/en-US/Articles/c3793d7a-cc6e-44b2-ab82-cbe8e2a86732/Coffee-processing-techniques
... just put an additional character to the GUID segment of the URL:
http://your-fqdn/your-dg-site/en-US/Articles/c3793d7a-cc6e-44b2-ab82-cbe8e2a86732x/Coffee-processing-techniques
This will submit direct page hits, without the HTTP referer value. To test out sending of the header value, add a few bad hyperlinks to your home page. In Views > Home > Index.cshtml, add a few hard-coded anchor tags with malformed URLs:
<a href="http://your-fqdn/your-dg-site/en-US/Articles/c3793d7a-cc6e-44b2-ab82-cbe8e2a86732-bad/Coffee-processing-techniques">Bad link 1</a>
<a href="http://your-fqdn/your-dg-site/en-US/Articles/a57115be-452b-419f-8ec5-c6305a44fb61-bad/Origins-of-Arabica-Bourbon">Bad link 2</a>
Click through those bad links a few times to gather some data. After every click, check out GA to see whether an event was processed under Realtime > Events. If nothing happens, double-check the GTM preview bar to see whether the invalidUrl event was triggered. If it was, click the event in the preview bar to see whether the event caused the Invalid URL tag to be fired in GTM. Also, make sure your browser does not block tracking, especially in anonymous mode.
/ff-tracking-protection.png "ff-tracking-protection.png")
Configuring the report in GA
Building your Invalid URLs report from the ground up wouldn't be exactly the most efficient way. You can actually custom-tailor an existing report. Then you can save it in GA under Customization > Saved Reports.
First, navigate to Behavior > Events > Top Events. In the bottom-right corner of the page, click Refresh Report a few times until the Invalid URL event pops up. Be patient. Only the Realtime > Events report has fresh data. The other reports are re-processed with delays. Once the event appears, click through to see the Event Action value (Invalid URL hit). Click through once more to get to the Event Label dimension. Once that's done, you can add your HTTP Referrer secondary dimension.
/ga-behavior-events-top-events-invalid-url-hit-secondary-dimension.png "ga-behavior-events-top-events-invalid-url-hit-secondary-dimension.png")
After that, you can simply use the SAVE button in the top right corner of the page to save the current configuration into your report. You can also save the report with a filter that omits other HTTP Referrer values than your own FQDN. Just click the advanced link and set a proper filter on that dimension. You can both include and exclude hits that meet the criteria.
/ga-saved-reports-invalid-urls-advanced-filter.png "ga-saved-reports-invalid-urls-advanced-filter.png")
Download the report
You can also download a simplified version of the report in the GA Solution Gallery. However, please bear in mind that the gallery only allows us to store the report itself, without the GTM configuration it needs. Although there is a GTM Template Gallery, it does not support custom triggers and modifications to the Google Analytics Settings with custom dimensions. We cannot publish the whole GTM configuration in the gallery.
Conclusion
You now have a report telling you where the weak points in your intra-site navigation are. When implementing your report, you gained lots of knowledge about GTM and GA that would help you in gathering other arbitrary data about your website, collect other kinds of events and configure reports. We hope you enjoyed the ride.
We'd love to get your feedback on our series. Feel free to praise and criticize, either via the comments below, or at training@kentico.com.