Drupal, heat maps and user flows

I can make some user flow assumptions based on the the structure of this document. I can assume that when you entered this page the first thing you noticed was Arnold. Most likely the second thing you did was to read the title text to verify that this is actually a developers site and not a movie site. And since you are reading this sentence I can quite happily assume that you also understood that you had to click "Read the article" or scroll down to actually read the content on this web page. But how do I verify those assumptions? As far as I know, no one might read this text at all. So how can I find out how people are using my Drupal site?

tl;dr

Look up to your right - try using the button named Heatmap. Is this wow? Then download the HeatMap Module for Drupal for your own site.

Why Heat Map tracking?

Business Intelligence (BI) tools are basically tools that transforms raw user pattern data to make you understand how people are using your website and how it might convert in form of clicks, registrations, sales or whatnot. You probably know of one of those tools already, which has an great Drupal module - the traffic statistical tool called Google Analytics. It's awesome and it's free. With Google Analytics I can get some basic information about how the users are viewing this page since I will have information about session times, click-throughs etc. It actually caters to most basic needs. But it's only detailed enough to make me aware that something might be wrong in the design or text of a page. But not exactly what. To get more information there are many different BI tools available, as well as plugins to Google Analytics. Many of these are available to Drupal site owners for free, while some uses external data crunching and might be based on monthly subscription/one-time costs. The best tools available for knowing how users react to the website is something called Eye-tracking. The basically means that you can with the help of hardware (a web camera for instance) understand how the users eyes looks through your website. While this is really useful, it's also really expensive. Because you need software, hardware and an actual test group. It doesn't make sense to test it on yourself since you already know the functionality of your website. The second best thing might be a service that claims to do virtual Eye-tracking. But this is still starting at 199€ a month so it's anyway pretty expensive for private persons, smaller companies and entrepreneurs.

What is Heat Map tracking?

So Heat Map mouse tracking is probably the best tool that you might be able to afford on a tight budget. Basically the theory goes like this - many users will use the arrow of the mouse to point where they are looking at. But this is of course not true for everyone, I usually put the mouse pointer in the upper left corner for instance. But while it might not be the perfect tool to do eye-tracking it will give you basic understanding of how the user perceives your pages. In the case with the HeatMap module for Drupal it also give you click tracking so you can get a sense of where people assume that they should click as call to action. And last but not least it gives you an scroll map. This is so you can see if people actually have read all of your text or where they quit.

What exists for Drupal?

There exists a multitude of service online where you only need to embed a JavaScript - some of them even built a module wrapping the JavaScript. But as far as I know they all cost money. Outside of these services the module Click Heatmap exists. This is a really awesome module, but it does not integrate internally with the Drupal site as I want it to do. But if it fits your need or if you love the ClickHeat library then this is the module for you.

HeatMap Module

I created the HeatMap module for Drupal as the one heat map functionality that fits my needs. If you check in the top right corner you will actually see the viewing interface for the HeatMap module. Usually this should only be seen by the administrator/business analyst of course, but I wanted to showcase it for you. There you can get information directly from any page that you might track. For this specific web site I only track this page for showcasing.

Installing HeatMap Module

1. The installation is pretty straight forward. The only requirement is the Libraries module. When you have that installed just download and enable the module.

Install Heatmap Drupal

2. When that is installed you need to download the heatmap.js library to your Libraries directory (usually /sites/all/libraries). In the end it should look something like /sites/*/libraries/heatmap.js-develop/build/heatmap.js.

3. If you want to be able to filter on country level you also need to install the GeoIP module, but this is no requirement.

Setup HeatMap Module

Once this is done you can go to the configuration of the HeatMap Module under admin/config/system/heatmap. You have the following possibilities here:

Enable Heatmapping

This is pretty self-explanatory. If this is checked the module starts doing it's work. If it's disabled it doesn't. It's disabled by default.

Enable Theme

Here you can choose the themes that you want the HeatMap Module to collect data for. So basically a theme filter. Multichoice 

Only record specific pages

This works like the block visibility. Basically you can use opt-in, opt-out or PHP code to choose exactly where the HeatMap should collect data or not.

Only record specific user roles

Most likely you only want to record the anonymous users on an content page or logged in users on a community or whatever. You get the prerogative as the administrator.

Fragment the traffic

This give you the possibility to do a workaround when you have heavy traffic. What it does is that it lets you choose what percentage should be tracked. If you for instance have 100000 visitors, it's very likely that even one percent of them - 1000 - would be enough to get a correct assessment on how people uses your website in general.

Types of storage

The module comes with plugins so you can use other types of storage if you do not want to use Drupals built in storage. Read more under performance for information about why the built in storage might not be that good for you. If you choose another storage type, some new options might occure here.

Memcache layer

This lets you store the tracking in memcache for a certain amount of time and then do aggregated batch inserts. This is recommended from performance point of view. Requires Memcache module.

Rendering settings for each theme

Since we can't know without your help where the actual content starts on your website it is imperative that you let us know if you want to be able to do a assessment of all the tracking independent of the screen size of the visitors browser.  By default the HeatMap only shows you heatmaps depending on different sizes. This is good for responsive designs since it might look different on different devices. But to get overview of all sizes you need to fill out information here. The information you have to fill out is if you site is left, center or right aligned (for full width designs choose left). You also need to fill out the container identifier of the main content. This can either be a class or a id and you provide it like a css selector a.k.a. #maincontainer or .maincontainer.

About user roles and permissions

The HeatMap Module for Drupal comes with two different permissions. The first one is the right to Setup HeatMaps - this should naturally only be given to administrators.

The second one is View Heatmap. This permission should be given to a user role for people that needs to do the BI. What is more important is that they otherwise have equal roles as the user they want to learn about. The problem otherwise might be that if you for instance use the administrator account to view the heatmaps your page will look different then for the normal user, so because of for instance the administration module some of the mouse movement map data might be some pixels off. So please create a specific role for this.

Performance

The problem with the HeatMap module is that it does alot of writes. Every five seconds or user click it inserts or updates a record per each user that you are tracking that is visiting the website. Please be aware that this is done with post requests, so any type of caching you have will naturallly not work for the HeatMap module (and should not). For a popular website too many inserts might be unbearable, so I built in some tricks that you can use.

First of all you should install Memcache. Not just for this module, by for any Drupal site. Heck - for any website that is. Memcache lets you keep cache in RAM instead of writing it to disk. The later is usually a tiresome operation for all servers even with fast SSD disks. Also depending on what storage engine you use for MySQL it might write and lock the disk while writing, meaning that people reading your website might get access problems. If you enable Memcache option in HeatMap this will help the load a little bit since it will store all the inserts in RAM memory and then on a batch job it will bulk write it all to disk. The danger with this - as with for instance MyISAM storage engine - is that if your server dies before writing to disk, the data will be gone. But the last minutes of HeatMap data is probably not that crucial for you :)

Secondly if you still have problems and you have a popular website you can use the option to fragment the traffic. So you can for instance say that you only track 10% of the visitors. This would lead to 90% less inserts and if your site has enough visitors the 10% should statistically show results that are in line with what 100% would have shown.

The third option is to install another database type. I've create two plugins for a nosql database (CouchDB) and a search server (Elasticsearch). The main reason for chosing these two is that they use distributed scaling meaning that it scales horizontally on inserts and updates as well, not just reading, something that can be hard to fix in a MySQL database. The plugin system uses it's own hooks, so if you prefer some other system just develop it! I might add a MongoDB plugin in the future, as well as heatmap server as a service (I own a cloud storage company, so it would be easy for me to fix).

What is missing

This module is still a module in the working - if something is missing that should be in the module please let me know and I'll do the damndest to fix it. If you need very specific fixes to it I'm also available as a freelancer for consultancy.