Fix everything!
These amazing widgets! will fix everything.
Let widgets! work for you.
{% endblock %} When the time comes to add in the A/B testing features, we'll change this a little, but most of this is good as it is. We'll also need a page to direct the users to if they want to buy a widget. We'll first define a route for this page in the src/web_ab/routes/home.clj file. The following is the route and the controller: (defn purchase-page [] (layout/render "purchase.html" {})) (defroutes home-routes (GET "/" [] (home-page)) (GET "/purchase/" [] (purchase-page))) Chapter 8 [ 217 ] The view is defined in the src/web_ab/views/templates/purchase.html file. This file is very similar to the preceding template file, except that it's considerably simpler. It just contains a thank you message for the left panel, and there's no button or link on the right-hand side. For more details about this page, see the code download. In fact, this is enough to define the base, control site in this project. Now let's look at what we need to do to define the A/B testing features. Implementing A/B testing Adding A/B testing into the site that we have so far will be pretty straightforward web development. We'll need to define a model and functions that implement the test framework's basic functionality. We can then incorporate them into the site's existing controllers and views: 1. The code that defines the data and the database settings will go into the src/web_ab/models/schema.clj file. It will start with the following namespace declaration: (ns web-ab.models.schema (:require [clojure.java.jdbc :as sql] [noir.io :as io])) 2. The first facet of this section of the site that we'll define is the model. We'll add a table to the database schema that defines a table to track the A/B participants: (defn create-abtracking-table [] (sql/with-connection db-spec (sql/create-table :abtracking [:id "INTEGER IDENTITY"] [:testgroup "INT NOT NULL"] [:startat "TIMESTAMP NOT NULL DEFAULT NOW()"] [:succeed "TIMESTAMP DEFAULT NULL"]))) A/B Testing – Statistical Experiments for the Web [ 218 ] 3. Now, in the src/web_ab/models/db.clj file, we'll define some low-level functions to work with the rows in this table. For this file, we'll use the following namespace declaration: (ns web-ab.models.db (:use korma.core [korma.db :only (defdb)]) (:require [web-ab.models.schema :as schema] [taoensso.timbre :as timbre])) 4. The first function in this namespace will take a group keyword (:control or :test) and insert a row into the database with a code that represents that group and the default values for the starting time (the current time) and the time in which the interaction succeeds (NULL): (defn create-abtracking [group] (get (insert abtracking (values [{:testgroup (group-code group)}])) (keyword "scope_identity()"))) 5. Next, we'll create a function that sets an abtracking object's succeed field to the current time. This will mark the interaction as a success: (defn mark-succeed [id] (update abtracking (set-fields {:succeed (sqlfn :now)}) (where {:id id}))) These, along with a few other functions that you can find in the code download for this chapter, will form a low-level interface with this data table. Most of the time, however, we'll deal with A/B testing using a slightly higher-level interface. This interface will live in the src/web_ab/ab_testing.clj file. It will contain the following namespace declaration: (ns web-ab.ab-testing (:require [noir.cookies :as c] [taoensso.timbre :as timbre] [web-ab.models.db :as db] [incanter.stats :as s] [clojure.set :as set] [web-ab.util :as util]) (:import [java.lang Math])) Chapter 8 [ 219 ] To understand the code in this module, we need to first talk about how the A/B testing system will work. We have the following number of requirements that we need to make sure are implemented: • If the users have visited the site before the A/B test, they should see the control version of the site. We assume that there's a tracking cookie already being used for this. In this case, the cookie will be named visits, and it will simply track the number of times a user has visited the home page of the site. • If this is the users' first visit to the site, they will be randomly assigned to the control group or the test group, and they'll be shown the appropriate page for that group. Also, they'll receive a tracking cookie for the observation that they are, and we'll insert the tracking information for them into the database. • If the users have visited the site earlier and are participants in the A/B test, they should see the same version of the site that they saw previously. • Finally, when a user who is a participant in the experiment visits the purchase page, that observation in the experiment will be marked as a success. We'll write functions for most of these cases as well as a function to route the user to the right branch whenever one visits the front page. We'll write another function to handle item number four. For the first function, we'll implement what's necessary to start a new observation in the experiment. We'll enter the functions into the database and insert the tracking cookie into the session: (defn new-test [test-cases] (let [[group text] (rand-nth (seq test-cases)) ab-tracking (db/get-abtracking (db/create-abtracking group))] (c/put! :abcode (:id ab-tracking)) text)) The functions in the db namespace (aliased from web-ab.models.db) are from the low-level model interface that we just defined. In fact, the implementation for create-abtracking is listed on the preceding page. The c/put! function is from the noir.cookies namespace. It inserts the cookie value into the session. In this case, it inserts the tracking instance's database ID under the abcode key. Finally, new-test returns the text that should be used on the page. A/B Testing – Statistical Experiments for the Web [ 220 ] The next function for this level of abstraction is get-previous-copy. This is used whenever a user who is already a participant in the experiment visits the page again. It takes a database ID and the different versions of the site that are being used in the current test, and it retrieves the row from the database and looks up the right copy text to be used on the page, given whether the observation is in the control group or the test group: (defn get-previous-copy [ab-code test-cases] (-> ab-code db/get-abtracking :testgroup db/code->group test-cases)) This function simply runs the input through a number of conversions. First, this function converts it to a full data row tuple based on the database ID. Next, it selects the testgroup field, and it translates it into a group keyword. This is finally translated into the appropriate text for the page, based on the group keyword. The next function that we're going to look at ties the two previous functions together with item number one from the preceding list (where the returning visitors are shown the control page without being entered into the experiment): (defn start-test [counter default test-cases] (let [c (Long/parseLong (c/get counter "0")) ab-code (get-abcode-cookie)] (c/put! counter (inc c)) (cond (and (>= ab-code 0) (> c 0)) (get-previous-copy ab-code test-cases) (and (< ab-code 0) (> c 0)) default :else (new-test test-cases)))) First, this function expects three parameters: the cookie name for the counter, the default text for the control page, and a map from the group keywords to page text. This function looks at the value of the counter cookie and the abtest cookie, both of which will be -1 or 0 if they're not set, and it decides what should be displayed for the user as well as inserts whatever needs to be inserted into the database. Chapter 8 [ 221 ] In the preceding code snippet, we can see that the calls to the two functions that we've just looked at are highlighted in the code listing. Also, here we define a function that looks for the abtest cookie and, if it's found, we mark it as having succeeded, shown as follows: (defn mark-succeed [] (let [ab-code (get-abcode-cookie)] (when (> ab-code -1) (db/mark-succeed ab-code)))) Finally, once the experiment is over, we need to perform the analysis that determines whether the control page performed better or the test page: (defn perform-test ([ab-testing] (perform-test ab-testing 0.05)) ([ab-testing p] (let [groups (group-by-group ab-testing) t (-> (s/t-test (to-flags (:test groups)) :y (to-flags (:control groups)) :alternative :less) (select-keys [:p-value :t-stat :x-mean :y-mean :n1 :n2]) (set/rename-keys {:x-mean :test-p, :y-mean :control-p, :n1 :test-n, :n2 :control-n}))] (assoc t :p-target p :significant (<= (:p-value t) p))))) To perform the actual analysis, we use the t-test function from incanter.stats in the Incanter library (http://incanter.org/). We'll get into this analysis in more detail later in the chapter. For now, let's just pay attention to how the data flows through this function. The t-test function returns a map that contains a lot of numbers. For the output, we need to select some of this information and rename the keys for some of the data that we will use. We use the core select-keys function to select only the information that we need, and we use clojure.set/rename-keys to give the rest of the names that will fit our current domain in a better manner. To the results of the analysis, we'll also add a couple of other pieces of data. One will be the alpha value, that is, the target value for p that we're trying to improve upon. The other depends on whether the results are significant or not. This is found by testing the value of p against the significance level that we're trying for. A/B Testing – Statistical Experiments for the Web [ 222 ] With the low-level and high-level interfaces to the A/B testing process in place, we can turn our attention to actually using it. First, we need to update the view template for the home page from what we listed in the preceding snippet. Remember, the file is in src/web_ab/views/templates/home.html. We want to simply change the name of the link to go to the purchase page. It needs to be a parameter that we can use to insert a value into the template. For instance, the following snippet contains the updated version of the right-hand panel, including the highlighted line that we can use to insert the text into the page: 