How to Set Up a Split Test

1. Ensure that Google Analytics is in use on your webpage. You can either use the standard Google Analytics Javascript code, or create a Trigger Set containing the "Google Analytics: Load" Action.
2. Create two or more Trigger Sets to test against each other.
   • In each Trigger Set, include a "Google Analytics: Send Event" Action to count impressions.
     • If the Trigger Set displays its content when the page loads (eg. in a floating bar), put it in an OnLoad Trigger.
     • If the Trigger Set displays its content in response to some Trigger (eg. a click, scroll, exit intent, etc.), put it in the list of actions under that Trigger. Set it to execute only once.
     • We recommend setting the "Category" for the event to "TriggerNote", and the "Action" to something like "trigger", "impression", etc. The "Label" should identify the Trigger Set.
   • Use another event to track conversions.
     • If the conversion action leaves the visitor on the same page, use another "Google Analytics: Send Event" Action when the desired action is taken.
     • To record clicks that take the visitor to another page, use the "Google Analytics Click Tracker" Recipe.
     • We recommend setting the "Category" for the event to "TriggerNote", and the "Action" to something like "click", "conversion", etc. The "Label" should be the same as for the impression counting event.
3. Create a Selector Set to load each of the Trigger Sets you created.
   • Use the same Selector Group and Selector Priority for each alternative.
   • If you wish to load the split test manually into specific pages rather than using Selectors, do not specify any Selectors.
   • If you want to run the split test on every page that auto-loads Trigger Sets, use the "Always select" Selector.
   • Otherwise, use other Selectors as needed.
   • Either way, be sure to click the "Add Selector" button (even with the "Always select" Selector).
4. Create a Split Test with the same Selector Group and Selector Priority as the Selector Sets you created.
5. Enter a name for a cookie that will be used to ensure that each person sees the same alternative each time (if they see different alternatives, your test results will be less valid, because their actions may be influenced by things they saw on a prior page view).
6. If, in step 1, you used the "Google Analytics: Load" Action to load Google Anaytics, be sure the Trigger Set you created in that step is checked at the bottom of the form.
7. Click "Save Split Test".
8. In the webpages where you are running the split test, if you wish to load the test using Selectors, be sure to include the code "TriggerNoteAutoSelect();". If you didn't specify any Selector criteria, load the split test manually using code like "TriggerNoteUseSplitTest(3);", where the number is the number of the Split Test.

How to Analyze Split Test Results

• If you've set up your split test as recommended above, TriggerNote will show you a statistical analysis of the results.
  1. To view the results, first, you'll need to authorize TriggerNote to access your Google Analytics data.
     1. Log into your TriggerNote Control Panel.
     2. Click "Analytics".
     3. Click "Authorize Google Analytics Access".
     4. Click the link on the page that is displayed. A new window will open up where you can grant authorization to TriggerNote.
     5. Once you've finished granting authorization, copy the code from the page that is displayed, paste it into the form shown in step c, and click the button to save it.
  2. Next, you'll need to create a report.
     1. If you don't already see the form for creating a report, click "View/New Report".
     2. Enter a name for the report.
     3. Usually, you'll want to select the date that you started collecting data for the split test. But if you want to, for example, just analyze the past week's data, you could specify "7 days ago" instead.
     4. Usually, you'll want to leave the end date as "0 days ago", but you may specify a different number of days, or a specific ending date if you wish.
     5. Enter the "Action" values you entered in your Trigger Sets for impressions and conversions (see step 2 at the top of this page).
     6. Check the checkboxes for all of your websites that you want to include in the analysis. For example, if you're running the same split test on 3 different sites, you could check them all to analyze their total results, or check just one or two to see how the test is going on them. This could be useful if different traffic on different sites leads to different test results.
     7. Click "Save Report".
  3. Once the preceding steps are complete, you can view your test results at any time.
     1. When you first save a report, the results are displayed immediately. To return to a previously saved report, click "View/New Report" and click either the name of the report, or the refresh icon next to the report name. Clicking the refresh icon will always get the latest data from Google Analytics. Clicking the report name will only refresh data that is at least 12 hours old.
     2. At this point, you'll see the total number of impressions and conversions for each alternative, as well as the number of unique impressions and conversions for each. The alternative with the highest conversion rate will appear on top.
     3. To perform a statistical analysis of the the results, use the radio buttons on the left to select two alternatives. An analysis of them will appear below the list. Along with confidence levels, a traffic light indicator will be displayed.
        • If the confidence level for both total and unique conversions is over 95%, you will see a green light, indicating that you can probably end the test and declare a winner.
        • If both confidence levels are over 90%, or either is over 95%, you'll see a yellow light. You'll usually want to wait for a bit more data and see if the confidence levels go above 95% before ending the test.
        • Otherwise, you'll see a red light -- you definitely need more data before declaring a winner.
• If you've set up your split test in some other way, and have a way to get the impression and conversion numbers, you can perform a statistical analysis by plugging the numbers into the White Hat Crew Split Test Analyzer.
  • Any data collected by sending events to Google Analytics can be found in the "Behavior / Events" section of Google Analytics.

Why You Need Statistical Analysis of Your Split Test Results

If you search online for information about how long to run split tests, you'll find a variety of recommendations. Some people say to run it till you have at least 100 conversions. Others say 300, 400, or 1000 conversions.

They're all wrong.

The two main factors that determine how long you should run a test are:

1. Your business cycle.
2. Statistical significance.

Your Business Cycle

Your business cycle is the length of time needed to ensure that your test results aren't skewed by, for example, only running your test during the week, and not on the weekend.

Seasonality and other factors may be involved too. Unless you're aware of other such factors, you'll generally get the best results if you run your tests for at least a week, and end them on that same day of the week that you started.

Statistical Significance

Where most of the recommendations you'll find fail is in picking a specific minimum number of conversions. The reason people do this is because it's easy to understand and easy to do -- easier than understanding and performing a proper statistical analysis.

The reason why you can't simply pick a specific number of conversions is that the number of conversions required varies depending on how much better the winning alternative is than the loser. If there's only a 2% difference, you need a lot more data than if there's a 20% difference.

Technically, it's impossible to know with 100% certainty that your test results are valid. What is possible is to have high confidence in your results.

What statistical analysis does for you is to calculate a confidence level that the version that's currently winning your test really is better than the other version.

Statisticians recommend running tests until you have at least 95% confidence -- and you'd do well to listen to them. I've personally seen tests where the confidence level got up into the high 80s, or maybe even the low 90s, only to turn the other direction as more data came in. Don't give in to the temptation to quit too soon.

On the other hand, if you've run the test for a long time, and the confidence level isn't getting to 95%, that may mean that there's no difference between versions, and you should just pick one and end the test.

Home | About Us | Contact

Copyright © 2015-2024 Gecko Tribe, LLC, All rights reserved.
PO Box 5835, Grand Island, NE 68802