SPLIT TEST ACCELERATOR
 

STA Technical Support FAQ

See if your question is here before contacting support.





General Questions

1.   What happens if my question isn't in this list yet?
2.   Should I expect a reply to my techical support question.
3.   Are you going to have more questions here eventually?


Specific Errors

4.   STA is not registering sales, and when someone hits the thankyou page, what is recorded is a "no cookie set, unique ip" or "no cookie set, dup ip" event.
5.   Database Installation Question: Having trouble getting the installation script to run successfully.
6.   STA doesn't seem to be tracking all my sales.


Test Design And Interpretation Questions:

7.   How do the adcodes work?
8.   Can I use STA to split test different affiliate offers?
9.   What do the numbers mean in the Factor by Factor split test report?
10.   How do I switch from the taguchi mode of the test to the "final" or confirmation mode?
11.   How do you determine when your results are statistically significant.







General Questions


1.   What happens if my question isn't in this list yet?

Answer: Post your question on the forum, or contact Jim via email.  top

Answer: Yes. You should always expect a reply to your technical support email messages.

However, if you are suggesting a new feature or making other suggestions on the forum, you may not get a reply, though your feedback is definitely much appreciated.  
top

Answer: Yes. Of course. We just got this new FAQ system going, and we will add content here so you can troubleshoot your problems without having to email us.

Until we get your question up here, though, please, do not hesitate to post questions on the forum, or send an email message for support.  
top



Specific Errors


4.   STA is not registering sales, and when someone hits the thankyou page, what is recorded is a "no cookie set, unique ip" or "no cookie set, dup ip" event.

Answer: The problem you're facing is a cookie problem.

The cookie isn't being read on your thankyou page.

This can have two main causes.

1. The cookie is never set on your landing page.

2. The cookie is set on your landing page, but not read on your thankyou page.

#1 can have many causes but the most common is that your top code snippet is placed incorrectly, and isn't the very first thing on the page.

Another possibility is that the user is not accepting cookies. Though, if you have, say, more than 4 or 5 of these in a row, you can probably rule out the idea that none of these visitors were accepting cookies, since about 95% of visitors accept cookies.

#2 can have a few causes, and the most common are these:

a. The landing page and thankyou page are on different domains. This is not supported with STA.

b. The landing page and thankyou page top snippets have different project id values. Check to make sure the value of '$id' is the same on both pages.   top

Answer: Most of the time setting up a new database and getting the parameters into dbinfo.php is straightforward and easy.

However, sometimes there are some unusual cirucumstances that make it more difficult. Here are some of them:

  top

6.   STA doesn't seem to be tracking all my sales.

Answer: First, make sure you're showing all the stats you think you're showing. Make sure the filter fields on the report page aren't filtering more results than you want. If you want to show ALL results, make sure your date range is consistent with this.

Second, ask yourself whether there's more than one way to get to your conversion page. If your landing page receives only a portion of the traffic that can result in the conversion you're looking for, it stands to reason that STA will track only a portion of the conversion events.

Third, STA shows results according to the impression time (the time the visitor first hits your landing page). So, if someone visits YESTERDAY, and takes the action TODAY, their conversion would show on YESTERDAY'S stats, and not today's.

That's going to be out of sync with some other tracking reports, but it's the best way to report results for split testing, because you want to know how effective your page was for visits that originated within a date range, and you want to credit visits within that date range with conversions that happen outside of that date range.

Fourth, make sure you have all tracking snippets placed correctly.

Fifth. Once you've checked all of the above, feel free to post a question at the forum (or if it's an emergency, and you're within your support window, send an email message for email support).   
top



Test Design And Interpretation Questions:


7.   How do the adcodes work?

Answer: You can track how your specific ads perform by using adcodes.

Setting this up properly requires two steps:

  • First, you must decide which variable name you want to use in the destination url of the ad, and then modify your destination urls to use this variable.

    For instance, you could use the variable name "adcode", and your destination URL for one ad could be something like: http://www.mysite.com/mypage.php?adcode=1, and the url for another ad could be http://www.mysite.com/mypage.php?adcode=2
  • Second, once you decide on a variable name, you must tell STA to look for that variable name. You do that on the project creation page for the project you are sending the ad traffic to.

    You can specify up to three variables to track at a time. Typically you will need only one, but in some instances it's useful to track more than one parameter at a time (for instance, if you're using SpeedPPC).
Once you start tracking adcodes there are two places you're likely to want to view reports related to adcodes.
  • First, from the main support page, there is a filter that will show you your taguchi results based on particular ads.
  • Second, the conversion summary report has an "adcode" setting, and this will tell you how each ad is converting.
  
top

8.   Can I use STA to split test different affiliate offers?

Answer: Yes.

Just make your landing page a redirect page, and have STA rotate 6 different redirect commands.

Suppose you want to test 6 different offers to the same traffic. Here's some sample code you could use:

<?php $id=2; include('/my_domain/public_html/sta/topper.php');?>
<?php if ($f[1]==1){header("Location: first_URL");}?>
<?php if ($f[1]==2){header("Location: second_URL");}?>
<?php if ($f[1]==3){header("Location: third_URL");}?>
<?php if ($f[1]==4){header("Location: fourth_URL");}?>
<?php if ($f[1]==5){header("Location: fifth_URL");}?>
<?php if ($f[1]==6){header("Location: sixth_URL");}?>

basically you just place the landing page snippet on the page. This determines which page the visitor will see.

Then you modify the factor code so the visitors get directed to the right page for the option they are being shown.

You still have to figure out how to track the sales.

If you can place a tracking pixel on the thankyou page, that will work.

If you don't have access to the thankyou page, and can't get the owner to place your tracking pixel, you still have another option.

If you are working with an affiiliate network that allows TIDs or SIDs in the affiliate link, you can place the value of $visid as your TID value. $visid is a value that gets set in the landing page code, and is available on the page. So, if you just use a little PHP to append the $visid number to the affiliate link, it will pass the STA visitor id number to the affiliate network.

Then, when you get your affiliate report, you can manually enter the visitor ids back into STA and it will match up the sales with the visits.

There are more instructions on the "manual conversions" page (it's a button on the top of the main report page).   
top

9.   What do the numbers mean in the Factor by Factor split test report?

Answer:

  • Influence: This number tells you how important this factor is on this page. A higher percentage means that it's an important factor, and might even merit more fine-tuning in a future test.

  • Option: This number tells you which option the row pertains to.

  • Actions/Impressions:These numbers tell you how many times the option showed up on a page (impression) and how many times a desirable action resulted (action).

  • Conversion Rate Range: The conversion rate is just the actions divided by the impressions. But this number might not represent the long range conversion rate. For instance, if you've only had one impression, and one action, your conversion rate is 100%, but you are unlikely to see this percentage over time. The more impressions you have, the closer you should expect to be to the long term conversion rate. Thus as you get more traffic, the range of the predicted conversion rate will get narrower and narrower.

    Here's how you read the number. Example: 3.5% +/- 1.5% -- Your predicted conversion rate is 3.5%, and there's a 95% chance that the long term conversion rate will fall between 2.0% and 5.0% (3.5% + 1.5% and 3.5% - 1.5%).

  • 95% intervals: This is a graphical representation of the conversion rate range, plus some other information indicated by colors.

    The colors indicate how well the option is doing versus the control option (always option 1). Red indicates the odds that the option will fare worse in the long run. Green indicates how confident you should be that the option will fare better in the long run. The more red or green you see, the more confident you should be of the relative outcomes. A lot of grey means that there's not enough information to tell the difference.

  • Beat Control?: This number tells you the likelihood, on a percentage basis, that the given option will beat the control in the long run. When the number is highlited yellow, this indicates that there is a greater than 95% chance that the option will beat the control in the long run. This calculation is based on assumptions that may not play out completely well in the real world, but it is the number everyone uses, and gets things pretty close to right.

  • Beat Leader?: This number tells you the likelihood, on a percentage basis, that the given option will beat the currently leading option in the long run. When the number is shown in a light grey font, the option is trailing by a significant margin, and has a less than 5% chance of beating the leader in the long run. This calculation is based on the same assumptions as the above calculations, and the same caveats apply.
  
top

10.   How do I switch from the taguchi mode of the test to the "final" or confirmation mode?

Answer: There is a video under the "video tutorials" tab that demonstrates how to run a confirmation test.  top

Answer: This is a little bit tricky. It's more complicated than many people assume, because they've been exposed to overly simplified rules of thumb.

But stick with me, because I think I can help you get a good handle on it.

Let's take the single variable case first. How do you determine if a simple split test has significant results?

The old rule of thumb was that you wait for the leading option to have 30 actions, and then you choose the winner.

But here's the problem: Suppose headline A has a 30/1000 conversion ratio, and headline B has a 29/1000 conversion ratio. Can you really be sure that Headline A is better? No. In fact, there's probably a 40% chance or so that headline B will prove to be better in the long run.

On the other hand, if headline A has a 20/1000 conversion ratio, and headline B has a 5/1000 conversion ratio, this is extremely signficant, and headline A is almost certainly better than B, even though you haven't had 30 conversions for headline A yet.

In spite of this counterexample, I believe the problem with the 30 sale rule is actually that it invites you to end tests too early on average.

That's why the Split Test Accelerator has confidence numbers built in. These numbers are arrived at by doing pairwise comparisons of two creatives, and running a normal approximation to the binomial distribution test on them.

With these numbers you don't have to worry about rules of thumb. You just wait until one option is better than the other with 95% confidence, and then you can consider ending the test. (You should wait until all the options have at least one action, though.)



The image gives results for a multivariate test, but focus on just one of the factors, and you can see how this works in the single variable case.

For instance, you can see that the "headline color" factor is significant, as is the "testimonials" factor. They both come in at over 95%, and this is incdicated with yellow highlighting.

Now, the multi-variate case is a little more complicated.

The main reason it's more complicated is because there are many variables, and they don't all reach the state of significance at the same time.

One factor can show signficant results after 1000 impressions, and another might not show significance until you have 100,000 impressions.

So, what do you do about this?

Here's the multivariate rule of thumb (I'm not against rules of thumb, but some are more useful than others):

DON'T WAIT FOR MORE THAN 1/4 OF YOUR FACTORS TO BECOME SIGNIFICANT.

Typically, if you can get 4-5 factors to be significant in a 20 factor test, you're doing great. In fact, just 2 significant factors, where an alternative version of the factor beats the control, is cause for celebration. You've found two needles in a haystack of ideas, and those two factors might well give you a 50% boost.

Yes, it's the Pareto principle in action again. 90% of your gains will come from 10% of your factors.

And the name of the game is to find those factors quickly, and move on to another test.

So here are the rules for ending a multi-variate test:

  1. Make sure every ad (in the ad by ad section) has at least one action (this helps insure you against the possiblity that you set the test up wrong)
  2. Make sure you run your test for at least a week, so you can average out any "day of week" effects.
  3. THEN, try to get at least 1/10 of the factors to be significant, but don't wait for more than 1/4 of them to be significant.

    For instance, in a 20 factor test, you are looking for 2-5 significant factors. In a 10 factor test, you are looking for 1-3, and in a 7 factor test, you are looking for 1-2 significant factors.
  4. If you have more than 400 actions (not impressions) in the test, and no factors are significant, you should consider locking in the leaders and starting over. You probably won't see much improvement in this case, but you won't see much improvement waiting longer either.
Now, how long should this take?

It depends on how well you designed your test.

The more WILDLY DIFFERENT your options, the faster you should reach signficance.

That's the weird thing. When you hit a home run with a Taguchi test, it happens very quickly. A test that doubles your conversion rate can be over with 100 actions. One that gives you a 20% boost might take 400 actions to determine winners in enough factors.

That's why I advise you to just start over if the test is taking too long. There's no point taking 3 months to find out which headline will give you a 1% boost, when you could try new factors that might give you a 50% boost in two weeks.   
top





If you have any additional questions, Here is an email contact form.

Thank you.

Sincerely,

Jim Stone, Ph.D.

(c) Copyright 2004-2012 by Jim Stone Enterprises. - All Rights Reserved
20027 13th Drive SE Bothell, WA 98012
Disclaimer¦ Contact ¦ Free Adwords Accelerator Spreadsheet ¦ Free Multivariate Testing Course ¦ Free Confidence Measuring tool ¦ Split Test Accelerator ¦ Blog ¦