Split Test: Too Early To Decide?

Status
Not open for further replies.
F

ForgottenMonk

New member
Feb 10, 2008
14
0
0
I've been split testing to different ads. But todays results even though my impressions are low I'm wondering if its ok to declare a winner. My results are

clicks impressions
7 469
1 288

is it too early to declare the first one as a winner?

Thanks in advance
 


In my opinion too early...I usually do about 30 clicks. If you're not getting a lot of volume wait around till about 20 at least. Not necessarily on both ads though, it depends on how they're doing.

For example, if one has 30 clicks and the other 2 then it's pretty obvious, but if they are at 25 clicks each and there is only a slight variation in CTR I'll let em run for another 10 or 20 and go from there.
 
As a very rough rule of thumb, a sample of 30 is statistically significant, so m0rtal's advice sounds right.
 
Adwords, Adcenter, and Yahoo (when ad optimization is turned on), will give the better performing ad more display time once it's determined that the ad is actually performing better on a statistically significant level.

For example... you upload two ads... they should each get equal display time 50%/50%. As the campaign runs further, you'll see the system adjust the display rate in favor of the better performing ad. After a week, you might notice a 70/30% display rate. From this, you can tell which is the better performing ad, and by how much--as deemed by the system.
 
this is true but i still like to do my own split testing...i found that they start showing the better performing ad too soon


Someone said:
Adwords, Adcenter, and Yahoo (when ad optimization is turned on), will give the better performing ad more display time once it's determined that the ad is actually performing better on a statistically significant level.

For example... you upload two ads... they should each get equal display time 50%/50%. As the campaign runs further, you'll see the system adjust the display rate in favor of the better performing ad. After a week, you might notice a 70/30% display rate. From this, you can tell which is the better performing ad, and by how much--as deemed by the system.
Click to expand...
 
Here's a quick rule of thumb for determining whether you have statistical significance:

Assume 100 clicks total
54 clicks on ad "A"
46 clicks on ad "B"

"A" outperformed, but is it statistically significant?

1. Take the square root of the total clicks (SQRT(100)=10)
2. Take difference between clicks on "A" and clicks on "B" (54-46=8)
3. If result from #2 is greater than result from #1, you have statistical significance. In this example, the results are not significant.

In plain language, yes "A" did better but it may be due to chance only.

In your case, your numbers are way too low to draw any statistical conclusions. Having said all that, watch the trends. If one ad is increasing at twice the rate of the other, you'll get statistical significance by the time you reach 30 so pull the plug on the loser early. No need to wait. The above formulas are best when used in situations where the numbers are greater and closer together.
 
Update: new numbers

Ad Clicks Impr
=== ====== =====
A 28 2,518
B 7 1,852

Thanks for each of your replies, but I have other questions.

I used the split tester someone listed here. It does seem ad A does have a significant ctr. The problem I am working through now is... while I had no conversions with Ad A, I had 2 conversions with Ad B! Ad A links to a landing page with no conversions, Ad B is a direct link with 2 conversions. The only difference between the ads is the Display URL.

So I don't know what the heck to do! Improve the landing page? Just do a direct link? What do you guys think?
 
Your landing page is probably not doing a good job of pre selling. Test different layouts/content on the LP. Figure out what questions these people want answered and go from there. Hit up a forum of the product/niche and see what all the fuss is about. See what your competitors are doing.

Are people even going through to the offer page from your LP? What %? Can you improve it?

Use crazyegg to test your click through rates and see which links people are clicking on your page. Also, that amount of clicks isn't necessarily significant. Did they come from the same keywords as those who went directly? You are tracking right? :) If you consistently got this level of performance from direct linking over let's say the next 100 clicks then there may be something to it but at this point I wouldn't get excited about it.

Anyhow, there's more to it but there's some stuff to think about.
 
Hi forgottenmonk

there is some good advise there for you

i would leave it longer too

for better results
wol
 
I agree with m0rtal. You need to know for sure what people are doing on your landing page. Given your stats it would appear that people are bailing out on the landing page, so it is not pre-selling. BUT, you have to track to know for sure. I use GoTryThis to track EVERY link on all of my sites. A lot of people would consider this overkill, but once it is set up, I know exactly what people are doing on my pages as soon as they are doing it.

By tracking everything you can reduce this down to a pretty exact science. Don't deal with assumptions or maybe's. If you deal only with the numbers you're listening to what your customers are telling you. And that is the only thing that matters, imho.
 
Status
Not open for further replies.
Top Bottom