Optimizely is the best website a/b testing tool I’ve used in my career so far and its is one of my favorite growth hacking tools.
Optimizely is also quite expensive.
But is it worth the price? Keep reading.
Want to know how much it costs? View Optimizely Pricing.
Why Should You Trust Me?
You should be skeptical of software reviews on the internet. Many of them are paid for, either directly by brands who sponsor the content and pay bloggers like me in exchange for a favorable review, or indirectly because the blogger is part of an affiliate program.
This post is not sponsored and there are no affiliate links.
If that changes I will update that disclaimer.
Now, who am I? And more importantly, why should you listen to me?
Hey, I’m Nick, and I do this shit for a living. For the past 7 years I’ve worked as a professional digital marketer for brands like Modernize, Mailgun, Mailjet, Osmosis, PhoneBurner, Jotform, and now Loom.
I’ve actively used Optimizely, and their competitors VWO and Google Optimize, as part of my full-time job.
Because I’ve actually used the product I’m more qualified to write about it than frankly most internet reviews you’ll read.
With that out of the way, let’s get into the review of Optimizely.
But first a short primer on ab testing.
What Is A/B Testing(Split Testing)?
A/B testing is a form of experimentation that compares two versions of something against each other. It’s also known as split testing. In digital marketing, A/B testing is most commonly associated with testing two versions of a webpage.
AB Testing vs Multivariate Testing
Multivariate testing is the process of comparing more than two versions of something, also known as A/B/C testing or A/B/X testing.
Statistical Significance in AB Testing
Statistical significance, often shortened to stat sig, is the likelihood the difference between your control version (version A) and your test version (version B) isn’t due to random chance.
This is often expressed as a percent: 95% significance level.
That means you can be 95% confident your results are not due to random chance.
The higher the percent the greater the chance your results are statistically significant.
In human terms, sometimes results happen due to purely random chance. Saying a result is stat sig means you’re confident in your result, and you did the path to prove it.
AB Testing Tools for Websites
The 3 most common website testing tools are:
How Much Traffic Do You Need For AB Testing?
At a minimum your webpage needs 500 visitors/day, which is about 15,000 visitors/month.
Not 500 visitors to your entire site, but 500 visitors to a specific page on your website every day.
That number assumes two things:
- Your test will run for at least 30 days.
- The minimum improvement in conversion rate you want to see is 10%.
If you want to assume different numbers, plug them into this ab testing duration calculator.
Outside of the number of daily visitors, the two most important data points to consider if you can run statistically significant a/b tests are test duration and minimum improvement.
AB Test Duration
If you have to run a test for 90 days to get a significant result, then that’s too long.
The shorter your testing duration, the faster you get results and you can move onto the next test.
In my professional experience this is often the biggest drag on a/b testing at the Enterprise level.
Let me give you a real world scenario.
You’re spending millions on Google Ads each month. You’ve been converting that traffic at 10%.
But last month your conversion rate dropped to 8%. Your boss starts to freak out.
You want to do a good job so you launch an A/B test. Great job, right?
Not so fast. Your test will take 30 days to get a result because you don’t have enough traffic.
Is your boss going to be ok with that? Probably not because he’ll have to explain that to his boss.
That’s what I mean when I say testing duration is the largest issue you’ll encounter. Often you can’t just magic more traffic out of thin air.
Minimum Conversion Improvement
What’s the least increase in conversion rate you want to measure?
This is another way of measuring how precise you want your test to be.
Here’s an example. If your minimum conversion improvement is 30%, that means you don’t care about results that improve it from 1% to 29%.
The higher the conversion improvement, the less data you need to collect before you get a significant result.
Try playing around with the calculator. Leave all the other details the same but increase the minimum conversion rate.
What happens? The test duration decreases.
Now try dropping the minimum to 1%.
The duration is now hundreds or thousands of days.
What’s the right number to use? Personally, I like 10%.
10% means if my conversion rate is currently 10%, and the test measures an increase to 11%, then I want to know that.
Anything less than 10% is meaningless to me.
Optimizely Review
I said this at the beginning, but I’ll say it again: Optimizely is best-in-class for website a/b testing..
Why? It has the best suite of features to create, measure, and report on your website and landing page optimization tests.
They also have a great support team that I’ve personally noticed is very responsive and timely. You’re also typically assigned a Customer Success Manager(CSM) to help with smaller questions.
My CSM invited me to a private slack group so I could get ahold of her faster. I thought it was a nice touch.
Optimizely Pricing
The most frustrating thing to me about shopping enterprise software is the hoops you have to jump through.
- Fill out a form
- Talk to an SDR so they can qualify you
- Talk to an AE so they can ask you the exact same questions as the SDR
- Get a demo from the AE but they won’t discuss pricing
- Another 30 minute call where pricing is discussed at the end
That sounds terrible, and it is, and it’s also unfortunately a real example of what the B2B software buying process is like right now.
I know because I’ve done this many, many times. And I’m actively doing it right now with a vendor as of June 2021.
Thanks for coming to my TED Talk on B2B Pricing.
So, how much does Optimizely cost?
$60k/year.
That was the amount my company paid them annually based on a two-year contract.
Some software vendors won’t let you sign anything less than a two year contract. Sometimes you can do 18 months, but 24 months is becoming standard.
Optimizely’s pricing is based mostly on how many monthly visitors your website gets, plus additional add-on services.
Optimizely Alternatives
While Optimizely is great, it’s expensive as shit and not every company has the budget for that.
Here are two great Optimizely alternatives.
Optimizely vs VWO
VWO is a somewhat cost-effective alternative to Optimizely.
VWO costs about $30k/year.
Which is still a lot of money, but at 50% the cost of Optimizely it’s more affordable in comparison. If that number still shocks you then I have bad news, it’s a really common price point for B2B software across a variety of industries.
The price may be attractive but VWO comes with some downsides.
They’re based in India which can create communication problems when interacting with their support and sales teams.
I’ve used VWO at two separate companies and compared to Optimizely their support is just not as good.
The other downside is how VWO tracks custom conversions. Custom conversions are when you fire their Javascript code on conditions you create, commonly in Google Tag Manager.
Optimizely makes this really easy. You create custom conversions at the Project level and can assign them to any test.
VWO’s implementation is more complex. You can only create custom conversions inside an individual test. You can assign the custom conversions to other tests, but you have to scroll through a cumbersome list of ‘Frequently Used Goals’.
Optimizely vs Google Optimize
Here’s where Google Optimize wins against Optimizely: it’s free.
If you need a simple website testing tool to get something out the door soon, Google Optimize is 100% the tool I would recommend.
It’s very capable and plays really nice with other Google tools like Analytics and Google Ads.
If you have zero budget, or don’t want to go through a long sales process, start with Google Optimize. Then when you outgrow it you can make a better case for VWO or Optimizely.
I could go on and on, comparing features to make my post longer and to improve my SEO performance, but here’s the facts.
Google Optimize is free. It’s a freaking great free tool.
The end.
Conclusion
Optimizely is a great tool, and if I had unlimited budget it would be my #1 choice every time.
But sometimes budget is a concern, and VWO is more attractive at that 50% price point.
If I have no money then I can get by with Google Optimize for a long time. Especially if I’m at an early-stage company and need to start testing yesterday.