A/B testing email service providers

Can changing your email service provider improve open and click rates? If you've ever spent any time in the dashboard of your email service provider (ESP), you've probably focused on the open or click rate of you email. Why aren't people opening your email? It's tempting to immediately start re-writing your emails, using stronger headlines perhaps, or maybe toning down your copy to sound less aggressive. 

While those certainly may be problems, there is often a much simpler answer. A lot of issues with low open/click rates are really just delivery issues.  Your customers can’t open/click on an email they never receive. If you’re seeing unusually low open/click rates for your industry, it’s very possible that it’s caused by delivery issues.

A lot of issues with low open/click rates are really just delivery issues. Your customers can’t open/click on an email they never receive.

Instead of spending time and resources A/B testing subject lines or revamping the design of your CTA buttons, you might be better off making sure your emails are actually being received. One customer, Childcare.co.uk, which helps connects parents and childcare providers and tutors, recently migrated from Amazon SES to Postmark. When they switched from Amazon SES to Postmark, they saw their open rates go up 11 percent without changing anything else about their emails. That's a huge win, but it's one you can only make if you're monitoring where your mail ends up. Inbox providers are increasingly leaning on domain reputation for filtering, but IP addresses still matter. If your ESP isn’t closely policing their IP reputations, you could end up sending from an IP whose reputation has been tainted by other senders.

In our post about monitoring email delivery, we covered some ways to monitor common issues that might stop your email from being delivered. And here, by delivered we really mean accepted by your customer's mail server (or ISP). That's half the battle. Once it's on your customer's server though there's quite a few ways to monitor how that server delivers that email to the customer. That is, monitoring the so-called last mile—does it get delivered to the inbox, sent to spam or hung up by some filter the customer never gets to see.

Unfortunately, deliverability is hard to measure directly because you can’t login to everyone’s mailbox to see if an email was delivered. Your ESP doesn't know either. When it says an email was "delivered" it only means that the mail server reported back that it accepted the email. It could well mean that it was delivered to the spam folder, or in the case of Gmail, the promotions folder. There are however, some services that can help you get a better idea of where your email is going.

The two big services in this space are 250ok and ReturnPath. Using these services can give you data about your delivery rate to a large list of ISP “seed accounts”. When you send to these seed accounts from your ESP, the services will track if the message went to the inbox, spam folder, or if it was missing entirely. Alternatively, you could use our MailHandler gem to setup your own custom monitoring.

Once you've got a mailbox delivery monitoring system set up, you can start looking at the data. The big question will likely be how much of your email is ending up in the spam folder and why. With seed accounts and services like 250ok, you want to view the data as a trend instead of an absolute delivery rate. In some cases factors like content and broken seed accounts can skew the results, so use it as a starting point in monitoring inbox placement instead of a definitive measure of your success or failure.

Now you know how much is ending up in spam folders—the industry standard is somewhere around 20 percent, depending on who you ask, though we think you should aim for much lower than that—the question becomes, how do you fix it?

Before you start hiring copy writers and running textual analysis on your keywords, consider that it might be something much lower level, like the quality of your ESP. Now that you have the tools to monitor delivery it's well worth your time to start comparing services. Just like you'd A/B test headlines, it's worth the time and effort to A/B test ESPs. We've even put together a thorough guide for things to consider when evaluating transactional email service providers. 

All delivery and IP address reputations are not created equal. While upgrading to a more expensive dedicated IP address can sometimes help fix problems, unless you’re sending significant volumes of email, you may not be able to keep the IP address “warm,” and more of your emails may end up in spam. Moreover, dedicated IPs can be recycled after a customer leaves a service, and even with a dedicated IP, it's possible that you may suffer from reputation issues as a result of the previous IP address owner's indiscretions. In any case, never assume that great delivery is a given, even with dedicated IP addresses.