Quick explanation of issues today

This is an update to our earlier reported queueing of emails.

At around 10:50am EST our datacenter was doing some IP address mapping on our entire environment (Beanstalk, Postmark and dploy.io). Unfortunately, they set the wrong IP mappings, which caused Postmark to send emails from an IP that is not intended for sending mass emails.

For about 40 minutes, Postmark used an IP that does not have the reputation for sending large volume of transactional emails, which meant that email delivery was negatively affected. As soon as we realized the issue we put Postmark offline so that we wouldn’t continue to send using the wrong IP. You will see bounces for emails that were rejected by ISPs. Sadly, we can’t resend those emails programmatically, but you can send them again.

We’re working on getting everyone reimbursed for those emails that were sent in that amount of time. We’re running a query now and we’re going to add them back for everyone today.

We can’t know for sure what emails were accepted and which weren’t, and we’re extremely sorry for that. We’re working with our datacenter team to figure out how that could have happened and make sure it never, ever happens again.

I’m sorry, we’re so disappointed this happened.