Monday, May 16, 2011

Oooh Barracuda!

Earlier this year I was working on an issue setting up TLS encrypted email between my company and a new client using our Barracuda Spam Firewall. The process should have been a piece of cake, especially since we use STARTTLS commands on all outbound email. What we were running into with this particular client was TLS would never negotiate even when we forced encryption to their domain.

The Problem

I had opened and closed 2 separate tickets, each time with the rep claiming the problem would be resolved if we implemented their suggestion. Each time I had to contact the client and schedule off-hours resources on their end to test the changes, and each time it failed. We were rapidly approaching the go-live date, and no encrypted email between our sites would effectively mean the agreed upon work flows would need to be modified.

I once again called in, created a new ticket, and escalated with the prior 2 tickets as reference. After speaking with a top tier tech at Barracuda I was told that what we needed was not possible. The issue had to do with the fact that this client was running an ESMTP inspect process on their ASA, and were masking the banner information returned from the EHLO command. As it was explained to me the Barracuda was defaulting to non-extended commands and issuing a HELO as soon as it did not see EHLO in the response. Because of this we never issued a STARTTLS, and the encryption negotiation would never take place. Not being able to force a STARTTLS when choosing to encrypt all email to a domain regardless of banner reply was a bug in my mind, and there was no way I was going back to this new (and very large) client to tell them we could not encrypt email between our sites in a seamless fashion.

The Solution

I learned a long time ago that engineers often times only see things from the technical side of things, and in this case it was clear, the system as it was built today would not work the way we needed it to work. This is why I was getting nowhere fast with the standard support channel. In order to get a resolution I laid everything out in a lengthy email, explained the urgency of the issue, and sent it to the sales department. Involving the sales team brings in a different line of thinking (and motivation), and it was my hope they could escalate in a way that was not possible through standard methods.

What happened next was nothing short of amazing. I instantly received replies from members of the sales team, and within a few hours I received a call from the CTO and co-founder of the company Zachary Levow. Mr. Levow wanted to understand the exact problem we were having, had me send him some logs (while working with an engineer on their side), and even emailed me his cell phone stating he would be my point of contact until the issue was resolved. Barracuda is a decent sized company with over 130,000 organizations using their products so working directly with their CTO certainly caught me off guard.

Needless to say the work done in the next 48 hours far exceeded my expectations. Barracuda wrote custom code for our system, tested without our client being involved, and promised to roll the changes into a permanent patch. Within a couple weeks we saw this: [BNSF-15994] Enhancement: If TLS Encryption is required per the DOMAINS > Manage Domain > ADVANCED > Email Protocol page, the Barracuda Spam & Virus Firewall will always issue an EHLO, regardless of welcome banner containing ESMTP. So there you have it, our little feature request made it into a full production release in a matter of weeks (so we could continue patching our system without fear of breaking the custom code), the client was very happy we were able to work the issue on our side (as opposed to them making firewall changes or coming up with a different work flow), and it solved a very real issue that probably affected other users.

In the end we purchased 2 more Barracuda Spam Firewalls, and I'm sure we will continue to partner with this company long term. What could have been a bad experience was turned completely around by the hard work of the sales, technical, and executive team at a great tech company. My hats off to you Barracuda, your product comes strongly recommended! For all my VMware tweeps make sure to check out their new Spam Firewall Virtual Appliance.

0 comments: