Guide to Messaging Security

Best practices for implementing messaging security

By Joel Snyder, Network World Lab Alliance

Most enterprises already have some form of messaging security in place in the form of spam and virus filtering. When installing a new or (more likely) replacement messaging security system, key areas to pay attention to are performance, user experience, and management and operational costs.

Performance is generally easy to manage. Aside from simply having enough hardware to do the job, there are three keys to keep performance up to par.

First, some sort of reputation-based filtering should occur very early in the transaction, certainly before the entire email message has been accepted by the messaging security gateway. This single step, alone, will block between 50% and 75% of incoming messages in a typical enterprise, giving a dramatic reduction in total load. Proper use of good reputation-based filtering products has an almost vanishingly low false positive rate-far lower than the anti-spam engines themselves. If reputation-based filtering is done properly, it also only results in detectable false positives, because the sender gets notification from their own mail gateway that the message was blocked. This allows for quick remediation, and is preferable to a typical anti-spam/anti-virus false positive, where the message goes into a black hole and is unlikely to be discovered. Anti-spam engines that can return their verdict before the message is fully accepted are even better, but for performance reasons this is not yet a common strategy.

A second performance optimization is to ensure the messaging security gateway has access to the directory of legitimate email messages, either through dynamic lookups or a regular transfer of directory information to the gateway. By only accepting email for users who actually exist, the system performance is again increased. This also solves the problem of what to do with messages that have been wrongly addressed. If they're simply dropped, then a legitimate correspondent won't know that their message was never received. But if they're placed into a queue to be returned, the load of attempting to bounce both simply misaddressed and spam email will quickly overload even the largest systems. Some enterprises have been reluctant to deploy directory information to the edge because of a misguided belief that this measure aids attackers in directory harvest attacks. In fact, protecting against such attacks is easily done, and all enterprise-class products have has this feature for years.

The third key strategy for best performance is to ensure that a high-availability configuration is in place from the very start. Unlike other security appliances such as firewalls built on fanless, diskless, custom-made platforms, messaging security gateways are all simply Linux (or occasionally Windows) servers, with the attendant potential for hardware, disk subsystem, and even operating system failure. Some vendors have selected poorly-engineered platforms in an effort to cut costs, which further reduces overall reliability. While a replacement systems is rarely more than a FedEx transaction away, being without spam protection for even 24 hours can cripple an email system-and, subsequently, being without email for 24 hours can cripple a business. The solution to all of these vulnerabilities is to have redundant, load-sharing, hardware in place from the start so that a problem in one system does not take the entire gateway function offline.

User experience is another area to be careful with when implementing enterprise-class products. Users are especially sensitive to changes in their email systems, and a critical factor to ensuring the highest user satisfaction is the perception of empowerment. While enterprise-class email gateways have a low spam/virus false positive rate, the rate will never be zero. As users detect these false positives, they will be angered, frustrated, and lose trust in the email system as a business tool. The best way to reduce anger and frustration and increase trust is to empower the user to see and handle their own false positives. In some environments, users may also want to manage their own anti-spam sensitivity settings and whitelists, although this is more likely to be a waste of time in enterprise environments, contributing to a higher "fiddle factor" with little attendant benefit. Buying and managing end-user spam quarantines for the daily false positive may seem like a poor use of IT resources, but it gives the end users a much greater feeling of control over their email flow, and thus contributes to better overall satisfaction with the product.

The implementation area to be very concerned about is operational and management costs. Many messaging security gateways treat themselves as "black boxes", accepting email and either passing it along, deleting it or quarantining it. An enterprise requires visibility into the box, with the ability to identify and track messages quickly and efficiently. In a high-volume environment, this typically requires a separate server or application which can aggregate log files and provide searching and reporting functions to help desk and operations teams.

When deploying a messaging security gateway, it is critical to have these tools and services up and running before the gateway goes into service-because it is when a system is first installed that the question of "what happened to my email" is most likely to be asked. As a simple litmus test, you should have the capability to answer the question "what was the disposition of all messages from the company president's son's AOL account last weekend" in less than a minute. If you don't have quick and easy visibility into the black box, you'll end up angered and frustrated yourself-something to avoid in a product designed to protect us and make our lives easier.

Top tips for buying messaging security products

By Joel Snyder, Network World Lab Alliance

Since every enterprise already has some form of messaging security (thinks, anti-spam and/or anti-virus) in place, the decision to consider a new gateway is generally prompted  by problems with an existing system.  Obviously, the key tip for buying is to make sure your new gateway is at least as good as your old one.  This means identifying what you like-and don't like-about your existing gateway and using that information to guide your evaluation criteria for the new gateway. 

Moving on from what you have to where you are going,  you'll also want to look in five specific evaluation areas: anti-spam features, anti-virus features, end-user features, system architecture features, and additional security features.  We take them one by one here.

Obviously, the biggest differentiator between products is the quality of the anti-spam engine when its applied to your own mail flow.  To determine that, you'll need to test any potential gateway in your own environment.  Once you've found an engine that meets your goals for catch and false positive rates, you'll want to consider at least the following as ways of differentiating products and identifying ones that meet your needs best:

  • Does the anti-spam engine offer multiple verdict levels (such as "definitely spam" and "probably spam") that you can use to help reduce undetected false positives?
  • Does the messaging security gateway have reputation-based filtering that allows you to refuse a message at SMTP time to reduce total system load?
  • Can the messaging security gateway integrate easily with your existing email directory infrastructure?

Most products have a single anti-virus engine, selected by the product vendor.  Unfortunately, this engine choice is usually subject to a set of forces that lie outside of your control, such as current partnerships and acquisition strategies.  Since it is a best practice to have a different anti-virus engine in the messaging security gateway from the one you use on the desktop, this can adversely affect your deployment.  You may want to consider:

  • Can the product use multiple anti-virus engines, either in parallel or separately?
  • Does the product's anti-virus engine properly complement installed infrastructure in your enterprise to offer best coverage?
  • What long-term commitment do you have from the vendor on the choice of anti-virus engine?

Some messaging security gateways operate entirely without any end-user interaction, and this may be your preferred deployment scenario.  However, you should examine products that at least have the option of end-user features:

  • Does the product have the option for an end-user anti-spam/anti-malware quarantine?  Can the quarantine be enabled for users individually or must it be done for everyone?
  • Does the product have per-user settings for sensitivity, block-list, and whitelist?  Can these features be managed at the group level as well as the individual user level?
  • Can the product link to your existing authentication infrastructure, or does it have some method to reasonably authenticate end users?  (Note that a very rigorous authentication is likely not necessary, since most of what's in the quarantine will be spam.)

In a simple single-system deployment, system architecture is not that critical.  But in the enterprise, with scalability and high availability requirements, you should consider:

  • Can the product be centrally managed, with settings for gateways and groups of gateways handled without resorting to element management?
  • Can the product scale easily, simply by adding gateways into a management group or cluster?
  • Does the vendor offer built-in or off-the-shelf log management tools that can aggregate information from multiple gateways for help desk support and reporting purposes?
  • In the event of a total system failure, how hard is it to "restore to factory defaults" the gateway and re-apply your configuration?

Most security gateways have focused on anti-spam/anti-malware features.  However, messaging security goes far beyond these two buckets.  Unfortunately, enterprise requirements for additional security features are all over the map and it's difficult to identify any single product as being "best" in all additional features.  Instead, you'll have to figure out what you want and make sure that it's supported in the products you're looking at.  Some of the key features you may want to use include:

  • Message encryption using TLS, under tight policy control; also other integration with encryption and message protection systems
  • Content filtering capabilities, including your own and vendor-supplied dictionaries
  • Message archiving capabilities


Top Trends in the Enterprise Messaging Security market

By Joel Snyder, Network World Lab Alliance

The messaging security gateway market has dropped to a mild simmer since the heady growth phase we saw from it in 2003 and 2004.  From a peak of over 125 vendors then, a handful of key players in the market have consolidated much of the business and offer products with similar features and common functionality.  A recent wave of acquisitions has helped to separate out companies with long-term financial resources and broader technology from those who are likely to slowly fade away.  While there remain critical differences between products and clear evaluation criteria to distinguish them, most enterprise customers will find the same set of product vendors on their short lists out of these technology leaders:  Ironport (now part of Cisco Systems), Secure Computing (which bought Ciphertrust), SonicWALL, Symantec (which purchased Brightmail), Trend Micro, and Tumbleweed Communications.  Barracuda Networks and Proofpoint, neither of which are public companies, also have gained substantial market and mind-share even if they don't have the same financial resources of their publicly-traded competitors.

No other business security requirement translates as nicely to a hosted service as anti-spam/anti-malware filtering.  While on-site messaging security gateway appliances are cost-effective, the benefits of having a focused team doing nothing but anti-spam on your behalf have certainly kept the hosted service providers growing and profitable.  Key players in this hosted service provider space, including MessageLabs, Microsoft (through its Frontbridge acquisition), and Postini (now part of Google) are as strong as they ever were and have legions of happy customers. 

There is no clear trend to or from hosted services; it seems that companies go one direction or the other based on a wide variety of factors.  Most of the reasons our security clients cite for changing to or from hosted anti-spam/anti-virus have little to do with product technology and much more to do with other factors, such as internal staffing costs and expertise, strategic IT direction, and vendor relationships. 

Based on our continuous testing of anti-spam solutions for five years, we see no signs that the amount of spam is dropping or that spammers are giving in to the nearly-universal presence of anti-spam gateways.  In fact, because enterprise-class anti-spam gateways are successful at blocking 95% or more of incoming spam, this has simply increased the number of spam messages that are being sent as spammers and (increasingly) phishers push harder to get their message through the gate. 

Heavy use of messaging security gateways has also pressured spammers to be more ingenious in how they deliver their messages.  The implications of this trend are that any enterprise buying a security gateway needs to be assured that the vendor has the R&D resources needed to keep abreast of spammers' rapidly-changing techniques.  In addition, security gateways should be purchased with significant performance reserves, as new spam filtering techniques (along with the ever-escalating load of email and spam in general) are requiring more and more resources in the gateway. 

Messaging security gateways, despite the best efforts of the vendor community, are being increasingly perceived as a commodity product, with almost all of the emphasis being placed on their anti-spam features as the biggest commodity.  Our testing suggests there are substantial differences in the functionality of different anti-spam engines.  Nevertheless, many vendors are content to propagate the myth that the engines perform equally well, since this works to their advantage because it puts them on similar footing with the leading vendors.  Enterprise buyers, because of the difficulty of actually measuring differences in anti-spam engines, are also happy to go along with this perception.


To try and differentiate themselves above their anti-spam engines,  vendors have turned to putting a large numbers of bells and whistles in their products.  Some of these additions represent features that may not be appropriate for a messaging security gateway (such as IM logging and filtering); others are messaging-specific features, such as encryption, that focus on particular niches or vertical markets.  The most critical features for the enterprise, including centralized management and reporting, scalability, and high availability, are still only available in a handful of high-end products.


How Do Messaging Security Gateways Work?

By Joel Snyder, Network Lab Alliance

Messaging security gateways sit at the edge of the corporate network and act as a first barrier between the Internet and the enterprise messaging system. 

Although there is considerable variation in features and architecture, the majority of message security systems sits facing the Internet and receives incoming mail directly from the outside world that is destined for the enterprise.  As a first step they usually provide rate-control and reputation-based filtering for incoming mail.  For mail that gets rightly pass through these controls, messaging security gateways will then scan for spam and viruses, and apply further controls and filters on the stream of mail coming in.  Once the mail has been "cleaned" (sometimes these products are called "email hygiene" because of this cleaning process), the messages are passed onto the enterprise email system on the inside of the enterprise. 

The same gateway can also be used for outbound message delivery, usually with a slightly different set of security controls in place (often filtering, archiving, and anti-virus are applied to outbound email).  In this scenario, the enterprise mail systems simply hands all Internet-bound mail to the gateway, which then takes responsibility for delivering it.  The most common feature used in outbound delivery is footer stamping, the nearly ubiquitous practice in certain professions of placing a long addendum onto each message suggesting that anyone reading the message who shouldn't be must either delete it or, at the very least, gnaw off their own right arm. 

Messaging security gateways are a refinement on the older "email gateway" product space, which was originally put in place in large scale networks to convert Internet messaging formats (SMTP and MIME) to and from proprietary formats and addressing schemes used in the enterprise (such as MS Mail, cc:Mail, or GroupWise).  This new crop of messaging security gateways, driven to market by the need for anti-spam/anti-virus functionality at the edge of the network, have lost a lot of the functionality and features of their older brothers, but have taken on the appliance form factor and dramatic increases in performance more appropriate to their sharpened focus on a few specific functions. 

While scanning for spam and viruses can be done elsewhere in the message flow, such as on the email servers themselves, most email managers have found messaging security gateway appliances the perfect match for an unpleasant job.  By separating the filtering function out and keeping spam and viruses out of the mission-critical mail servers themselves, they are able to keep performance levels up and keep worries about interoperability and software integration down.  The appliance-like nature of most gateways also means that a poorly performing gateway can easily be upgraded or replaced with a beefier model without placing an impact on production mail streams. 

Although the gateways are largely independent of the core email system, some integration is needed for best operation.  For example, the messaging security gateway must be linked to the enterprise directory-normally via LDAP-so that it knows what mail to receive, what messages to refuse and how to further route the mail inside the enterprise network (especially if there are multiple internal email systems).

Some vendors in this space, notably Symantec, are experimenting with breaking the messaging security gateway into two parts: one piece specifically designed for rate control and reputation-based email filtering, and a second honed to handle the filtering, archiving, and scanning functions.  The idea is that in truly enormous message streams--a million messages an hour would be where this starts to kick in-having these functions separated offers the opportunity for greater scalability. 

While anti-spam and anti-virus scanning are the commonalities that most vendors put in their gateways, a wide variety of other messaging-oriented functions show up in these systems as well.  Content filtering-looking for specific words or phrases-is a frequent feature, as is message archiving-the ability to copy the incoming or outgoing message stream to an archiving server.  As part of the anti-spam functionality, some devices include their own spam or virus quarantine servers.  And email encryption services, ranging from transport-based encryption (such as enforcing TLS encryption with certain business partners) to application-layer encryption (such as signing and encrypting messages so that only the designated end-user can read them), are also found fairly frequently. 

In their quest for greater differentiation in an increasingly commoditized market, vendors are also branching off into other "messaging" security functions, such as Instant Messaging (IM) security. 


Subscribe to the Best of PCWorld Newsletter