Building Scalable Email Systems

Implementing Scalable Email Systems using Freely Available Components

Introduction

Many organizations are struggling with the problem of providing secure and scalable email systems to their users. This paper describes how we solved the problem of providing secure and scalable email to the users of Southern Illinois University School of Medicine in Springfield. This system is implemented using IMAP, SMTP, SSL and LDAP on the UNIX platform.

Problem

Our Requirements

As we started to design a system to replace our current mix of Novell GroupWise and UNIX POP mail clients and servers numerous design criteria were developed.

Regardless of how the new system was designed, the user had to be insulated from the system implementation. We wanted to provide a simple configuration for all clients that would work both inside and outside our firewall, plus we wanted to continue toward our goal of "standard" email addresses for all users. We had to continue in the framework of first initial last name @ domain for all accounts regardless of how the mail system was implemented.

A scalable solution means different things to different people. We already had a pair of Sun Enterprise 3000 servers that could have easily scaled to support our 2000 users. The downside to this was the initial and incremental cost involved in scaling the large SUN systems. We wanted a system that could be scaled/upgraded using cheap commodity PCs.

The system design had to be fault tolerant. We wanted to try and eliminate as many single points of failure as possible.

 Our department tries to avoid implementing proprietary systems. The new system had to use standards compliant components, such as IMAP and SMTP. Working with limited educational budgets made freeware solutions very attractive.

 We wanted to avoid making modifications to critical systems, such as the IMAP daemon itself.

 Our department was already taking steps to standardize on UNIX for many of our back-end systems.

 For obvious reasons, we wanted to implement a system that would eliminate plain-text passwords from our network. At minimum, we needed the ability to control where on the network plain-text passwords would travel.

Currently Available Systems

Many different email systems are available. Most commercial systems don’t meet the design criteria of being Open Standards based. This eliminated Netscape Messenger, Microsoft Exchange and Lotus Notes. Of the freely available UNIX systems, the most popular, UW IMAP, was not known for it security. The Cyrus IMAP system from Carnegie Mellon was attractive because of its speed and the security provided by the sealed server design (eventually Cyrus was incorporated into our design)

General Solution

The solution we finally implemented is an intelligent IMAP proxy. Rather than modify or develop an IMAP daemon, we decided to leave the daemon untouched and build a front-end to meet our design goals. The proxy works by implementing the non-authenticated states of the IMAP protocol. (LOGIN, LOGOUT, NOOP, CAPABILITY). The plain-text goal is achieved by implementing the SSL protocol in the IMAP proxy. Plain-text password exchanges are limited to our secured switched server network.

The client connects to the proxy and attempts to login. The proxy accept the username and password and looks the username up in a username à host database. The proxy then connects to the host on which the user’s mail exists passing the login information to the actual server and the proxy becomes just a "bit passer" for the IMAP connection.

Users are unaware they are connecting to a proxy which allows us to distribute mail across several servers providing both fault tolerance and scalability. More than one proxy can be deployed using Round-Robin DNS or more sophisticated load balancing schemes. For example, an IMAP cluster could be front-ended with a load-balanced cluster of machines running the proxy. A single failure would not down the entire system. This system doesn’t provide for backup if a back-end IMAP message store fails, but damage is contained to the lost mail store.

The proxy does make several assumptions. First, the IMAP servers must have the same capabilities. The proxy must be able to return the IMAP server’s capability before the username is available. Reporting the wrong capabilities for an IMAP server can be very dangerous and could result in lost or corrupt email. Second, the proxy must have access either locally or via the network to a mapping of usernames to hosts. Our implementation builds a database (hash) on the local machine from our LDAP directory every day. The decision was made to build a local database to reduce the possibility that an overloaded network interface could cause database lookups to fail. Third, a delivery mechanism must be built that maps usernames to their final mail storage server. Our implementation uses information in a LDAP directory to distribute mail delivery data to several inbound mail handlers.

This code requires the following pieces:

Future Growth

This system can be expanded in several ways. First, bigger faster IMAP servers. Second, add additional IMAP servers. Third, implement more sophisticated load balancers for the proxy front-end.

Our Specific Solution

In our environment, we were already building a fault-tolerance Kerberos and LDAP infrastructure. We have several geographically distributed Kerberos servers that provide password information for many of our services and several geographically distributed redundant LDAP servers, which we reference by a round-robin DNS host name. LDAP provides the distributed username to IMAP mail store host mappings used by our proxies and mail exchangers. Our individual IMAP mail servers use CMU’s Cyrus mail server. Cyrus was chosen because it’s fast, sealed and integrated nicely with our Kerberos environment. At present we have 2 IMAP servers running Cyrus. Our proxies are actually implemented on our IMAP servers. The proxy runs on the official IMAP and IMAP-SSL ports and the Cyrus daemon runs on a different port, to which only the proxies can connect. Users connect to a round-robin DNS host named. Imap.foo.bar which resolves to both IMAP servers. Each server runs the proxy which in turn connects the user to the correct IMAP mail store.

pfleming@siumed.edu
November 1,1999