Sendmail relaying control and SMTP AUTH with SuSE Linux 7.2

(04-Apr-2003)

This small page describes my experience with trying to update the sendmail on my SuSE Linux 7.2 box and create a useful configuration to prevent relaying, but allow mail submission from remote clients.

My hoster provides Linux "root" servers with a standard SuSE Linux installation. With all the security holes appearing it is important to keep up to date with critical software. Several exploits have become known about sendmail, and that is when you might want to update it. If you want to have specific features like a usable relaying control you might want to install a new version of sendmail. Some options require recompiling of sendmail. That is what I tried here: getting a new source distribution of sendmail 8.12 from sendmail.org, recompiling and installing this version. Most likely this is not the fastest way to get it to work. The purpose of this page is to give some explanation why something is done so you can understand the steps and modify them if your system setup is a bit different or you have additional requirements.

The first step is unpacking the distribution to a suitable directory, I usually just untar to /usr/local/src/ which will create the whole source tree. For me the main source directory of sendmail is /usr/local/src/sendmail-8.12.9/ let's call this SENDMAILSRCDIR for now. The usual way to compile sendmail is running

sh Build

in SENDMAILSRCDIR. This produced some errors and suggested defining NEWDB. Sendmail uses a bunch of configuration files which usually are in /etc/mail, including the main configuration file sendmail.cf. Most of these files are used in the form of Berkely DB databases, a standard way of using a fast small file-based database. So the next step is to find out where NEWDB (and also SMTP AUTH config info) has to be defined. The main config file for compile-time options for sendmail is

SENDMAILSRCDIR/devtools/Site/site.config.m4

(probably one of the reasons why people hate configuring sendmail). Sendmail uses the macro processor m4 to read some configuration files, and m4 is wierd itself. Anyway, you put the define for NEWDB there, and the define for SMTP AUTH using the SASL library as well:

APPENDDEF(`confMAPDEF',`-DNEWDB')
APPENDDEF(`confENVDEF',`-DSASL')
APPENDDEF(`conf_sendmail_LIBS', `-lsasl')

This means use 'NEW' Berkely DB library to access config files in /etc/mail and use the SASL library for authentication with SMTP AUTH (define and additional library for linking). The NEWDB define should *not* be necessary as it is default, but in a standard SuSE installation a few required DB header files might be missing. You will notice this when you compile with NEWDB and gcc complains about not finding the required include file db.h. This is part of the db-devel rpm package of SuSE which is normally not installed. You can get this rpm from the suse.com ftp-server and install it with

rpm -ihv db-devel.rpm

Depending on your SuSE version the filename might be a bit different. Leaving out SASL for now, you should now be able to build the executeable. The next step is installing the sendmail executable in the proper place and replace the old version. Stop the old sendmail by running

/etc/rc.d/sendmail stop

I suggest a backup of the old executable /usr/sbin/sendmail in case you run into serious trouble and want to revert. Try to run

make install

in SENDMAILSRCDIR will most likely fail because the install script complains about a nonexisting unix group. This is because the default installation is now to use separate programs for submitting and processing mail and running these partially as unprivileged users. This might be a good idea, but complicates matters. However you can still install sendmail the traditional way by running

sh ./Build install-set-user-id

in SENDMAILSRCDIR/sendmail. See SENDMAILSRCDIR/sendmail/SECURITY for a discussion of this option. Remove the file /etc/mail/submit.cf if there is one. The new sendmail should now be installed, and you can restart sendmail with

/etc/rc.d/sendmail start

It will probably run, but might complain about an old config file version. At this point you should learn a bit about configuring sendmail.cf. While it is possible to directly hack this file and its ruleset (and mess up sendmail completely) the normal way is to edit the source for it, a .m4 file usually placed in SENDMAILSRCDIR/cf/cf/. Start the configuration by copying generic-linux.mc to yourname.mc. Edit the file and change DOMAIN(generic) to DOMAIN(yourdomain), and copy SENDMAILSRCDIR/cf/domain/generic.m4 to SENDMAILSCRDIR/cf/domain/yourdomain.m4. (Replace yourname and yourdomain with useful names). This standard setup should be fine for a server with permanent Internet access. Go back into SENDMAILSRC/cf/cf and compile the mc file with

make yourname.cf

Copy the resulting file yourname.cf to /etc/sendmail.cf or /etc/mail/sendmail.cf. Which one depends on your setup. /etc/sendmail.cf is the old place, and /etc/mail/sendmail.cf might just be a link to /etc/sendmail.cf. Put the old file in a safe place first. Restart sendmail. You don't need to stop and start it if you didn't recompile, you can just send the SIGHUP signal to the sendmail process

kill -HUP pid

and sendmail will re-read its configuration files. You can get the pid with ps -ax | grep sendmail.