HOWTO_Email_Content_Filtering_with_mimedefang

1 Introduction
2 Purpose
3 Installation
- 3.1 MIMEDefang
- 3.2 Extra packages
4 Configuration
5 Results
6 Known Issues
7 Related Links

Introduction

This document will try to explain how to do server-side e-mail content filtering with a milter tool such as MIMEDefang.

Note: This article describes a practical approach to my personal needs and is not a general purpose content filtering howto. However, please contribute so that it may eventually become a more extensive guide.

Warning: MIMEDefang officially supports Sendmail via the MILTER protocol. Other mailers such as Postfix seem to implement MILTER but as of today there is no specific reference to Postfix in the MIMEDefang documentation. Use at your own risk.

Purpose

The author's goal is to implement a mail filtering gateway/relay for outgoing SMTP traffic. In particular, whenever a mail client sends messages through this relay, the filter will:

check for viri
check attached files' extensions and sizes
manipulate attached files
deliver filtered message to the next hop

The e-mail network layout this guide is based on is as follows:

Mail Client (LAN)
       |
Corporate Mail Server (LAN)
  |    |
  |   Email Content Filtering Relay (LAN)
  |    |
Outgoing Mail Server (LAN)
       |
Recipient Mail Servers (WAN)

Note: The proposed MIMEDefang configuration is meant for lazy Windows users who seldom compress their attachments and sometimes decide to send out a Christmas greeting as a 5MB pps file to about 10-15 recipients.

Installation

MIMEDefang

Code: emerge -pv mimedefang

  mail-filter/mimedefang-2.63

If Portage only has an earlier masked version then you can create (if necessary) or edit /etc/portage/package.keywords and change the following:

File: /etc/portage/package.keywords

mail-filter/mimedefang

Then you can create your own overlay and do a version bump to 2.63. Edit /etc/make.conf and specify the custom overlay directory:

File: /etc/make.conf

PORTDIR_OVERLAY=/usr/local/portage

# mkdir /usr/local/portage/mail-filter
# cp -r /usr/portage/mail-filter/mimedefang /usr/local/portage/mail-filter/
# cd /usr/local/portage/mail-filter/mimedefang
# mv mimedefang-***.ebuild mimedefang-2.63.ebuild
# ebuild mimedefang-2.63.ebuild digest
# emerge sendmail mimedefang

Note: If you get another mailer working successfully (eg. postfix, qmail, exim) then please post your code here.

Extra packages

If you have Apache and want the email recipients to download the attachments or if you want to automatically compress attachments then you need to install extra packages

Adjust /etc/portage/package.keywords if necessary and

# emerge mod_perl File-MMagic Archive-Zip

Configuration

MIMEDefang

mimedefang-filter

You should customize /etc/mail/mimedefang-filter. If you want to compress uncompressed outgoing attachments then you should use Archive::Zip. Also read the MIMEDefang comments about inline warnings.

Code: Settings

$AddWarningsInline = 1;
$Stupidity{"NoMultipleInlines"} = 1;

You need a custom procedure if you want to append a list of URLs which point to the user's attachments on your web server in case they were removed from the original message.

Code: Add the following procedure

#***********************************************************************
# %PROCEDURE: custom_list_replacement_urls
# %ARGUMENTS:
#  entity
# %RETURNS:
#  nothing
# %DESCRIPTION:
#  Lists replacement URLs at end of body message for all removed attachments.
#  Must be called from filter_end.
#***********************************************************************
sub custom_list_replacement_urls ($;$$) {
   my($entity, $header, $footer) = @_;
   my $plain = $header.join("\n", @custom_attachURLlist).$footer;
   my $html = $plain;
   $html =~ s|(http://\S*)|<a href="$1">$1</a>|g;
   $html =~ s/\n/<br>\n/g;
   append_text_boilerplate($entity, $plain, 0);
   append_html_boilerplate($entity, $html, 0);
}

If you do not want to list the attachment URLs at the bottom of the message body (boilerplate) then do not call custom_list_replacement_urls. Otherwise, you can call it from filter_end (you probably won't need to call Spamaassassin since e-mails are coming from a known internal source; so replace the Spamassassin call with this one).

Code: In "sub filter_end" add this

   if (@custom_attachURLlist > 0) {
	&custom_list_replacement_urls($entity,"Click on each link to download the attachment.\n","\nData is available during one week.")
   }
   if ($Sender eq '<>') {
	append_text_boilerplate($entity, "\nPlease do not send this message again. Please consult the IT dept. with ref. msg id $MessageID (unknown sender/reply-path).", 0);
	my($custom_bkfname) = "/SAMBA/Trouble_email/msg_".time()."_".int(rand(100)).".eml";
	copy_or_link("./INPUTMSG", $custom_bkfname);
	chmod 0666, $custom_bkfname;
   }

And you need to initialize URL list array.

Code: In "sub filter_begin" on the second line add this

   @custom_attachURLlist = ();

The following code in sub filter is used to:

avoid filtering if sender is my_boss@wherever
zip attachment if it's not already compressed
replace attachment with url if attached file size or message bandwidth are greater than a given threshold

Code: In "sub filter" add this

   # If it's the postmaster (or unknown sender) then remove all attachments
   if ( ($Sender eq '<>') && ($fname ne '') ) {
       md_graphdefang_log('mail', 'CUSTOM_UNKNOWN_SENDER', $fname);
       return action_drop_with_warning("Attachment removed due to unknown sender: $fname\n");
   }

   # Some senders do not get their mail filtered
   if ( ($fname ne '') && ($Sender ne '<>') && ($Sender ne '<me@domain1.org>') && ($Sender ne '<me@domain2.org>') ) {

	# If it's not a compressed file then zip it
	my($comp_exts, $re);
	$comp_exts = '(zip|gz|tgz|bz2|Z|\{[^\}]+\})';
	$re = '\.' . $comp_exts . '\.*$';
	my $zipped;

	if ( ($Features{"Archive::Zip"}) && (!re_match($entity, $re)) ) {
	    md_graphdefang_log('mail', 'CUSTOM_COMPRESS', $fname);
	    my $zip = Archive::Zip->new();
	    my $member;
	    $member = $zip->addFile($entity->bodyhandle->path, $fname);
	    # compress (DEFLATED) or not (STORED)
	    $member->desiredCompressionMethod( COMPRESSION_DEFLATED );
	    $member->desiredCompressionLevel( COMPRESSION_LEVEL_BEST_COMPRESSION );
	    $member->setLastModFileDateTimeFromUnix( 318211200 );
	    $fname = "$fname.zip" unless $fname =~ s/\.[^.]*$/\.zip/;
	    $zip->writeToFileNamed("./Work/CUSTOM_$fname");
	    custom_action_replace_with_file($entity, "./Work/CUSTOM_$fname", $fname);
	    $zipped = 1;
	}

	my($custom_size, $custom_bandwidth);
	$custom_bandwidth = (stat("./INPUTMSG"))[7]*scalar(@Recipients);
	md_graphdefang_log('mail', 'CUSTOM_BANDWIDTH', $custom_bandwidth);
	$custom_size = (stat($entity->bodyhandle->path))[7];
	md_graphdefang_log('mail', 'CUSTOM_MSG_PART_SIZE', $custom_size);
	if ( ($custom_size > 800*1024) || ($custom_bandwidth > 1300*1024) ) {
   	    action_notify_sender("Your e-mail attachments have been replaced with web links. No further action required.\n");
   	    return custom_action_replace_with_url($entity,"/var/www/localhost/htdocs/dload","http://download.domain3.org/dload","You can download the attachment from:<br>\n<a href='_URL_'>_URL_</a><br>\nNOTICE: the attached file ($fname - $custom_size bytes) is available for about 1 week.\n","text/html","html","attachment",$fname);
	}

	if ($zipped) {
	    return 1;
	}

   }

   return action_accept();

Some more custom procedures should be added.

Code: Add the following procedures to your filter file

#***********************************************************************
# %PROCEDURE: custom_action_replace_with_warning
# %ARGUMENTS:
#  msg -- warning message
#  mtype -- mime type
#  mext -- extension
#  mname -- file name
# %RETURNS:
#  Nothing
# %DESCRIPTION:
#  Makes a note to drop the current part and replace it with a warning
#***********************************************************************
sub custom_action_replace_with_warning ($;$$$) {
   my($msg, $mtype, $mext, $mname) = @_;
   return 0 if (!in_filter_context("custom_action_replace_with_warning"));
   $Actions{'replace_with_warning'}++;
   $Action = "replace";
   $mtype = "text/plain" unless defined($mtype);
   $mext = "txt" unless defined($mext);
   $mname = "warning" unless defined($mname);
   $ReplacementEntity = MIME::Entity->build(Type => $mtype,
					     Encoding => "-suggest",
					     Data => [ "$msg\n" ]);
   $WarningCounter++;
   $ReplacementEntity->head->mime_attr("Content-Type.name" => "$mname$WarningCounter.$mext");
   $ReplacementEntity->head->mime_attr("Content-Disposition" => "inline");
   $ReplacementEntity->head->mime_attr("Content-Disposition.filename" => "$mname$WarningCounter.$mext");
   return 1;
}

#***********************************************************************
# %PROCEDURE: custom_action_replace_with_file
# %ARGUMENTS:
#  entity -- mime entity
#  nfname -- new file path
#  nfdesc -- description
#  nftype -- mime type
#  nfencode -- encoding
# %RETURNS:
#  Nothing
# %DESCRIPTION:
#  Makes a note to drop the current part and replace it with a file
#***********************************************************************
sub custom_action_replace_with_file ($$;$$$$$) {
   my($entity, $nfpath, $nfname, $nftype, $nfencode, $nfdispo) = @_;
   return 0 if (!in_filter_context("custom_action_replace_with_file"));
   $Actions{'replace_with_file'}++;
   $Action = "replace";
   $nftype = "application/octet-stream" unless defined($nftype);
   $nfname = "" unless defined($nfname);
   $nfencode = "base64" unless defined($nfencode);
   $nfdispo = "attachment" unless defined($nfdispo);
   $ReplacementEntity = MIME::Entity->build(Type => $nftype,
					     Encoding => $nfencode,
					     Path => $nfpath,
					     Filename => $nfname,
					     Disposition => $nfdispo);
   copy_or_link($nfpath, $entity->bodyhandle->path) or return 0;
   return 1;
}
 
#***********************************************************************
# %PROCEDURE: custom_action_replace_with_url
# %ARGUMENTS:
#  entity -- part to replace
#  doc_root -- document root in which to place file
#  base_url -- base URL for retrieving document
#  msg -- message to replace document with.  The string "_URL_" is
#         replaced with the actual URL of the part.
#  mtype -- message mime type (for the warning msg)
#  mext -- message extension (for the warning msg)
#  mname -- message name (for the warning msg)
#  cd_data -- optional Content-Disposition filename data to save
#  salt    -- optional salt to add to SHA1 hash.
# %RETURNS:
#  1 on success, 0 on failure
# %DESCRIPTION:
#  Places the part in doc_root/{sha1_of_part}.ext and replaces it with
#  an mtype part giving the URL for pickup.
#***********************************************************************
sub custom_action_replace_with_url ($$$$;$$$$$) {
   my($entity, $doc_root, $base_url, $msg, $mtype, $mext, $mname, $cd_data, $salt) = @_;
   my($ctx);
   my($path);
   my($fname, $ext, $name, $url);
   my $extension = "";

   return 0 unless in_filter_context("custom_action_replace_with_url");
   return 0 unless defined($entity->bodyhandle);
   $path = $entity->bodyhandle->path;
   return 0 unless defined($path);
   open(IN, "<$path") or return 0;

   $ctx = Digest::SHA1->new;
   $ctx->addfile(*IN);
   $ctx->add($salt) if defined($salt);
   close(IN);

   $fname = takeStabAtFilename($entity);
   $fname = "" unless defined($fname);
   $extension = $1 if ($fname =~ /(\.[^.]*)$/);

   # Use extension if it is .[alpha,digit,underscore]
   $extension = "" unless ($extension =~ /^\.[A-Za-z0-9_]*$/);

   # Filename to save
   $name = $ctx->hexdigest . $extension;
   $fname = $doc_root . "/" . $name;
   $url = $base_url . "/" . $name;

   if (-r $fname) {
	# If file exists, then this is either a duplicate or someone
	# has defeated SHA1.  Just update the mtime on the file.
	my($now);
	$now = time;
	utime($now, $now, $fname);
   } else {
	copy_or_link($path, $fname) or return 0;
	# In case umask is whacked...
	chmod 0644, $fname;
   }

   # save optional Content-Disposition data
   if (defined($cd_data) and ($cd_data ne "")) {
	if (open CDF, ">$doc_root/.$name") {
	    print CDF $cd_data;
	    close CDF;
	    chmod 0644, "$doc_root/.$name";
	}
   }

   push(@custom_attachURLlist, $cd_data."\n".$url."\n");

   $msg =~ s/_URL_/$url/g;
   if (defined($mtype) and ($mtype ne "") and defined($mext) and ($mext ne "") and defined($mname) and ($mname ne "")) {
	custom_action_replace_with_warning($msg, $mtype, $mext, $mname);
	}
   else {
	custom_action_replace_with_warning($msg);
	}

   return 1;
}

mimedefang conf.d

You can define several parameters in /etc/conf.d/mimedefang such as not to include the MIMEDefang version in outgoing headers (check out the man page).

Code: Add the following procedures to your filter file

SYSLOG_FACILITY=mail
MD_EXTRA="-X"

Antivirus

If you have ClamAV installed then you need to adjust its configuration file /etc/clamd.conf.

Code:

 LocalSocket /var/spool/MIMEDefang/clamd.sock
 User defang

Set file permissions for the defang user.

# chown defang:root /var/run/clamav
# chown defang:root /var/log/clamav/clamd.log

Mailer

Note: Sendmail configuration will be described briefly in this section but if you succeeded with another mail daemon then please post your method here.

/etc/mail/sendmail.mc

divert(-1)
divert(0)dnl
include(`/usr/share/sendmail-cf/m4/cf.m4')dnl
VERSIONID(`$Id: sendmail-procmail.mc,v 1.2 2004/12/07 01:59:31 g2boojum Exp $')dnl
OSTYPE(linux)dnl
DOMAIN(generic)dnl
FEATURE(access_db)
FEATURE(accept_unqualified_senders)
#FEATURE(accept_unresolvable_domains)
MAILER(smtp)dnl
define(`SMART_HOST', `hostname.mydomain.org')
#define(`confLOG_LEVEL', `15')
INPUT_MAIL_FILTER(`mimedefang', `S=unix:/var/spool/MIMEDefang/mimedefang.sock, F=T, T=S:5m;R:5m')

/etc/mail/access

 Connect:localhost       RELAY
 Connect:127.0.0.1       RELAY
 Connect:10.215.144.16   RELAY

/etc/mail/submit.mc

divert(0)dnl
VERSIONID(`$Id: submit.mc,v 8.14 2006/04/05 05:54:41 ca Exp $')
define(`confCF_VERSION', `Submit')dnl
define(`__OSTYPE__',`')dnl dirty hack to keep proto.m4 from complaining
define(`_USE_DECNET_SYNTAX_', `1')dnl support DECnet
define(`confTIME_ZONE', `USE_TZ')dnl
define(`confDONT_INIT_GROUPS', `True')dnl
define(`confDIRECT_SUBMISSION_MODIFIERS', `C')dnl
dnl
dnl If you use IPv6 only, change [127.0.0.1] to [IPv6:::1]
FEATURE(`msp', `[10.215.144.7]')dnl

In my case, 10.215.144.7 is hostname.mydomain.org (a QMAIL server in my LAN in charge of sending out to the Internet) and 10.215.144.16 is the Corporate Mail Server which is passing e-mails to this box.

# m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf
# m4 /etc/mail/submit.mc > /etc/mail/submit.cf
# makemap hash /etc/mail/access.db < /etc/mail/access
# makemap hash /etc/mail/aliases.db < /etc/mail/aliases

Check permissions.

Code: ls -all /var/spool/clientmqueue

drwxrwx---  2 smmsp smmsp 4096 dic 17 03:43 .

Code: If you want to send notifications: ls -l /usr/sbin/sendmail*

-r-xr-sr-x 1 root smmsp 702312 dic 14 15:07 /usr/sbin/sendmail
-r-xr-sr-x 1 root smmsp 702312 dic 14 15:07 /usr/sbin/sendmail.sendmail

If you want sender notifications to arrive within, say, 2 minutes then you need to change /etc/conf.d/sendmail

SENDMAIL_OPTS="-bd -q30m -L sm-mta"
CLIENTMQUEUE_OPTS="-Ac -q2m -L sm-cm"

Apache and mod_perl

/etc/apache2/modules.d/apache2-mod_perl-startup.pl

use lib qw(/var/www/localhost/perl);
# enable if the mod_perl 1.0 compatibility is needed
# use Apache2::compat ();

You should verify that /etc/apache2/modules.d/75_mod_perl.conf has the following entries (ref. bug).

   <Location  /perl/*.pl>
       SetHandler perl-script
       PerlResponseHandler ModPerl::Registry
       Options -Indexes ExecCGI
       PerlSendHeader On
       # AllowOverride All
       Order allow,deny
       Allow from all
   </Location>

   #set Apache::PerlRun Mode for /cgi-perl Alias
   <Location /cgi-perl/*.pl>
       SetHandler perl-script
       PerlResponseHandler ModPerl::PerlRun
       Options -Indexes ExecCGI
       PerlSendHeader On
       # AllowOverride All
       Order allow,deny
       Allow from all
   </Location>

/etc/apache2/vhosts.d/default_vhost.include or whichever Apache conf file you use should contain something like this:

   <Directory "/var/www/localhost/htdocs/dload">
       AllowOverride None
       Options None
       php_flag engine off
       SetHandler perl-script
       PerlTypeHandler Apache::AddContentDisposition
       AddDefaultCharset off
       DefaultType application/octet-stream
       Order allow,deny
       Allow from all
   </Directory>

   ErrorDocument 400 "not allowed"
   ErrorDocument 401 "not allowed"
   ErrorDocument 403 "not allowed"
   ErrorDocument 404 "not allowed"
   ErrorDocument 405 "not allowed"
   ErrorDocument 408 "not allowed"
   ErrorDocument 410 "not allowed"
   ErrorDocument 411 "not allowed"
   ErrorDocument 412 "not allowed"
   ErrorDocument 413 "not allowed"
   ErrorDocument 414 "not allowed"
   ErrorDocument 415 "not allowed"
   ErrorDocument 500 "not allowed"
   ErrorDocument 501 "not allowed"
   ErrorDocument 502 "not allowed"
   ErrorDocument 503 "not allowed"
   ErrorDocument 506 "not allowed"

Of course if you do not have PHP installed you can remove the php_flag.

/var/www/localhost/perl/Apache/AddContentDisposition.pm

# Apache 1.3 mod_perl PerlTypeHandler to add the Content-Disposition
# header to outgoing files from a ".filename" file in the same directory
# containing the real file.  This allows files with names such as
# "341ee566.gif" to have a human-friendly name associated with them.
#
# See RFC 2183 for more information on the Content-Disposition header.
#
# Enable this module with a PerlTypeHandler directive for the
# <Directory> or similar areas in question after installing this file
# under an Apache directory in @INC.
#
# PerlTypeHandler Apache::AddContentDisposition
#
# The author disclaims all copyrights and releases this module into the
# public domain.

package Apache::AddContentDisposition; 

use Apache::Constants qw(OK DECLINED);

# to determine the Content-Type of the file
use File::MMagic; 

#       rename  from    =>              to
my %map = (
       'image/pjpeg' => 'image/jpeg'
);
my %needMagic = (
        'application/octet-stream' => 1
        , 'application/binary' => 1
);

sub handler {
        my $r = shift;

#LOG!  $r->warn("In AddContentDisposition" . $r->filename); 

        # /foo/bar file from Apache -> /foo/.bar metafile
        (my $metafile = $r->filename) =~ s,/([^/]+)$,/.$1,;

        # first line of data should contain mime type, tab, then the
        # recommended filename
        unless(open(FILE, $metafile)) {
                $r->log_reason("No such metafile \"$metafile\": $! " . $r->filename);
                return DECLINED;
        }
        my $filename = <FILE>;
        close FILE;

#LOG!  $r->warn("$metafile -> $filename");

        return DECLINED unless $filename;

        my $mime_type = undef;
        if($filename =~ s!^([a-z0-9-]+/\S+)\s+!!i) {
                $mime_type = lc($1);
        }

        # sanitize out characters not listed and .. runs to mitigate potential
        # security problems on clients
        $filename =~ s/[^\w.-]//g;
        $filename =~ s/\.\.+/\./g;
        # Lowercase extension
        $filename =~ s/\.([^\.]*[A-Z][^\.]*)$/\.\L$1/g;

#LOG!        $r->warn("1: MIME type = '$mime_type', filename = '$filename'");

        return DECLINED unless $filename;

        if(!$mime_type || defined $needMagic{$mime_type}) {
                # need to set Content-Type as is set to text/plain once our custom
                # Content-Disposition header is set, which trips up Mozilla
                my $type = File::MMagic->new->checktype_filename($r->filename);

                $mime_type = $type if $type;

#LOG!                $r->warn("File/MMagic: MIME type = '$mime_type', filename = '$filename'");
        }

        return DECLINED unless $mime_type;

        if(my $type = $map{lc($mime_type)}) {
                $mime_type = $type;
        }

        $r->content_type($mime_type);
        $r->headers_out->set("Content-Disposition" => "inline; filename=$filename");

        return OK;
        }

1;

Set permissions.

# chown apache:apache /var/www/localhost/perl/Apache/AddContentDisposition.pm
# chmod a+rx /var/www/localhost/perl/Apache/AddContentDisposition.pm
# chmod 777 /var/www/localhost/htdocs/dload

Results

Always launch mimedefang before sendmail. Always stop sendmail before mimedefang.

# /etc/init.d/mimedefang start
# /etc/init.d/sendmail start

If a local client sends an e-mail then it will be scanned by MIMEDefang (you should check the mail log and/or add a generic custom header in filter_begin with action_add_header).

You might want to configure syslog-ng to send mail facility logs to a specific file. Add the following to /etc/syslog-ng/syslog-ng.conf.

filter f_mail { facility(mail); };
destination maillog { file("/var/log/maillog"); };
log { source(src); filter(f_mail); destination(maillog); };

Look out for perl compilation/run-time errors in the log file as messages get filtered.

If you followed this guide then you should get the following results:

an outgoing e-mail with a big attachment will be compressed if necessary and if its size is still too big it will be replaced by a URL link pointing to your Apache directory. The recipient should receive a "push here dummy" link which will retrieve the file data with the original file name.
an outgoing e-mail with a small (non-filtered) attachment but with plenty of recipients should get filtered and replaced by a URL (this should save you lots of bandwidth problems, at least on the outgoing SMTP line).
a message with a small uncompressed attachment will be delivered without URL replacement but zipped.

Known Issues

an e-mail with just one recipient and two attachments (a big one and a small one) will have a replaced URL even for the small attachment (unless you remove the INPUTMSG size condition in filter).
if several outgoing messages contain the same uncompressed file then your Apache directory will contain several copies of the same file with different SHA1 digests as the zip files are generated on the fly (different timestamps). It can be fixed by using a fixed date for the compressed file (setLastModFileDateTimeFromUnix).

Gentoo Wiki

HOWTO_Email_Content_Filtering_with_mimedefang

Contents

Introduction

Purpose

Installation

MIMEDefang

Extra packages

Configuration

MIMEDefang

mimedefang-filter

mimedefang conf.d

Antivirus

Mailer

Apache and mod_perl

Results

Known Issues

Related Links