Basic Mail Setup
Enhanced Mail Services
Wrapping it Up
No installation would be complete without a Logfile analyzer. I decided to use AWStats since it will do both my web server and the mail logfile.
Installation and Configuration
We will start by emerging AWStats. As a reminder, I am using the vhosts flag as I have Apache configured for virtual hosts so I do not want emerge to try and install this in the Apache base directory. I will use webapp-config instead.
|Code: emerging AWstats|
# USE="vhosts" emerge awstats
Since I will be setting up the log analyzer in the same domain as I put squirrelmail, I will put it in an /awstats subdirectory. We will then secure access to this so that no unauthorized users will be able to access it.
The output of webapp-config will be important to capture. It will contain a mitt-full of variables, etc. that will be needed to configure the apache server correctly, so be sure to copy the output so you can paste it into your apache config files later.
|Code: Installing with webapp-config|
# webapp-config –d /awstats -I -h example.com awstats 6.7-r2
Although we are going to copy the output, we are not going to use it directly in our config files, with good reason. Many of the mini-howto’s will have you paste the output in directly … this is wrong. Some will even recommend that you pipe the output of webapp-config directly into your httpd.conf file. NO … Bad …. STOP IT.
The generated output points all of your Alias values into other areas of your system (“/usr/share/webapps/awstats ….”). This is not the copy of the application you should be using. We installed it in our domain (“example.com”) and so that is where we should be pointing to, not some directory off elsewhere in our system (like that is real secure).
The output is going to look some like this:
|Code: webapp-config Output|
================================================================= POST-INSTALL INSTRUCTIONS ================================================================= ### # INSTRUCTIONS: # # If you would like to require authentication to access AWStats, then use ONE # of the example configurations below. There are many other modules and # options for authentication which will not be discussed here. # # NOTE: Related AWStats configuration directives are: # # AllowAccessFromWebToAuthenticatedUsersOnly=1 # AllowAccessFromWebToFollowingAuthenticatedUsers="user1 [user2 ...]" # # You can use these to set a per-domain user access when needed for virtual # hosting. That means: only the selected user(s) will have access to stats for # the particular domain. All other users will not be allowed to see the domain # stats even though they have authenticated successfully. ### ### # SECTION I - Basic Authentication # # The following example requires mod_auth (apache 2.0) or mod_auth_basic # (apache 2.2) to work. Make sure you have installed and enabled it in # /etc/apache2/httpd.conf # # Add your AWStats users to /etc/awstats/.htpasswd file. # Please see 'man htpasswd2' for more details if you need. # # htpasswd2 -c /etc/awstats/.htpasswd username1 htpasswd2 # /etc/awstats/.htpasswd username2 etc... ### #<Location "/cgi-bin/awstats.pl"> # AuthType Basic # AuthName "AWStats authenticated zone" # AuthUserFile /etc/awstats/.htpasswd # Require valid-user #</Location> ### # SECTION II - Digest Authentication # # The following example requires mod_auth_digest to work. Make sure you have # installed and enabled it in /etc/apache2/httpd.conf # # Do not forget to replace www.example.com as appropriate. You can also add as # many domains as you need to this line. # # Add your AWStats users to /etc/awstats/.htdigest file. Please see 'man # htdigest2' and http://httpd.apache.org/docs-2.0/mod/mod_auth_digest.html for # more details if you need. # # htdigest2 -c /etc/awstats/.htdigest "AWStats authenticated zone" username1 # htdigest2 /etc/awstats/.htdigest "AWStats authenticated zone" username2 # etc... ### #<Location "/cgi-bin/awstats.pl"> # AuthType Digest # AuthName "AWStats authenticated zone" # AuthDigestFile /etc/awstats/.htdigest # AuthDigestDomain http://www.example.com https://www.example.com # require valid-user # # The following line is REQUIRED to work around a bug in MSIE. # # See http://httpd.apache.org/docs-2.0/mod/mod_auth_digest.html # BrowserMatch "MSIE" AuthDigestEnableQueryStringHack=On #</Directory> =================================================================
For now I will only focus on changing the necessary parts. All the stuff below the Note, labeled as INSTRUCTIONS is optional and is used for security. We will get to that, but for now, let’s just make the necessary changes to get this fired up.
Rather than putting our config into the httpd.conf file, we are going to put it all in vhost config file
# nano /etc/apache2/vhosts.d/example.conf.include ... <IfModule alias_module> ... ScriptAlias /cgi-bin/ "/var/www/example.com/cgi-bin/" ScriptAlias /awstats "/var/www/example.com/cgi-bin/awstats.pl" ScriptAlias /awstats.pl "/var/www/example.com/cgi-bin/awstats.pl" </IfModule> ...
We now need to setup an awstats configuration file so that it will know where to find its log files, what formats to read, etc. To keep things simple for now, we will setup a config file to read the apache log for our domain, www.example.com. (I will assume that if you have come this far, you have setup apache and said logs will be readily available).
For each domain or different log file you will read, you will need to setup a separate config file. By the time we have finished this section, we will have two setup, one for the web server and one for postfix. Create the base config files by copying the model that came with awstats.
|Code: Additional Config Files|
#cp /etc/awstats/awstats.model.conf /etc/awstats/awstats.www.example.com.conf #cp /etc/awstats/awstats.model.conf /etc/awstats/awstats.mail.example.com.conf
For now we will only modify that www.example.com config which will be used for the web server. You will only need to adjust the parameters under the MAIN SETUP SECTION that I have identified below to your specific settings. If it is not shown below, you should be able to leave it as defaulted with no problems.
# nano /etc/awstats/awstats.www.example.com.conf #----------------------------------------------------------------------------- # MAIN SETUP SECTION (Required to make AWStats work) #----------------------------------------------------------------------------- # "LogFile" contains the web, ftp or mail server log file to analyze. # Possible values: A full path, or a relative path from awstats.pl # directory. # Example: "/var/log/apache/access.log" # Example: "../logs/mycombinedlog.log" # You can also use tags in this filename if you need a dynamic file # name depending on date or time (Replacement is made by AWStats at the # beginning of its execution). The available tags : # %YYYY-n is replaced with 4 digits year we were n hours ago # %YY-n is replaced with 2 digits year we were n hours ago # %MM-n is replaced with 2 digits month we were n hours ago # %MO-n is replaced with 3 letters month we were n hours ago # %DD-n is replaced with day we were n hours ago # %HH-n is replaced with hour we were n hours ago # %NS-n is replaced with number of seconds at 00:00 since 1970 # %WM-n is replaced with the week number in month (1-5) # %Wm-n is replaced with the week number in month (0-4) # %WY-n is replaced with the week number in year (01-52) # %Wy-n is replaced with the week number in year (00-51) # %DW-n is replaced with the day number in week (1-7, 1=sunday) # use n=24 if you need (1-7, 1=monday) # %Dw-n is replaced with the day number in week (0-6, 0=sunday) # use n=24 if you need (0-6, 0=monday) # Use 0 for n if you need current year, month, day, hour... # Example: "/var/log/access_log.%YYYY-0%MM-0%DD-0.log" # Example: "C:/WINNT/system32/LogFiles/W3SVC1/ex%YY-24%MM-24%DD-24.log" # You can also use a pipe if log file come from a pipe : # Example: "gzip -d </var/log/apache/access.log.gz |" # If there is several log files from load balancing servers : # Example: "/pathtotools/logresolvemerge.pl *.log |" # LogFile="/var/log/apache2/access_log" # "SiteDomain" must contain the main domain name, or the main intranet # web server name, used to reach the web site. # If you share the same log file for several virtual web servers, this # parameter is used to tell AWStats to filter record that contains # records for this virtual host name only (So check that this virtual # hostname can be found in your log file and use a personalized log # format that include the %virtualname tag). # But for multi hosting a better solution is to have one log file for # each virtual web server. In this case, this parameter is only used # to generate full URL's links when ShowLinksOnUrl option is set to 1. # If analysing mail log, enter here the domain name of mail server. # Example: "myintranetserver" # Example: "www.domain.com" # Example: "ftp.domain.com" # Example: "domain.com" # SiteDomain="www.example.com" # When AWStats updates its statistics, it stores results of its # analysis in files (AWStats database). All those files are written in # the directory defined by the "DirData" parameter. Set this value to # the directory where you want AWStats to save its database and # working files into. # Warning: If you want to be able to use the # "AllowToUpdateStatsFromBrowser" feature (see later), you need # "Write" permissions by web server user on this directory (and # "Modify" for Windows NTFS file systems). # Example: "/var/lib/awstats" # Example: "../data" # Example: "C:/awstats_data_dir" # Default: "." (means same directory as awstats.pl) # DirData="../datadir"
Setting DirData wasn’t technically needed to make this work, but if you don’t the next step will dump data files into your cgi-bin directory, which is not where they belong. A datadir directory should have been created when we installed using webapp-config, so we should use it.
Our final step before testing is to have the AWStats utility rip through the Apache logfile created the analyzed data for output.
|Code: Preparing Logfiles|
# cd /var/www/example.com/cgi-bin/ # ./awstats.pl -config=www.example.com –update
This will dump a file into /var/www/example.com/datadir/ which will be read by AWStats when you access it from your browser.
Testing should be very straight forward. First of all, be sure to restart your Apache server so that the configuration changes we made take effect:
|Code: Restarting Apache|
# /etc/init.d/apache2 restart * Stopping Apache2... [ ok ] * Starting Apache2... [ ok ]
Now all you should have to do is point your browser at www.example.com/awstats and you should be seeing the stats for example.com on your web server.
If you did not get this working at this point, I suggest you go back over the instructions and get this working. The next steps are only going to further complicate the setup so as always you want to make sure the foundation is settled before building any higher.
Accepting Postfix Logs
It’s great that we can analyze apache logs, unfortunately we setout to analyze postfix logs. No worries, only a few minor configuration changes and we will be good to go.
First, we must setup AWStats to use the mail log file preprocessor mlc. (This is a drop-in-replacement for maillogconvert.pl that works with the newer postfix log format used in standard-gentoo installations nowadays.) For this we will download and install mlc, which is a pretty easy thing to do (of course you could also compile it if you think you need it).
|Code: Download and install mlc|
wget http://mlc.anzac.at/mlc-1.2.1.tar.gz tar -xzf mlc-1.2.1.tar.gz mv mlc-1.2.1/mlc /usr/local/bin/ rm -rf mlc-1.2.1 # put mlc-1.2.1.tar.gz to reasonable place or delete it at this point.
Then we modify the second configuration file we made, awstats.mail.example.com.conf. You will need to set each of the parameters as outlined below. Espceially you have to set "/var/log/postfix/*" to the correct path of your log files. All others can be left to their default settings.
# nano /etc/awstats/awstats.mail.example.com.conf LogFile="cat `ls -tr /var/log/postfix/*` | /usr/local/bin/mlc 2> /var/log/maillogconvert.err | " LogType=M LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd" SiteDomain="www.example.com" DirData="/var/lib/awstats" LevelForBrowsersDetection=0 LevelForOSDetection=0 LevelForRefererAnalyze=0 LevelForRobotsDetection=0 LevelForWormsDetection=0 LevelForSearchEnginesDetection=0 LevelForFileTypesDetection=0 ShowMenu=1 ShowSummary=HB ShowMonthStats=HB ShowDaysOfMonthStats=HB ShowDaysOfWeekStats=HB ShowHoursStats=HB ShowDomainsStats=0 ShowHostsStats=HBL ShowAuthenticatedUsers=0 ShowRobotsStats=0 ShowEMailSenders=HBML ShowEMailReceivers=HBML ShowSessionsStats=0 ShowPagesStats=0 ShowFileTypesStats=0 ShowFileSizesStats=0 ShowBrowsersStats=0 ShowOSStats=0 ShowOriginStats=0 ShowKeyphrasesStats=0 ShowKeywordsStats=0 ShowMiscStats=0 ShowHTTPErrorsStats=0 ShowSMTPErrorsStats=1
You need to remember that in my case, because my mail server is on a different box. I had to transfer the postfix log (which was my /var/log/messages file) over and place in the /postfix directory specified above. For now I have a cron job setup to automatically transfer it, but in the future, I will install NFS so I can remotely mount to the other system to get the log files.
The last step for the log file is to run the AWStats analyzer:
|Code: Preparing Logfiles|
# /var/www/example.com/cgi-bin/awstats.pl -config=mail.example.com –update
This time when you want to test, point your browser at www.example.com/awstats?config=mail.example.com and you should get the log analysis for the your postfix server. If you want to add more domains, mail servers, etc. just create new awstats.domain.conf files as we did above and then change the value after “config=” to point to the specific .conf file you want to use.
The last part of this setup some security so that only authorized users can access your statistics. This is where the rest of the webapp-config output that we copied above comes into play.
In my case I decided to use the Basic Authentication outlined in Section I. To activate this I added the necessary fields to my awstats.conf file.
# nano /etc/apache2/vhosts.d/example.com.include ... <Directory "/var/www/example.com/cgi-bin"> AllowOverride None Options None Order allow,deny Allow from all AuthType Basic AuthName "AWStats authenticated zone" AuthUserFile /etc/awstats/.htpasswd Require valid-user </Directory>
Set the AWStats config files so that it knows the user must be authenticated and specify which users are allowed:
# nano /etc/awstats/awstats.mail.example.com.conf # The following two parameters allow you to protect a config file from being # read by AWStats when called from a browser if web user has not been # authenticated. Your AWStats program must be in a web protected "realm" (With # Apache, you can use .htaccess files to do so. With other web servers, see # your server setup manual). # Change : Effective immediately # Possible values: 0 or 1 # Default: 0 # AllowAccessFromWebToAuthenticatedUsersOnly=1 # This parameter give the list of all authorized authenticated users to view # statistics for this domain/config file. This parameter is used only if # AllowAccessFromWebToAuthenticatedUsersOnly is set to 1. # Change : Effective immediately # Example: "user1 user2" # Example: "__REMOTE_USER__" # Default: "" # AllowAccessFromWebToFollowingAuthenticatedUsers="userid"
Finally we will create the .htpasswd file that will contain the password.
|Code: Creating .htpasswd|
# htpasswd2 –c /etc/awstats/.htpasswd userid New password: Re-type new password:
Now when you try to access AWStats, your browser should prompt you for a userid and password.
Remember, that if you do not enable "AllowToUpdateStatsFromBrowser", you have to add the following file:
cd /var/www/example.com/cgi-bin/ ./awstats.pl -config=www.example.com –update ./awstats.pl -config=mail.example.com –update
Created by NickStallman.net, Luxury Homes Australia
Real estate agents should list their apartments, townhouses and units in Australia.