piler installation guide

(updated ) 9 min
Linux Mail Server Storage & Backup Email mailpiler Tutorial

Note

This article is more than 6 year(s) old – information may be outdated.

piler installation guide

Want to update piler? Then you can check out my piler update guide.

This is my installation guide for mailpiler I have written and optimized in the last days. If there are any questions or ideas, please let me know in the comments below or via email.

ABOUT

PILER

piler is a open source email archiving solution with benefits and many features. The main usecase is for compliance and legal reasons of companies. Piler can encrypt and compress, deduplicate mails to reduce disk usage.

THIS GUIDE

This should guide you through the setup of mailpiler on a fresh Ubuntu installation. For your convenience I have added quite some sed commands for easily adjusting most configuration files as needed and also variables for easier replacing specific values throughout the guide.

However note, that the goal of this is not automating the installation and you might need to adjust the configuration to your own needs during or after steps provided. Consulting documentation and guides linked below is recommended.

This guide should take approx. 45 minutes.

ASSISTANCE

If assistance is required or having questions:

REQUIREMENTS

Installation documentation tested under following circumstances:

  • You have general and advanced Linux experience.
  • A clean Ubuntu 20.04.1 installation.
  • A fresh and dedicated machine for mailpiler.
  • Using user root throughout the guide. (piler runs under user piler)
  • Hardware requirements
    • Memory: 1 GB should be sufficient for small setups (~300 MB in idle after setup)
    • Disk: 10 GB for basic installation + your mail data
  • Network requirements
    • DNS record like piler.domain.tld already set accordingly.
    • Your mailserver must have access to Piler via SMTP (TCP/25) to send mails to archive via BCC.
    • Your piler installation requires access to your mailserver with either SMTP (TCP/25) or IMAPS (TCP/993) for restoring mails.

Important to note:

  • Do ONLY use this on clean machines dedicated for mailpiler. Also for security-purposes you would like to have your email archiving solution separated.
  • In this guide we do not enable clamd antivirus as we assume emails are already checked from the mailserver at the point where they arrived in the mailbox.
  • While this guide is quite universal, it was tested against a mailcow-powered email server.

Installation of dependencies

Preparation

To make life easier, we set variables in the current shell: (replace with your own values!)

SH
PILER_DOMAIN="piler.domain.tld"
MAILSERVER_DOMAIN="mail.domain.tld"

Install general dependencies

  1. First, we install general dependencies needed later on.

    SH
    apt install sysstat build-essential libwrap0-dev libpst-dev tnef libytnef0-dev unrtf catdoc libtre-dev tre-agrep poppler-utils libzip-dev unixodbc libpq5 software-properties-common libpoppler-dev openssl libssl-dev python3-mysqldb memcached pwgen telnet

  2. Remove packages pre-installed with Ubuntu 20.04 we do not need: (optional)

    SH
    apt remove --yes snapd multipath-tools
apt autoremove --yes

Install database (MariaDB)

  1. Prepare and install MariaDB database.

    SH
    apt-key adv --fetch-keys 'https://mariadb.org/mariadb_release_signing_key.asc'
add-apt-repository 'deb [arch=amd64] https://mirror.netcologne.de/mariadb/repo/10.4/ubuntu focal main'
apt install mariadb-{server,client} libmariadb-client-lgpl-dev-compat
  2. Optimize database: (as per mailpiler FAQ)
SH
cat > /etc/mysql/conf.d/mailpiler.conf <<EOF
innodb_buffer_pool_size=256M
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=64M
innodb_log_file_size=16M
query_cache_size=0
query_cache_type=0
query_cache_limit=2M
EOF

systemctl restart mariadb

Install PHP

Install PHP and required modules for mailpiler GUI from ondrej’s awesome PHP repo.

SH
add-apt-repository --yes ppa:ondrej/php
apt install php7.4-{fpm,common,ldap,mysql,cli,opcache,phpdbg,gd,memcache,json,readline,zip}

Install webserver

Install nginx webserver from ondrej’s awesome nginx repo.

SH
add-apt-repository --yes ppa:ondrej/nginx-mainline
apt install nginx
systemctl enable nginx

Install sphinxsearch

Install sphinxsearch, needed for indexing by mailpiler.

Note: Check version and download link on sphinxsearch download site and adjust version in download link below, if necessary.

Download and install binaries.

SH
mkdir -p /root/mailpiler/sphinxsearch/
cd /root/mailpiler/sphinxsearch/
wget http://sphinxsearch.com/files/sphinx-3.3.1-b72d67b-linux-amd64.tar.gz
tar xfz sphinx-*-linux-amd64.tar.gz
cp -v sphinx-*/bin/* /usr/local/bin/
rm /etc/cron.d/sphinxsearch

Install xlhtml

Prepare dependency for mailpiler.

Note: Check version and download link on mailpiler installation guide and adjust version in download link below, if necessary.

Download, configure, compile and install:

SH
mkdir -p /root/mailpiler/xlhtml/
cd /root/mailpiler/xlhtml/
wget https://bitbucket.org/jsuto/piler/downloads/xlhtml-0.5.1-sj-mod.tar.gz
tar xzf xlhtml-*-sj-mod.tar.gz
cd xlhtml-*-sj-mod/
./configure
make
make install
ldconfig

Installation of mailpiler

Install mailpiler

Prepare and install mailpiler itself.

  1. Create dedicated user and set permission accordingly.

    SH
    groupadd piler
useradd -g piler -m -s /bin/bash -d /var/piler piler
usermod -L piler
chmod 755 /var/piler

  2. Then: Download, configure, compile and install.

    Note: Check version and download link on mailpiler download and adjust version in download link below, if necessary.

    SH
    mkdir -p /root/mailpiler/piler/
cd /root/mailpiler/piler/
wget https://bitbucket.org/jsuto/piler/downloads/piler-1.3.9.tar.gz
tar xzf piler-*.tar.gz
cd piler-*/
./configure --localstatedir=/var --with-database=mysql --enable-memcached
make
make install
ldconfig

    Note: The install guide was only tested with 1.3.9. Newer versions – see the website – can work, but were not tested. Check the website to pick the latest version.

mailpiler post-install

After mailpiler was installed, we prepare everything.

  1. Generate password and prepare database:

    SH
    PILER_MYSQL_USER_PW="$(pwgen -cnsB 32 1)"
echo; echo "---"; echo "MYSQL PILER PASSWORD: $PILER_MYSQL_USER_PW"; echo "---"; echo

cp util/postinstall.sh util/postinstall.sh.bak
sed -i "s/   SMARTHOST=.*/   SMARTHOST="\"$MAILSERVER_DOMAIN\""/" util/postinstall.sh
sed -i 's/   WWWGROUP=.*/   WWWGROUP="www-data"/' util/postinstall.sh

    You will need this password in just a second during post-install.

  2. Start post-installation script of mailpiler:

    SH
    make postinstall

    The default values prompted from the post-installation script should be fine for this guide as we adjusted them with sed just right above. But still verify them before proceeding!

    At this point you have following key questions to answer:

    • The webserver groupname is www-data.
    • The mysql password for piler is the generated password above. (Copy-pasted values will not be visible.)
    • At mysql root password you can simply type in anything, as socket will be used.
    • Value smtp relay will be mail.domain.tld (same as $MAILSERVER_DOMAIN)

    Example output:

    TEXT
    This is the postinstall utility for piler
It should be run only at the first install. DO NOT run on an existing piler installation!

Continue? [Y/N] [N] y

Please enter the webserver groupname [www-data]

Please enter mysql hostname [localhost]

Please enter mysql socket path [/var/run/mysqld/mysqld.sock]

Please enter mysql database [piler]

Please enter mysql user name [piler]

Please enter mysql password for piler []

Please enter mysql root password. If its a recent version of mysql and uses socket authentication, then any string would do here []
mysql connection successful

Please enter smtp relay [mail.domain.tld]

Please enter smtp relay port [25]
no crontab for piler

INSTALLATION SUMMARY:

[...]

Correct? [Y/N] [N] Y

Continue and modify system? [Y/N] [N] Y

  3. Adjust piler and sphinx configuration:

    SH
    cp /usr/local/etc/piler/piler.conf /usr/local/etc/piler/piler.conf.bak
cp /usr/local/etc/piler/sphinx.conf /usr/local/etc/piler/sphinx.conf.bak

sed -i "s/hostid=.*/hostid=$PILER_DOMAIN/" /usr/local/etc/piler/piler.conf
sed -i "s/update_counters_to_memcached=.*/update_counters_to_memcached=1/" /usr/local/etc/piler/piler.conf
sed -i "s/spam_header_line=.*/spam_header_line=X-Spam-Flag: YES/" /usr/local/etc/piler/piler.conf # rspamd in mailcow setup.

sed -i "s/define('SPHINX_VERSION', .*/define('SPHINX_VERSION', 331);/" /usr/local/etc/piler/sphinx.conf
sed -i "s/define('SPHINX_STRICT_SCHEMA', 0);/define('SPHINX_STRICT_SCHEMA', 1);/" /usr/local/etc/piler/sphinx.conf # required for Sphinx 3.3.1

  4. Start mailpiler and searchd and enable autostart:

    SH
    /etc/init.d/rc.piler start
/etc/init.d/rc.searchd start

update-rc.d rc.piler defaults
update-rc.d rc.searchd defaults

Set up mailpiler WebUI

  1. Generate self-signed certificate (or use a different existing one):

    SH
    mkdir -p /etc/nginx/ssl
openssl req -x509 -nodes -days 3650 -newkey rsa:4096 -keyout /etc/nginx/ssl/piler.key -out /etc/nginx/ssl/piler.crt -subj "/CN=$PILER_DOMAIN" -addext "subjectAltName=DNS:$PILER_DOMAIN"

  2. Configure webserver and PHP:

    SH
    cp contrib/webserver/piler-nginx.conf /etc/nginx/sites-enabled/piler
sed -i "s|PILER_HOST|$PILER_DOMAIN|g" /etc/nginx/sites-enabled/piler
sed -i "s|/var/run/php/php7.2-fpm.sock|/var/run/php/php7.4-fpm.sock|g" /etc/nginx/sites-enabled/piler

  3. Let’s make the piler vhost more secure:

    SH
    sed -i "/server_name.*/a \\
        listen 443 ssl http2;\n\n\
        ssl_certificate /etc/nginx/ssl/piler.crt;\n\
        ssl_certificate_key /etc/nginx/ssl/piler.key;\n\n\
        ssl_session_timeout 1d;\n\
        ssl_session_cache shared:SSL:15m;\n\
        ssl_session_tickets off;\n\n\
        # modern configuration of Mozilla SSL configurator. Tweak to your needs.\n\
        ssl_protocols TLSv1.2 TLSv1.3;\n\
        ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;\n\
        ssl_prefer_server_ciphers off;\n\n\
        add_header X-Frame-Options SAMEORIGIN;\n\
        add_header X-Content-Type-Options nosniff;" /etc/nginx/sites-enabled/piler

sed -i "/^server {.*/i\
\# HTTP to HTTPS redirect.\n\
server {\n\
        listen 80;\n\
        server_name $PILER_DOMAIN;\n\
        return 301 https://\$host\$request_uri;\n\
}" /etc/nginx/sites-enabled/piler

  4. Now we test and reload the nginx configuration:

    SH
    nginx -t && systemctl reload nginx

Configure mailpiler webui

Now we adjust configuration file the mailpiler webui.

Furthermore we do enable authentication over IMAP. Meaning: All users with a mailbox can directly login into mailpiler, no manual intervention nor manually creating accounts is required. Each user can only see mails where the email address was either recipient or sender.

SH
cp /usr/local/etc/piler/config-site.php /usr/local/etc/piler/config-site.bak.php

sed -i "s|\$config\['SITE_URL'\] = .*|\$config\['SITE_URL'\] = 'https://$PILER_DOMAIN/';|" /usr/local/etc/piler/config-site.php

cat >> /usr/local/etc/piler/config-site.php <<EOF

// CUSTOM
\$config['PROVIDED_BY'] = '$MAILSERVER_DOMAIN';
\$config['SUPPORT_LINK'] = 'https://$MAILSERVER_DOMAIN';
\$config['COMPATIBILITY'] = '';

// fancy features.
\$config['ENABLE_INSTANT_SEARCH'] = 1;
\$config['ENABLE_TABLE_RESIZE'] = 1;

\$config['ENABLE_DELETE'] = 1;
\$config['ENABLE_ON_THE_FLY_VERIFICATION'] = 1;

// general settings.
\$config['TIMEZONE'] = 'UTC';

// authentication
// Enable authentication against an imap server
\$config['ENABLE_IMAP_AUTH'] = 1;
\$config['RESTORE_OVER_IMAP'] = 1;
\$config['IMAP_RESTORE_FOLDER_INBOX'] = 'INBOX';
\$config['IMAP_RESTORE_FOLDER_SENT'] = 'Sent';
\$config['IMAP_HOST'] = '$MAILSERVER_DOMAIN';
\$config['IMAP_PORT'] =  993;
\$config['IMAP_SSL'] = true;

// special settings.
\$config['MEMCACHED_ENABLED'] = 1;
\$config['SPHINX_STRICT_SCHEMA'] = 1; // required for Sphinx 3.3.1, see https://bitbucket.org/jsuto/piler/issues/1085/sphinx-331.
EOF

Finish line

  1. Clean up:

    SH
    apt autoremove --yes
apt clean

  2. Login to WebUI at https://$PILER_DOMAIN:

    TEXT
    Username: admin@local
Password: pilerrocks

And we’re done with mailpiler!

Next steps

  • Adjust mailpiler configuration to your personal needs.
  • Secure your mailpiler instance (specially SMTP TCP/25) against unintended access. All mails retrieved at SMTP TCP/25 will be imported by mailpiler. Could be problematic with public-reachable instances.
  • Configure mailserver to BCC mails from your mailserver to Pilers SMTP.
  • Import mails to mailpiler, see: mailpiler.org/wiki/current:importing-emails

INTEGRATION

Special tasks

These are special tasks you might or might not need at some point.

Retrieve aliases and set realname from mailcow

As IMAP authentication is used here, mailpiler is not aware to which alias addresses the user has access to nor knows the real name of the user, which can be set in mailcow too.

As mailpiler provides a functionality to hook into the authentication process, we are able to query the relevant details from our mailcow instance via the API and set this during the authentication procedure for the duration of the session.

I have built an implementation for this scenario, see the instructions in the official mailcow docs here.

Reverse proxy

It might be useful to make mailpiler accessible under a specific subdomain like https://archive.mail.domain.tld. For example useful when you have mailpiler running internally but would like to have the WebUI accessible from outside.

This can also be used for mailcow – the path could be, for example, /opt/mailcow-dockerized/data/conf/nginx/z_mailpiler.conf.

SH
# adjust these both variables:
PILER_DOMAIN_INTERNAL="piler.domain.tld"
PILER_DOMAIN_OUTSIDE="mailarchive.domain.tld"

cat > nginx_mailpiler.conf <<EOF
# HTTP to HTTPS redirect.
server {
    listen 80;
    server_name $PILER_DOMAIN_OUTSIDE;
    server_tokens off;
    return 301 https://\$host\$request_uri;
}
# Main server block
server {
    listen 443 http2;
    server_name $PILER_DOMAIN_OUTSIDE;
    server_tokens off;

    access_log off;
    error_log off;

    ssl_certificate /etc/ssl/mail/cert.pem;
    ssl_certificate_key /etc/ssl/mail/key.pem;

    ssl_session_timeout 1d;
    #ssl_session_cache shared:SSL:15m; # dont use for mailcow
    ssl_session_tickets off;

    # modern configuration of Mozilla SSL configurator. Tweak to your needs.
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    add_header X-Frame-Options SAMEORIGIN;
    add_header X-Content-Type-Options nosniff;
    add_header Strict-Transport-Security "max-age=15768000;";
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;
    add_header Referrer-Policy strict-origin;

    location / {
        proxy_set_header X-Real-IP         \$remote_addr;
        proxy_set_header X-Forwarded-For   \$proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto \$scheme;
        proxy_set_header Connection        "";
        proxy_read_timeout                 900;
        proxy_buffers                      32 4k;
        proxy_pass https://$PILER_DOMAIN_INTERNAL;
    }
}
EOF

In case of mailcow, to add ADDITIONAL_SAN automatically, you can use:

SH
cd /opt/mailcow-dockerized/
cp mailcow.conf mailcow.conf.bak
# let's add the $PILER_DOMAIN_OUTSIDE at the end of ADDITIONAL_SAN
# so that mailcow manages certs for mailpiler this way.
sed -i "/^ADDITIONAL_SAN=.*/ s/$/,$PILER_DOMAIN_OUTSIDE/" mailcow.conf

Restart services

SH
/etc/init.d/rc.searchd stop
/etc/init.d/rc.piler stop

/etc/init.d/rc.piler start
/etc/init.d/rc.searchd start

Checking logs

Log can be found at /var/log/mail.log.

SH
tail -f /var/log/mail.log

Piler-Folder Support

If folder support is required (see docs, should be enabled at day-one). It’s not about IMAP folders.

SH
sed -i '/^source base$/,/}/ s/}/   sql_attr_uint = folder\n}/' /usr/local/etc/piler/sphinx.conf
echo "\$config['ENABLE_FOLDER_RESTRICTIONS'] = 1;" >> /usr/local/etc/piler/config-site.php
sed -i "s/enable_folders=0/enable_folders=1/" /usr/local/etc/piler/piler.conf

And then restart piler and searchd.

Reset and reindex sphinx

searchd should not run while manually re-indexing!

SH
/etc/init.d/rc.searchd stop
rm -rf /var/piler/sphinx/*
sudo -u piler indexer --all --config /usr/local/etc/piler/sphinx.conf
/etc/init.d/rc.searchd start

Reset everything

SH
/etc/init.d/rc.searchd stop
/etc/init.d/rc.piler stop

rm -rf /var/piler/store/00/*

mysql --defaults-file=/etc/mysql/debian.cnf -e "DROP DATABASE piler;";
mysql --defaults-file=/etc/mysql/debian.cnf -e "CREATE DATABASE piler CHARACTER SET utf8;";
mysql --defaults-file=/etc/mysql/debian.cnf piler < /root/mailpiler/piler/piler-1.3.9/util/db-mysql.sql

rm -rf /var/piler/sphinx/*
sudo -u piler indexer --all --config /usr/local/etc/piler/sphinx.conf

/etc/init.d/rc.piler start
/etc/init.d/rc.searchd start

Sources

Comments

Loading comments...
You can use Markdown to format your comment.
0 / 5000 characters
Comments are moderated and may take some time to appear.