Compare commits

..

10 Commits

Author SHA1 Message Date
link2xt
4b04aae83b feat: reduce maximal_queue_lifetime from 5d to 2d
If the message is not delivered within 2 days,
it is unlikely to be delivered in 5 days either.
2026-05-20 19:27:58 +00:00
link2xt
0eed92171c fix: reduce maxproc for filtermail-transport LMTP client to 500
This further reduces it from 1000.
For small servers this may be needed if they have low memory.
For large servers may be increased manually for now.
2026-05-20 15:39:11 +00:00
link2xt
a5b9a98baa fix: limit the number of LMTP clients for filtermail-transport to 1000
Postfix does not have jitter for deferred mails
and scans the queue periodically every
queue_run_delay (<https://www.postfix.org/postconf.5.html#queue_run_delay>).
As a result it is likely
to try delivering many deferred messages
at the same time.

Normally the number of outgoing connections
should be low even with unreachable destinations,
but after the server downtime
or if admin flushes the queue manually
it is possible that a lot of messages
to the same unreachable destination
expire at once and are moved
from "deferred" into the "active" queue.

Trying to deliver them all at once
may make the server run out of memory
by starting many LMTP clients.
Limiting the number of LMTP processes
turns OOM problem into head of line blocking problem.
Messages sent to reachable destinations
will be delayed as well,
but at least deferred messages will
get distributed over time.

In this case "active" queue may grow
(up to qmgr_message_active_limit defaulting to 20000),
but then admin may notice the problem
and solve it e.g. by making the destinations reachable
or setting up a transport map to route
messages for known dead servers into discard transport.

Eventually the problem should be solved
by filtermail-transport quickly returning temporary errors
for destinations which already have many messages queued,
then we can reduce "maxproc" further.
2026-05-19 22:17:04 +00:00
link2xt
ab2d807084 fix: set relay restrictions per smtpd service with default reject
We never want to defer email with a tepporary error when it has destination
that we cannot deliver locally and don't want to relay.
To avoid doing this accidentally, set default action to "reject"
and then override it with the minimal restrictions per smtpd.

Submission ports already had smtpd_relay_restrictions=permit_sasl_authenticated,reject override.

Each smtpd port must have at least one of
reject, reject_unauth_destination, defer, defer_if_permit, defer_unauth_destination
according to <https://www.postfix.org/postconf.5.html#smtpd_relay_restrictions>.

I have set smtpd_relay_restrictions=reject_unauth_destination for port 25 and incoming reinject port,
and smtpd_relay_restrictions=permit_mynetworks,reject for outgoing reinject port.
2026-05-19 15:54:15 +00:00
j4n
ce05b26c77 ci: auto-trigger docker build on release tag push
docker-dispatch.yaml previously only fired on push to main and manual
workflow_dispatch, so tagging 1.11.0 did not build the release image.
This change adds matching of X.Y.Z tag.
2026-05-19 14:58:05 +02:00
missytake
77ed93fb7a docs: add scripts/initenv.sh to upgrade instructions 2026-05-18 10:35:25 +02:00
missytake
39d1ecaa03 chore(release): prepare for 1.11.0 2026-05-15 17:13:58 +02:00
holger krekel
a266ffd060 fix: fix #972 by increasing file descriptors for filtermail 2026-05-14 22:40:25 +02:00
holger krekel
a47bb94143 feat: warn about any unused chatmail.ini parameter at the end of "cmdeploy run" 2026-05-14 20:58:47 +02:00
holger krekel
43ae9fee5c feat!: ignore passthrough_sender and passthrough_recipients to eliminate one more source of unencrypted messages
When running "cmdeploy run" operators will see a warning if their chatmail.ini contains these unused options.
2026-05-14 20:58:47 +02:00
12 changed files with 150 additions and 52 deletions

View File

@@ -9,6 +9,7 @@ name: Trigger Docker build
on:
push:
branches: [main]
tags: ['[0-9]+.[0-9]+.[0-9]+']
workflow_dispatch:
permissions: {}

View File

@@ -1,5 +1,47 @@
# Changelog for chatmail deployment
## [1.11.0] - 2026-05-15
### Breaking Changes
- [**breaking**] Drop passthrough_sender and passthrough_recipients chatmail.ini options to eliminate one more source of unencrypted messages
### Features
- Use filtermail for delivery to remote MTAs
- Expose metadata "maxsmtprecipients" value
- Support setup without domain, with only an IPv4 address (#963)
- *(doc/docker)* Introduce docker images in documentation
- DKIM-sign bounce messages (mainly "user does not exist")
- *(config)* Load default values from Config(), not chatmail.ini.f (#853)
- Make turn_socket_path configurable, and cleanup tests and turnserver code.
- Warn about any unused chatmail.ini parameter at the end of "cmdeploy run"
### Bug Fixes
- Make www tests work with editable instead of just plain installs
- Use path with no leading slash for mxdeliv
- Increase filtermail-transport concurrency limit
- Fix #972 by increasing file descriptors for filtermail
- *(mtail)* Correct boot ordering and deploy restart logic
- *(cmdeploy)* Stop and disable unbound-resolvconf
- *(nginx)* Properly redirect www to mail_domain
- *(dns)* Query correct NS if MNAME server is hidden (#954)
- Legacy token metadata storage used list type, but if no new setmetadata happened, the user would not be notified at all.
- *(logging)* Log all http requests to syslog
### Documentation
- Document how to upgrade to new version (#965)
### Other
- *(deps)* Upgrade to filtermail v0.6.4
### Refactor
- Introduce automated change-tracking across deployers
## 1.10.0 2026-04-30
* start mtail after networking is fully up <https://github.com/chatmail/relay/pull/942>

View File

@@ -16,7 +16,8 @@ def read_config(inipath):
class Config:
def __init__(self, inipath, params):
self._inipath = inipath
raw_domain = params["mail_domain"]
params = dict(params)
raw_domain = params.pop("mail_domain")
self.mail_domain_bare = raw_domain
if is_valid_ipv4(raw_domain):
@@ -29,60 +30,59 @@ class Config:
self.mail_domain = raw_domain
self.postfix_myhostname = raw_domain
self.max_user_send_per_minute = int(params.get("max_user_send_per_minute", 60))
self.max_user_send_burst_size = int(params.get("max_user_send_burst_size", 10))
self.max_mailbox_size = params.get("max_mailbox_size", "500M")
self.max_message_size = int(params.get("max_message_size", 31457280))
self.delete_mails_after = params.get("delete_mails_after", "20")
self.delete_large_after = params.get("delete_large_after", "7")
self.max_user_send_per_minute = int(params.pop("max_user_send_per_minute", 60))
self.max_user_send_burst_size = int(params.pop("max_user_send_burst_size", 10))
self.max_mailbox_size = params.pop("max_mailbox_size", "500M")
self.max_message_size = int(params.pop("max_message_size", 31457280))
self.delete_mails_after = params.pop("delete_mails_after", "20")
self.delete_large_after = params.pop("delete_large_after", "7")
self.delete_inactive_users_after = int(
params.get("delete_inactive_users_after", 90)
params.pop("delete_inactive_users_after", 90)
)
self.username_min_length = int(params.get("username_min_length", 9))
self.username_max_length = int(params.get("username_max_length", 9))
self.password_min_length = int(params.get("password_min_length", 9))
self.passthrough_senders = params.get("passthrough_senders", "").split()
self.passthrough_recipients = params.get("passthrough_recipients", "").split()
self.www_folder = params.get("www_folder", "")
self.filtermail_smtp_port = int(params.get("filtermail_smtp_port", "10080"))
self.username_min_length = int(params.pop("username_min_length", 9))
self.username_max_length = int(params.pop("username_max_length", 9))
self.password_min_length = int(params.pop("password_min_length", 9))
self.www_folder = params.pop("www_folder", "")
self.filtermail_smtp_port = int(params.pop("filtermail_smtp_port", "10080"))
self.filtermail_smtp_port_incoming = int(
params.get("filtermail_smtp_port_incoming", "10081")
params.pop("filtermail_smtp_port_incoming", "10081")
)
self.filtermail_http_port_incoming = int(
params.get("filtermail_http_port_incoming", "10082")
params.pop("filtermail_http_port_incoming", "10082")
)
self.filtermail_lmtp_port_transport = int(
params.get("filtermail_lmtp_port_transport", "10083")
params.pop("filtermail_lmtp_port_transport", "10083")
)
self.postfix_reinject_port = int(params.get("postfix_reinject_port", "10025"))
self.postfix_reinject_port = int(params.pop("postfix_reinject_port", "10025"))
self.postfix_reinject_port_incoming = int(
params.get("postfix_reinject_port_incoming", "10026")
params.pop("postfix_reinject_port_incoming", "10026")
)
self.mtail_address = params.get("mtail_address")
self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
self.acme_email = params.get("acme_email", "")
self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
self.imap_compress = params.get("imap_compress", "false").lower() == "true"
self.turn_socket_path = params.get(
self.mtail_address = params.pop("mtail_address", None)
self.disable_ipv6 = params.pop("disable_ipv6", "false").lower() == "true"
self.acme_email = params.pop("acme_email", "")
self.imap_rawlog = params.pop("imap_rawlog", "false").lower() == "true"
self.imap_compress = params.pop("imap_compress", "false").lower() == "true"
self.turn_socket_path = params.pop(
"turn_socket_path", "/run/chatmail-turn/turn.socket"
)
if "iroh_relay" not in params:
iroh_relay = params.pop("iroh_relay", None)
if iroh_relay is None:
self.iroh_relay = "https://" + raw_domain
self.enable_iroh_relay = True
else:
self.iroh_relay = params["iroh_relay"].strip()
self.iroh_relay = iroh_relay.strip()
self.enable_iroh_relay = False
self.privacy_postal = params.get("privacy_postal")
self.privacy_mail = params.get("privacy_mail")
self.privacy_pdo = params.get("privacy_pdo")
self.privacy_supervisor = params.get("privacy_supervisor")
self.privacy_postal = params.pop("privacy_postal", None)
self.privacy_mail = params.pop("privacy_mail", None)
self.privacy_pdo = params.pop("privacy_pdo", None)
self.privacy_supervisor = params.pop("privacy_supervisor", None)
# TLS certificate management.
# If tls_external_cert_and_key is set, use externally managed certs.
# Otherwise derived from the domain name:
# - Domains starting with "_" use self-signed certificates
# - All other domains use ACME.
external = params.get("tls_external_cert_and_key", "").strip()
external = params.pop("tls_external_cert_and_key", "").strip()
if external:
parts = external.split()
@@ -103,11 +103,12 @@ class Config:
self.tls_key_path = f"/var/lib/acme/live/{raw_domain}/privkey"
# deprecated option
mbdir = params.get("mailboxes_dir", f"/home/vmail/mail/{raw_domain}")
mbdir = params.pop("mailboxes_dir", f"/home/vmail/mail/{raw_domain}")
self.mailboxes_dir = Path(mbdir.strip())
# old unused option (except for first migration from sqlite to maildir store)
self.passdb_path = Path(params.get("passdb_path", "/home/vmail/passdb.sqlite"))
self.passdb_path = Path(params.pop("passdb_path", "/home/vmail/passdb.sqlite"))
self._unused_keys = list(params)
@property
def max_mailbox_size_mb(self):

View File

@@ -42,13 +42,6 @@ mail_domain = {mail_domain}
# minimum length a password must have
#password_min_length = 9
# list of chatmail addresses which can send outbound un-encrypted mail
#passthrough_senders =
# list of e-mail recipients for which to accept outbound un-encrypted mails
# (space-separated, item may start with "@" to whitelist whole recipient domains)
#passthrough_recipients =
# Use externally managed TLS certificates instead of built-in acmetool.
# Paths refer to files on the deployment server (not the build machine).
# Both files must already exist before running cmdeploy.

View File

@@ -45,8 +45,12 @@ def test_read_config_basic_using_defaults(tmp_path, maildomain):
assert example_config.username_min_length == 9
assert example_config.username_max_length == 9
assert example_config.password_min_length == 9
assert example_config.passthrough_recipients == []
assert example_config.passthrough_senders == []
assert example_config._unused_keys == []
def test_config_unused_keys(make_config):
config = make_config("chat.example.org", {"passthrough_senders": "x@y.org"})
assert config._unused_keys == ["passthrough_senders"]
def test_config_userstate_paths(make_config, tmp_path):

View File

@@ -84,6 +84,15 @@ def run_cmd_options(parser):
add_ssh_host_option(parser)
def _warn_unused_settings(unused_keys, out):
if unused_keys:
names = ", ".join(unused_keys)
out.red(
f"WARNING: chatmail.ini contains settings that have no effect: {names}\n"
"Please remove them from chatmail.ini."
)
def run_cmd(args, out):
"""Deploy chatmail services on the remote server."""
@@ -125,6 +134,7 @@ def run_cmd(args, out):
out.green("Deploy completed.")
else:
out.green("Deploy completed, call `cmdeploy dns` next.")
_warn_unused_settings(args.config._unused_keys, out)
return 0
except subprocess.CalledProcessError:
out.red("Deploy failed")

View File

@@ -6,6 +6,7 @@ ExecStart={{ bin_path }} {{ config_path }} transport
Restart=always
RestartSec=30
User=vmail
LimitNOFILE=524288
[Install]
WantedBy=multi-user.target

View File

@@ -1,5 +1,3 @@
/^From:/ DUNNO
/^Message-Id:/ DUNNO
/^Chat-Is-Post-Message:/ DUNNO
/^Content-Type:/ DUNNO
/.*/ IGNORE
/^DKIM-Signature:/ IGNORE
/^Authentication-Results:/ IGNORE
/^Received:/ IGNORE

View File

@@ -53,7 +53,8 @@ smtpd_tls_exclude_ciphers = aNULL, RC4, MD5, DES
# See <https://www.postfix.org/FORWARD_SECRECY_README.html#server_fs>.
tls_preempt_cipherlist = yes
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
# Reject by default, override per smtpd in master.cf
smtpd_relay_restrictions = reject
myhostname = {{ config.postfix_myhostname }}
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
@@ -101,9 +102,24 @@ smtpd_peername_lookup = no
# so instead this is handled in filtermail.
# We use LMTP instead SMTP so we can communicate per-recipient errors back to postfix.
default_transport = lmtp-filtermail:inet:[127.0.0.1]:{{ config.filtermail_lmtp_port_transport }}
# All deliveries over lmtp-filtermail are treated
# as having the same destination [127.0.0.1],
# so it is not possible to limit per-destination concurrency here,
# it is a job for filtermail-transport.
# Total number of parallel deliveries is limited
# by "maxproc" column in /etc/postfix/master.cf for lmtp-filtermail.
# Settings below are to prevent Postfix queue manager
# from limiting the number of LMTP connections to filtermail-transport.
# Read <https://www.postfix.org/TUNING_README.html#rope> and
# <https://www.postfix.org/SCHEDULER_README.html> for the details
# of the Postfix algorithm that we effectively disable here.
lmtp-filtermail_initial_destination_concurrency=10000
lmtp-filtermail_destination_concurrency_limit=10000
# Do not try to deliver messages for more than 2 days.
maximal_queue_lifetime = 2d
{% if not config.ipv4_relay %}
# DKIM-sign locally generated mail (bounces, DSNs).
# These bypass smtpd, so they need explicit milter configuration.

View File

@@ -17,6 +17,7 @@ smtp inet n - y - - smtpd
-o smtpd_tls_security_level=encrypt
-o smtpd_tls_mandatory_protocols=>=TLSv1.2
-o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port_incoming }}
-o smtpd_relay_restrictions=reject_unauth_destination
submission inet n - y - 5000 smtpd
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
@@ -81,12 +82,14 @@ filter unix - n n - - lmtp
-o syslog_name=postfix/reinject
-o milter_macro_daemon_name=ORIGINATING
-o cleanup_service_name=authclean
-o smtpd_relay_restrictions=permit_mynetworks,reject
{% if not config.ipv4_relay %} -o smtpd_milters=unix:opendkim/opendkim.sock
{% endif %}
# Local SMTP server for reinjecting incoming filtered mail
127.0.0.1:{{ config.postfix_reinject_port_incoming }} inet n - n - 100 smtpd
-o syslog_name=postfix/reinject_incoming
-o smtpd_relay_restrictions=reject_unauth_destination
# Cleanup `Received` headers for authenticated mail
# to avoid leaking client IP.
@@ -102,7 +105,15 @@ filter unix - n n - - lmtp
authclean unix n - - - 0 cleanup
-o header_checks=regexp:/etc/postfix/submission_header_cleanup
lmtp-filtermail unix - - y - 10000 lmtp
# Reducing `maxproc` here may result in a head of line blocking
# when there are many messages sent to unreachable destinations
# at the same time.
# LMTP clients here talk to filtermail-transport.
# LMTP has no pipelining,
# so while filtermail-transport tries to deliver the message,
# possibly waiting for a long connection timeout
# or talking to a slow server, LMTP client cannot be reused.
lmtp-filtermail unix - - y - 500 lmtp
-o syslog_name=postfix/lmtp-filtermail
-o lmtp_header_checks=
-o lmtp_tls_security_level=none

View File

@@ -232,6 +232,26 @@ def try_n_times(n, f):
return f()
def test_rewrite_subject(cmsetup, maildata):
"""Test that subject gets replaced with [...]."""
user1, user2 = cmsetup.gen_users(2)
sent_msg = maildata(
"encrypted.eml",
from_addr=user1.addr,
to_addr=user2.addr,
subject="Unencrypted subject",
).as_string()
user1.smtp.sendmail(from_addr=user1.addr, to_addrs=[user2.addr], msg=sent_msg)
# The message may need some time to get delivered by postfix.
messages = try_n_times(5, user2.imap.fetch_all_messages)
assert len(messages) == 1
rcvd_msg = messages[0]
assert "Subject: [...]" not in sent_msg
assert "Subject: [...]" in rcvd_msg
assert "Subject: Unencrypted subject" in sent_msg
assert "Subject: Unencrypted subject" not in rcvd_msg
def test_exceed_rate_limit(cmsetup, gencreds, maildata, chatmail_config):

View File

@@ -60,6 +60,7 @@ and run the following commands:
::
git pull origin main --rebase --autostash
scripts/initenv.sh
scripts/cmdeploy run
If you don't want the latest development version,