Compare commits

..

5 Commits

Author SHA1 Message Date
missytake
39d1ecaa03 chore(release): prepare for 1.11.0 2026-05-15 17:13:58 +02:00
holger krekel
a266ffd060 fix: fix #972 by increasing file descriptors for filtermail 2026-05-14 22:40:25 +02:00
holger krekel
a47bb94143 feat: warn about any unused chatmail.ini parameter at the end of "cmdeploy run" 2026-05-14 20:58:47 +02:00
holger krekel
43ae9fee5c feat!: ignore passthrough_sender and passthrough_recipients to eliminate one more source of unencrypted messages
When running "cmdeploy run" operators will see a warning if their chatmail.ini contains these unused options.
2026-05-14 20:58:47 +02:00
holger krekel
42dc781d7d feat: make turn_socket_path configurable, and cleanup tests and turnserver code.
this is originally motivated by https://github.com/chatmail/relay/pull/840
2026-05-13 21:02:28 +02:00
14 changed files with 235 additions and 177 deletions

View File

@@ -1,5 +1,47 @@
# Changelog for chatmail deployment
## [1.11.0] - 2026-05-15
### Breaking Changes
- [**breaking**] Drop passthrough_sender and passthrough_recipients chatmail.ini options to eliminate one more source of unencrypted messages
### Features
- Use filtermail for delivery to remote MTAs
- Expose metadata "maxsmtprecipients" value
- Support setup without domain, with only an IPv4 address (#963)
- *(doc/docker)* Introduce docker images in documentation
- DKIM-sign bounce messages (mainly "user does not exist")
- *(config)* Load default values from Config(), not chatmail.ini.f (#853)
- Make turn_socket_path configurable, and cleanup tests and turnserver code.
- Warn about any unused chatmail.ini parameter at the end of "cmdeploy run"
### Bug Fixes
- Make www tests work with editable instead of just plain installs
- Use path with no leading slash for mxdeliv
- Increase filtermail-transport concurrency limit
- Fix #972 by increasing file descriptors for filtermail
- *(mtail)* Correct boot ordering and deploy restart logic
- *(cmdeploy)* Stop and disable unbound-resolvconf
- *(nginx)* Properly redirect www to mail_domain
- *(dns)* Query correct NS if MNAME server is hidden (#954)
- Legacy token metadata storage used list type, but if no new setmetadata happened, the user would not be notified at all.
- *(logging)* Log all http requests to syslog
### Documentation
- Document how to upgrade to new version (#965)
### Other
- *(deps)* Upgrade to filtermail v0.6.4
### Refactor
- Introduce automated change-tracking across deployers
## 1.10.0 2026-04-30
* start mtail after networking is fully up <https://github.com/chatmail/relay/pull/942>

View File

@@ -26,7 +26,6 @@ chatmail-expire = "chatmaild.expire:daily_expire_main"
chatmail-quota-expire = "chatmaild.expire:quota_expire_main"
chatmail-fsreport = "chatmaild.fsreport:main"
lastlogin = "chatmaild.lastlogin:main"
turnserver = "chatmaild.turnserver:main"
[project.entry-points.pytest11]
"chatmaild.testplugin" = "chatmaild.tests.plugin"

View File

@@ -16,9 +16,9 @@ def read_config(inipath):
class Config:
def __init__(self, inipath, params):
self._inipath = inipath
raw_domain = params["mail_domain"]
params = dict(params)
raw_domain = params.pop("mail_domain")
self.mail_domain_bare = raw_domain
self.ssh_host = params.get("ssh_host", raw_domain)
if is_valid_ipv4(raw_domain):
self.ipv4_relay = raw_domain
@@ -30,57 +30,59 @@ class Config:
self.mail_domain = raw_domain
self.postfix_myhostname = raw_domain
self.max_user_send_per_minute = int(params.get("max_user_send_per_minute", 60))
self.max_user_send_burst_size = int(params.get("max_user_send_burst_size", 10))
self.max_mailbox_size = params.get("max_mailbox_size", "500M")
self.max_message_size = int(params.get("max_message_size", 31457280))
self.delete_mails_after = params.get("delete_mails_after", "20")
self.delete_large_after = params.get("delete_large_after", "7")
self.max_user_send_per_minute = int(params.pop("max_user_send_per_minute", 60))
self.max_user_send_burst_size = int(params.pop("max_user_send_burst_size", 10))
self.max_mailbox_size = params.pop("max_mailbox_size", "500M")
self.max_message_size = int(params.pop("max_message_size", 31457280))
self.delete_mails_after = params.pop("delete_mails_after", "20")
self.delete_large_after = params.pop("delete_large_after", "7")
self.delete_inactive_users_after = int(
params.get("delete_inactive_users_after", 90)
params.pop("delete_inactive_users_after", 90)
)
self.username_min_length = int(params.get("username_min_length", 9))
self.username_max_length = int(params.get("username_max_length", 9))
self.password_min_length = int(params.get("password_min_length", 9))
self.passthrough_senders = params.get("passthrough_senders", "").split()
self.passthrough_recipients = params.get("passthrough_recipients", "").split()
self.www_folder = params.get("www_folder", "")
self.filtermail_smtp_port = int(params.get("filtermail_smtp_port", "10080"))
self.username_min_length = int(params.pop("username_min_length", 9))
self.username_max_length = int(params.pop("username_max_length", 9))
self.password_min_length = int(params.pop("password_min_length", 9))
self.www_folder = params.pop("www_folder", "")
self.filtermail_smtp_port = int(params.pop("filtermail_smtp_port", "10080"))
self.filtermail_smtp_port_incoming = int(
params.get("filtermail_smtp_port_incoming", "10081")
params.pop("filtermail_smtp_port_incoming", "10081")
)
self.filtermail_http_port_incoming = int(
params.get("filtermail_http_port_incoming", "10082")
params.pop("filtermail_http_port_incoming", "10082")
)
self.filtermail_lmtp_port_transport = int(
params.get("filtermail_lmtp_port_transport", "10083")
params.pop("filtermail_lmtp_port_transport", "10083")
)
self.postfix_reinject_port = int(params.get("postfix_reinject_port", "10025"))
self.postfix_reinject_port = int(params.pop("postfix_reinject_port", "10025"))
self.postfix_reinject_port_incoming = int(
params.get("postfix_reinject_port_incoming", "10026")
params.pop("postfix_reinject_port_incoming", "10026")
)
self.mtail_address = params.get("mtail_address")
self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
self.acme_email = params.get("acme_email", "")
self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
self.imap_compress = params.get("imap_compress", "false").lower() == "true"
if "iroh_relay" not in params:
self.mtail_address = params.pop("mtail_address", None)
self.disable_ipv6 = params.pop("disable_ipv6", "false").lower() == "true"
self.acme_email = params.pop("acme_email", "")
self.imap_rawlog = params.pop("imap_rawlog", "false").lower() == "true"
self.imap_compress = params.pop("imap_compress", "false").lower() == "true"
self.turn_socket_path = params.pop(
"turn_socket_path", "/run/chatmail-turn/turn.socket"
)
iroh_relay = params.pop("iroh_relay", None)
if iroh_relay is None:
self.iroh_relay = "https://" + raw_domain
self.enable_iroh_relay = True
else:
self.iroh_relay = params["iroh_relay"].strip()
self.iroh_relay = iroh_relay.strip()
self.enable_iroh_relay = False
self.privacy_postal = params.get("privacy_postal")
self.privacy_mail = params.get("privacy_mail")
self.privacy_pdo = params.get("privacy_pdo")
self.privacy_supervisor = params.get("privacy_supervisor")
self.privacy_postal = params.pop("privacy_postal", None)
self.privacy_mail = params.pop("privacy_mail", None)
self.privacy_pdo = params.pop("privacy_pdo", None)
self.privacy_supervisor = params.pop("privacy_supervisor", None)
# TLS certificate management.
# If tls_external_cert_and_key is set, use externally managed certs.
# Otherwise derived from the domain name:
# - Domains starting with "_" use self-signed certificates
# - All other domains use ACME.
external = params.get("tls_external_cert_and_key", "").strip()
external = params.pop("tls_external_cert_and_key", "").strip()
if external:
parts = external.split()
@@ -101,11 +103,12 @@ class Config:
self.tls_key_path = f"/var/lib/acme/live/{raw_domain}/privkey"
# deprecated option
mbdir = params.get("mailboxes_dir", f"/home/vmail/mail/{raw_domain}")
mbdir = params.pop("mailboxes_dir", f"/home/vmail/mail/{raw_domain}")
self.mailboxes_dir = Path(mbdir.strip())
# old unused option (except for first migration from sqlite to maildir store)
self.passdb_path = Path(params.get("passdb_path", "/home/vmail/passdb.sqlite"))
self.passdb_path = Path(params.pop("passdb_path", "/home/vmail/passdb.sqlite"))
self._unused_keys = list(params)
@property
def max_mailbox_size_mb(self):

View File

@@ -3,9 +3,6 @@
# mail domain (MUST be set to fully qualified chat mail domain)
mail_domain = {mail_domain}
# Where to deploy the relay - if unspecified, mail_domain will be used.
ssh_host = localhost
#
# If you only do private test deploys, you don't need to modify any settings below
#
@@ -45,13 +42,6 @@ ssh_host = localhost
# minimum length a password must have
#password_min_length = 9
# list of chatmail addresses which can send outbound un-encrypted mail
#passthrough_senders =
# list of e-mail recipients for which to accept outbound un-encrypted mails
# (space-separated, item may start with "@" to whitelist whole recipient domains)
#passthrough_recipients =
# Use externally managed TLS certificates instead of built-in acmetool.
# Paths refer to files on the deployment server (not the build machine).
# Both files must already exist before running cmdeploy.

View File

@@ -1,4 +1,5 @@
import logging
import socket
import sys
import time
from contextlib import contextmanager
@@ -7,7 +8,14 @@ from .config import read_config
from .dictproxy import DictProxy
from .filedict import FileDict
from .notifier import Notifier
from .turnserver import turn_credentials
def turn_credentials(turn_socket_path):
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client_socket:
client_socket.settimeout(5)
client_socket.connect(turn_socket_path)
with client_socket.makefile("rb") as file:
return file.readline().decode("utf-8").strip()
def _is_valid_token_timestamp(timestamp, now):
@@ -79,12 +87,20 @@ class Metadata:
class MetadataDictProxy(DictProxy):
def __init__(self, notifier, metadata, iroh_relay=None, turn_hostname=None):
def __init__(
self,
notifier,
metadata,
iroh_relay=None,
turn_hostname=None,
turn_socket_path=None,
):
super().__init__()
self.notifier = notifier
self.metadata = metadata
self.iroh_relay = iroh_relay
self.turn_hostname = turn_hostname
self.turn_socket_path = turn_socket_path
def handle_lookup(self, parts):
# Lpriv/43f5f508a7ea0366dff30200c15250e3/devicetoken\tlkj123poi@c2.testrun.org
@@ -101,7 +117,7 @@ class MetadataDictProxy(DictProxy):
return f"O{self.iroh_relay}\n"
case "turn":
try:
res = turn_credentials()
res = turn_credentials(self.turn_socket_path)
except Exception:
logging.exception("failed to get TURN credentials")
return "N\n"
@@ -135,6 +151,7 @@ def main():
config = read_config(config_path)
iroh_relay = config.iroh_relay
mail_domain = config.mail_domain
socket_path = config.turn_socket_path
vmail_dir = config.mailboxes_dir
if not vmail_dir.exists():
@@ -152,6 +169,7 @@ def main():
metadata=metadata,
iroh_relay=iroh_relay,
turn_hostname=mail_domain,
turn_socket_path=socket_path,
)
dictproxy.serve_forever_from_socket(socket)

View File

@@ -45,8 +45,12 @@ def test_read_config_basic_using_defaults(tmp_path, maildomain):
assert example_config.username_min_length == 9
assert example_config.username_max_length == 9
assert example_config.password_min_length == 9
assert example_config.passthrough_recipients == []
assert example_config.passthrough_senders == []
assert example_config._unused_keys == []
def test_config_unused_keys(make_config):
config = make_config("chat.example.org", {"passthrough_senders": "x@y.org"})
assert config._unused_keys == ["passthrough_senders"]
def test_config_userstate_paths(make_config, tmp_path):

View File

@@ -324,7 +324,7 @@ def test_turn_credentials_exception_returns_N(notifier, metadata, monkeypatch):
turn_hostname="turn.example.org",
)
def mock_turn_credentials():
def mock_turn_credentials(turn_socket_path):
raise ConnectionRefusedError("socket not available")
monkeypatch.setattr(chatmaild.metadata, "turn_credentials", mock_turn_credentials)
@@ -348,7 +348,9 @@ def test_turn_credentials_success(notifier, metadata, monkeypatch):
turn_hostname="turn.example.org",
)
monkeypatch.setattr(chatmaild.metadata, "turn_credentials", lambda: "user:pass")
monkeypatch.setattr(
chatmaild.metadata, "turn_credentials", lambda path: "user:pass"
)
transactions = {}
res = dictproxy.handle_dovecot_request(

View File

@@ -0,0 +1,46 @@
import socket
import threading
import pytest
from chatmaild.metadata import turn_credentials
@pytest.fixture
def turn_socket(tmp_path):
sock_path = str(tmp_path / "turn.socket")
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
server.bind(sock_path)
server.listen(1)
yield sock_path, server
server.close()
def test_turn_credentials_timeout(turn_socket):
sock_path, server = turn_socket
with pytest.raises(socket.timeout):
# Inside turn_credentials the kernel listen backlog (1)
# completes connect() without accept()
# so the client blocks on readline() until the 5s timeout fires.
turn_credentials(sock_path)
def test_turn_credentials_connection_refused_on_not_existing_socket(tmp_path):
missing = str(tmp_path / "nonexistent.socket")
with pytest.raises((ConnectionRefusedError, FileNotFoundError)):
turn_credentials(missing)
def test_turn_credentials_socket_success(turn_socket):
sock_path, server = turn_socket
def respond():
conn, _ = server.accept()
conn.sendall(b"testuser:testpass\n")
conn.close()
t = threading.Thread(target=respond, daemon=True)
t.start()
result = turn_credentials(sock_path)
assert result == "testuser:testpass"

View File

@@ -1,73 +0,0 @@
import socket
import threading
import time
from unittest.mock import patch
import pytest
from chatmaild.turnserver import turn_credentials
SOCKET_PATH = "/run/chatmail-turn/turn.socket"
@pytest.fixture
def turn_socket(tmp_path):
"""Create a real Unix socket server at a temp path."""
sock_path = str(tmp_path / "turn.socket")
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
server.bind(sock_path)
server.listen(1)
yield sock_path, server
server.close()
def _call_turn_credentials(sock_path):
"""Call turn_credentials but connect to sock_path instead of hardcoded path."""
original_connect = socket.socket.connect
def patched_connect(self, address):
if address == SOCKET_PATH:
address = sock_path
return original_connect(self, address)
with patch.object(socket.socket, "connect", patched_connect):
return turn_credentials()
def test_turn_credentials_timeout(turn_socket):
"""Server accepts but never responds — must raise socket.timeout."""
sock_path, server = turn_socket
def accept_and_hang():
conn, _ = server.accept()
time.sleep(30)
conn.close()
t = threading.Thread(target=accept_and_hang, daemon=True)
t.start()
with pytest.raises(socket.timeout):
_call_turn_credentials(sock_path)
def test_turn_credentials_connection_refused(tmp_path):
"""Socket file doesn't exist — must raise ConnectionRefusedError or FileNotFoundError."""
missing = str(tmp_path / "nonexistent.socket")
with pytest.raises((ConnectionRefusedError, FileNotFoundError)):
_call_turn_credentials(missing)
def test_turn_credentials_success(turn_socket):
"""Server responds with credentials — must return stripped string."""
sock_path, server = turn_socket
def respond():
conn, _ = server.accept()
conn.sendall(b"testuser:testpass\n")
conn.close()
t = threading.Thread(target=respond, daemon=True)
t.start()
result = _call_turn_credentials(sock_path)
assert result == "testuser:testpass"

View File

@@ -1,10 +0,0 @@
#!/usr/bin/env python3
import socket
def turn_credentials() -> str:
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client_socket:
client_socket.settimeout(5)
client_socket.connect("/run/chatmail-turn/turn.socket")
with client_socket.makefile("rb") as file:
return file.readline().decode("utf-8").strip()

View File

@@ -84,10 +84,19 @@ def run_cmd_options(parser):
add_ssh_host_option(parser)
def _warn_unused_settings(unused_keys, out):
if unused_keys:
names = ", ".join(unused_keys)
out.red(
f"WARNING: chatmail.ini contains settings that have no effect: {names}\n"
"Please remove them from chatmail.ini."
)
def run_cmd(args, out):
"""Deploy chatmail services on the remote server."""
ssh_host = args.ssh_host if args.ssh_host else args.config.ssh_host
ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain_bare
sshexec = get_sshexec(ssh_host)
require_iroh = args.config.enable_iroh_relay
strict_tls = args.config.tls_cert_mode == "acme"
@@ -107,7 +116,7 @@ def run_cmd(args, out):
pyinf = "pyinfra --dry" if args.dry_run else "pyinfra"
cmd = f"{pyinf} --ssh-user root {ssh_host} {deploy_path} -y"
if ssh_host in ["localhost", "@local"]:
if ssh_host == "localhost":
cmd = f"{pyinf} @local {deploy_path} -y"
if version.parse(pyinfra.__version__) < version.parse("3"):
@@ -125,6 +134,7 @@ def run_cmd(args, out):
out.green("Deploy completed.")
else:
out.green("Deploy completed, call `cmdeploy dns` next.")
_warn_unused_settings(args.config._unused_keys, out)
return 0
except subprocess.CalledProcessError:
out.red("Deploy failed")
@@ -148,7 +158,7 @@ def dns_cmd(args, out):
ipv4 = args.config.ipv4_relay
print(f"[WARNING] {ipv4} is not a domain, skipping DNS checks.")
return 0
ssh_host = args.ssh_host if args.ssh_host else args.config.ssh_host
ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain
sshexec = get_sshexec(ssh_host, verbose=args.verbose)
tls_cert_mode = args.config.tls_cert_mode
strict_tls = tls_cert_mode == "acme"
@@ -185,7 +195,7 @@ def status_cmd_options(parser):
def status_cmd(args, out):
"""Display status for online chatmail instance."""
ssh_host = args.ssh_host if args.ssh_host else args.config.ssh_host
ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain_bare
sshexec = get_sshexec(ssh_host, verbose=args.verbose)
out.green(f"chatmail domain: {args.config.mail_domain}")

View File

@@ -6,6 +6,7 @@ ExecStart={{ bin_path }} {{ config_path }} transport
Restart=always
RestartSec=30
User=vmail
LimitNOFILE=524288
[Install]
WantedBy=multi-user.target

View File

@@ -62,8 +62,8 @@ def maildomain(chatmail_config):
@pytest.fixture(scope="session")
def sshdomain(chatmail_config):
return os.environ.get("CHATMAIL_SSH", chatmail_config.ssh_host)
def sshdomain(maildomain):
return os.environ.get("CHATMAIL_SSH", maildomain)
@pytest.fixture

View File

@@ -14,14 +14,21 @@ Minimal requirements and prerequisites
You will need the following:
- Control over a domain through a DNS provider of your choice.
(there is experimental support for :ref:`IP-only relays <iponly>`).
- A Debian 12 server with reachable SMTP/SUBMISSIONS/IMAPS/HTTPS ports.
- A Debian 12 **deployment server** with reachable SMTP/SUBMISSIONS/IMAPS/HTTPS ports.
IPv6 is encouraged if available. Chatmail relay servers only require
1GB RAM, one CPU, and perhaps 10GB storage for a few thousand active
chatmail addresses.
- A Linux or Unix **build machine** with key-based SSH access to the root
user of the deployment server.
You must add a passphrase-protected private key to your local ssh-agent because you
cant type in your passphrase during deployment.
(An ed25519 private key is required due to an `upstream bug in
paramiko <https://github.com/paramiko/paramiko/issues/2191>`_)
- Control over a domain through a DNS provider of your choice
(there is experimental support for :ref:`IP-only relays <iponly>`).
.. _setup:
@@ -31,7 +38,7 @@ Setup with ``scripts/cmdeploy``
We use ``chat.example.org`` as the chatmail domain in the following
steps. Please substitute it with your own domain.
1. Setup the initial DNS records for your relay.
1. Setup the initial DNS records for your deployment server.
The following is an example in the
familiar BIND zone file format with a TTL of 1 hour (3600 seconds).
Please substitute your domain and IP addresses.
@@ -51,25 +58,22 @@ steps. Please substitute it with your own domain.
The ``mta-sts`` CNAME and ``_mta-sts`` TXT records
are not needed for such domains.
2. Login to the server with SSH, clone the repository and bootstrap the Python
2. On your local PC, clone the repository and bootstrap the Python
virtualenv.
::
ssh root@chat.example.org
git clone https://github.com/chatmail/relay
cd relay
scripts/initenv.sh
3. Then, create a chatmail configuration file
3. On your local build machine (PC), create a chatmail configuration file
``chatmail.ini``:
::
scripts/cmdeploy init chat.example.org # <-- use your domain
.. note::
To use self-signed TLS certificates
instead of Let's Encrypt,
use a domain name starting with ``_``
@@ -80,7 +84,13 @@ steps. Please substitute it with your own domain.
See the :doc:`overview`
for details on certificate provisioning.
4. Now run the deployment script to install the relay to the server:
4. Verify that SSH root login to the deployment server server works:
::
ssh root@chat.example.org # <-- use your domain
5. From your local build machine, setup and configure the remote deployment server:
::
@@ -92,6 +102,7 @@ steps. Please substitute it with your own domain.
public).
Docker installation
-------------------
@@ -99,32 +110,26 @@ There is experimental support for running chatmail via Docker.
A monolithic image based on the above cmdeploy method is available `through a separate repository <https://github.com/chatmail/docker/pkgs/container/docker>`_.
See the `chatmail/docker README <https://github.com/chatmail/docker>`_ for full setup instructions.
Next Steps
----------
Now you should display and check all recommended DNS records
to enable federation with other relays:
::
scripts/cmdeploy dns
You should also test whether your chatmail service is working correctly:
::
scripts/cmdeploy test
Other Helpful Commands
Other helpful commands
----------------------
To check the status of your chatmail relay:
To check the status of your deployment server running the chatmail service:
::
scripts/cmdeploy status
To display and check all recommended DNS records:
::
scripts/cmdeploy dns
To test whether your chatmail service is working correctly:
::
scripts/cmdeploy test
To measure the performance of your chatmail service:
@@ -166,9 +171,8 @@ This starts a local live development cycle for chatmail web pages:
directory and generating HTML files and copying assets to the
``www/build`` directory.
- if you are running scripts/cmdeploy webdev on the relay itself,
you need to configure a route in /etc/nginx/nginx.conf
to expose the build directory.
- Starts a browser window automatically where you can “refresh” as
needed.
Custom web pages
----------------
@@ -186,7 +190,7 @@ Disable automatic address creation
--------------------------------------------------------
If you need to stop address creation, e.g. because some script is wildly
creating addresses, login with ssh to the relay and run:
creating addresses, login with ssh to the deployment machine and run:
::
@@ -242,3 +246,25 @@ The deploy will verify that both files exist on the server.
If you use such a setup, you must trigger the reload explicitly after renewal::
systemctl start tls-cert-reload.service
Migrating to a new build machine
----------------------------------
To move or add a build machine,
clone the relay repository on the new build machine, and copy the ``chatmail.ini`` file from the old build machine.
Make sure ``rsync`` is installed, then initialize the environment:
::
./scripts/initenv.sh
Run safety checks before a new deployment:
::
./scripts/cmdeploy dns
./scripts/cmdeploy status
If you keep multiple build machines (ie laptop and desktop), keep ``chatmail.ini`` in sync between
them.