Compare commits

..

1 Commits

Author SHA1 Message Date
link2xt f896ce6c75 feat: add tool to analyze deferred queue
It prints all destinations with the number of recipients
and all the reasons. Operator can then try
to fix the problems for destinations,
e.g. by manually adding reverse proxy
addresses to /etc/hosts for failing domains
or routing IP addresses to another interface.
2026-05-12 19:24:20 +02:00
42 changed files with 418 additions and 739 deletions
-40
View File
@@ -1,40 +0,0 @@
name: No-DNS
on:
# Triggers when a PR is merged into main or a direct push occurs
push:
branches: [ "main" ]
# Triggers for any PR (and its subsequent commits) targeting the main branch
pull_request:
branches: [ "main" ]
permissions: {}
# Newest push wins: Prevents multiple runs from clashing and wasting runner efforts
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
no-dns:
name: LXC deploy and test
uses: chatmail/cmlxc/.github/workflows/lxc-test.yml@main
with:
cmlxc_version: main
cmlxc_commands: |
cmlxc init
# single cmdeploy relay test
cmlxc -v deploy-cmdeploy --source ./repo --type ipv4 cm0
cmlxc -v test-cmdeploy cm0
# cross cmdeploy relay test (two ipv4 relays)
cmlxc -v deploy-cmdeploy --source ./repo --ipv4-only --type ipv4 cm1
cmlxc -v test-cmdeploy cm0 cm1
# cross cmdeploy/madmail relay tests
cmlxc -v deploy-madmail mad0
cmlxc -v test-cmdeploy cm0 mad0
cmlxc -v test-mini mad0 cm0
cmlxc -v test-mini cm0 mad0
+4 -5
View File
@@ -1,4 +1,4 @@
name: CI
name: Run unit-tests and container-based deploy+test verification
on:
# Triggers when a PR is merged into main or a direct push occurs
@@ -29,7 +29,7 @@ jobs:
ref: ${{ github.event.pull_request.head.sha }}
persist-credentials: false
- name: download filtermail
run: curl -L https://github.com/chatmail/filtermail/releases/download/v0.7.0/filtermail-x86_64 -o /usr/local/bin/filtermail && chmod +x /usr/local/bin/filtermail
run: curl -L https://github.com/chatmail/filtermail/releases/download/v0.6.4/filtermail-x86_64 -o /usr/local/bin/filtermail && chmod +x /usr/local/bin/filtermail
- name: run chatmaild tests
working-directory: chatmaild
run: pipx run tox
@@ -57,9 +57,9 @@ jobs:
lxc-test:
name: LXC deploy and test
uses: chatmail/cmlxc/.github/workflows/lxc-test.yml@main
uses: chatmail/cmlxc/.github/workflows/lxc-test.yml@v0.13.5
with:
cmlxc_version: main
cmlxc_version: v0.13.5
cmlxc_commands: |
cmlxc init
# single cmdeploy relay test
@@ -76,4 +76,3 @@ jobs:
cmlxc -v test-cmdeploy cm0 mad0
cmlxc -v test-mini cm0 mad0
cmlxc -v test-mini mad0 cm0
-1
View File
@@ -9,7 +9,6 @@ name: Trigger Docker build
on:
push:
branches: [main]
tags: ['[0-9]+.[0-9]+.[0-9]+']
workflow_dispatch:
permissions: {}
+1 -43
View File
@@ -1,46 +1,4 @@
# Changelog for chatmail deployment
## [1.11.0] - 2026-05-15
### Breaking Changes
- [**breaking**] Drop passthrough_sender and passthrough_recipients chatmail.ini options to eliminate one more source of unencrypted messages
### Features
- Use filtermail for delivery to remote MTAs
- Expose metadata "maxsmtprecipients" value
- Support setup without domain, with only an IPv4 address (#963)
- *(doc/docker)* Introduce docker images in documentation
- DKIM-sign bounce messages (mainly "user does not exist")
- *(config)* Load default values from Config(), not chatmail.ini.f (#853)
- Make turn_socket_path configurable, and cleanup tests and turnserver code.
- Warn about any unused chatmail.ini parameter at the end of "cmdeploy run"
### Bug Fixes
- Make www tests work with editable instead of just plain installs
- Use path with no leading slash for mxdeliv
- Increase filtermail-transport concurrency limit
- Fix #972 by increasing file descriptors for filtermail
- *(mtail)* Correct boot ordering and deploy restart logic
- *(cmdeploy)* Stop and disable unbound-resolvconf
- *(nginx)* Properly redirect www to mail_domain
- *(dns)* Query correct NS if MNAME server is hidden (#954)
- Legacy token metadata storage used list type, but if no new setmetadata happened, the user would not be notified at all.
- *(logging)* Log all http requests to syslog
### Documentation
- Document how to upgrade to new version (#965)
### Other
- *(deps)* Upgrade to filtermail v0.6.4
### Refactor
- Introduce automated change-tracking across deployers
# Changelog for chatmail deployment
## 1.10.0 2026-04-30
+2
View File
@@ -24,7 +24,9 @@ chatmail-metadata = "chatmaild.metadata:main"
chatmail-expire = "chatmaild.expire:daily_expire_main"
chatmail-quota-expire = "chatmaild.expire:quota_expire_main"
chatmail-fsreport = "chatmaild.fsreport:main"
chatmail-deferred = "chatmaild.deferred:main"
lastlogin = "chatmaild.lastlogin:main"
turnserver = "chatmaild.turnserver:main"
[project.entry-points.pytest11]
"chatmaild.testplugin" = "chatmaild.tests.plugin"
+67 -65
View File
@@ -1,4 +1,3 @@
import ipaddress
from pathlib import Path
import iniconfig
@@ -9,78 +8,67 @@ from chatmaild.user import User
def read_config(inipath):
assert Path(inipath).exists(), inipath
cfg = iniconfig.IniConfig(inipath)
return Config(inipath, params=cfg.sections["params"])
params = cfg.sections["params"]
default_config_content = get_default_config_content(params["mail_domain"])
df_params = iniconfig.IniConfig("ini", data=default_config_content)["params"]
new_params = dict(df_params.items())
new_params.update(params)
return Config(inipath, params=new_params)
class Config:
def __init__(self, inipath, params):
self._inipath = inipath
params = dict(params)
raw_domain = params.pop("mail_domain")
self.mail_domain_bare = raw_domain
if is_valid_ipv4(raw_domain):
self.ipv4_relay = raw_domain
self.mail_domain = f"[{raw_domain}]"
self.postfix_myhostname = ipaddress.IPv4Address(raw_domain).reverse_pointer
else:
self.ipv4_relay = None
self.mail_domain = raw_domain
self.postfix_myhostname = raw_domain
self.max_user_send_per_minute = int(params.pop("max_user_send_per_minute", 60))
self.max_user_send_burst_size = int(params.pop("max_user_send_burst_size", 10))
self.max_mailbox_size = params.pop("max_mailbox_size", "500M")
self.max_message_size = int(params.pop("max_message_size", 31457280))
self.delete_mails_after = params.pop("delete_mails_after", "20")
self.delete_large_after = params.pop("delete_large_after", "7")
self.delete_inactive_users_after = int(
params.pop("delete_inactive_users_after", 90)
)
self.username_min_length = int(params.pop("username_min_length", 9))
self.username_max_length = int(params.pop("username_max_length", 9))
self.password_min_length = int(params.pop("password_min_length", 9))
self.www_folder = params.pop("www_folder", "")
self.filtermail_smtp_port = int(params.pop("filtermail_smtp_port", "10080"))
self.mail_domain = params["mail_domain"]
self.max_user_send_per_minute = int(params.get("max_user_send_per_minute", 60))
self.max_user_send_burst_size = int(params.get("max_user_send_burst_size", 10))
self.max_mailbox_size = params["max_mailbox_size"]
self.max_message_size = int(params.get("max_message_size", "31457280"))
self.delete_mails_after = params["delete_mails_after"]
self.delete_large_after = params["delete_large_after"]
self.delete_inactive_users_after = int(params["delete_inactive_users_after"])
self.username_min_length = int(params["username_min_length"])
self.username_max_length = int(params["username_max_length"])
self.password_min_length = int(params["password_min_length"])
self.passthrough_senders = params["passthrough_senders"].split()
self.passthrough_recipients = params["passthrough_recipients"].split()
self.www_folder = params.get("www_folder", "")
self.filtermail_smtp_port = int(params.get("filtermail_smtp_port", "10080"))
self.filtermail_smtp_port_incoming = int(
params.pop("filtermail_smtp_port_incoming", "10081")
params.get("filtermail_smtp_port_incoming", "10081")
)
self.filtermail_http_port_incoming = int(
params.pop("filtermail_http_port_incoming", "10082")
params.get("filtermail_http_port_incoming", "10082")
)
self.filtermail_lmtp_port_transport = int(
params.pop("filtermail_lmtp_port_transport", "10083")
params.get("filtermail_lmtp_port_transport", "10083")
)
self.postfix_reinject_port = int(params.pop("postfix_reinject_port", "10025"))
self.postfix_reinject_port = int(params.get("postfix_reinject_port", "10025"))
self.postfix_reinject_port_incoming = int(
params.pop("postfix_reinject_port_incoming", "10026")
params.get("postfix_reinject_port_incoming", "10026")
)
self.mtail_address = params.pop("mtail_address", None)
self.disable_ipv6 = params.pop("disable_ipv6", "false").lower() == "true"
self.acme_email = params.pop("acme_email", "")
self.imap_rawlog = params.pop("imap_rawlog", "false").lower() == "true"
self.imap_compress = params.pop("imap_compress", "false").lower() == "true"
self.turn_socket_path = params.pop(
"turn_socket_path", "/run/chatmail-turn/turn.socket"
)
iroh_relay = params.pop("iroh_relay", None)
if iroh_relay is None:
self.iroh_relay = "https://" + raw_domain
self.mtail_address = params.get("mtail_address")
self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
self.acme_email = params.get("acme_email", "")
self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
self.imap_compress = params.get("imap_compress", "false").lower() == "true"
if "iroh_relay" not in params:
self.iroh_relay = "https://" + params["mail_domain"]
self.enable_iroh_relay = True
else:
self.iroh_relay = iroh_relay.strip()
self.iroh_relay = params["iroh_relay"].strip()
self.enable_iroh_relay = False
self.privacy_postal = params.pop("privacy_postal", None)
self.privacy_mail = params.pop("privacy_mail", None)
self.privacy_pdo = params.pop("privacy_pdo", None)
self.privacy_supervisor = params.pop("privacy_supervisor", None)
self.privacy_postal = params.get("privacy_postal")
self.privacy_mail = params.get("privacy_mail")
self.privacy_pdo = params.get("privacy_pdo")
self.privacy_supervisor = params.get("privacy_supervisor")
# TLS certificate management.
# If tls_external_cert_and_key is set, use externally managed certs.
# Otherwise derived from the domain name:
# - Domains starting with "_" use self-signed certificates
# - All other domains use ACME.
external = params.pop("tls_external_cert_and_key", "").strip()
external = params.get("tls_external_cert_and_key", "").strip()
if external:
parts = external.split()
@@ -91,22 +79,21 @@ class Config:
)
self.tls_cert_mode = "external"
self.tls_cert_path, self.tls_key_path = parts
elif raw_domain.startswith("_") or self.ipv4_relay:
elif self.mail_domain.startswith("_"):
self.tls_cert_mode = "self"
self.tls_cert_path = "/etc/ssl/certs/mailserver.pem"
self.tls_key_path = "/etc/ssl/private/mailserver.key"
else:
self.tls_cert_mode = "acme"
self.tls_cert_path = f"/var/lib/acme/live/{raw_domain}/fullchain"
self.tls_key_path = f"/var/lib/acme/live/{raw_domain}/privkey"
self.tls_cert_path = f"/var/lib/acme/live/{self.mail_domain}/fullchain"
self.tls_key_path = f"/var/lib/acme/live/{self.mail_domain}/privkey"
# deprecated option
mbdir = params.pop("mailboxes_dir", f"/home/vmail/mail/{raw_domain}")
mbdir = params.get("mailboxes_dir", f"/home/vmail/mail/{self.mail_domain}")
self.mailboxes_dir = Path(mbdir.strip())
# old unused option (except for first migration from sqlite to maildir store)
self.passdb_path = Path(params.pop("passdb_path", "/home/vmail/passdb.sqlite"))
self._unused_keys = list(params)
self.passdb_path = Path(params.get("passdb_path", "/home/vmail/passdb.sqlite"))
@property
def max_mailbox_size_mb(self):
@@ -163,13 +150,28 @@ def get_default_config_content(mail_domain, **overrides):
for name, value in extra.items():
new_line = f"{name} = {value}"
new_lines.append(new_line)
return "\n".join(new_lines)
content = "\n".join(new_lines)
def is_valid_ipv4(address: str) -> bool:
"""Check if a mail_domain is an IPv4 address."""
try:
ipaddress.IPv4Address(address)
return True
except ValueError:
return False
# apply testrun privacy overrides
if mail_domain.endswith(".testrun.org"):
override_inipath = inidir.joinpath("override-testrun.ini")
privacy = iniconfig.IniConfig(override_inipath)["privacy"]
lines = []
for line in content.split("\n"):
for key, value in privacy.items():
value_lines = value.format(mail_domain=mail_domain).strip().split("\n")
if not line.startswith(f"{key} =") or not value_lines:
continue
if len(value_lines) == 1:
lines.append(f"{key} = {value}")
else:
lines.append(f"{key} =")
for vl in value_lines:
lines.append(f" {vl}")
break
else:
lines.append(line)
content = "\n".join(lines)
return content
+39
View File
@@ -0,0 +1,39 @@
"""
Analyze deferred mails and print most common failing destinations.
Example:
python -m chatmaild.deferred
"""
import json
import subprocess
from collections import Counter, defaultdict
def main():
p = subprocess.Popen(["postqueue", "-j"], text=True, stdout=subprocess.PIPE)
domain_reasons = defaultdict(Counter)
domain_total = Counter()
for line in p.stdout:
item = json.loads(line)
if item["queue_name"] != "deferred":
continue
for recipient in item["recipients"]:
_, domain = recipient["address"].rsplit("@", 1)
reason = recipient["delay_reason"].removeprefix(
"host 127.0.0.1[127.0.0.1] said: "
)
domain_total[domain] += 1
domain_reasons[domain][reason] += 1
for domain, total in reversed(domain_total.most_common()):
print(f"{domain} ({total} recipients)")
for reason, count in domain_reasons[domain].most_common():
print(f" {count}: {reason}")
if __name__ == "__main__":
main()
-10
View File
@@ -168,16 +168,6 @@ class Expiry:
if mbox.last_login and mbox.last_login < cutoff_without_login:
self.remove_mailbox(mbox.basedir)
return
elif mbox.last_login is None:
try:
if not self.dry:
os.rmdir(mbox.basedir)
self.del_mboxes += 1
except OSError:
print_info(
f"Skipped deleting {mbox.basedir}, doesn't have last_login but isn't empty"
)
return
mboxname = os.path.basename(mbox.basedir)
if self.verbose:
+30 -15
View File
@@ -12,35 +12,42 @@ mail_domain = {mail_domain}
#
# email sending rate per user and minute
#max_user_send_per_minute = 60
max_user_send_per_minute = 60
# per-user max burst size for sending rate limiting (GCRA bucket capacity)
#max_user_send_burst_size = 10
max_user_send_burst_size = 10
# maximum mailbox size of a chatmail address
# (Oldest messages will be removed automatically, so mailboxes never run full)
#max_mailbox_size = 500M
# Oldest messages will be removed automatically, so mailboxes never run full.
max_mailbox_size = 500M
# maximum message size for an e-mail in bytes
#max_message_size = 31457280
max_message_size = 31457280
# days after which mails are unconditionally deleted
#delete_mails_after = 20
delete_mails_after = 20
# days after which large messages (>200k) are unconditionally deleted
#delete_large_after = 7
delete_large_after = 7
# days after which users without a successful login are deleted (database and mails)
#delete_inactive_users_after = 90
delete_inactive_users_after = 90
# minimum length a username must have
#username_min_length = 9
username_min_length = 9
# maximum length a username can have
#username_max_length = 9
username_max_length = 9
# minimum length a password must have
#password_min_length = 9
password_min_length = 9
# list of chatmail addresses which can send outbound un-encrypted mail
passthrough_senders =
# list of e-mail recipients for which to accept outbound un-encrypted mails
# (space-separated, item may start with "@" to whitelist whole recipient domains)
passthrough_recipients =
# Use externally managed TLS certificates instead of built-in acmetool.
# Paths refer to files on the deployment server (not the build machine).
@@ -56,11 +63,19 @@ mail_domain = {mail_domain}
# Deployment Details
#
# SMTP outgoing filtermail and reinjection
filtermail_smtp_port = 10080
postfix_reinject_port = 10025
# SMTP incoming filtermail and reinjection
filtermail_smtp_port_incoming = 10081
postfix_reinject_port_incoming = 10026
# if set to "True" IPv6 is disabled
#disable_ipv6 = False
disable_ipv6 = False
# Your email adress, which will be used in acmetool to manage Let's Encrypt SSL certificates
#acme_email =
acme_email =
# Defaults to https://iroh.{{mail_domain}} and running `iroh-relay` on the chatmail
# service.
@@ -93,13 +108,13 @@ mail_domain = {mail_domain}
# in per-maildir ".in/.out" files.
# Note that you need to manually cleanup these files
# so use this option with caution on production servers.
#imap_rawlog = false
imap_rawlog = false
# set to true if you want to enable the IMAP COMPRESS Extension,
# which allows IMAP connections to be efficiently compressed.
# WARNING: Enabling this makes it impossible to hibernate IMAP
# processes which will result in much higher memory/RAM usage.
#imap_compress = false
imap_compress = false
#
@@ -0,0 +1,16 @@
[privacy]
passthrough_recipients = privacy@testrun.org echo@{mail_domain}
privacy_postal =
Merlinux GmbH, Represented by the managing director H. Krekel,
Reichgrafen Str. 20, 79102 Freiburg, Germany
privacy_mail = privacy@testrun.org
privacy_pdo =
Prof. Dr. Fabian Schmieder, lexICT UG (limited), Ostfeldstr. 49, 30559 Hannover.
You can contact him at *delta-privacy@merlinux.eu* (Keyword: DPO)
privacy_supervisor =
State Commissioner for Data Protection and Freedom of Information of
Baden-Württemberg in 70173 Stuttgart, Germany.
+29 -46
View File
@@ -1,5 +1,4 @@
import logging
import socket
import sys
import time
from contextlib import contextmanager
@@ -8,14 +7,7 @@ from .config import read_config
from .dictproxy import DictProxy
from .filedict import FileDict
from .notifier import Notifier
def turn_credentials(turn_socket_path):
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client_socket:
client_socket.settimeout(5)
client_socket.connect(turn_socket_path)
with client_socket.makefile("rb") as file:
return file.readline().decode("utf-8").strip()
from .turnserver import turn_credentials
def _is_valid_token_timestamp(timestamp, now):
@@ -87,44 +79,38 @@ class Metadata:
class MetadataDictProxy(DictProxy):
def __init__(
self,
notifier,
metadata,
iroh_relay=None,
turn_hostname=None,
turn_socket_path=None,
):
def __init__(self, notifier, metadata, iroh_relay=None, turn_hostname=None):
super().__init__()
self.notifier = notifier
self.metadata = metadata
self.iroh_relay = iroh_relay
self.turn_hostname = turn_hostname
self.turn_socket_path = turn_socket_path
def handle_lookup(self, parts):
# Lpriv/43f5f508a7ea0366dff30200c15250e3/devicetoken\tlkj123poi@c2.testrun.org
match parts[0].split("/", 2):
case ["priv", _, keyname] if keyname == self.metadata.DEVICETOKEN_KEY:
addr = parts[1]
keyparts = parts[0].split("/", 2)
if keyparts[0] == "priv":
keyname = keyparts[2]
addr = parts[1]
if keyname == self.metadata.DEVICETOKEN_KEY:
res = " ".join(self.metadata.get_tokens_for_addr(addr))
return f"O{res}\n"
case ["shared", _, keyname]:
prefix = "vendor/vendor.dovecot/pvt/server/vendor/deltachat/"
if keyname.startswith(prefix):
match keyname[len(prefix) :]:
case "irohrelay" if self.iroh_relay:
return f"O{self.iroh_relay}\n"
case "turn":
try:
res = turn_credentials(self.turn_socket_path)
except Exception:
logging.exception("failed to get TURN credentials")
return "N\n"
return f"O{self.turn_hostname}:3478:{res}\n"
case "maxsmtprecipients":
# postfix default (see "postconf smtpd_recipient_limit")
return "O1000\n"
elif keyparts[0] == "shared":
keyname = keyparts[2]
if (
keyname == "vendor/vendor.dovecot/pvt/server/vendor/deltachat/irohrelay"
and self.iroh_relay
):
# Handle `GETMETADATA "" /shared/vendor/deltachat/irohrelay`
return f"O{self.iroh_relay}\n"
elif keyname == "vendor/vendor.dovecot/pvt/server/vendor/deltachat/turn":
try:
res = turn_credentials()
except Exception:
logging.exception("failed to get TURN credentials")
return "N\n"
port = 3478
return f"O{self.turn_hostname}:{port}:{res}\n"
logging.warning(f"lookup ignored: {parts!r}")
return "N\n"
@@ -134,13 +120,12 @@ class MetadataDictProxy(DictProxy):
# https://github.com/dovecot/core/blob/main/src/lib-storage/mailbox-attribute.h
keyname = parts[1].split("/")
value = parts[2] if len(parts) > 2 else ""
match keyname:
case ["priv", _, key] if key == self.metadata.DEVICETOKEN_KEY:
self.metadata.add_token_to_addr(addr, value)
return True
case ["priv", _, "messagenew"]:
self.notifier.new_message_for_addr(addr, self.metadata)
return True
if keyname[0] == "priv" and keyname[2] == self.metadata.DEVICETOKEN_KEY:
self.metadata.add_token_to_addr(addr, value)
return True
elif keyname[0] == "priv" and keyname[2] == "messagenew":
self.notifier.new_message_for_addr(addr, self.metadata)
return True
return False
@@ -151,7 +136,6 @@ def main():
config = read_config(config_path)
iroh_relay = config.iroh_relay
mail_domain = config.mail_domain
socket_path = config.turn_socket_path
vmail_dir = config.mailboxes_dir
if not vmail_dir.exists():
@@ -169,7 +153,6 @@ def main():
metadata=metadata,
iroh_relay=iroh_relay,
turn_hostname=mail_domain,
turn_socket_path=socket_path,
)
dictproxy.serve_forever_from_socket(socket)
+15 -12
View File
@@ -2,6 +2,7 @@
"""CGI script for creating new accounts."""
import ipaddress
import json
import secrets
import string
@@ -14,6 +15,16 @@ ALPHANUMERIC = string.ascii_lowercase + string.digits
ALPHANUMERIC_PUNCT = string.ascii_letters + string.digits + string.punctuation
def wrap_ip(host):
if host.startswith("[") and host.endswith("]"):
return host
try:
ipaddress.ip_address(host)
return f"[{host}]"
except ValueError:
return host
def create_newemail_dict(config: Config):
user = "".join(
secrets.choice(ALPHANUMERIC) for _ in range(config.username_max_length)
@@ -22,22 +33,16 @@ def create_newemail_dict(config: Config):
secrets.choice(ALPHANUMERIC_PUNCT)
for _ in range(config.password_min_length + 3)
)
return dict(email=f"{user}@{config.mail_domain}", password=f"{password}")
return dict(email=f"{user}@{wrap_ip(config.mail_domain)}", password=f"{password}")
def create_dclogin_url(config, email, password):
def create_dclogin_url(email, password):
"""Build a dclogin: URL with credentials and self-signed cert acceptance.
Uses ic=3 (AcceptInvalidCertificates) so chatmail clients
can connect to servers with self-signed TLS certificates.
"""
if config.ipv4_relay:
imap_host = "&ih=" + config.ipv4_relay
smtp_host = "&sh=" + config.ipv4_relay
else:
imap_host = ""
smtp_host = ""
return f"dclogin:{quote(email, safe='@[]')}?p={quote(password, safe='')}&v=1{imap_host}{smtp_host}&ic=3"
return f"dclogin:{quote(email, safe='@')}?p={quote(password, safe='')}&v=1&ic=3"
def print_new_account():
@@ -46,9 +51,7 @@ def print_new_account():
result = dict(email=creds["email"], password=creds["password"])
if config.tls_cert_mode == "self":
result["dclogin_url"] = create_dclogin_url(
config, creds["email"], creds["password"]
)
result["dclogin_url"] = create_dclogin_url(creds["email"], creds["password"])
print("Content-Type: application/json")
print("")
-5
View File
@@ -31,11 +31,6 @@ def example_config(make_config):
return make_config("chat.example.org")
@pytest.fixture
def ipv4_config(make_config):
return make_config("1.3.3.7")
@pytest.fixture
def maildomain(example_config):
return example_config.mail_domain
+20 -44
View File
@@ -1,10 +1,6 @@
import pytest
from chatmaild.config import (
is_valid_ipv4,
parse_size_mb,
read_config,
)
from chatmaild.config import parse_size_mb, read_config
def test_read_config_basic(example_config):
@@ -13,21 +9,10 @@ def test_read_config_basic(example_config):
assert not example_config.privacy_pdo and not example_config.privacy_postal
inipath = example_config._inipath
inipath.write_text(
inipath.read_text().replace(
"#max_user_send_per_minute = 60",
"max_user_send_per_minute = 37",
)
)
inipath.write_text(inipath.read_text().replace("60", "37"))
example_config = read_config(inipath)
assert example_config.max_user_send_per_minute == 37
assert example_config.mail_domain == "chat.example.org"
assert example_config.ipv4_relay is None
def test_read_config_ipv4(ipv4_config):
assert ipv4_config.ipv4_relay == "1.3.3.7"
assert ipv4_config.mail_domain == "[1.3.3.7]"
def test_read_config_basic_using_defaults(tmp_path, maildomain):
@@ -36,21 +21,26 @@ def test_read_config_basic_using_defaults(tmp_path, maildomain):
example_config = read_config(inipath)
assert example_config.max_user_send_per_minute == 60
assert example_config.filtermail_smtp_port_incoming == 10081
assert example_config.filtermail_smtp_port == 10080
assert example_config.postfix_reinject_port == 10025
assert example_config.max_user_send_per_minute == 60
assert example_config.max_mailbox_size == "500M"
assert example_config.delete_mails_after == "20"
assert example_config.delete_large_after == "7"
assert example_config.username_min_length == 9
assert example_config.username_max_length == 9
assert example_config.password_min_length == 9
assert example_config._unused_keys == []
def test_config_unused_keys(make_config):
config = make_config("chat.example.org", {"passthrough_senders": "x@y.org"})
assert config._unused_keys == ["passthrough_senders"]
def test_read_config_testrun(make_config):
config = make_config("something.testrun.org")
assert config.mail_domain == "something.testrun.org"
assert len(config.privacy_postal.split("\n")) > 1
assert len(config.privacy_supervisor.split("\n")) > 1
assert len(config.privacy_pdo.split("\n")) > 1
assert config.privacy_mail == "privacy@testrun.org"
assert config.filtermail_smtp_port == 10080
assert config.postfix_reinject_port == 10025
assert config.max_user_send_per_minute == 60
assert config.max_mailbox_size == "500M"
assert config.delete_mails_after == "20"
assert config.delete_large_after == "7"
assert config.username_min_length == 9
assert config.username_max_length == 9
assert config.password_min_length == 9
assert "privacy@testrun.org" in config.passthrough_recipients
assert config.passthrough_senders == []
def test_config_userstate_paths(make_config, tmp_path):
@@ -145,17 +135,3 @@ def test_max_mailbox_size_mb(make_config):
config = make_config("chat.example.org")
assert config.max_mailbox_size == "500M"
assert config.max_mailbox_size_mb == 500
@pytest.mark.parametrize(
["input", "result"],
[
("example.org", False),
("1.3.3.7", True),
("fe::1", False),
("ad.1e.dag.adf", False),
("12394142", False),
],
)
def test_is_valid_ipv4(input, result):
assert result == is_valid_ipv4(input)
@@ -1,7 +1,6 @@
import itertools
import os
import random
import shutil
import time
from datetime import datetime
from fnmatch import fnmatch
@@ -10,7 +9,6 @@ from pathlib import Path
import pytest
from chatmaild.expire import (
Expiry,
FileEntry,
MailboxStat,
expire_to_target,
@@ -106,32 +104,6 @@ def test_stats_mailbox(mbox1):
assert mbox3.last_login is None
def test_mbox_without_password(mbox1, example_config, capsys):
password = Path(mbox1.basedir).joinpath("password")
os.remove(password)
mbox_rescan = MailboxStat(mbox1.basedir)
assert mbox_rescan.last_login is None
exp = Expiry(
example_config, dry=False, now=datetime.now().timestamp(), verbose=False
)
exp.process_mailbox_stat(mbox_rescan)
out, err = capsys.readouterr()
assert "doesn't have last_login but isn't empty" in err
assert os.path.isdir(mbox_rescan.basedir)
for entry in os.scandir(mbox_rescan.basedir):
if os.path.isdir(entry):
shutil.rmtree(entry)
else:
os.remove(entry)
exp.process_mailbox_stat(mbox_rescan)
out, err = capsys.readouterr()
assert "doesn't have last_login but isn't empty" not in err
assert not os.path.isdir(mbox_rescan.basedir)
def test_report_no_mailboxes(example_config):
args = (str(example_config._inipath),)
report_main(args)
+11 -26
View File
@@ -324,7 +324,7 @@ def test_turn_credentials_exception_returns_N(notifier, metadata, monkeypatch):
turn_hostname="turn.example.org",
)
def mock_turn_credentials(turn_socket_path):
def mock_turn_credentials():
raise ConnectionRefusedError("socket not available")
monkeypatch.setattr(chatmaild.metadata, "turn_credentials", mock_turn_credentials)
@@ -348,9 +348,7 @@ def test_turn_credentials_success(notifier, metadata, monkeypatch):
turn_hostname="turn.example.org",
)
monkeypatch.setattr(
chatmaild.metadata, "turn_credentials", lambda path: "user:pass"
)
monkeypatch.setattr(chatmaild.metadata, "turn_credentials", lambda: "user:pass")
transactions = {}
res = dictproxy.handle_dovecot_request(
@@ -362,8 +360,15 @@ def test_turn_credentials_success(notifier, metadata, monkeypatch):
def test_iroh_relay(dictproxy):
key = b"Lshared/0123/vendor/vendor.dovecot/pvt/server/vendor/deltachat/irohrelay\tuser@example.org"
rfile, wfile = io.BytesIO(b"H\n" + key), io.BytesIO()
rfile = io.BytesIO(
b"\n".join(
[
b"H",
b"Lshared/0123/vendor/vendor.dovecot/pvt/server/vendor/deltachat/irohrelay\tuser@example.org",
]
)
)
wfile = io.BytesIO()
dictproxy.iroh_relay = "https://example.org/"
dictproxy.loop_forever(rfile, wfile)
assert wfile.getvalue() == b"Ohttps://example.org/\n"
@@ -378,23 +383,3 @@ def test_legacy_token_migration(metadata, testaddr):
tokens = mdict[metadata.DEVICETOKEN_KEY]
assert isinstance(tokens, dict)
assert "oldtoken1" in tokens and "oldtoken2" in tokens
@pytest.mark.parametrize(
"suffix, expected",
[
(b"vendor/deltachat/maxsmtprecipients", b"O1000\n"),
(b"wrong/prefix/key", b"N\n"),
(b"vendor/deltachat/unknown", b"N\n"),
],
ids=["maxsmtprecipients", "prefix_mismatch", "unknown_name"],
)
def test_shared_lookup(dictproxy, suffix, expected):
key = (
b"Lshared/0123/vendor/vendor.dovecot/pvt/server/"
+ suffix
+ b"\tuser@example.org"
)
rfile, wfile = io.BytesIO(b"H\n" + key), io.BytesIO()
dictproxy.loop_forever(rfile, wfile)
assert wfile.getvalue() == expected
+7 -18
View File
@@ -19,35 +19,24 @@ def test_create_newemail_dict(example_config):
assert ac1["password"] != ac2["password"]
def test_create_newemail_dict_ip(ipv4_config):
ac = create_newemail_dict(ipv4_config)
assert ac["email"].endswith("@[1.3.3.7]")
def test_create_newemail_dict_ip(make_config):
config = make_config("1.2.3.4")
ac = create_newemail_dict(config)
assert ac["email"].endswith("@[1.2.3.4]")
def test_create_dclogin_url(example_config):
addr = "user@example.org"
password = "p@ss w+rd"
url = create_dclogin_url(example_config, addr, password)
def test_create_dclogin_url():
url = create_dclogin_url("user@example.org", "p@ss w+rd")
assert url.startswith("dclogin:")
assert "v=1" in url
assert "ic=3" in url
assert addr in url
assert "user@example.org" in url
# password special chars must be encoded
assert "p%40ss" in url
assert "w%2Brd" in url
def test_create_dclogin_url_ipv4(ipv4_config):
addr = "user@[1.3.3.7]"
password = "p@ss w+rd"
url = create_dclogin_url(ipv4_config, addr, password)
assert url.startswith("dclogin:")
assert "v=1" in url
assert "ic=3" in url
assert addr in url
def test_print_new_account(capsys, monkeypatch, maildomain, tmpdir, example_config):
monkeypatch.setattr(chatmaild.newemail, "CONFIG_PATH", str(example_config._inipath))
print_new_account()
@@ -1,46 +0,0 @@
import socket
import threading
import pytest
from chatmaild.metadata import turn_credentials
@pytest.fixture
def turn_socket(tmp_path):
sock_path = str(tmp_path / "turn.socket")
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
server.bind(sock_path)
server.listen(1)
yield sock_path, server
server.close()
def test_turn_credentials_timeout(turn_socket):
sock_path, server = turn_socket
with pytest.raises(socket.timeout):
# Inside turn_credentials the kernel listen backlog (1)
# completes connect() without accept()
# so the client blocks on readline() until the 5s timeout fires.
turn_credentials(sock_path)
def test_turn_credentials_connection_refused_on_not_existing_socket(tmp_path):
missing = str(tmp_path / "nonexistent.socket")
with pytest.raises((ConnectionRefusedError, FileNotFoundError)):
turn_credentials(missing)
def test_turn_credentials_socket_success(turn_socket):
sock_path, server = turn_socket
def respond():
conn, _ = server.accept()
conn.sendall(b"testuser:testpass\n")
conn.close()
t = threading.Thread(target=respond, daemon=True)
t.start()
result = turn_credentials(sock_path)
assert result == "testuser:testpass"
@@ -0,0 +1,73 @@
import socket
import threading
import time
from unittest.mock import patch
import pytest
from chatmaild.turnserver import turn_credentials
SOCKET_PATH = "/run/chatmail-turn/turn.socket"
@pytest.fixture
def turn_socket(tmp_path):
"""Create a real Unix socket server at a temp path."""
sock_path = str(tmp_path / "turn.socket")
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
server.bind(sock_path)
server.listen(1)
yield sock_path, server
server.close()
def _call_turn_credentials(sock_path):
"""Call turn_credentials but connect to sock_path instead of hardcoded path."""
original_connect = socket.socket.connect
def patched_connect(self, address):
if address == SOCKET_PATH:
address = sock_path
return original_connect(self, address)
with patch.object(socket.socket, "connect", patched_connect):
return turn_credentials()
def test_turn_credentials_timeout(turn_socket):
"""Server accepts but never responds — must raise socket.timeout."""
sock_path, server = turn_socket
def accept_and_hang():
conn, _ = server.accept()
time.sleep(30)
conn.close()
t = threading.Thread(target=accept_and_hang, daemon=True)
t.start()
with pytest.raises(socket.timeout):
_call_turn_credentials(sock_path)
def test_turn_credentials_connection_refused(tmp_path):
"""Socket file doesn't exist — must raise ConnectionRefusedError or FileNotFoundError."""
missing = str(tmp_path / "nonexistent.socket")
with pytest.raises((ConnectionRefusedError, FileNotFoundError)):
_call_turn_credentials(missing)
def test_turn_credentials_success(turn_socket):
"""Server responds with credentials — must return stripped string."""
sock_path, server = turn_socket
def respond():
conn, _ = server.accept()
conn.sendall(b"testuser:testpass\n")
conn.close()
t = threading.Thread(target=respond, daemon=True)
t.start()
result = _call_turn_credentials(sock_path)
assert result == "testuser:testpass"
+10
View File
@@ -0,0 +1,10 @@
#!/usr/bin/env python3
import socket
def turn_credentials() -> str:
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client_socket:
client_socket.settimeout(5)
client_socket.connect("/run/chatmail-turn/turn.socket")
with client_socket.makefile("rb") as file:
return file.readline().decode("utf-8").strip()
+2 -20
View File
@@ -84,24 +84,13 @@ def run_cmd_options(parser):
add_ssh_host_option(parser)
def _warn_unused_settings(unused_keys, out):
if unused_keys:
names = ", ".join(unused_keys)
out.red(
f"WARNING: chatmail.ini contains settings that have no effect: {names}\n"
"Please remove them from chatmail.ini."
)
def run_cmd(args, out):
"""Deploy chatmail services on the remote server."""
ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain_bare
ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain
sshexec = get_sshexec(ssh_host)
require_iroh = args.config.enable_iroh_relay
strict_tls = args.config.tls_cert_mode == "acme"
if args.config.ipv4_relay:
args.dns_check_disabled = True
if not args.dns_check_disabled:
remote_data = dns.get_initial_remote_data(sshexec, args.config.mail_domain)
if not dns.check_initial_remote_data(remote_data, strict_tls=strict_tls, print=out.red):
@@ -130,11 +119,8 @@ def run_cmd(args, out):
elif not args.dns_check_disabled and strict_tls and not remote_data["acme_account_url"]:
out.red("Deploy completed but letsencrypt not configured")
out.red("Run 'cmdeploy run' again")
elif args.config.ipv4_relay:
out.green("Deploy completed.")
else:
out.green("Deploy completed, call `cmdeploy dns` next.")
_warn_unused_settings(args.config._unused_keys, out)
return 0
except subprocess.CalledProcessError:
out.red("Deploy failed")
@@ -154,10 +140,6 @@ def dns_cmd_options(parser):
def dns_cmd(args, out):
"""Check DNS entries and optionally generate dns zone file."""
if args.config.ipv4_relay:
ipv4 = args.config.ipv4_relay
print(f"[WARNING] {ipv4} is not a domain, skipping DNS checks.")
return 0
ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain
sshexec = get_sshexec(ssh_host, verbose=args.verbose)
tls_cert_mode = args.config.tls_cert_mode
@@ -195,7 +177,7 @@ def status_cmd_options(parser):
def status_cmd(args, out):
"""Display status for online chatmail instance."""
ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain_bare
ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain
sshexec = get_sshexec(ssh_host, verbose=args.verbose)
out.green(f"chatmail domain: {args.config.mail_domain}")
+15 -15
View File
@@ -171,14 +171,16 @@ class UnboundDeployer(Deployer):
"unbound-anchor -a /var/lib/unbound/root.key || true",
],
)
self.ensure_directory(
path="/etc/unbound/unbound.conf.d",
)
self.put_template(
"unbound/unbound.conf.j2",
"/etc/unbound/unbound.conf.d/chatmail.conf",
disable_ipv6=self.config.disable_ipv6,
)
if self.config.disable_ipv6:
self.ensure_directory(
path="/etc/unbound/unbound.conf.d",
)
self.put_template(
"unbound/unbound.conf.j2",
"/etc/unbound/unbound.conf.d/chatmail.conf",
)
else:
self.remove_file("/etc/unbound/unbound.conf.d/chatmail.conf")
def activate(self):
server.shell(
@@ -368,7 +370,7 @@ class ChatmailVenvDeployer(Deployer):
def configure(self):
_configure_remote_venv_with_chatmaild(self, self.config)
configure_remote_units(self, self.config.mail_domain_bare, self.units)
configure_remote_units(self, self.config.mail_domain, self.units)
def activate(self):
activate_remote_units(self, self.units)
@@ -467,7 +469,7 @@ def deploy_chatmail(config_path: Path, disable_mail: bool, website_only: bool) -
"""
config = read_config(config_path)
check_config(config)
bare_host = config.mail_domain_bare
mail_domain = config.mail_domain
if website_only:
Deployment().perform_stages([WebsiteDeployer(config)])
@@ -512,8 +514,6 @@ def deploy_chatmail(config_path: Path, disable_mail: bool, website_only: bool) -
(["master", "smtpd"], config.postfix_reinject_port_incoming),
("filtermail", config.filtermail_smtp_port),
("filtermail", config.filtermail_smtp_port_incoming),
("filtermail", config.filtermail_http_port_incoming),
("filtermail", config.filtermail_lmtp_port_transport),
]
for service, port in port_services:
print(f"Checking if port {port} is available for {service}...")
@@ -526,7 +526,7 @@ def deploy_chatmail(config_path: Path, disable_mail: bool, website_only: bool) -
)
exit(1)
tls_deployer = get_tls_deployer(config, bare_host)
tls_deployer = get_tls_deployer(config, mail_domain)
all_deployers = [
ChatmailDeployer(config),
@@ -534,13 +534,13 @@ def deploy_chatmail(config_path: Path, disable_mail: bool, website_only: bool) -
FiltermailDeployer(),
JournaldDeployer(),
UnboundDeployer(config),
TurnDeployer(bare_host),
TurnDeployer(mail_domain),
IrohDeployer(config.enable_iroh_relay),
tls_deployer,
WebsiteDeployer(config),
ChatmailVenvDeployer(config),
MtastsDeployer(),
*([] if config.ipv4_relay else [OpendkimDeployer(bare_host)]),
OpendkimDeployer(mail_domain),
# Dovecot should be started before Postfix
# because it creates authentication socket
# required by Postfix.
+1 -1
View File
@@ -68,7 +68,7 @@ class DovecotDeployer(Deployer):
)
def configure(self):
configure_remote_units(self, self.config.mail_domain_bare, self.units)
configure_remote_units(self, self.config.mail_domain, self.units)
_configure_dovecot(self, self.config)
def activate(self):
@@ -7,7 +7,6 @@ listen = 0.0.0.0
protocols = imap lmtp
auth_mechanisms = plain
auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@[]
{% if debug == true %}
auth_verbose = yes
+5 -2
View File
@@ -1,3 +1,4 @@
from pyinfra import host
from pyinfra.facts.files import File
@@ -20,8 +21,8 @@ class ExternalTlsDeployer(Deployer):
def configure(self):
# Verify cert and key exist on the remote host using pyinfra facts.
for path in (self.cert_path, self.key_path):
if host.get_fact(File, path=path) is None:
raise Exception(f"External TLS file not found on server: {path}")
if host.get_fact(File, path=path) is None:
raise Exception(f"External TLS file not found on server: {path}")
self.ensure_systemd_unit(
"external/tls-cert-reload.path.j2",
@@ -39,3 +40,5 @@ class ExternalTlsDeployer(Deployer):
running=True,
enabled=True,
)
+3 -3
View File
@@ -20,10 +20,10 @@ class FiltermailDeployer(Deployer):
return
arch = host.get_fact(facts.server.Arch)
url = f"https://github.com/chatmail/filtermail/releases/download/v0.7.0/filtermail-{arch}"
url = f"https://github.com/chatmail/filtermail/releases/download/v0.6.4/filtermail-{arch}"
sha256sum = {
"x86_64": "451f295a85b3b12dbb0f89e18ec319f742ee46dec218f20f7923bfb017a248bd",
"aarch64": "6833061b2a2028264fdeb32f0a6123e1ff73de57dace125364016300b748452e",
"x86_64": "5295115952c72e4c4ec3c85546e094b4155a4c702c82bd71fcdcb744dc73adf6",
"aarch64": "6892244f17b8f26ccb465766e96028e7222b3c8adefca9fc6bfe9ff332ca8dff",
}[arch]
self.download_executable(url, self.bin_path, sha256sum)
@@ -6,7 +6,6 @@ ExecStart={{ bin_path }} {{ config_path }} transport
Restart=always
RestartSec=30
User=vmail
LimitNOFILE=524288
[Install]
WantedBy=multi-user.target
@@ -1,23 +1,3 @@
# List of headers for incoming messages
# that must be retained for functionality and compatibility reasons
/^From:/ DUNNO
/^Message-Id:/ DUNNO
/^Chat-/ DUNNO
/^Content-Type:/ DUNNO
# For receiving clear-text messages (still supported in May 2026)
/^Subject:/ DUNNO
/^Date:/ DUNNO
/^To:/ DUNNO
/^CC:/ DUNNO
/^References:/ DUNNO
/^In-Reply-To:/ DUNNO
# Senders might support Autocrypt 1 but not RFC9788 (Header Protection)
/^Autocrypt:/ DUNNO
# SecureJoin V2 protocol headers (for backward compatibility)
/^Secure-Join/ DUNNO
# Ignore all other headers
/.*/ IGNORE
/^DKIM-Signature:/ IGNORE
/^Authentication-Results:/ IGNORE
/^Received:/ IGNORE
+8 -33
View File
@@ -53,19 +53,15 @@ smtpd_tls_exclude_ciphers = aNULL, RC4, MD5, DES
# See <https://www.postfix.org/FORWARD_SECRECY_README.html#server_fs>.
tls_preempt_cipherlist = yes
# Reject by default, override per smtpd in master.cf
smtpd_relay_restrictions = reject
myhostname = {{ config.postfix_myhostname }}
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = {{ config.mail_domain }}
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
# When postfix receives mail for $mydestination,
# it hands it over to dovecot via $local_transport.
# Note: IP literals must be handled via local delivery / mydestination.
mydestination = {{ config.mail_domain }}
local_transport = lmtp:unix:private/dovecot-lmtp
# postfix doesn't check whether local users exist or not:
local_recipient_maps =
# Postfix does not deliver mail for any domain by itself.
# Primary domain is listed in `virtual_mailbox_domains` instead
# and handed over to Dovecot.
mydestination =
relayhost =
{% if disable_ipv6 %}
@@ -83,6 +79,8 @@ inet_protocols = ipv4
inet_protocols = all
{% endif %}
virtual_transport = lmtp:unix:private/dovecot-lmtp
virtual_mailbox_domains = {{ config.mail_domain }}
lmtp_header_checks = regexp:/etc/postfix/lmtp_header_cleanup
mua_client_restrictions = permit_sasl_authenticated, reject
@@ -102,28 +100,5 @@ smtpd_peername_lookup = no
# so instead this is handled in filtermail.
# We use LMTP instead SMTP so we can communicate per-recipient errors back to postfix.
default_transport = lmtp-filtermail:inet:[127.0.0.1]:{{ config.filtermail_lmtp_port_transport }}
# All deliveries over lmtp-filtermail are treated
# as having the same destination [127.0.0.1],
# so it is not possible to limit per-destination concurrency here,
# it is a job for filtermail-transport.
# Total number of parallel deliveries is limited
# by "maxproc" column in /etc/postfix/master.cf for lmtp-filtermail.
# Settings below are to prevent Postfix queue manager
# from limiting the number of LMTP connections to filtermail-transport.
# Read <https://www.postfix.org/TUNING_README.html#rope> and
# <https://www.postfix.org/SCHEDULER_README.html> for the details
# of the Postfix algorithm that we effectively disable here.
lmtp-filtermail_initial_destination_concurrency=10000
lmtp-filtermail_destination_concurrency_limit=10000
# Do not try to deliver messages for more than 2 days.
maximal_queue_lifetime = 2d
{% if not config.ipv4_relay %}
# DKIM-sign locally generated mail (bounces, DSNs).
# These bypass smtpd, so they need explicit milter configuration.
non_smtpd_milters = unix:opendkim/opendkim.sock
internal_mail_filter_classes = bounce
milter_macro_daemon_name = ORIGINATING
{% endif %}
+2 -14
View File
@@ -17,7 +17,6 @@ smtp inet n - y - - smtpd
-o smtpd_tls_security_level=encrypt
-o smtpd_tls_mandatory_protocols=>=TLSv1.2
-o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port_incoming }}
-o smtpd_relay_restrictions=reject_unauth_destination
submission inet n - y - 5000 smtpd
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
@@ -81,15 +80,12 @@ filter unix - n n - - lmtp
127.0.0.1:{{ config.postfix_reinject_port }} inet n - n - 100 smtpd
-o syslog_name=postfix/reinject
-o milter_macro_daemon_name=ORIGINATING
-o smtpd_milters=unix:opendkim/opendkim.sock
-o cleanup_service_name=authclean
-o smtpd_relay_restrictions=permit_mynetworks,reject
{% if not config.ipv4_relay %} -o smtpd_milters=unix:opendkim/opendkim.sock
{% endif %}
# Local SMTP server for reinjecting incoming filtered mail
127.0.0.1:{{ config.postfix_reinject_port_incoming }} inet n - n - 100 smtpd
-o syslog_name=postfix/reinject_incoming
-o smtpd_relay_restrictions=reject_unauth_destination
# Cleanup `Received` headers for authenticated mail
# to avoid leaking client IP.
@@ -105,15 +101,7 @@ filter unix - n n - - lmtp
authclean unix n - - - 0 cleanup
-o header_checks=regexp:/etc/postfix/submission_header_cleanup
# Reducing `maxproc` here may result in a head of line blocking
# when there are many messages sent to unreachable destinations
# at the same time.
# LMTP clients here talk to filtermail-transport.
# LMTP has no pipelining,
# so while filtermail-transport tries to deliver the message,
# possibly waiting for a long connection timeout
# or talking to a slow server, LMTP client cannot be reused.
lmtp-filtermail unix - - y - 500 lmtp
lmtp-filtermail unix - - y - 10000 lmtp
-o syslog_name=postfix/lmtp-filtermail
-o lmtp_header_checks=
-o lmtp_tls_security_level=none
@@ -12,7 +12,7 @@ def test_init(tmp_path, maildomain):
inipath = tmp_path.joinpath("chatmail.ini")
main(["init", "--config", str(inipath), maildomain])
config = read_config(inipath)
assert config.mail_domain_bare == maildomain
assert config.mail_domain == maildomain
def test_capabilities(imap):
@@ -89,11 +89,12 @@ def test_concurrent_logins_same_account(
assert login_results.get()
def test_no_vrfy(cmfactory, chatmail_config, maildomain):
def test_no_vrfy(cmfactory, chatmail_config):
ac = cmfactory.get_online_account()
addr = ac.get_config("addr")
domain = chatmail_config.mail_domain
s = smtplib.SMTP(maildomain)
s = smtplib.SMTP(domain)
s.starttls()
s.putcmd("vrfy", f"wrongaddress@{chatmail_config.mail_domain}")
@@ -5,7 +5,6 @@ import subprocess
import time
import pytest
from chatmaild.config import is_valid_ipv4
from cmdeploy import remote
from cmdeploy.cmdeploy import get_sshexec
@@ -22,8 +21,6 @@ class TestSSHExecutor:
assert out == out2
def test_perform_initial(self, sshexec, maildomain):
if is_valid_ipv4(maildomain):
pytest.skip(f"{maildomain} is not a domain")
res = sshexec(
remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=maildomain)
)
@@ -64,10 +61,8 @@ class TestSSHExecutor:
else:
pytest.fail("didn't raise exception")
def test_opendkim_restarted(self, sshexec, maildomain):
def test_opendkim_restarted(self, sshexec):
"""check that opendkim is not running for longer than a day."""
if is_valid_ipv4(maildomain):
pytest.skip(f"{maildomain} is an IPv4 relay, opendkim is not installed")
cmd = "systemctl show opendkim --timestamp=utc --property=ActiveEnterTimestamp"
out = sshexec(call=remote.rshell.shell, kwargs=dict(command=cmd))
datestring = out.split("=")[1]
@@ -194,34 +189,6 @@ def test_reject_missing_dkim(cmsetup, maildata, from_addr):
s.sendmail(from_addr=from_addr, to_addrs=recipient.addr, msg=msg)
def test_bounces_are_dkim_signed(cmsetup, cmsetup2, maildata, maildomain):
# we send a message to non-existant user and expect a bounce message
# which will only get through if the bounce message was DKIM-signed
if is_valid_ipv4(maildomain):
pytest.skip("DKIM is not configured on IPv4-only relays")
sender = cmsetup2.gen_users(1)[0]
nonexistent = f"nosuchuser_test42@{cmsetup.maildomain}"
msg = maildata(
"encrypted.eml",
from_addr=sender.addr,
to_addr=nonexistent,
).as_string()
sender.smtp.sendmail(sender.addr, [nonexistent], msg)
def bounce_in_inbox():
messages = sender.imap.fetch_all_messages()
for m in messages:
if "mail delivery" in m.lower() or "undelivered" in m.lower():
return m
raise ValueError("bounce not yet in inbox")
bounce = try_n_times(30, bounce_in_inbox)
assert "nosuchuser_test42" in bounce
def try_n_times(n, f):
for _ in range(n - 1):
try:
@@ -323,6 +290,4 @@ def test_nginx_access_log_only_defined_once(sshdomain):
kwargs=dict(command="nginx -T 2>/dev/null"),
)
access_logs = [l for l in conf.splitlines() if l.strip().startswith("access_log")]
assert len(access_logs) == 1, (
f"expected 1 access_log, found {len(access_logs)}: {access_logs}"
)
assert len(access_logs) == 1, f"expected 1 access_log, found {len(access_logs)}: {access_logs}"
@@ -15,7 +15,7 @@ def imap_mailbox(cmfactory, ssl_context):
(ac1,) = cmfactory.get_online_accounts(1)
user = ac1.get_config("addr")
password = ac1.get_config("mail_pw")
host = user.split("@")[1].strip("[").strip("]")
host = user.split("@")[1]
mailbox = imap_tools.MailBox(host, ssl_context=ssl_context)
mailbox.login(user, password)
mailbox.dc_ac = ac1
@@ -178,7 +178,7 @@ def test_hide_senders_ip_address(cmfactory, ssl_context):
chat.send_text("testing submission header cleanup")
user2.wait_for_incoming_msg()
addr = user2.get_config("addr")
host = addr.split("@")[1].strip("[").strip("]")
host = addr.split("@")[1]
pw = user2.get_config("mail_pw")
mailbox = imap_tools.MailBox(host, ssl_context=ssl_context)
mailbox.login(addr, pw)
+26 -30
View File
@@ -1,4 +1,5 @@
import imaplib
import ipaddress
import itertools
import os
import random
@@ -9,18 +10,19 @@ import time
from pathlib import Path
import pytest
from chatmaild.config import is_valid_ipv4, read_config
def format_mail_domain(raw_domain: str) -> str:
if is_valid_ipv4(raw_domain):
return f"[{raw_domain}]"
return raw_domain
from chatmaild.config import read_config
conftestdir = Path(__file__).parent
def _is_ip(domain):
try:
ipaddress.ip_address(domain)
return True
except ValueError:
return False
def pytest_configure(config):
config._benchresults = {}
config.addinivalue_line(
@@ -56,7 +58,7 @@ def chatmail_config(pytestconfig):
@pytest.fixture(scope="session")
def maildomain(chatmail_config):
return chatmail_config.mail_domain_bare
return chatmail_config.mail_domain
@pytest.fixture(scope="session")
@@ -276,6 +278,7 @@ def gencreds(chatmail_config):
def gen(domain=None):
domain = domain if domain else chatmail_config.mail_domain
addr_domain = f"[{domain}]" if _is_ip(domain) else domain
while 1:
num = next(count)
alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
@@ -289,7 +292,7 @@ def gencreds(chatmail_config):
password = "".join(
random.choices(alphanumeric, k=chatmail_config.password_min_length)
)
yield f"{user}@{domain}", f"{password}"
yield f"{user}@{addr_domain}", f"{password}"
return lambda domain=None: next(gen(domain))
@@ -314,8 +317,7 @@ class ChatmailACFactory:
def _make_transport(self, domain):
"""Build a transport config dict for the given domain."""
domain_deliverable = format_mail_domain(domain)
addr, password = self.gencreds(domain_deliverable)
addr, password = self.gencreds(domain)
transport = {
"addr": addr,
"password": password,
@@ -324,7 +326,7 @@ class ChatmailACFactory:
"imapServer": domain,
"smtpServer": domain,
}
if domain.startswith("_") or is_valid_ipv4(domain):
if self.chatmail_config.tls_cert_mode == "self":
transport["certificateChecks"] = "acceptInvalidCertificates"
return transport
@@ -339,17 +341,16 @@ class ChatmailACFactory:
accounts = []
for _ in range(num):
account = self.dc.add_account()
domain_deliverable = format_mail_domain(domain)
addr, password = self.gencreds(domain_deliverable)
if is_valid_ipv4(domain):
addr, password = self.gencreds(domain)
if _is_ip(domain):
# Use DCLOGIN scheme with explicit server hosts,
# matching how madmail presents its addresses to users.
qr = (
f"dclogin:{addr}"
f"?p={password}&v=1"
f"&ih={domain}&ip=993&is=ssl"
f"&sh={domain}&sp=465&ss=ssl"
f"&ic=3"
f"&ih={domain}&ip=993"
f"&sh={domain}&sp=465"
f"&ic=3&ss=default"
)
future = account.add_transport_from_qr.future(qr)
else:
@@ -360,7 +361,7 @@ class ChatmailACFactory:
# ensure messages stay in INBOX so that they can be
# concurrently fetched via extra IMAP connections during tests
account.set_config("bcc_self", "1")
account.set_config("delete_server_after", "10")
accounts.append(account)
for future in futures:
@@ -415,10 +416,10 @@ class Remote:
def iter_output(self, logcmd="", ready=None):
getjournal = "journalctl -f" if not logcmd else logcmd
print(self.sshdomain)
if self.sshdomain in ("@local", "localhost"):
command = []
else:
command = ["ssh", f"root@{self.sshdomain}"]
match self.sshdomain:
case "@local": command = []
case "localhost": command = []
case _: command = ["ssh", f"root@{self.sshdomain}"]
[command.append(arg) for arg in getjournal.split()]
popen = subprocess.Popen(
command,
@@ -465,11 +466,6 @@ def cmsetup(maildomain, gencreds, ssl_context):
return CMSetup(maildomain, gencreds, ssl_context)
@pytest.fixture
def cmsetup2(maildomain2, gencreds, ssl_context):
return CMSetup(maildomain2, gencreds, ssl_context)
class CMSetup:
def __init__(self, maildomain, gencreds, ssl_context):
self.maildomain = maildomain
@@ -480,7 +476,7 @@ class CMSetup:
print(f"Creating {num} online users")
users = []
for i in range(num):
addr, password = self.gencreds(format_mail_domain(self.maildomain))
addr, password = self.gencreds()
user = CMUser(self.maildomain, addr, password, self.ssl_context)
assert user.smtp
users.append(user)
@@ -39,14 +39,6 @@ class TestCmdline:
out, err = capsys.readouterr()
assert "deleting config file" in out.lower()
def test_dns_skip_on_ip(self, capsys, tmp_path, monkeypatch):
monkeypatch.delenv("CHATMAIL_INI", raising=False)
inipath = tmp_path / "chatmail.ini"
assert main(["init", "--config", str(inipath), "1.3.3.7"]) == 0
assert main(["dns", "--config", str(inipath)]) == 0
out, err = capsys.readouterr()
assert out == "[WARNING] 1.3.3.7 is not a domain, skipping DNS checks.\n"
def test_www_folder(example_config, tmp_path):
reporoot = importlib.resources.files(__package__).joinpath("../../../../").resolve()
@@ -23,7 +23,8 @@ def make_host(*fact_pairs):
if cls not in facts:
registered = ", ".join(c.__name__ for c in facts)
raise LookupError(
f"unexpected get_fact({cls.__name__}); only registered: {registered}"
f"unexpected get_fact({cls.__name__}); "
f"only registered: {registered}"
)
return facts[cls]
@@ -1,7 +1,4 @@
# Managed by cmdeploy
# Managed by cmdeploy: disable IPv6 in unbound.
server:
{% if disable_ipv6 %}
interface: 127.0.0.1
do-ip6: no
{% endif %}
cache-max-negative-ttl: 0
-24
View File
@@ -15,7 +15,6 @@ goes beyond what classic email servers offer:
streaming, privacy-preserving Push Notifications for Apple, Google, and `Ubuntu Touch <https://docs.ubports.com/en/latest/appdev/guides/pushnotifications.html>`_;
- **Security Enforcement**: only strict TLS, DKIM and OpenPGP with minimized metadata accepted
(DKIM is not enforced on :ref:`IP-only relays <iponly>`)
- **Reliable Federation and Decentralization:** No spam or IP reputation checks, federating
depends on established IETF standards and protocols.
@@ -48,29 +47,6 @@ Dovecot, and are configured to run unattended without much maintenance
effort. Chatmail relays happily run on low-end hardware like a Raspberry
Pi.
.. _upgrade:
How can I upgrade my chatmail relay?
------------------------------------
To upgrade to the latest ``main`` branch,
``cd`` into your local checkout of `https://github.com/chatmail/relay/`_
and run the following commands:
::
git pull origin main --rebase --autostash
scripts/initenv.sh
scripts/cmdeploy run
If you don't want the latest development version,
but a specific tagged release like `1.10.0 <https://github.com/chatmail/relay/releases/tag/1.10.0>`_,
run ``git pull origin 1.10.0`` instead.
If you made local changes for your setup,
they will be reapplied as long as they don't conflict with the upgrade.
If a conflict arises, ``git status`` will tell you how to resolve it.
How trustable are chatmail relays?
----------------------------------
+2 -14
View File
@@ -14,6 +14,8 @@ Minimal requirements and prerequisites
You will need the following:
- Control over a domain through a DNS provider of your choice.
- A Debian 12 **deployment server** with reachable SMTP/SUBMISSIONS/IMAPS/HTTPS ports.
IPv6 is encouraged if available. Chatmail relay servers only require
1GB RAM, one CPU, and perhaps 10GB storage for a few thousand active
@@ -26,11 +28,6 @@ You will need the following:
(An ed25519 private key is required due to an `upstream bug in
paramiko <https://github.com/paramiko/paramiko/issues/2191>`_)
- Control over a domain through a DNS provider of your choice
(there is experimental support for :ref:`IP-only relays <iponly>`).
.. _setup:
Setup with ``scripts/cmdeploy``
-------------------------------------
@@ -101,15 +98,6 @@ steps. Please substitute it with your own domain.
configure at your DNS provider (it can take some time until they are
public).
Docker installation
-------------------
There is experimental support for running chatmail via Docker.
A monolithic image based on the above cmdeploy method is available `through a separate repository <https://github.com/chatmail/docker/pkgs/container/docker>`_.
See the `chatmail/docker README <https://github.com/chatmail/docker>`_ for full setup instructions.
Other helpful commands
----------------------
-1
View File
@@ -19,4 +19,3 @@ Contributions and feedback welcome through the https://github.com/chatmail/relay
reverse_dns
related
faq
iponly
-40
View File
@@ -1,40 +0,0 @@
.. _iponly:
Hosting without DNS records
===========================
.. note::
This option is experimental and might change without notice.
In case you don't have a domain,
for example in a local network,
you can run a chatmail relay with only an IPv4 address as well.
To deploy a relay without a domain,
run ``cmdeploy init`` with only the IPv4 address
during the :ref:`installation steps <setup>`,
for example ``cmdeploy init 13.12.23.42``.
Drawbacks
---------
- your transport encryption will only use self-signed TLS certificates,
which are vulnerable against MITM attacks.
the chatmail core's end-to-end encryption should suffice in most scenarios though.
- your messages will not be DKIM-signed;
experimentally, most chatmail relays accept non-DKIM-signed messages from IP-only relays,
but some relays might not accept messages from yours.
Email addresses
---------------
When running without a domain,
your chatmail addresses will use the IPv4 address
in brackets as the domain part,
for example ``user@[13.12.23.42]``.
This is a valid email address format
according to :rfc:`5321`.
+16 -58
View File
@@ -156,7 +156,6 @@ Chatmail relay dependency diagram
postfix --- |10083|filtermail-transport;
filtermail-outgoing --- |10025 reinject|postfix;
filtermail-incoming --- |10026 reinject|postfix;
postfix --- |milter opendkim.sock|OpenDKIM
dovecot --- |doveauth.socket|doveauth;
dovecot --- |message delivery|maildir["maildir
/home/vmail/.../user"];
@@ -180,66 +179,26 @@ Chatmail relay dependency diagram
style nginx-right fill:#f66;
style postfix fill:#f66;
style dovecot fill:#f66;
style OpenDKIM fill:#f66;
style notification-proxy fill:#f66;
Accepting and delivering mail
-----------------------------
Message between users on the same relay
---------------------------------------
.. mermaid::
:caption: This diagram shows all the paths a message can take.
:caption: This diagram shows the path a non-federated message takes.
flowchart LR
subgraph chatmail relay
subgraph postfix
qmgr .-> lmtp-filtermail["lmtp/lmtp-filtermail (default_transport)"]
qmgr .-> lmtp["lmtp (local_transport)"]
lmtp --> cleanup["cleanup (lmtp_header_cleanup)"]
bounce
smtpd-submission["smtpd/submission"]
smtpd-smtps["smtpd/smtps"]
smtpd-reinject-outgoing["smtpd/reinject-outgoing"] --> authclean["cleanup/authclean (submission_header_cleanup)"]
authclean --> qmgr
smtpd-smtp["smtpd/smtp"]
smtpd-reinject-incoming["smtpd/reinject-incoming"] --> qmgr
end
lmtp-filtermail --LMTP inet:10083--> filtermail-transport
cleanup --LMTP unix:private/dovecot-lmtp --> dovecot
dovecot --> maildir
smtpd-submission --SMTP inet:10080--> filtermail-outgoing
smtpd-smtps --SMTP inet:10080--> filtermail-outgoing
filtermail-outgoing --SMTP inet:10025--> smtpd-reinject-outgoing
open-dkim["OpenDKIM (signing only)"] <--milter unix:opendkim/opendkim.sock--> smtpd-reinject-outgoing
bounce <--milter unix:opendkim/opendkim.sock--> open-dkim
bounce --> qmgr
nginx
smtpd-smtp -.SMTP inet:10081.-> filtermail-incoming
nginx -.HTTP inet:10082.-> filtermail-incoming
filtermail-incoming --SMTP inet:10026--> smtpd-reinject-incoming
end
filtermail-transport -.SMTP inet:25.-> mta1[Remote relay]
filtermail-transport -.HTTPS /mxdeliv.-> mta1
client[Client] -.SMTP inet:587.-> smtpd-submission
client -.SMTP inet:465.-> smtpd-smtps
client -.SMTP inet:443.-> nginx
nginx -.SMTP inet:465.-> smtpd-smtps
mta2[Remote relay] -.SMTP inet:25.-> smtpd-smtp
mta2 -.HTTPS /mxdeliv.-> nginx
style postfix fill:#363
style qmgr fill:#252
style authclean fill:#252
style cleanup fill:#252
style lmtp-filtermail fill:#252
style lmtp fill:#252
style bounce fill:#252
style smtpd-submission fill:#252
style smtpd-smtps fill:#252
style smtpd-reinject-outgoing fill:#252
style smtpd-reinject-incoming fill:#252
style smtpd-smtp fill:#252
style filtermail-outgoing fill:#225
style filtermail-incoming fill:#225
style filtermail-transport fill:#225
graph LR;
sender --> |465|smtps/smtpd;
sender --> |587|submission/smtpd;
smtps/smtpd --> |10080|filtermail;
submission/smtpd --> |10080|filtermail;
filtermail --> |10025|smtpd_reinject;
smtpd_reinject --> cleanup;
cleanup --> qmgr;
qmgr --> smtpd_accepts_message;
qmgr --> |lmtp|dovecot;
dovecot --> recipient;
dovecot --> sender's_other_devices;
Operational details of a chatmail relay
----------------------------------------
@@ -306,8 +265,7 @@ from the chatmail relay server.
Email domain authentication (DKIM)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Chatmail relays enforce :rfc:`DKIM <6376>` to authenticate incoming emails
(except for :ref:`IP-only relays <iponly>`).
Chatmail relays enforce :rfc:`DKIM <6376>` to authenticate incoming emails.
Incoming emails must have a valid DKIM signature with
Signing Domain Identifier (SDID, ``d=`` parameter in the DKIM-Signature
header) equal to the ``From:`` header domain. This property is checked