Compare commits

...
Author SHA1 Message Date
holger krekel 6cd943afb5 test: allow a repo-root level "pytest -n6" to succeed.
the fixtures from chatmaild and cmdeploy test plugins were clashing.
2026-08-26 18:00:37 +02:00
holger krekel d25e8a8ee8 chore(doc): use sphinx roles for referencing repository files and dirs
this allows PRs to add references without having to point to main where
a file might not be, and thus fail CI.
2026-08-26 15:32:16 +02:00
j4n cae03e2714 chore(cmdeploy): refactor all pins into pins.py 2026-08-26 15:10:34 +02:00
j4n 7db16cc716 feat(mtail): validate programs during deploy
Compile /etc/mtail before activate() restarts the service, to catch errors
early.
2026-08-26 13:19:54 +02:00
j4n 4cdccee63b feat(mtail): deploy filtermail.mtail and gate mtail rule copy on mtail_address
Deploy filtermail.mtail program along delivered_mail.mtail, fetched from
upstream; for this, refactor download_executable to accept mode, so we
can use it to upload non-binaries.
refactor: make hashes (for uniformity) and mtail version (for use by
mtail deployer) module constants.

Additionally, gate both mtail programs on mtail_address being set.
2026-08-26 13:19:54 +02:00
holger krekel 12664d9188 chore: follow the new mtail release source, upgrade 3.0.8 to 3.4.9
see https://github.com/google/mtail/issues/929 for the move.
2026-08-25 19:36:20 +02:00
holger krekel 1f0ddb7e5b chore: un-hardcode executable paths in some systemd service files
makes it consistent with the other services.
2026-08-25 17:30:32 +02:00
holger krekel 2af8d0e7b5 test: integrate lua testing into regular pytest run for push notifications
turns out Python has the nice https://pypi.org/project/lupa/
that allows us to quite easily test dovecot LUA parts
without figuring out errors on deploy.
2026-08-25 11:38:17 +02:00
holger krekel e489a1ea29 ci: try to fix lack of RFC822 item support in madmail and make CI pass
some local debugging revealed madmail v2 does not support fetching RFC822 items
(gives an empty body) so try BODY.PEEK[] instead.
2026-08-24 18:27:14 +02:00
adbenitezandholger krekel 455da45d36 update gplay to 2.59.1 2026-08-24 17:51:05 +02:00
KamyarATandholger krekel 48ad92bf24 docs: describe Madmail v2 as a Rust chatmail relay
The previous blurb still called Madmail an experimental Maddy fork.
That applied to v1. v2 is a Rust rewrite that ships SMTP, IMAP,
encryption enforcement, and real-time services in a single binary.
2026-08-20 15:36:27 +02:00
holger krekel 33f9cddb1b feat: serve an APPVERSIONS.json index file to clients via IMAP metadata
This is designed to help implement self-updating APKs (and later other clients),
see counterpart https://github.com/chatmail/core/pull/8557
2026-08-13 12:54:47 +02:00
missytakeandGitHub dc8e0a34a2 chore(release): prepare for 1.12.0 (#1034) 2026-07-31 11:48:16 +02:00
holger krekel efc24fcdf3 cleanup: contents:read not needed for relay repo
public repos need no contents::read and there were permissions: {}
2026-07-30 20:44:04 +02:00
missytake 9a9bda80b1 fix: ss -tulpn can sometimes show dovecot first 2026-07-30 12:12:27 +02:00
missytake 74f4721f2b ci: fix docs upload path 2026-07-30 09:38:41 +02:00
holger krekel 14f829003d fix: List Iroh proxy endpoints used by 0.35 and 1.0, drop stale /relay/probe from earlier versions 2026-07-29 22:44:21 +02:00
link2xt 5da9aaeb37 refactor: install dns-root-data instead of using unbound-anchor
unbound-anchor retrieves root trust anchor directly via HTTPS,
but Debian already distributes the same files in dns-root-data package.
2026-07-29 19:28:06 +00:00
holger krekel d068052f0c restrict usable workflow to read permissions 2026-07-29 14:15:35 +02:00
holger krekel fa5afadaaa feat!: introduce configurable system limits to reject new address creation and limit imap/smtp connections.
The default values are geared towards minimal-requirements server.
If you have a big server, you will need to set chatmail.ini parameters.
2026-07-28 19:08:19 +02:00
j4n eb862f1645 chore(ci): update doc staging upload path 2026-07-27 17:38:50 +02:00
holger krekel 6f3039509c feat(dovecot): advertise privacy_mail as admin contact, drop server comment
- The hardcoded root@relaydomain address was unusued/unrooted
  and no one knew about it, it was used nowhere.
  Better to use privacy_mail which operators can set
  which might eventually be shown in the UI's relay list.

- remove generic useless mail_server_comment string.
  Clients receive these values as IMAP METADATA /shared/admin and
  /shared/comment only for internal debugging/info purposes.
2026-07-26 14:13:08 +02:00
link2xt 504fb249a9 refactor(postfix): remove unused "filter" lmtp service
It looks like the reason it exists in the default configuration
is an example for <https://www.postfix.org/FILTER_README.html>
Using this filter requires configuration such as
"-o content_filter=filter:dummy" and we don't have it.
2026-07-22 20:28:36 +00:00
link2xt deca1ef2a3 feat(postfix): disable processing of MIME headers
Default behavior is processing headers
that are actually part of the message body.
This is unlikely to be needed
as headers such as Received are not added into MIME parts,
but may result in broken DKIM signatures
if the body is modified.
2026-07-13 09:46:32 +00:00
W0rmsyandGitHub 0323e757a9 chore: update Let's Encrypt Subscriber Agreement to 1.8
The changes are described in the announcement:
https://community.letsencrypt.org/t/updating-the-let-s-encrypt-subscriber-agreement-to-v1-8/248355
2026-07-13 09:45:41 +00:00
j4n 752957dae8 feat(mtail): add incoming_mailer_daemon_mail_count
Track volume of mailer-daemon messages separately from unencrypted.
2026-07-13 11:43:44 +02:00
Jagoda Estera ŚlązakandGitHub 3a553f2286 chore(deps): Upgrade filtermail to v0.7.4 (#1014)
## 0.7.4 - 2026-07-01

### Features

- *(logs)* Log incoming mailer-daemon message sources

### Miscellaneous Tasks

- Update filtermail.mtail

### Testing

- Place #[tokio::test] after rstest case macros

## 0.7.3 - 2026-06-27

### Features

- *(transport)* Worker eviction

## 0.7.2 - 2026-06-26

### Documentation

- *(readme)* Disable colors in mermaid diagrams
- *(readme)* Update Transport mode doc

### Features

- *(transport)* Destination worker pool

### Refactor

- Implement Display for AddressDomain

### Testing

- Test filtermail-transport

Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
2026-07-04 15:01:06 +09:00
HocuriandGitHub cb1e4ff5bb docs(README.md): Clarify security enforcement (#1011)
Make it clear which security enforcement properties a chatmail server has
2026-06-17 12:38:37 +02:00
54 changed files with 884 additions and 134 deletions
+1 -1
View File
@@ -29,7 +29,7 @@ jobs:
ref: ${{ github.event.pull_request.head.sha }}
persist-credentials: false
- name: download filtermail
run: curl -L https://github.com/chatmail/filtermail/releases/download/v0.7.1/filtermail-x86_64 -o /usr/local/bin/filtermail && chmod +x /usr/local/bin/filtermail
run: curl -L https://github.com/chatmail/filtermail/releases/download/v0.7.4/filtermail-x86_64 -o /usr/local/bin/filtermail && chmod +x /usr/local/bin/filtermail
- name: run chatmaild tests
working-directory: chatmaild
run: pipx run tox
+6 -1
View File
@@ -13,6 +13,11 @@ jobs:
scripts:
name: build
runs-on: ubuntu-latest
env:
# Pin the repository links in the docs to this pull request's head commit
# so that linkcheck resolves files which only exist on the branch so far.
# see doc/conf.py
DOC_GITHUB_REF: ${{ github.event.pull_request.head.sha }}
environment:
name: 'staging.chatmail.at/doc/relay/'
url: https://staging.chatmail.at/doc/relay/${{ steps.prepare.outputs.prid }}
@@ -51,7 +56,7 @@ jobs:
mkdir -p "$HOME/.ssh"
echo "${{ secrets.CHATMAIL_STAGING_SSHKEY }}" > "$HOME/.ssh/key"
chmod 600 "$HOME/.ssh/key"
rsync -rILvh -e "ssh -i $HOME/.ssh/key -o StrictHostKeyChecking=no" $GITHUB_WORKSPACE/doc/build/ "${{ secrets.USERNAME }}@chatmail.at:/var/www/html/staging.chatmail.at/doc/relay/${STEPS_PREPARE_OUTPUTS_PRID}/"
rsync -rILvh -e "ssh -i $HOME/.ssh/key -o StrictHostKeyChecking=no" $GITHUB_WORKSPACE/doc/build/ "${{ secrets.USERNAME }}@chatmail.at:${STEPS_PREPARE_OUTPUTS_PRID}/"
env:
STEPS_PREPARE_OUTPUTS_PRID: ${{ steps.prepare.outputs.prid }}
+1 -1
View File
@@ -47,5 +47,5 @@ jobs:
mkdir -p "$HOME/.ssh"
echo "${{ secrets.CHATMAIL_STAGING_SSHKEY }}" > "$HOME/.ssh/key"
chmod 600 "$HOME/.ssh/key"
rsync -rILvh -e "ssh -i $HOME/.ssh/key -o StrictHostKeyChecking=no" $GITHUB_WORKSPACE/doc/build/ "${{ secrets.USERNAME }}@chatmail.at:/var/www/html/chatmail.at/doc/relay/"
rsync -rILvh -e "ssh -i $HOME/.ssh/key -o StrictHostKeyChecking=no" $GITHUB_WORKSPACE/doc/build/ "${{ secrets.USERNAME }}@chatmail.at:"
+55
View File
@@ -1,5 +1,60 @@
# Changelog for chatmail deployment
## [1.12.0] - 2026-07-31
### Breaking Changes
- [**breaking**] Introduce configurable system limits to reject new address creation and limit imap/smtp connections.
Dovecot default connection limit lowered from 50k to 10k,
Postfix default connection limit lowered from 5k to 1k,
larger relays need to adjust their settings.
### Features
- Reduce maximal_queue_lifetime from 5d to 2d
- Disable negative cache in unbound (#992)
- *(mtail)* Add incoming_mailer_daemon_mail_count
- *(postfix)* Disable processing of MIME headers
- *(dovecot)* Advertise privacy_mail as admin contact, drop server comment
### Bug Fixes
- Set relay restrictions per smtpd service with default reject
- Reduce maxproc for filtermail-transport LMTP client to 500
- Core 2.50.0 does not have delete_server_after config anymore.
- Check if all required ports are available for filtermail (#983)
- Always deploy unbound.conf.d/chatmail.conf (#993)
- Expire empty directories (#994)
- Crypt-r dependency was declared for wrong Python version
- Always overwrite /etc/resolv.conf, even if it is a symbolic link
- Pass kwargs to files.put()
- List Iroh proxy endpoints used by 0.35 and 1.0, drop stale /relay/probe from earlier versions
- Fix port discovery when ss -tulpn shows dovecot before stats
### Documentation
- Add scripts/initenv.sh to upgrade instructions
- Update overview diagrams (#995)
- *(overview)* Remove mermaid styles from 'Accepting and delivering mail' (#1009)
- *(README.md)* Clarify security enforcement (#1011)
### Miscellaneous Tasks
- *(ci)* Auto-trigger docker build on release tag push
- *(acmetool)* Update let's encrypt ToS link to 1.8
- *(ci)* Update doc staging upload path
- *(ci)* Fix docs upload path
### Refactor
- *(postfix)* Remove unused "filter" lmtp service
- Install dns-root-data instead of using unbound-anchor
- *(deps)* Remove domain-validator dependency
### Testing
- Set socket security for IMAP and SMTP to "TLS" in "dclogin"
## [1.11.0] - 2026-05-15
### Breaking Changes
+3
View File
@@ -5,3 +5,6 @@ We use [git-cliff] to generate the changelog from commit messages before the rel
[Conventional Commits]: https://www.conventionalcommits.org/
[git-cliff]: https://git-cliff.org/
To update client app version information,
edit [chatmaild/src/chatmaild/defaults/appversions.json](chatmaild/src/chatmaild/defaults/appversions.json).
+8 -3
View File
@@ -1,19 +1,24 @@
# Chatmail relays for end-to-end encrypted email
Chatmail relay servers are interoperable Mail Transport Agents (MTAs) designed for:
Chatmail relay servers are interoperable Mail Transport Agents (MTAs) designed for:
- **Zero State:** no private data or metadata collected, messages are auto-deleted, low disk usage
- **Instant/Realtime:** sub-second message delivery, realtime P2P
streaming, privacy-preserving Push Notifications for Apple, Google, and Huawei;
- **Security Enforcement**: only strict TLS, DKIM and OpenPGP with minimized metadata accepted
- **Security Enforcement**: Only connections with strict TLS are accepted;
all messages must be correctly signed with DKIM and OpenPGP-encrypted with minimized metadata.
There are experimental exceptions for no-DNS relays,
which are allowed use self-signed TLS certificates
and which do not need to DKIM-sign their messages.
Unencrypted messages are allowed in neither case.
- **Reliable Federation and Decentralization:** No spam or IP reputation checks, federating
depends on established IETF standards and protocols.
This repository contains everything needed to setup a ready-to-use chatmail relay on an ssh-reachable host.
This repository contains everything needed to setup a ready-to-use chatmail relay on an ssh-reachable host.
For getting started and more information please refer to the web version of this repositories' documentation at
[https://chatmail.at/doc/relay](https://chatmail.at/doc/relay)
+4 -6
View File
@@ -1,15 +1,13 @@
# Releasing a new version of chatmail relay
For example, to release version 1.9.0 of chatmail relay, do the following steps.
For example, to release version 1.13.0 of chatmail relay, do the following steps.
1. Update the changelog: `git cliff --unreleased --tag 1.9.0 --prepend CHANGELOG.md` or `git cliff -u -t 1.9.0 -p CHANGELOG.md`.
1. Update the changelog: `git cliff --unreleased --tag 1.13.0 --prepend CHANGELOG.md` or `git cliff -u -t 1.13.0 -p CHANGELOG.md`.
2. Open the changelog in the editor, edit it if required.
3. Commit the changes to the changelog with a commit message `chore(release): prepare for 1.9.0`.
3. Tag the release: `git tag --annotate 1.9.0`.
4. Open a PR with the new commit, merge it to main after review.
4. Push the release tag: `git push origin 1.9.0`.
5. Create a GitHub release: `gh release create 1.9.0`.
5. In the web interface, create a GitHub release, tell it to create a new tag.
+1
View File
@@ -1,3 +1,4 @@
include src/chatmaild/defaults/*.json
include src/chatmaild/ini/*.ini.f
include src/chatmaild/ini/*.ini
include src/chatmaild/tests/mail-data/*
+1
View File
@@ -8,6 +8,7 @@ version = "0.3"
dependencies = [
"iniconfig",
"filelock",
"psutil",
"requests",
"crypt-r >= 3.13.1 ; python_version >= '3.13'",
]
+10
View File
@@ -75,6 +75,16 @@ class Config:
self.privacy_pdo = params.pop("privacy_pdo", None)
self.privacy_supervisor = params.pop("privacy_supervisor", None)
self.max_load_1m = float(params.pop("max_load_1m", 5))
self.min_available_memory_mb = parse_size_mb(
params.pop("min_available_memory", "200M")
)
self.min_free_disk_space_mb = parse_size_mb(
params.pop("min_free_disk_space", "1G")
)
self.max_imap_connections = int(params.pop("max_imap_connections", 10000))
self.max_smtp_connections = int(params.pop("max_smtp_connections", 1000))
# TLS certificate management.
# If tls_external_cert_and_key is set, use externally managed certs.
# Otherwise derived from the domain name:
@@ -0,0 +1,15 @@
{
"clients": [
{
"clientId": "deltachat",
"sources": [
{
"sourceId": "gplay",
"versionInteger": 757,
"versionString": "2.59.1",
"downloadUrl": "https://github.com/deltachat/deltachat-android/releases/download/v2.59.1/deltachat-gplay-release-2.59.1.apk"
}
]
}
]
}
+3
View File
@@ -14,6 +14,7 @@ except ImportError:
from .config import Config, read_config
from .dictproxy import DictProxy
from .migrate_db import migrate_from_db_to_maildir
from .syslimits import has_sufficient_resources
NOCREATE_FILE = "/etc/chatmail-nocreate"
VALID_LOCALPART_RE = re.compile(r"^[a-z0-9._-]+$")
@@ -147,6 +148,8 @@ class AuthDictProxy(DictProxy):
return userdata
if not is_allowed_to_create(self.config, addr, cleartext_password):
return
if not has_sufficient_resources(self.config):
return
lock = filelock.FileLock(str(user.password_path) + ".lock", timeout=5)
with lock:
@@ -42,6 +42,33 @@ mail_domain = {mail_domain}
# minimum length a password must have
#password_min_length = 9
#
# System resource limits
#
# The following three limits refuse creation of new addresses
# while existing addresses keep working.
# Rejections are logged by the doveauth service.
# Maximum 1-minute load average, as reported by "uptime";
# it counts processes waiting for disk I/O as well as for CPU.
#max_load_1m = 5
# Minimum memory available without swapping.
#min_available_memory = 200M
# Minimum free disk space on the file system holding the mailboxes.
#min_free_disk_space = 1G
# Maximum number of concurrent IMAP connections
# (the Dovecot imap process limit).
#max_imap_connections = 10000
# Maximum number of concurrent SMTP connections
# on each of the submission and smtps ports (the Postfix process limit).
# A single client IP may use up to a fifth of this.
#max_smtp_connections = 1000
# Use externally managed TLS certificates instead of built-in acmetool.
# Paths refer to files on the deployment server (not the build machine).
# Both files must already exist before running cmdeploy.
+18
View File
@@ -1,8 +1,10 @@
import json
import logging
import socket
import sys
import time
from contextlib import contextmanager
from importlib.resources import files
from .config import read_config
from .dictproxy import DictProxy
@@ -18,6 +20,18 @@ def turn_credentials(turn_socket_path):
return file.readline().decode("utf-8").strip()
def read_appversions(path):
try:
data = json.loads(path.read_bytes())
except FileNotFoundError:
return None
except (OSError, ValueError):
logging.exception(f"failed to read {path}")
return None
# the dict protocol is line-based, keep the value single-line
return json.dumps(data, separators=(",", ":"))
def _is_valid_token_timestamp(timestamp, now):
# Token if invalid after 90 days
# or if the timestamp is in the future.
@@ -101,6 +115,7 @@ class MetadataDictProxy(DictProxy):
self.iroh_relay = iroh_relay
self.turn_hostname = turn_hostname
self.turn_socket_path = turn_socket_path
self.appversions_path = files(__package__).joinpath("defaults/appversions.json")
def handle_lookup(self, parts):
# Lpriv/43f5f508a7ea0366dff30200c15250e3/devicetoken\tlkj123poi@c2.testrun.org
@@ -125,6 +140,9 @@ class MetadataDictProxy(DictProxy):
case "maxsmtprecipients":
# postfix default (see "postconf smtpd_recipient_limit")
return "O1000\n"
case "appversions":
value = read_appversions(self.appversions_path)
return f"O{value}\n" if value else "N\n"
logging.warning(f"lookup ignored: {parts!r}")
return "N\n"
+32
View File
@@ -0,0 +1,32 @@
"""Detect whether the system is at its limits."""
import logging
import psutil
MB = 1024 * 1024
def read_value(getter):
try:
return getter()
except Exception as e:
logging.warning("ignoring unreadable system limit: %s", e)
return None
def has_sufficient_resources(config):
"""Return False if load, memory or disk exceeds a configured limit."""
load = read_value(lambda: psutil.getloadavg()[0])
mem = read_value(lambda: psutil.virtual_memory().available // MB)
disk = read_value(lambda: psutil.disk_usage(str(config.mailboxes_dir)).free // MB)
if load is not None and load > config.max_load_1m:
msg = f"load avg {load:.2f} > {config.max_load_1m:.2f}"
elif mem is not None and mem < config.min_available_memory_mb:
msg = f"available memory {mem}MB < {config.min_available_memory_mb}MB"
elif disk is not None and disk < config.min_free_disk_space_mb:
msg = f"free disk {disk}MB < {config.min_free_disk_space_mb}MB"
else:
return True
logging.warning("registration rejected: %s", msg)
return False
+9 -5
View File
@@ -20,6 +20,10 @@ def make_config(tmp_path):
basedir.mkdir(parents=True, exist_ok=True)
overrides = settings.copy() if settings else {}
overrides["mailboxes_dir"] = str(basedir)
# permissive resource limits so tests never depend on host load/memory/disk
overrides.setdefault("max_load_1m", "99999")
overrides.setdefault("min_available_memory", "0")
overrides.setdefault("min_free_disk_space", "0")
write_initial_config(inipath, mail_domain, overrides=overrides)
return read_config(inipath)
@@ -37,22 +41,22 @@ def ipv4_config(make_config):
@pytest.fixture
def maildomain(example_config):
def example_maildomain(example_config):
return example_config.mail_domain
@pytest.fixture
def testaddr(maildomain):
return f"user.name@{maildomain}"
def testaddr(example_maildomain):
return f"user.name@{example_maildomain}"
@pytest.fixture
def gencreds(maildomain):
def example_gencreds(example_maildomain):
count = itertools.count()
next(count)
def gen(domain=None):
domain = domain if domain else maildomain
domain = domain if domain else example_maildomain
while 1:
num = next(count)
alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
@@ -0,0 +1,96 @@
import json
import pytest
from chatmaild.metadata import MetadataDictProxy
ALLOWED_URL_PREFIXES = (
"https://github.com/deltachat/",
"https://download.delta.chat/",
)
def check_string(value):
assert isinstance(value, str), value
assert value
def check_version_integer(value):
# core parses this as u32, see https://github.com/chatmail/core/pull/8557
assert isinstance(value, int) and not isinstance(value, bool), value
assert 0 <= value < 2**32, value
def check_appversions(data):
"""Verifies the file the way core parses it.
core deserializes into typed structs and drops the whole payload
of a relay if a single value has an unexpected type,
while missing or misspelled keys silently turn into defaults.
"""
assert set(data) == {"clients"}, data
assert isinstance(data["clients"], list)
assert data["clients"]
client_ids = []
for client in data["clients"]:
assert set(client) == {"clientId", "sources"}, client
check_string(client["clientId"])
client_ids.append(client["clientId"])
assert isinstance(client["sources"], list)
assert client["sources"]
source_ids = []
for source in client["sources"]:
assert set(source) == {
"sourceId",
"versionInteger",
"versionString",
"downloadUrl",
}, source
check_string(source["sourceId"])
source_ids.append(source["sourceId"])
check_version_integer(source["versionInteger"])
check_string(source["versionString"])
check_string(source["downloadUrl"])
assert source["downloadUrl"].startswith(ALLOWED_URL_PREFIXES)
# core takes the first matching source, later duplicates never surface
assert len(set(source_ids)) == len(source_ids), source_ids
assert len(set(client_ids)) == len(client_ids), client_ids
@pytest.fixture
def appversions():
# check the file which chatmail-metadata actually serves
path = MetadataDictProxy(notifier=None, metadata=None).appversions_path
return json.loads(path.read_text())
def test_appversions_schema(appversions):
check_appversions(appversions)
@pytest.mark.parametrize("value", [True, -1, 2**32, "754", 754.0, None])
def test_version_integer_rejected(appversions, value):
appversions["clients"][0]["sources"][0]["versionInteger"] = value
with pytest.raises(AssertionError):
check_appversions(appversions)
@pytest.mark.parametrize("key", ["clientId", "sources"])
def test_misspelled_client_key_rejected(appversions, key):
client = appversions["clients"][0]
client[key + "s"] = client.pop(key)
with pytest.raises(AssertionError):
check_appversions(appversions)
def test_duplicate_source_id_rejected(appversions):
sources = appversions["clients"][0]["sources"]
sources.append(dict(sources[0]))
with pytest.raises(AssertionError):
check_appversions(appversions)
def test_foreign_download_url_rejected(appversions):
appversions["clients"][0]["sources"][0]["downloadUrl"] = "https://example.org/x.apk"
with pytest.raises(AssertionError):
check_appversions(appversions)
+4 -2
View File
@@ -30,9 +30,9 @@ def test_read_config_ipv4(ipv4_config):
assert ipv4_config.mail_domain == "[1.3.3.7]"
def test_read_config_basic_using_defaults(tmp_path, maildomain):
def test_read_config_basic_using_defaults(tmp_path, example_maildomain):
inipath = tmp_path.joinpath("chatmail.ini")
inipath.write_text(f"[params]\nmail_domain = {maildomain}")
inipath.write_text(f"[params]\nmail_domain = {example_maildomain}")
example_config = read_config(inipath)
assert example_config.max_user_send_per_minute == 60
assert example_config.filtermail_smtp_port_incoming == 10081
@@ -45,6 +45,8 @@ def test_read_config_basic_using_defaults(tmp_path, maildomain):
assert example_config.username_min_length == 9
assert example_config.username_max_length == 9
assert example_config.password_min_length == 9
assert example_config.max_imap_connections == 10000
assert example_config.max_smtp_connections == 1000
assert example_config._unused_keys == []
+19 -5
View File
@@ -19,8 +19,8 @@ def dictproxy(example_config):
return AuthDictProxy(config=example_config)
def test_basic(dictproxy, gencreds):
addr, password = gencreds()
def test_basic(dictproxy, example_gencreds):
addr, password = example_gencreds()
dictproxy.lookup_passdb(addr, password)
data = dictproxy.lookup_userdb(addr)
assert data
@@ -107,7 +107,7 @@ def test_handle_dovecot_protocol_user_not_exists(example_config):
assert wfile.getvalue() == b"N\n"
def test_handle_dovecot_protocol_iterate(gencreds, example_config):
def test_handle_dovecot_protocol_iterate(example_config):
dictproxy = AuthDictProxy(config=example_config)
dictproxy.lookup_passdb("asdf00000@chat.example.org", "q9mr3faue")
dictproxy.lookup_passdb("asdf11111@chat.example.org", "q9mr3faue")
@@ -174,14 +174,14 @@ def test_concurrent_creation_same_account(dictproxy):
assert len(passwords_seen) == 1
def test_50_concurrent_lookups_different_accounts(gencreds, dictproxy):
def test_50_concurrent_lookups_different_accounts(example_gencreds, dictproxy):
num_threads = 50
req_per_thread = 5
results = queue.Queue()
def lookup():
for i in range(req_per_thread):
addr, password = gencreds()
addr, password = example_gencreds()
try:
dictproxy.lookup_passdb(addr, password)
except Exception:
@@ -202,3 +202,17 @@ def test_50_concurrent_lookups_different_accounts(gencreds, dictproxy):
res = results.get()
if res is not None:
pytest.fail(f"concurrent lookup failed\n{res}")
def test_insufficient_resources_block_creation_not_existing_logins(
dictproxy, example_gencreds, monkeypatch
):
addr, password = example_gencreds()
assert dictproxy.lookup_passdb(addr, password)
monkeypatch.setattr(
chatmaild.doveauth, "has_sufficient_resources", lambda config: False
)
newaddr, newpassword = example_gencreds()
assert not dictproxy.lookup_passdb(newaddr, newpassword)
assert dictproxy.lookup_passdb(addr, password)
@@ -1,4 +1,5 @@
import io
import json
import time
import pytest
@@ -7,6 +8,7 @@ import requests
from chatmaild.metadata import (
Metadata,
MetadataDictProxy,
read_appversions,
)
from chatmaild.notifier import (
Notifier,
@@ -369,6 +371,32 @@ def test_iroh_relay(dictproxy):
assert wfile.getvalue() == b"Ohttps://example.org/\n"
def test_read_appversions(tmp_path):
path = tmp_path.joinpath("appversions.json")
assert read_appversions(path) is None
path.write_text('{\n "clients": []\n}')
assert read_appversions(path) == '{"clients":[]}'
# the value travels as a single dict protocol line
path.write_text('{"clients": [{"clientId": "one\\ntwo"}]}')
assert read_appversions(path) == '{"clients":[{"clientId":"one\\ntwo"}]}'
path.write_text("bad json")
assert read_appversions(path) is None
def test_appversions_lookup(dictproxy):
# the version information shipped with chatmaild is served as a single line
key = b"Lshared/0123/vendor/vendor.dovecot/pvt/server/vendor/deltachat/appversions"
key += b"\tuser@example.org"
rfile, wfile = io.BytesIO(b"H\n" + key), io.BytesIO()
dictproxy.loop_forever(rfile, wfile)
value = wfile.getvalue()
assert value.startswith(b"O") and value.endswith(b"\n")
assert json.loads(value[1:])["clients"]
def test_legacy_token_migration(metadata, testaddr):
with metadata.get_metadata_dict(testaddr).modify() as data:
data[metadata.DEVICETOKEN_KEY] = ["oldtoken1", "oldtoken2"]
@@ -48,7 +48,7 @@ def test_create_dclogin_url_ipv4(ipv4_config):
assert addr in url
def test_print_new_account(capsys, monkeypatch, maildomain, tmpdir, example_config):
def test_print_new_account(capsys, monkeypatch, tmpdir, example_config):
monkeypatch.setattr(chatmaild.newemail, "CONFIG_PATH", str(example_config._inipath))
print_new_account()
out, err = capsys.readouterr()
@@ -0,0 +1,45 @@
import shutil
import psutil
from chatmaild.syslimits import has_sufficient_resources
PERMISSIVE = {
"max_load_1m": "99999",
"min_available_memory": "0",
"min_free_disk_space": "0",
}
def test_rejects_constrained_system(make_config, caplog):
assert has_sufficient_resources(make_config("chat.example.org", PERMISSIVE))
for settings in (
{"max_load_1m": "-1.0"},
{"min_available_memory": "99999999G"},
{"min_free_disk_space": "99999999G"},
):
config = make_config("chat.example.org", PERMISSIVE | settings)
caplog.clear()
assert not has_sufficient_resources(config), settings
assert "registration rejected" in caplog.text
def test_unreadable_disk_does_not_reject(make_config, caplog):
config = make_config(
"chat.example.org", PERMISSIVE | {"min_free_disk_space": "99999999G"}
)
shutil.rmtree(config.mailboxes_dir)
assert has_sufficient_resources(config)
assert "ignoring" in caplog.text
def test_one_unreadable_value_keeps_other_checks(make_config, monkeypatch, caplog):
def raise_error(*args):
raise psutil.Error("dud")
monkeypatch.setattr(psutil, "getloadavg", raise_error)
config = make_config(
"chat.example.org", PERMISSIVE | {"min_free_disk_space": "99999999G"}
)
assert not has_sufficient_resources(config)
assert "ignoring" in caplog.text
+1
View File
@@ -19,6 +19,7 @@ dependencies = [
"pytest-xdist",
"execnet",
"imap_tools",
"lupa",
"deltachat-rpc-client",
"deltachat-rpc-server",
]
+9 -3
View File
@@ -4,6 +4,8 @@ from ..basedeploy import Deployer
class AcmetoolDeployer(Deployer):
bin_path = "/usr/bin/acmetool"
def __init__(self, email, domains):
self.domains = domains
self.email = email
@@ -41,8 +43,12 @@ class AcmetoolDeployer(Deployer):
domains=self.domains,
)
self.ensure_systemd_unit("acmetool/acmetool-redirector.service")
self.ensure_systemd_unit("acmetool/acmetool-reconcile.service")
self.ensure_systemd_unit(
"acmetool/acmetool-redirector.service.j2", bin_path=self.bin_path
)
self.ensure_systemd_unit(
"acmetool/acmetool-reconcile.service.j2", bin_path=self.bin_path
)
self.ensure_systemd_unit("acmetool/acmetool-reconcile.timer")
def activate(self):
@@ -52,5 +58,5 @@ class AcmetoolDeployer(Deployer):
server.shell(
name=f"Reconcile certificates for: {', '.join(self.domains)}",
commands=["acmetool --batch --xlog.severity=debug reconcile"],
commands=[f"{self.bin_path} --batch --xlog.severity=debug reconcile"],
)
@@ -4,5 +4,5 @@ After=network.target
[Service]
Type=oneshot
ExecStart=/usr/bin/acmetool --batch reconcile
ExecStart={{ bin_path }} --batch reconcile
@@ -3,7 +3,7 @@ Description=acmetool HTTP redirector
[Service]
Type=notify
ExecStart=/usr/bin/acmetool redirector --service.uid=daemon --bind=127.0.0.1:402
ExecStart={{ bin_path }} redirector --service.uid=daemon --bind=127.0.0.1:402
Restart=always
RestartSec=30
@@ -1,2 +1,2 @@
"acme-enter-email": "{{ email }}"
"acme-agreement:https://letsencrypt.org/documents/LE-SA-v1.7-June-04-2026.pdf": true
"acme-agreement:https://letsencrypt.org/documents/LE-SA-v1.8-July-06-2026.pdf": true
+4 -3
View File
@@ -51,7 +51,7 @@ def get_resource(arg, pkg=__package__):
return importlib.resources.files(pkg).joinpath(arg)
def configure_remote_units(deployer, mail_domain, units) -> None:
def configure_remote_units(deployer, mail_domain, units, **kwargs) -> None:
remote_base_dir = "/usr/local/lib/chatmaild"
remote_venv_dir = f"{remote_base_dir}/venv"
remote_chatmail_inipath = f"{remote_base_dir}/chatmail.ini"
@@ -63,6 +63,7 @@ def configure_remote_units(deployer, mail_domain, units) -> None:
config_path=remote_chatmail_inipath,
remote_venv_dir=remote_venv_dir,
mail_domain=mail_domain,
**kwargs,
)
basename = fn if "." in fn else f"{fn}.service"
@@ -226,7 +227,7 @@ class Deployer:
res = files.directory(name=name, path=path, present=False, **kwargs)
return self._update_restart_signals(path, res)
def download_executable(self, url, dest, sha256sum, extract=None):
def download_executable(self, url, dest, sha256sum, extract=None, mode="755"):
existing = host.get_fact(Sha256File, dest)
if existing == sha256sum:
return
@@ -243,7 +244,7 @@ class Deployer:
f"({dl_cmd}"
f" && echo '{sha256sum} {tmp}' | sha256sum -c"
f" && mv {tmp} {dest})",
f"chmod 755 {dest}",
f"chmod {mode} {dest}",
],
)
self.need_restart = True
+44 -34
View File
@@ -33,6 +33,7 @@ from .filtermail.deployer import FiltermailDeployer
from .mtail.deployer import MtailDeployer
from .nginx.deployer import NginxDeployer
from .opendkim.deployer import OpendkimDeployer
from .pins import IROH_ARTIFACTS, TURN_ARTIFACTS
from .postfix.deployer import PostfixDeployer
from .selfsigned.deployer import SelfSignedTlsDeployer
from .www import build_webpages, find_merge_conflict, get_paths
@@ -138,9 +139,22 @@ class UnboundDeployer(Deployer):
# On an IPv4-only system, if unbound is started but not configured,
# it causes subsequent steps to fail to resolve hosts.
with blocked_service_startup():
# dns-root-data is an optional package
# that contains /usr/share/dns/root.key
#
# This file is copied into /var/lib/unbound/root.key
# at the start of "unbound" systemd unit
# by /usr/libexec/unbound-helper shell script
# from the "unbound" package as of version 1.17.1-2+deb12u4
#
# The same /var/lib/unbound/root.key can be retrieved directly
# following the procedure from
# <https://www.rfc-editor.org/info/rfc7958/#section-3.1>
# with "unbound-anchor -a /var/lib/unbound/root.key"
# We don't install and use "unbound-anchor".
apt.packages(
name="Install unbound",
packages=["unbound", "unbound-anchor", "dnsutils"],
packages=["unbound", "dns-root-data", "dnsutils"],
)
def configure(self):
@@ -166,12 +180,6 @@ class UnboundDeployer(Deployer):
dest="/etc/resolv.conf",
force=True,
)
server.shell(
name="Generate root keys for validating DNSSEC",
commands=[
"unbound-anchor -a /var/lib/unbound/root.key || true",
],
)
self.ensure_directory(
path="/etc/unbound/unbound.conf.d",
)
@@ -246,6 +254,13 @@ class LegacyRemoveDeployer(Deployer):
def install(self):
apt.packages(name="Remove rspamd", packages="rspamd", present=False)
# unbound-anchor was used to download /var/lib/unbound/root.key
# It is replaced by dns-root-data which contains /usr/share/dns/root.key.
# unbound systemd unit copies /usr/share/dns/root.key
# into /var/lib/unbound/root.key automatically on start
# as long as /usr/share/dns/root.key is present.
apt.packages(name="Remove unbound-anchor", packages="unbound-anchor", present=False)
# remove historic expunge script
# which is now implemented through a systemd timer (chatmail-expire)
self.remove_file("/etc/cron.d/expunge")
@@ -287,55 +302,48 @@ def check_config(config):
class TurnDeployer(Deployer):
bin_path = "/usr/local/bin/chatmail-turn"
def __init__(self, mail_domain):
self.mail_domain = mail_domain
self.units = ["turnserver"]
def install(self):
(url, sha256sum) = {
"x86_64": (
"https://github.com/chatmail/chatmail-turn/releases/download/v0.4/chatmail-turn-x86_64-linux",
"1ec1f5c50122165e858a5a91bcba9037a28aa8cb8b64b8db570aa457c6141a8a",
),
"aarch64": (
"https://github.com/chatmail/chatmail-turn/releases/download/v0.4/chatmail-turn-aarch64-linux",
"0fb3e792419494e21ecad536464929dba706bb2c88884ed8f1788141d26fc756",
),
}[host.get_fact(facts.server.Arch)]
self.download_executable(url, "/usr/local/bin/chatmail-turn", sha256sum)
(url, sha256sum) = TURN_ARTIFACTS[host.get_fact(facts.server.Arch)]
self.download_executable(url, self.bin_path, sha256sum)
def configure(self):
configure_remote_units(self, self.mail_domain, self.units)
configure_remote_units(
self, self.mail_domain, self.units, bin_path=self.bin_path
)
def activate(self):
activate_remote_units(self, self.units)
class IrohDeployer(Deployer):
bin_path = "/usr/local/bin/iroh-relay"
config_path = "/etc/iroh-relay.toml"
def __init__(self, enable_iroh_relay):
self.enable_iroh_relay = enable_iroh_relay
def install(self):
(url, sha256sum) = {
"x86_64": (
"https://github.com/n0-computer/iroh/releases/download/v0.35.0/iroh-relay-v0.35.0-x86_64-unknown-linux-musl.tar.gz",
"45c81199dbd70f8c4c30fef7f3b9727ca6e3cea8f2831333eeaf8aa71bf0fac1",
),
"aarch64": (
"https://github.com/n0-computer/iroh/releases/download/v0.35.0/iroh-relay-v0.35.0-aarch64-unknown-linux-musl.tar.gz",
"f8ef27631fac213b3ef668d02acd5b3e215292746a3fc71d90c63115446008b1",
),
}[host.get_fact(facts.server.Arch)]
(url, sha256sum) = IROH_ARTIFACTS[host.get_fact(facts.server.Arch)]
self.download_executable(
url,
"/usr/local/bin/iroh-relay",
self.bin_path,
sha256sum,
extract="gunzip | tar -xf - ./iroh-relay -O",
)
def configure(self):
self.ensure_systemd_unit("iroh-relay.service")
self.put_file("iroh-relay.toml", "/etc/iroh-relay.toml")
self.ensure_systemd_unit(
"iroh-relay.service.j2",
bin_path=self.bin_path,
config_path=self.config_path,
)
self.put_file("iroh-relay.toml", self.config_path)
def activate(self):
self.ensure_service(
@@ -356,6 +364,8 @@ class ChatmailVenvDeployer(Deployer):
def __init__(self, config):
self.config = config
self.units = (
# doveauth must restart when chatmaild/ini file changes
"doveauth",
"chatmail-metadata",
"lastlogin",
"chatmail-expire",
@@ -504,10 +514,10 @@ def deploy_chatmail(config_path: Path, disable_mail: bool, website_only: bool) -
("nginx", 443),
(["master", "smtpd"], 465),
(["master", "smtpd"], 587),
(["imap-login", "dovecot"], 993),
(["dovecot", "imap-login"], 993),
("iroh-relay", 3340),
("mtail", 3903),
("stats", 3904),
(["dovecot", "stats"], 3904),
("nginx", 8443),
(["master", "smtpd"], config.postfix_reinject_port),
(["master", "smtpd"], config.postfix_reinject_port_incoming),
+4 -13
View File
@@ -14,18 +14,9 @@ from cmdeploy.basedeploy import (
configure_remote_units,
is_in_container,
)
from cmdeploy.pins import DOVECOT_SHA256, DOVECOT_VERSION
DOVECOT_ARCHIVE_VERSION = "2.3.21+dfsg1-3"
DOVECOT_PACKAGE_VERSION = f"1:{DOVECOT_ARCHIVE_VERSION}"
DOVECOT_SHA256 = {
("core", "amd64"): "dd060706f52a306fa863d874717210b9fe10536c824afe1790eec247ded5b27d",
("core", "arm64"): "e7548e8a82929722e973629ecc40fcfa886894cef3db88f23535149e7f730dc9",
("imapd", "amd64"): "8d8dc6fc00bbb6cdb25d345844f41ce2f1c53f764b79a838eb2a03103eebfa86",
("imapd", "arm64"): "178fa877ddd5df9930e8308b518f4b07df10e759050725f8217a0c1fb3fd707f",
("lmtpd", "amd64"): "2f69ba5e35363de50962d42cccbfe4ed8495265044e244007d7ccddad77513ab",
("lmtpd", "arm64"): "89f52fb36524f5877a177dff4a713ba771fd3f91f22ed0af7238d495e143b38f",
}
DOVECOT_PACKAGE_VERSION = f"1:{DOVECOT_VERSION}"
class DovecotDeployer(Deployer):
@@ -34,7 +25,7 @@ class DovecotDeployer(Deployer):
def __init__(self, config, disable_mail):
self.config = config
self.disable_mail = disable_mail
self.units = ["doveauth"]
self.units = []
def install(self):
arch = host.get_fact(Arch)
@@ -117,7 +108,7 @@ def _download_dovecot_package(package: str, arch: str) -> tuple[str | None, bool
if DOVECOT_PACKAGE_VERSION in installed_versions:
return None, False
url_version = DOVECOT_ARCHIVE_VERSION.replace("+", "%2B")
url_version = DOVECOT_VERSION.replace("+", "%2B")
deb_base = f"{pkg_name}_{url_version}_{arch}.deb"
primary_url = f"https://download.delta.chat/dovecot/{deb_base}"
fallback_url = f"https://github.com/chatmail/dovecot/releases/download/upstream%2F{url_version}/{deb_base}"
@@ -37,11 +37,15 @@ default_client_limit = 20000
# the following warning will be logged:
# Warning: service(imap): process_limit (1024) reached, client connections are being dropped
service imap {
process_limit = 50000
process_limit = {{ config.max_imap_connections }}
}
mail_server_admin = mailto:root@{{ config.mail_domain }}
mail_server_comment = Chatmail server
{% if config.privacy_mail %}
# Advertised to clients as IMAP METADATA /shared/admin (RFC 5464).
# The privacy_mail contact from chatmail.ini is used because it is
# the only address set by an operator.
mail_server_admin = mailto:{{ config.privacy_mail }}
{% endif %}
# `zlib` enables compressing messages stored in the maildir.
# See
+2 -5
View File
@@ -3,6 +3,7 @@ import os
from pyinfra import facts, host
from cmdeploy.basedeploy import Deployer
from cmdeploy.pins import FILTERMAIL_ARTIFACTS
class FiltermailDeployer(Deployer):
@@ -20,11 +21,7 @@ class FiltermailDeployer(Deployer):
return
arch = host.get_fact(facts.server.Arch)
url = f"https://github.com/chatmail/filtermail/releases/download/v0.7.1/filtermail-{arch}"
sha256sum = {
"x86_64": "fc2d8141166f8561b9711fb68c5327fc9421f814c46dc69671a4605a95b175c0",
"aarch64": "37e52c5ddb373ef29b5ead89658407c53f48d10ce055a2dbd9c606fa1ebd5f7f",
}[arch]
url, sha256sum = FILTERMAIL_ARTIFACTS[arch]
self.download_executable(url, self.bin_path, sha256sum)
def configure(self):
@@ -2,7 +2,7 @@
Description=Iroh relay
[Service]
ExecStart=/usr/local/bin/iroh-relay --config-path /etc/iroh-relay.toml
ExecStart={{ bin_path }} --config-path {{ config_path }}
Restart=on-failure
RestartSec=5s
User=iroh
@@ -28,6 +28,13 @@ counter created_nonci_accounts
}
}
# doveauth refusing new addresses because a chatmail.ini
# system resource limit is exceeded.
counter rejected_registrations
/registration rejected: / {
rejected_registrations++
}
counter postfix_timeouts
/timeout after DATA/ {
postfix_timeouts++
@@ -73,6 +80,11 @@ counter incoming_unencrypted_mail_count
filtered_incoming_mail_count++
}
counter incoming_mailer_daemon_mail_count
/Incoming: Filtering mailer-daemon message from/ {
incoming_mailer_daemon_mail_count++
filtered_incoming_mail_count++
}
counter rejected_unencrypted_mail_count
/Rejected unencrypted mail/ {
+31 -13
View File
@@ -1,10 +1,14 @@
from pyinfra import facts, host
from pyinfra.operations import apt
from pyinfra.operations import apt, server
from cmdeploy.basedeploy import Deployer
from cmdeploy.pins import FILTERMAIL_ARTIFACTS, MTAIL_ARTIFACTS
class MtailDeployer(Deployer):
bin_path = "/usr/local/bin/mtail"
progs_dir = "/etc/mtail"
def __init__(self, mtail_address):
self.mtail_address = mtail_address
@@ -12,19 +16,10 @@ class MtailDeployer(Deployer):
# Uninstall mtail package to install a static binary.
apt.packages(name="Uninstall mtail", packages=["mtail"], present=False)
(url, sha256sum) = {
"x86_64": (
"https://github.com/google/mtail/releases/download/v3.0.8/mtail_3.0.8_linux_amd64.tar.gz",
"d55cb601049c5e61eabab29998dbbcea95d480e5448544f9470337ba2eea882e",
),
"aarch64": (
"https://github.com/google/mtail/releases/download/v3.0.8/mtail_3.0.8_linux_arm64.tar.gz",
"f748db8ad2a1e0b63684d4c8868cf6a373a20f7e6922e5ece601fff0ee00eb1a",
),
}[host.get_fact(facts.server.Arch)]
(url, sha256sum) = MTAIL_ARTIFACTS[host.get_fact(facts.server.Arch)]
self.download_executable(
url,
"/usr/local/bin/mtail",
self.bin_path,
sha256sum,
extract="gunzip | tar -xf - mtail -O",
)
@@ -36,8 +31,31 @@ class MtailDeployer(Deployer):
"mtail/mtail.service.j2",
address=self.mtail_address or "127.0.0.1",
port=3903,
bin_path=self.bin_path,
progs_dir=self.progs_dir,
)
self.put_file("mtail/delivered_mail.mtail", "/etc/mtail/delivered_mail.mtail")
if self.mtail_address:
self.put_file(
"mtail/delivered_mail.mtail", f"{self.progs_dir}/delivered_mail.mtail"
)
url, sha256sum = FILTERMAIL_ARTIFACTS['mtail']
self.download_executable(
url,
f"{self.progs_dir}/filtermail.mtail",
sha256sum,
mode="644",
)
if self.need_restart:
# Check if all installed mtail rules compile or fail early
# --one_shot to exit, --port 0 to not clash with running mtail.
server.shell(
name="Validate mtail programs",
commands=[
f"timeout 30 {self.bin_path} --compile_only --one_shot"
f" --progs {self.progs_dir} --logs /dev/null"
" --address 127.0.0.1 --port 0"
],
)
def activate(self):
active = bool(self.mtail_address)
+1 -1
View File
@@ -5,7 +5,7 @@ Wants=network-online.target
[Service]
Type=simple
ExecStart=/bin/sh -c "journalctl -f -o short-iso -n 0 | /usr/local/bin/mtail --address={{ address }} --port={{ port }} --progs /etc/mtail --logtostderr --logs -"
ExecStart=/bin/sh -c "journalctl -f -o short-iso -n 0 | {{ bin_path }} --address={{ address }} --port={{ port }} --progs {{ progs_dir }} --logtostderr --logs -"
Restart=on-failure
RestartSec=2s
+4 -1
View File
@@ -128,7 +128,10 @@ http {
proxy_set_header Connection "upgrade";
}
location /relay/probe {
# Endpoints Iroh uses for net_report probes and that clients
# probe to tell whether this relay works. Both paths are served
# by iroh-relay 0.35 and by the 1.0 line.
location /ping {
proxy_pass http://127.0.0.1:3340;
proxy_http_version 1.1;
}
+62
View File
@@ -0,0 +1,62 @@
"""Versions, hashes, and download URLs for pre-built artifacts fetched during deploy."""
FILTERMAIL_VERSION = "v0.7.4"
FILTERMAIL_ARTIFACTS = {
"x86_64": (
f"https://github.com/chatmail/filtermail/releases/download/{FILTERMAIL_VERSION}/filtermail-x86_64",
"484cb8dff083134aefba9fce4a6b7ef4784a0f0e28e5108ecf8bb9e58a44fd2c",
),
"aarch64": (
f"https://github.com/chatmail/filtermail/releases/download/{FILTERMAIL_VERSION}/filtermail-aarch64",
"66aa0ca2ca9add7a12d92883d76f8786384092adfde24a3d3a1d0b1f30d23a9e",
),
"mtail": (
f"https://raw.githubusercontent.com/chatmail/filtermail/{FILTERMAIL_VERSION}/contrib/filtermail.mtail",
"948f688bb89ad47e6eb0fc8fa107e201a689f5adc264ff926be487a2a8562b51",
),
}
MTAIL_VERSION = "3.4.9"
MTAIL_ARTIFACTS = {
"x86_64": (
f"https://github.com/jaqx0r/mtail/releases/download/v{MTAIL_VERSION}/mtail_{MTAIL_VERSION}_linux_amd64.tar.gz",
"55f64a87f71955bb871c724b4aadf19fe9d854e6327196919c7fe44943427eab",
),
"aarch64": (
f"https://github.com/jaqx0r/mtail/releases/download/v{MTAIL_VERSION}/mtail_{MTAIL_VERSION}_linux_arm64.tar.gz",
"e0a2b66b372ca257d7daeb7ba10f9233a2192a1f9057618fccc6be5c854a2a3c",
),
}
DOVECOT_VERSION = "2.3.21+dfsg1-3"
DOVECOT_SHA256 = {
("core", "amd64"): "dd060706f52a306fa863d874717210b9fe10536c824afe1790eec247ded5b27d",
("core", "arm64"): "e7548e8a82929722e973629ecc40fcfa886894cef3db88f23535149e7f730dc9",
("imapd", "amd64"): "8d8dc6fc00bbb6cdb25d345844f41ce2f1c53f764b79a838eb2a03103eebfa86",
("imapd", "arm64"): "178fa877ddd5df9930e8308b518f4b07df10e759050725f8217a0c1fb3fd707f",
("lmtpd", "amd64"): "2f69ba5e35363de50962d42cccbfe4ed8495265044e244007d7ccddad77513ab",
("lmtpd", "arm64"): "89f52fb36524f5877a177dff4a713ba771fd3f91f22ed0af7238d495e143b38f",
}
TURN_VERSION = "v0.4"
TURN_ARTIFACTS = {
"x86_64": (
f"https://github.com/chatmail/chatmail-turn/releases/download/{TURN_VERSION}/chatmail-turn-x86_64-linux",
"1ec1f5c50122165e858a5a91bcba9037a28aa8cb8b64b8db570aa457c6141a8a",
),
"aarch64": (
f"https://github.com/chatmail/chatmail-turn/releases/download/{TURN_VERSION}/chatmail-turn-aarch64-linux",
"0fb3e792419494e21ecad536464929dba706bb2c88884ed8f1788141d26fc756",
),
}
IROH_VERSION = "v0.35.0"
IROH_ARTIFACTS = {
"x86_64": (
f"https://github.com/n0-computer/iroh/releases/download/{IROH_VERSION}/iroh-relay-{IROH_VERSION}-x86_64-unknown-linux-musl.tar.gz",
"45c81199dbd70f8c4c30fef7f3b9727ca6e3cea8f2831333eeaf8aa71bf0fac1",
),
"aarch64": (
f"https://github.com/n0-computer/iroh/releases/download/{IROH_VERSION}/iroh-relay-{IROH_VERSION}-aarch64-unknown-linux-musl.tar.gz",
"f8ef27631fac213b3ef668d02acd5b3e215292746a3fc71d90c63115446008b1",
),
}
+8
View File
@@ -85,6 +85,14 @@ inet_protocols = all
lmtp_header_checks = regexp:/etc/postfix/lmtp_header_cleanup
# Do not apply header checks to MIME headers
# and other headers that are actually part of the message body.
# Reference:
# <https://www.postfix.org/postconf.5.html#disable_mime_input_processing>
# <https://www.postfix.org/header_checks.5.html>
# <https://stump.io/blog/2020/11/29/a-little-gotcha-with-postfixs-header_checks/>
disable_mime_input_processing = yes
mua_client_restrictions = permit_sasl_authenticated, reject
mua_sender_restrictions = reject_sender_login_mismatch, permit_sasl_authenticated, reject
mua_helo_restrictions = permit_mynetworks, reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname, permit
+4 -5
View File
@@ -18,7 +18,7 @@ smtp inet n - y - - smtpd
-o smtpd_tls_mandatory_protocols=>=TLSv1.2
-o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port_incoming }}
-o smtpd_relay_restrictions=reject_unauth_destination
submission inet n - y - 5000 smtpd
submission inet n - y - {{ config.max_smtp_connections }} smtpd
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_tls_mandatory_protocols=>=TLSv1.3
@@ -32,9 +32,9 @@ submission inet n - y - 5000 smtpd
-o smtpd_sender_restrictions=$mua_sender_restrictions
-o smtpd_recipient_restrictions=
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
-o smtpd_client_connection_count_limit=1000
-o smtpd_client_connection_count_limit={{ config.max_smtp_connections // 5 }}
-o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
smtps inet n - y - 5000 smtpd
smtps inet n - y - {{ config.max_smtp_connections }} smtpd
-o syslog_name=postfix/smtps
-o smtpd_tls_wrappermode=yes
-o smtpd_tls_security_level=encrypt
@@ -48,7 +48,7 @@ smtps inet n - y - 5000 smtpd
-o smtpd_sender_restrictions=$mua_sender_restrictions
-o smtpd_recipient_restrictions=
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
-o smtpd_client_connection_count_limit=1000
-o smtpd_client_connection_count_limit={{ config.max_smtp_connections // 5 }}
-o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
#628 inet n - y - - qmqpd
pickup unix n - y 60 1 pickup
@@ -76,7 +76,6 @@ lmtp unix - - y - - lmtp
anvil unix - - y - 1 anvil
scache unix - - y - 1 scache
postlog unix-dgram n - n - 1 postlogd
filter unix - n n - - lmtp
# Local SMTP server for reinjecting outgoing filtered mail.
127.0.0.1:{{ config.postfix_reinject_port }} inet n - n - 100 smtpd
-o syslog_name=postfix/reinject
@@ -5,5 +5,5 @@ After=network.target
[Service]
Type=oneshot
User=vmail
ExecStart=/usr/local/lib/chatmaild/venv/bin/chatmail-expire /usr/local/lib/chatmaild/chatmail.ini -v --remove
ExecStart={execpath} {config_path} -v --remove
@@ -5,5 +5,5 @@ After=network.target
[Service]
Type=oneshot
User=vmail
ExecStart=/usr/local/lib/chatmaild/venv/bin/chatmail-fsreport /usr/local/lib/chatmaild/chatmail.ini
ExecStart={execpath} {config_path}
@@ -5,7 +5,7 @@ After=network.target
[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/chatmail-turn --realm {mail_domain} --socket /run/chatmail-turn/turn.socket
ExecStart={bin_path} --realm {mail_domain} --socket /run/chatmail-turn/turn.socket
# Create /run/chatmail-turn
RuntimeDirectory=chatmail-turn
+25
View File
@@ -0,0 +1,25 @@
"""Run the lua scripts we ship under lupa, which bundles Lua 5.4 like dovecot."""
import pytest
from lupa import lua54
from cmdeploy.basedeploy import get_resource
class Lua:
"""A Lua runtime to load shipped scripts and mocks into."""
def __init__(self):
self.rt = lua54.LuaRuntime(unpack_returned_tuples=True)
self.g = self.rt.globals()
def load(self, path):
self.rt.execute(get_resource(path).read_text())
def table(self, **kwargs):
return self.rt.table(**kwargs)
@pytest.fixture
def lua():
return Lua()
@@ -1,10 +1,12 @@
import ipaddress
import json
import re
import time
import imap_tools
import pytest
import requests
from chatmaild.tests.test_appversions import check_appversions
from cmdeploy.cmdeploy import get_sshexec
from cmdeploy.remote import rshell
@@ -51,6 +53,17 @@ class TestMetadataTokens:
assert res == b"1111 2222"
assert b"Getmetadata completed" in client.readline()
def test_get_appversions(self, imap_mailbox):
"get app version information shipped with the relay"
client = imap_mailbox.client
client.send(b'a01 GETMETADATA "" /shared/vendor/deltachat/appversions\n')
res = client.readline()
assert res[:1] == b"*"
res = client.readline().strip().rstrip(b")")
# the served value is a single line and passes the shipped file's schema
check_appversions(json.loads(res))
assert b"Getmetadata completed" in client.readline()
class TestEndToEndDeltaChat:
"Tests that use Delta Chat accounts on the chat mail instance."
+1 -1
View File
@@ -211,7 +211,7 @@ class ImapConn:
status, res = self.conn.select()
if int(res[0]) == 0:
raise ValueError("no messages in imap folder")
status, results = self.conn.fetch("1:*", "(RFC822)")
status, results = self.conn.fetch("1:*", "(BODY.PEEK[])")
assert status == "OK"
return results
@@ -111,7 +111,7 @@ def test_download_dovecot_package_uses_archive_version_for_url_and_filename(
deb, changed = dovecot_deployer._download_dovecot_package("core", "amd64")
archive_version = dovecot_deployer.DOVECOT_ARCHIVE_VERSION.replace("+", "%2B")
archive_version = dovecot_deployer.DOVECOT_VERSION.replace("+", "%2B")
expected_deb = f"/root/dovecot-core_{archive_version}_amd64.deb"
# Verify the returned path uses archive version, not package version (with epoch)
@@ -0,0 +1,85 @@
"""Test the push_notification.lua we ship, against a mocked dovecot mail API."""
import textwrap
import pytest
USER1 = "user12345@chat.example.org"
USER2 = "user67890@chat.example.org"
DOVECOT_MOCKS = textwrap.dedent("""
function make_user(username)
local function mailbox(_, name)
record("mailbox " .. name)
return {
sync = function() record("sync") end,
metadata_set = function(_, k, v)
record("metadata_set " .. k .. "=" .. v)
end,
free = function() record("free") end,
}
end
return {username = username, mailbox = mailbox}
end
""")
@pytest.fixture
def script(lua):
lua.rt.execute(DOVECOT_MOCKS)
lua.load("dovecot/push_notification.lua")
return lua
@pytest.fixture
def deliver(script):
def deliver(recipient, sender):
calls = []
script.g.record = calls.append
user = script.g.make_user(recipient)
ctx = script.g.dovecot_lua_notify_begin_txn(user)
event = script.table(mailbox="INBOX", from_address=sender)
script.g.dovecot_lua_notify_event_message_new(ctx, event)
script.g.dovecot_lua_notify_end_txn(ctx, True)
return calls
return deliver
def test_entry_points_have_the_names_dovecot_calls(script):
assert script.g.dovecot_lua_notify_begin_txn is not None
assert script.g.dovecot_lua_notify_event_message_new is not None
assert script.g.dovecot_lua_notify_end_txn is not None
def test_begin_txn_returns_the_user_as_event_context(script):
user = script.g.make_user(USER1)
ctx = script.g.dovecot_lua_notify_begin_txn(user)
ctx.marker = "seen"
assert user.marker == "seen"
def test_incoming_message_notifies_metadata_server(deliver):
assert deliver(USER1, sender=USER2) == [
"mailbox INBOX",
"sync",
"metadata_set /private/messagenew=",
"free",
]
def test_own_message_does_not_wake_the_sending_device(deliver):
assert deliver(USER1, sender=USER1) == [
"mailbox INBOX",
"sync",
"free",
]
def test_message_without_from_address_is_notified(deliver):
assert deliver(USER1, sender=None) == [
"mailbox INBOX",
"sync",
"metadata_set /private/messagenew=",
"free",
]
+14
View File
@@ -3,6 +3,8 @@
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
import os
# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
@@ -16,12 +18,24 @@ author = 'chatmail collective'
extensions = [
#'sphinx.ext.autodoc',
#'sphinx.ext.viewdoc',
'sphinx.ext.extlinks',
'sphinxcontrib.mermaid',
]
templates_path = ['_templates']
exclude_patterns = []
# Repository links go through the roles below.
# CI sets DOC_GITHUB_REF to the head commit of a pull request,
gh_ref = os.environ.get("DOC_GITHUB_REF", "main")
extlinks = {
"repofile": (f"https://github.com/chatmail/relay/blob/{gh_ref}/%s", "%s"),
"repodir": (f"https://github.com/chatmail/relay/tree/{gh_ref}/%s", "%s"),
}
# Warn about repository links spelled out in full instead of using the roles.
extlinks_detect_hardcoded_links = True
# -- Options for HTML output -------------------------------------------------
+1 -1
View File
@@ -54,7 +54,7 @@ How can I upgrade my chatmail relay?
------------------------------------
To upgrade to the latest ``main`` branch,
``cd`` into your local checkout of `https://github.com/chatmail/relay/`_
``cd`` into your local checkout of https://github.com/chatmail/relay/
and run the following commands:
::
+87
View File
@@ -199,6 +199,93 @@ creating addresses, login with ssh to the deployment machine and run:
Chatmail address creation will be denied while this file is present.
.. _system-limits:
Configurable System Limits
--------------------------
Limits for auto-rejecting address creation
..........................................
A relay refuses creation of new addresses
when the machine runs low on resources,
but existing addresses keep working.
Three ``chatmail.ini`` settings control this,
shown here with their defaults::
max_load_1m = 5
min_available_memory = 200M
min_free_disk_space = 1G
- ``max_load_1m`` is the maximum 1-minute load average,
as reported by ``uptime``;
it counts processes waiting for disk I/O as well as for CPU.
It is deliberately not scaled by the number of CPUs
because I/O rather than CPU is what typically limits a relay.
- ``min_available_memory`` is the minimum memory available without swapping.
- ``min_free_disk_space`` is the minimum free disk space
on the file system holding the mailboxes.
The defaults suit the small machine described in
`Minimal requirements and prerequisites`_.
.. note::
If you run a bigger machine,
raise ``max_load_1m`` after watching ``uptime`` under typical load.
Rejections are logged by the ``doveauth`` service,
so you can check whether a limit is set too tightly::
journalctl -u doveauth --grep 'registration rejected'
If ``mtail_address`` is set, rejections are also counted
in the ``rejected_registrations`` metric.
Overall IMAP and SMTP connection limits
.......................................
Two further settings bound how many connections
the relay accepts at all, again shown with their defaults::
max_imap_connections = 10000
max_smtp_connections = 1000
``max_imap_connections`` becomes the Dovecot imap process limit,
and ``max_smtp_connections`` the Postfix process limit
on each of the submission and smtps ports.
A single client IP may use up to a fifth of ``max_smtp_connections``.
Each connection costs memory,
so these limits defend the relay against running out of RAM.
Unless ``imap_compress`` is enabled,
an IMAP connection that is idle for ``imap_hibernate_timeout``
is handed over to the ``imap-hibernate`` process
and does not count towards ``max_imap_connections``,
which is why a relay can serve far more IMAP clients
than this setting suggests.
If you run a large relay with 10k or 100k's of addresses,
check current connection counts before upgrading
and set the limits accordingly.
To see how close a running relay is to these two limits,
copy ``scripts/check-connections.sh`` from the relay repository
onto the relay and run it there::
imap 5 ports 143,993 (max_imap_connections)
5 dovecot sessions, 0 of them in an active imap process
submission 0 ports 465,587 (max_smtp_connections per port)
incoming 0 port 25 (from other relays, no chatmail.ini limit)
It counts established sockets with ``ss``
and cross-checks the IMAP number against ``doveadm who``.
Running a relay with self-signed certificates
----------------------------------------------
+39 -14
View File
@@ -6,13 +6,13 @@ Technical overview
Directories of the relay repository
-----------------------------------
The `chatmail relay repository <https://github.com/chatmail/relay/tree/main/>`_
The `chatmail relay repository <https://github.com/chatmail/relay>`_
has four main directories.
``scripts/``
~~~~~~~~~~~~~
`scripts <https://github.com/chatmail/relay/tree/main/scripts>`_
:repodir:`scripts`
offers two convenience tools for beginners:
- ``initenv.sh`` installs a local virtualenv Python environment and
@@ -71,7 +71,7 @@ The deployed system components of a chatmail relay are:
``chatmaild/``
~~~~~~~~~~~~~~
`chatmaild <https://github.com/chatmail/relay/tree/main/chatmaild>`_
:repodir:`chatmaild`
is a Python package containing several small services which handle
authentication, trigger push notifications on new messages, ensure
that outbound mails are encrypted, delete inactive users, and some
@@ -83,25 +83,24 @@ that integrate with Dovecot and Postfix to achieve instant-onboarding
and only relaying OpenPGP end-to-end messages encrypted messages. A
short overview of ``chatmaild`` services:
- `doveauth <https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/doveauth.py>`_
- :repofile:`doveauth <chatmaild/src/chatmaild/doveauth.py>`
implements create-on-login address semantics and is used by Dovecot
during IMAP login and by Postfix during SMTP/SUBMISSION login which
in turn uses `Dovecot SASL
<https://doc.dovecot.org/2.3/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket>`_
to authenticate logins.
- `chatmail-metadata <https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/metadata.py>`_
is contacted by a `Dovecot lua
script <https://github.com/chatmail/relay/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua>`_
to store user-specific relay-side config. On new messages, it `passes
the users push notification
token <https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/notifier.py>`_
- :repofile:`chatmail-metadata <chatmaild/src/chatmaild/metadata.py>`
is contacted by a
:repofile:`Dovecot lua script <cmdeploy/src/cmdeploy/dovecot/push_notification.lua>`
to store user-specific relay-side config. On new messages, it
:repofile:`passes the users push notification token <chatmaild/src/chatmaild/notifier.py>`
to
`notifications.delta.chat <https://delta.chat/en/help#instant-delivery>`_
so the push notifications on the users phone can be triggered by
Apple/Google/Huawei.
- `chatmail-expire <https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/expire.py>`_
- :repofile:`chatmail-expire <chatmaild/src/chatmaild/expire.py>`
deletes old messages, large messages, and entire mailboxes
of users who have not logged in for longer than
``delete_inactive_users_after`` days.
@@ -109,15 +108,14 @@ short overview of ``chatmaild`` services:
- ``chatmail-quota-expire`` is called by Dovecot's ``quota_warning`` mechanism
and will automatically remove oldest messages to keep mailboxes well under ``max_mailbox_size``.
- `lastlogin <https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/lastlogin.py>`_
- :repofile:`lastlogin <chatmaild/src/chatmaild/lastlogin.py>`
is contacted by Dovecot when a user logs in and stores the date of
the login.
``www/``
~~~~~~~~~
`www <https://github.com/chatmail/relay/tree/main/www>`_ contains
the html, css, and markdown files which make up a chatmail relays
:repodir:`www` contains the html, css, and markdown files which make up a chatmail relays
web page. Edit them before deploying to make your chatmail relay
stand out.
@@ -249,6 +247,33 @@ Fresh chatmail addresses have a mailbox directory that contains:
directories will typically be empty unless the user of that address
hasnt been online for a while.
App version information (experimental)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A chatmail relay ships the
:repofile:`appversions.json <chatmaild/src/chatmaild/defaults/appversions.json>`
file of the ``chatmaild`` package
and serves its content under the IMAP METADATA key
``/shared/vendor/deltachat/appversions``.
Chat apps installed outside of app stores read this key
to learn about updates and where to download them.
The mechanism is experimental and may change.
The file travels with the normal deploy:
update the repository checkout and run ``cmdeploy run``.
Local modifications of ``appversions.json`` are deployed as-is,
so you can serve your own app version information,
including links to app downloads.
There is no automatic refresh:
version information changes only when you deploy again.
.. note::
Note that as of August 2026, only Delta Chat Android Google Play version
is beginning to support discovering app versions from relays.
Generally, consumers of relay-provided app version information
need to verify themselves that downloaded app files are valid.
Active ports
~~~~~~~~~~~~
+5 -4
View File
@@ -14,10 +14,11 @@ We know of three work-in-progress alternative implementation efforts:
it to support all of the features and configuration settings required
to operate as a chatmail relay.
- `Madmail <https://github.com/themadorg/madmail>`_: an
experimental fork of `Maddy Mail Server <https://maddy.email/>`_, modified
for chatmail deployments. It provides a single binary solution
for running a chatmail relay.
- `Madmail <https://github.com/themadorg/madmail>`_: a Rust-based
single-binary chatmail relay. Madmail v2 is a rewrite of an earlier
experimental fork of `Maddy Mail Server <https://maddy.email/>`_.
It includes SMTP, IMAP, encryption enforcement, and real-time
services (TURN/Iroh), and runs on Linux and Windows.
- `Chatmail Cookbook <https://github.com/feld/chatmail-cookbook>`_:
A Chef Cookbook implementing a relay server. The project follows the
+26
View File
@@ -0,0 +1,26 @@
#!/bin/bash
#
# Show current IMAP and SMTP connections on a chatmail relay,
# to compare against the max_imap_connections and max_smtp_connections
# settings in chatmail.ini. Run this on the relay itself.
set -e
# Count established TCP connections whose local port is one of the arguments.
established() {
filter=$(printf 'sport = :%s or ' "$@")
ss -Htn state established "( ${filter% or } )" | wc -l
}
# doveadm prints a header line and then one line per logged-in user,
# with that user's number of connections in the second column.
sessions=$(doveadm who | awk 'NR > 1 { n += $2 } END { print n + 0 }')
# Unless imap_compress is enabled, connections idle for
# imap_hibernate_timeout are handed over to the imap-hibernate
# process, so they cost no imap process while idle.
active=$(pgrep -x imap | wc -l)
printf 'imap %6d ports 143,993 (max_imap_connections)\n' "$(established 143 993)"
printf ' %6d dovecot sessions, %d of them in an active imap process\n' "$sessions" "$active"
printf 'submission %6d ports 465,587 (max_smtp_connections per port)\n' "$(established 465 587)"
printf 'incoming %6d port 25 (from other relays, no chatmail.ini limit)\n' "$(established 25)"