mirror of
https://github.com/chatmail/relay.git
synced 2026-09-02 13:33:14 +00:00
Compare commits
28
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6cd943afb5 | ||
|
|
d25e8a8ee8 | ||
|
|
cae03e2714 | ||
|
|
7db16cc716 | ||
|
|
4cdccee63b | ||
|
|
12664d9188 | ||
|
|
1f0ddb7e5b | ||
|
|
2af8d0e7b5 | ||
|
|
e489a1ea29 | ||
|
|
455da45d36 | ||
|
|
48ad92bf24 | ||
|
|
33f9cddb1b | ||
|
|
dc8e0a34a2 | ||
|
|
efc24fcdf3 | ||
|
|
9a9bda80b1 | ||
|
|
74f4721f2b | ||
|
|
14f829003d | ||
|
|
5da9aaeb37 | ||
|
|
d068052f0c | ||
|
|
fa5afadaaa | ||
|
|
eb862f1645 | ||
|
|
6f3039509c | ||
|
|
504fb249a9 | ||
|
|
deca1ef2a3 | ||
|
|
0323e757a9 | ||
|
|
752957dae8 | ||
|
|
3a553f2286 | ||
|
|
cb1e4ff5bb |
@@ -29,7 +29,7 @@ jobs:
|
||||
ref: ${{ github.event.pull_request.head.sha }}
|
||||
persist-credentials: false
|
||||
- name: download filtermail
|
||||
run: curl -L https://github.com/chatmail/filtermail/releases/download/v0.7.1/filtermail-x86_64 -o /usr/local/bin/filtermail && chmod +x /usr/local/bin/filtermail
|
||||
run: curl -L https://github.com/chatmail/filtermail/releases/download/v0.7.4/filtermail-x86_64 -o /usr/local/bin/filtermail && chmod +x /usr/local/bin/filtermail
|
||||
- name: run chatmaild tests
|
||||
working-directory: chatmaild
|
||||
run: pipx run tox
|
||||
|
||||
@@ -13,6 +13,11 @@ jobs:
|
||||
scripts:
|
||||
name: build
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
# Pin the repository links in the docs to this pull request's head commit
|
||||
# so that linkcheck resolves files which only exist on the branch so far.
|
||||
# see doc/conf.py
|
||||
DOC_GITHUB_REF: ${{ github.event.pull_request.head.sha }}
|
||||
environment:
|
||||
name: 'staging.chatmail.at/doc/relay/'
|
||||
url: https://staging.chatmail.at/doc/relay/${{ steps.prepare.outputs.prid }}
|
||||
@@ -51,7 +56,7 @@ jobs:
|
||||
mkdir -p "$HOME/.ssh"
|
||||
echo "${{ secrets.CHATMAIL_STAGING_SSHKEY }}" > "$HOME/.ssh/key"
|
||||
chmod 600 "$HOME/.ssh/key"
|
||||
rsync -rILvh -e "ssh -i $HOME/.ssh/key -o StrictHostKeyChecking=no" $GITHUB_WORKSPACE/doc/build/ "${{ secrets.USERNAME }}@chatmail.at:/var/www/html/staging.chatmail.at/doc/relay/${STEPS_PREPARE_OUTPUTS_PRID}/"
|
||||
rsync -rILvh -e "ssh -i $HOME/.ssh/key -o StrictHostKeyChecking=no" $GITHUB_WORKSPACE/doc/build/ "${{ secrets.USERNAME }}@chatmail.at:${STEPS_PREPARE_OUTPUTS_PRID}/"
|
||||
env:
|
||||
STEPS_PREPARE_OUTPUTS_PRID: ${{ steps.prepare.outputs.prid }}
|
||||
|
||||
|
||||
@@ -47,5 +47,5 @@ jobs:
|
||||
mkdir -p "$HOME/.ssh"
|
||||
echo "${{ secrets.CHATMAIL_STAGING_SSHKEY }}" > "$HOME/.ssh/key"
|
||||
chmod 600 "$HOME/.ssh/key"
|
||||
rsync -rILvh -e "ssh -i $HOME/.ssh/key -o StrictHostKeyChecking=no" $GITHUB_WORKSPACE/doc/build/ "${{ secrets.USERNAME }}@chatmail.at:/var/www/html/chatmail.at/doc/relay/"
|
||||
rsync -rILvh -e "ssh -i $HOME/.ssh/key -o StrictHostKeyChecking=no" $GITHUB_WORKSPACE/doc/build/ "${{ secrets.USERNAME }}@chatmail.at:"
|
||||
|
||||
|
||||
@@ -1,5 +1,60 @@
|
||||
# Changelog for chatmail deployment
|
||||
|
||||
## [1.12.0] - 2026-07-31
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
- [**breaking**] Introduce configurable system limits to reject new address creation and limit imap/smtp connections.
|
||||
Dovecot default connection limit lowered from 50k to 10k,
|
||||
Postfix default connection limit lowered from 5k to 1k,
|
||||
larger relays need to adjust their settings.
|
||||
|
||||
### Features
|
||||
|
||||
- Reduce maximal_queue_lifetime from 5d to 2d
|
||||
- Disable negative cache in unbound (#992)
|
||||
- *(mtail)* Add incoming_mailer_daemon_mail_count
|
||||
- *(postfix)* Disable processing of MIME headers
|
||||
- *(dovecot)* Advertise privacy_mail as admin contact, drop server comment
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Set relay restrictions per smtpd service with default reject
|
||||
- Reduce maxproc for filtermail-transport LMTP client to 500
|
||||
- Core 2.50.0 does not have delete_server_after config anymore.
|
||||
- Check if all required ports are available for filtermail (#983)
|
||||
- Always deploy unbound.conf.d/chatmail.conf (#993)
|
||||
- Expire empty directories (#994)
|
||||
- Crypt-r dependency was declared for wrong Python version
|
||||
- Always overwrite /etc/resolv.conf, even if it is a symbolic link
|
||||
- Pass kwargs to files.put()
|
||||
- List Iroh proxy endpoints used by 0.35 and 1.0, drop stale /relay/probe from earlier versions
|
||||
- Fix port discovery when ss -tulpn shows dovecot before stats
|
||||
|
||||
### Documentation
|
||||
|
||||
- Add scripts/initenv.sh to upgrade instructions
|
||||
- Update overview diagrams (#995)
|
||||
- *(overview)* Remove mermaid styles from 'Accepting and delivering mail' (#1009)
|
||||
- *(README.md)* Clarify security enforcement (#1011)
|
||||
|
||||
### Miscellaneous Tasks
|
||||
|
||||
- *(ci)* Auto-trigger docker build on release tag push
|
||||
- *(acmetool)* Update let's encrypt ToS link to 1.8
|
||||
- *(ci)* Update doc staging upload path
|
||||
- *(ci)* Fix docs upload path
|
||||
|
||||
### Refactor
|
||||
|
||||
- *(postfix)* Remove unused "filter" lmtp service
|
||||
- Install dns-root-data instead of using unbound-anchor
|
||||
- *(deps)* Remove domain-validator dependency
|
||||
|
||||
### Testing
|
||||
|
||||
- Set socket security for IMAP and SMTP to "TLS" in "dclogin"
|
||||
|
||||
## [1.11.0] - 2026-05-15
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
@@ -5,3 +5,6 @@ We use [git-cliff] to generate the changelog from commit messages before the rel
|
||||
|
||||
[Conventional Commits]: https://www.conventionalcommits.org/
|
||||
[git-cliff]: https://git-cliff.org/
|
||||
|
||||
To update client app version information,
|
||||
edit [chatmaild/src/chatmaild/defaults/appversions.json](chatmaild/src/chatmaild/defaults/appversions.json).
|
||||
|
||||
@@ -1,19 +1,24 @@
|
||||
|
||||
# Chatmail relays for end-to-end encrypted email
|
||||
|
||||
Chatmail relay servers are interoperable Mail Transport Agents (MTAs) designed for:
|
||||
Chatmail relay servers are interoperable Mail Transport Agents (MTAs) designed for:
|
||||
|
||||
- **Zero State:** no private data or metadata collected, messages are auto-deleted, low disk usage
|
||||
|
||||
- **Instant/Realtime:** sub-second message delivery, realtime P2P
|
||||
streaming, privacy-preserving Push Notifications for Apple, Google, and Huawei;
|
||||
|
||||
- **Security Enforcement**: only strict TLS, DKIM and OpenPGP with minimized metadata accepted
|
||||
- **Security Enforcement**: Only connections with strict TLS are accepted;
|
||||
all messages must be correctly signed with DKIM and OpenPGP-encrypted with minimized metadata.
|
||||
There are experimental exceptions for no-DNS relays,
|
||||
which are allowed use self-signed TLS certificates
|
||||
and which do not need to DKIM-sign their messages.
|
||||
Unencrypted messages are allowed in neither case.
|
||||
|
||||
- **Reliable Federation and Decentralization:** No spam or IP reputation checks, federating
|
||||
depends on established IETF standards and protocols.
|
||||
|
||||
This repository contains everything needed to setup a ready-to-use chatmail relay on an ssh-reachable host.
|
||||
This repository contains everything needed to setup a ready-to-use chatmail relay on an ssh-reachable host.
|
||||
For getting started and more information please refer to the web version of this repositories' documentation at
|
||||
|
||||
[https://chatmail.at/doc/relay](https://chatmail.at/doc/relay)
|
||||
|
||||
+4
-6
@@ -1,15 +1,13 @@
|
||||
# Releasing a new version of chatmail relay
|
||||
|
||||
For example, to release version 1.9.0 of chatmail relay, do the following steps.
|
||||
For example, to release version 1.13.0 of chatmail relay, do the following steps.
|
||||
|
||||
1. Update the changelog: `git cliff --unreleased --tag 1.9.0 --prepend CHANGELOG.md` or `git cliff -u -t 1.9.0 -p CHANGELOG.md`.
|
||||
1. Update the changelog: `git cliff --unreleased --tag 1.13.0 --prepend CHANGELOG.md` or `git cliff -u -t 1.13.0 -p CHANGELOG.md`.
|
||||
|
||||
2. Open the changelog in the editor, edit it if required.
|
||||
|
||||
3. Commit the changes to the changelog with a commit message `chore(release): prepare for 1.9.0`.
|
||||
|
||||
3. Tag the release: `git tag --annotate 1.9.0`.
|
||||
4. Open a PR with the new commit, merge it to main after review.
|
||||
|
||||
4. Push the release tag: `git push origin 1.9.0`.
|
||||
|
||||
5. Create a GitHub release: `gh release create 1.9.0`.
|
||||
5. In the web interface, create a GitHub release, tell it to create a new tag.
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
include src/chatmaild/defaults/*.json
|
||||
include src/chatmaild/ini/*.ini.f
|
||||
include src/chatmaild/ini/*.ini
|
||||
include src/chatmaild/tests/mail-data/*
|
||||
|
||||
@@ -8,6 +8,7 @@ version = "0.3"
|
||||
dependencies = [
|
||||
"iniconfig",
|
||||
"filelock",
|
||||
"psutil",
|
||||
"requests",
|
||||
"crypt-r >= 3.13.1 ; python_version >= '3.13'",
|
||||
]
|
||||
|
||||
@@ -75,6 +75,16 @@ class Config:
|
||||
self.privacy_pdo = params.pop("privacy_pdo", None)
|
||||
self.privacy_supervisor = params.pop("privacy_supervisor", None)
|
||||
|
||||
self.max_load_1m = float(params.pop("max_load_1m", 5))
|
||||
self.min_available_memory_mb = parse_size_mb(
|
||||
params.pop("min_available_memory", "200M")
|
||||
)
|
||||
self.min_free_disk_space_mb = parse_size_mb(
|
||||
params.pop("min_free_disk_space", "1G")
|
||||
)
|
||||
self.max_imap_connections = int(params.pop("max_imap_connections", 10000))
|
||||
self.max_smtp_connections = int(params.pop("max_smtp_connections", 1000))
|
||||
|
||||
# TLS certificate management.
|
||||
# If tls_external_cert_and_key is set, use externally managed certs.
|
||||
# Otherwise derived from the domain name:
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"clients": [
|
||||
{
|
||||
"clientId": "deltachat",
|
||||
"sources": [
|
||||
{
|
||||
"sourceId": "gplay",
|
||||
"versionInteger": 757,
|
||||
"versionString": "2.59.1",
|
||||
"downloadUrl": "https://github.com/deltachat/deltachat-android/releases/download/v2.59.1/deltachat-gplay-release-2.59.1.apk"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -14,6 +14,7 @@ except ImportError:
|
||||
from .config import Config, read_config
|
||||
from .dictproxy import DictProxy
|
||||
from .migrate_db import migrate_from_db_to_maildir
|
||||
from .syslimits import has_sufficient_resources
|
||||
|
||||
NOCREATE_FILE = "/etc/chatmail-nocreate"
|
||||
VALID_LOCALPART_RE = re.compile(r"^[a-z0-9._-]+$")
|
||||
@@ -147,6 +148,8 @@ class AuthDictProxy(DictProxy):
|
||||
return userdata
|
||||
if not is_allowed_to_create(self.config, addr, cleartext_password):
|
||||
return
|
||||
if not has_sufficient_resources(self.config):
|
||||
return
|
||||
|
||||
lock = filelock.FileLock(str(user.password_path) + ".lock", timeout=5)
|
||||
with lock:
|
||||
|
||||
@@ -42,6 +42,33 @@ mail_domain = {mail_domain}
|
||||
# minimum length a password must have
|
||||
#password_min_length = 9
|
||||
|
||||
#
|
||||
# System resource limits
|
||||
#
|
||||
|
||||
# The following three limits refuse creation of new addresses
|
||||
# while existing addresses keep working.
|
||||
# Rejections are logged by the doveauth service.
|
||||
|
||||
# Maximum 1-minute load average, as reported by "uptime";
|
||||
# it counts processes waiting for disk I/O as well as for CPU.
|
||||
#max_load_1m = 5
|
||||
|
||||
# Minimum memory available without swapping.
|
||||
#min_available_memory = 200M
|
||||
|
||||
# Minimum free disk space on the file system holding the mailboxes.
|
||||
#min_free_disk_space = 1G
|
||||
|
||||
# Maximum number of concurrent IMAP connections
|
||||
# (the Dovecot imap process limit).
|
||||
#max_imap_connections = 10000
|
||||
|
||||
# Maximum number of concurrent SMTP connections
|
||||
# on each of the submission and smtps ports (the Postfix process limit).
|
||||
# A single client IP may use up to a fifth of this.
|
||||
#max_smtp_connections = 1000
|
||||
|
||||
# Use externally managed TLS certificates instead of built-in acmetool.
|
||||
# Paths refer to files on the deployment server (not the build machine).
|
||||
# Both files must already exist before running cmdeploy.
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
import json
|
||||
import logging
|
||||
import socket
|
||||
import sys
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from importlib.resources import files
|
||||
|
||||
from .config import read_config
|
||||
from .dictproxy import DictProxy
|
||||
@@ -18,6 +20,18 @@ def turn_credentials(turn_socket_path):
|
||||
return file.readline().decode("utf-8").strip()
|
||||
|
||||
|
||||
def read_appversions(path):
|
||||
try:
|
||||
data = json.loads(path.read_bytes())
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
except (OSError, ValueError):
|
||||
logging.exception(f"failed to read {path}")
|
||||
return None
|
||||
# the dict protocol is line-based, keep the value single-line
|
||||
return json.dumps(data, separators=(",", ":"))
|
||||
|
||||
|
||||
def _is_valid_token_timestamp(timestamp, now):
|
||||
# Token if invalid after 90 days
|
||||
# or if the timestamp is in the future.
|
||||
@@ -101,6 +115,7 @@ class MetadataDictProxy(DictProxy):
|
||||
self.iroh_relay = iroh_relay
|
||||
self.turn_hostname = turn_hostname
|
||||
self.turn_socket_path = turn_socket_path
|
||||
self.appversions_path = files(__package__).joinpath("defaults/appversions.json")
|
||||
|
||||
def handle_lookup(self, parts):
|
||||
# Lpriv/43f5f508a7ea0366dff30200c15250e3/devicetoken\tlkj123poi@c2.testrun.org
|
||||
@@ -125,6 +140,9 @@ class MetadataDictProxy(DictProxy):
|
||||
case "maxsmtprecipients":
|
||||
# postfix default (see "postconf smtpd_recipient_limit")
|
||||
return "O1000\n"
|
||||
case "appversions":
|
||||
value = read_appversions(self.appversions_path)
|
||||
return f"O{value}\n" if value else "N\n"
|
||||
|
||||
logging.warning(f"lookup ignored: {parts!r}")
|
||||
return "N\n"
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Detect whether the system is at its limits."""
|
||||
|
||||
import logging
|
||||
|
||||
import psutil
|
||||
|
||||
MB = 1024 * 1024
|
||||
|
||||
|
||||
def read_value(getter):
|
||||
try:
|
||||
return getter()
|
||||
except Exception as e:
|
||||
logging.warning("ignoring unreadable system limit: %s", e)
|
||||
return None
|
||||
|
||||
|
||||
def has_sufficient_resources(config):
|
||||
"""Return False if load, memory or disk exceeds a configured limit."""
|
||||
load = read_value(lambda: psutil.getloadavg()[0])
|
||||
mem = read_value(lambda: psutil.virtual_memory().available // MB)
|
||||
disk = read_value(lambda: psutil.disk_usage(str(config.mailboxes_dir)).free // MB)
|
||||
if load is not None and load > config.max_load_1m:
|
||||
msg = f"load avg {load:.2f} > {config.max_load_1m:.2f}"
|
||||
elif mem is not None and mem < config.min_available_memory_mb:
|
||||
msg = f"available memory {mem}MB < {config.min_available_memory_mb}MB"
|
||||
elif disk is not None and disk < config.min_free_disk_space_mb:
|
||||
msg = f"free disk {disk}MB < {config.min_free_disk_space_mb}MB"
|
||||
else:
|
||||
return True
|
||||
logging.warning("registration rejected: %s", msg)
|
||||
return False
|
||||
@@ -20,6 +20,10 @@ def make_config(tmp_path):
|
||||
basedir.mkdir(parents=True, exist_ok=True)
|
||||
overrides = settings.copy() if settings else {}
|
||||
overrides["mailboxes_dir"] = str(basedir)
|
||||
# permissive resource limits so tests never depend on host load/memory/disk
|
||||
overrides.setdefault("max_load_1m", "99999")
|
||||
overrides.setdefault("min_available_memory", "0")
|
||||
overrides.setdefault("min_free_disk_space", "0")
|
||||
write_initial_config(inipath, mail_domain, overrides=overrides)
|
||||
return read_config(inipath)
|
||||
|
||||
@@ -37,22 +41,22 @@ def ipv4_config(make_config):
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def maildomain(example_config):
|
||||
def example_maildomain(example_config):
|
||||
return example_config.mail_domain
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def testaddr(maildomain):
|
||||
return f"user.name@{maildomain}"
|
||||
def testaddr(example_maildomain):
|
||||
return f"user.name@{example_maildomain}"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gencreds(maildomain):
|
||||
def example_gencreds(example_maildomain):
|
||||
count = itertools.count()
|
||||
next(count)
|
||||
|
||||
def gen(domain=None):
|
||||
domain = domain if domain else maildomain
|
||||
domain = domain if domain else example_maildomain
|
||||
while 1:
|
||||
num = next(count)
|
||||
alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from chatmaild.metadata import MetadataDictProxy
|
||||
|
||||
ALLOWED_URL_PREFIXES = (
|
||||
"https://github.com/deltachat/",
|
||||
"https://download.delta.chat/",
|
||||
)
|
||||
|
||||
|
||||
def check_string(value):
|
||||
assert isinstance(value, str), value
|
||||
assert value
|
||||
|
||||
|
||||
def check_version_integer(value):
|
||||
# core parses this as u32, see https://github.com/chatmail/core/pull/8557
|
||||
assert isinstance(value, int) and not isinstance(value, bool), value
|
||||
assert 0 <= value < 2**32, value
|
||||
|
||||
|
||||
def check_appversions(data):
|
||||
"""Verifies the file the way core parses it.
|
||||
|
||||
core deserializes into typed structs and drops the whole payload
|
||||
of a relay if a single value has an unexpected type,
|
||||
while missing or misspelled keys silently turn into defaults.
|
||||
"""
|
||||
assert set(data) == {"clients"}, data
|
||||
assert isinstance(data["clients"], list)
|
||||
assert data["clients"]
|
||||
client_ids = []
|
||||
for client in data["clients"]:
|
||||
assert set(client) == {"clientId", "sources"}, client
|
||||
check_string(client["clientId"])
|
||||
client_ids.append(client["clientId"])
|
||||
assert isinstance(client["sources"], list)
|
||||
assert client["sources"]
|
||||
source_ids = []
|
||||
for source in client["sources"]:
|
||||
assert set(source) == {
|
||||
"sourceId",
|
||||
"versionInteger",
|
||||
"versionString",
|
||||
"downloadUrl",
|
||||
}, source
|
||||
check_string(source["sourceId"])
|
||||
source_ids.append(source["sourceId"])
|
||||
check_version_integer(source["versionInteger"])
|
||||
check_string(source["versionString"])
|
||||
check_string(source["downloadUrl"])
|
||||
assert source["downloadUrl"].startswith(ALLOWED_URL_PREFIXES)
|
||||
# core takes the first matching source, later duplicates never surface
|
||||
assert len(set(source_ids)) == len(source_ids), source_ids
|
||||
assert len(set(client_ids)) == len(client_ids), client_ids
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def appversions():
|
||||
# check the file which chatmail-metadata actually serves
|
||||
path = MetadataDictProxy(notifier=None, metadata=None).appversions_path
|
||||
return json.loads(path.read_text())
|
||||
|
||||
|
||||
def test_appversions_schema(appversions):
|
||||
check_appversions(appversions)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", [True, -1, 2**32, "754", 754.0, None])
|
||||
def test_version_integer_rejected(appversions, value):
|
||||
appversions["clients"][0]["sources"][0]["versionInteger"] = value
|
||||
with pytest.raises(AssertionError):
|
||||
check_appversions(appversions)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("key", ["clientId", "sources"])
|
||||
def test_misspelled_client_key_rejected(appversions, key):
|
||||
client = appversions["clients"][0]
|
||||
client[key + "s"] = client.pop(key)
|
||||
with pytest.raises(AssertionError):
|
||||
check_appversions(appversions)
|
||||
|
||||
|
||||
def test_duplicate_source_id_rejected(appversions):
|
||||
sources = appversions["clients"][0]["sources"]
|
||||
sources.append(dict(sources[0]))
|
||||
with pytest.raises(AssertionError):
|
||||
check_appversions(appversions)
|
||||
|
||||
|
||||
def test_foreign_download_url_rejected(appversions):
|
||||
appversions["clients"][0]["sources"][0]["downloadUrl"] = "https://example.org/x.apk"
|
||||
with pytest.raises(AssertionError):
|
||||
check_appversions(appversions)
|
||||
@@ -30,9 +30,9 @@ def test_read_config_ipv4(ipv4_config):
|
||||
assert ipv4_config.mail_domain == "[1.3.3.7]"
|
||||
|
||||
|
||||
def test_read_config_basic_using_defaults(tmp_path, maildomain):
|
||||
def test_read_config_basic_using_defaults(tmp_path, example_maildomain):
|
||||
inipath = tmp_path.joinpath("chatmail.ini")
|
||||
inipath.write_text(f"[params]\nmail_domain = {maildomain}")
|
||||
inipath.write_text(f"[params]\nmail_domain = {example_maildomain}")
|
||||
example_config = read_config(inipath)
|
||||
assert example_config.max_user_send_per_minute == 60
|
||||
assert example_config.filtermail_smtp_port_incoming == 10081
|
||||
@@ -45,6 +45,8 @@ def test_read_config_basic_using_defaults(tmp_path, maildomain):
|
||||
assert example_config.username_min_length == 9
|
||||
assert example_config.username_max_length == 9
|
||||
assert example_config.password_min_length == 9
|
||||
assert example_config.max_imap_connections == 10000
|
||||
assert example_config.max_smtp_connections == 1000
|
||||
assert example_config._unused_keys == []
|
||||
|
||||
|
||||
|
||||
@@ -19,8 +19,8 @@ def dictproxy(example_config):
|
||||
return AuthDictProxy(config=example_config)
|
||||
|
||||
|
||||
def test_basic(dictproxy, gencreds):
|
||||
addr, password = gencreds()
|
||||
def test_basic(dictproxy, example_gencreds):
|
||||
addr, password = example_gencreds()
|
||||
dictproxy.lookup_passdb(addr, password)
|
||||
data = dictproxy.lookup_userdb(addr)
|
||||
assert data
|
||||
@@ -107,7 +107,7 @@ def test_handle_dovecot_protocol_user_not_exists(example_config):
|
||||
assert wfile.getvalue() == b"N\n"
|
||||
|
||||
|
||||
def test_handle_dovecot_protocol_iterate(gencreds, example_config):
|
||||
def test_handle_dovecot_protocol_iterate(example_config):
|
||||
dictproxy = AuthDictProxy(config=example_config)
|
||||
dictproxy.lookup_passdb("asdf00000@chat.example.org", "q9mr3faue")
|
||||
dictproxy.lookup_passdb("asdf11111@chat.example.org", "q9mr3faue")
|
||||
@@ -174,14 +174,14 @@ def test_concurrent_creation_same_account(dictproxy):
|
||||
assert len(passwords_seen) == 1
|
||||
|
||||
|
||||
def test_50_concurrent_lookups_different_accounts(gencreds, dictproxy):
|
||||
def test_50_concurrent_lookups_different_accounts(example_gencreds, dictproxy):
|
||||
num_threads = 50
|
||||
req_per_thread = 5
|
||||
results = queue.Queue()
|
||||
|
||||
def lookup():
|
||||
for i in range(req_per_thread):
|
||||
addr, password = gencreds()
|
||||
addr, password = example_gencreds()
|
||||
try:
|
||||
dictproxy.lookup_passdb(addr, password)
|
||||
except Exception:
|
||||
@@ -202,3 +202,17 @@ def test_50_concurrent_lookups_different_accounts(gencreds, dictproxy):
|
||||
res = results.get()
|
||||
if res is not None:
|
||||
pytest.fail(f"concurrent lookup failed\n{res}")
|
||||
|
||||
|
||||
def test_insufficient_resources_block_creation_not_existing_logins(
|
||||
dictproxy, example_gencreds, monkeypatch
|
||||
):
|
||||
addr, password = example_gencreds()
|
||||
assert dictproxy.lookup_passdb(addr, password)
|
||||
|
||||
monkeypatch.setattr(
|
||||
chatmaild.doveauth, "has_sufficient_resources", lambda config: False
|
||||
)
|
||||
newaddr, newpassword = example_gencreds()
|
||||
assert not dictproxy.lookup_passdb(newaddr, newpassword)
|
||||
assert dictproxy.lookup_passdb(addr, password)
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import io
|
||||
import json
|
||||
import time
|
||||
|
||||
import pytest
|
||||
@@ -7,6 +8,7 @@ import requests
|
||||
from chatmaild.metadata import (
|
||||
Metadata,
|
||||
MetadataDictProxy,
|
||||
read_appversions,
|
||||
)
|
||||
from chatmaild.notifier import (
|
||||
Notifier,
|
||||
@@ -369,6 +371,32 @@ def test_iroh_relay(dictproxy):
|
||||
assert wfile.getvalue() == b"Ohttps://example.org/\n"
|
||||
|
||||
|
||||
def test_read_appversions(tmp_path):
|
||||
path = tmp_path.joinpath("appversions.json")
|
||||
assert read_appversions(path) is None
|
||||
|
||||
path.write_text('{\n "clients": []\n}')
|
||||
assert read_appversions(path) == '{"clients":[]}'
|
||||
|
||||
# the value travels as a single dict protocol line
|
||||
path.write_text('{"clients": [{"clientId": "one\\ntwo"}]}')
|
||||
assert read_appversions(path) == '{"clients":[{"clientId":"one\\ntwo"}]}'
|
||||
|
||||
path.write_text("bad json")
|
||||
assert read_appversions(path) is None
|
||||
|
||||
|
||||
def test_appversions_lookup(dictproxy):
|
||||
# the version information shipped with chatmaild is served as a single line
|
||||
key = b"Lshared/0123/vendor/vendor.dovecot/pvt/server/vendor/deltachat/appversions"
|
||||
key += b"\tuser@example.org"
|
||||
rfile, wfile = io.BytesIO(b"H\n" + key), io.BytesIO()
|
||||
dictproxy.loop_forever(rfile, wfile)
|
||||
value = wfile.getvalue()
|
||||
assert value.startswith(b"O") and value.endswith(b"\n")
|
||||
assert json.loads(value[1:])["clients"]
|
||||
|
||||
|
||||
def test_legacy_token_migration(metadata, testaddr):
|
||||
with metadata.get_metadata_dict(testaddr).modify() as data:
|
||||
data[metadata.DEVICETOKEN_KEY] = ["oldtoken1", "oldtoken2"]
|
||||
|
||||
@@ -48,7 +48,7 @@ def test_create_dclogin_url_ipv4(ipv4_config):
|
||||
assert addr in url
|
||||
|
||||
|
||||
def test_print_new_account(capsys, monkeypatch, maildomain, tmpdir, example_config):
|
||||
def test_print_new_account(capsys, monkeypatch, tmpdir, example_config):
|
||||
monkeypatch.setattr(chatmaild.newemail, "CONFIG_PATH", str(example_config._inipath))
|
||||
print_new_account()
|
||||
out, err = capsys.readouterr()
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
import shutil
|
||||
|
||||
import psutil
|
||||
|
||||
from chatmaild.syslimits import has_sufficient_resources
|
||||
|
||||
PERMISSIVE = {
|
||||
"max_load_1m": "99999",
|
||||
"min_available_memory": "0",
|
||||
"min_free_disk_space": "0",
|
||||
}
|
||||
|
||||
|
||||
def test_rejects_constrained_system(make_config, caplog):
|
||||
assert has_sufficient_resources(make_config("chat.example.org", PERMISSIVE))
|
||||
for settings in (
|
||||
{"max_load_1m": "-1.0"},
|
||||
{"min_available_memory": "99999999G"},
|
||||
{"min_free_disk_space": "99999999G"},
|
||||
):
|
||||
config = make_config("chat.example.org", PERMISSIVE | settings)
|
||||
caplog.clear()
|
||||
assert not has_sufficient_resources(config), settings
|
||||
assert "registration rejected" in caplog.text
|
||||
|
||||
|
||||
def test_unreadable_disk_does_not_reject(make_config, caplog):
|
||||
config = make_config(
|
||||
"chat.example.org", PERMISSIVE | {"min_free_disk_space": "99999999G"}
|
||||
)
|
||||
shutil.rmtree(config.mailboxes_dir)
|
||||
assert has_sufficient_resources(config)
|
||||
assert "ignoring" in caplog.text
|
||||
|
||||
|
||||
def test_one_unreadable_value_keeps_other_checks(make_config, monkeypatch, caplog):
|
||||
def raise_error(*args):
|
||||
raise psutil.Error("dud")
|
||||
|
||||
monkeypatch.setattr(psutil, "getloadavg", raise_error)
|
||||
config = make_config(
|
||||
"chat.example.org", PERMISSIVE | {"min_free_disk_space": "99999999G"}
|
||||
)
|
||||
assert not has_sufficient_resources(config)
|
||||
assert "ignoring" in caplog.text
|
||||
@@ -19,6 +19,7 @@ dependencies = [
|
||||
"pytest-xdist",
|
||||
"execnet",
|
||||
"imap_tools",
|
||||
"lupa",
|
||||
"deltachat-rpc-client",
|
||||
"deltachat-rpc-server",
|
||||
]
|
||||
|
||||
@@ -4,6 +4,8 @@ from ..basedeploy import Deployer
|
||||
|
||||
|
||||
class AcmetoolDeployer(Deployer):
|
||||
bin_path = "/usr/bin/acmetool"
|
||||
|
||||
def __init__(self, email, domains):
|
||||
self.domains = domains
|
||||
self.email = email
|
||||
@@ -41,8 +43,12 @@ class AcmetoolDeployer(Deployer):
|
||||
domains=self.domains,
|
||||
)
|
||||
|
||||
self.ensure_systemd_unit("acmetool/acmetool-redirector.service")
|
||||
self.ensure_systemd_unit("acmetool/acmetool-reconcile.service")
|
||||
self.ensure_systemd_unit(
|
||||
"acmetool/acmetool-redirector.service.j2", bin_path=self.bin_path
|
||||
)
|
||||
self.ensure_systemd_unit(
|
||||
"acmetool/acmetool-reconcile.service.j2", bin_path=self.bin_path
|
||||
)
|
||||
self.ensure_systemd_unit("acmetool/acmetool-reconcile.timer")
|
||||
|
||||
def activate(self):
|
||||
@@ -52,5 +58,5 @@ class AcmetoolDeployer(Deployer):
|
||||
|
||||
server.shell(
|
||||
name=f"Reconcile certificates for: {', '.join(self.domains)}",
|
||||
commands=["acmetool --batch --xlog.severity=debug reconcile"],
|
||||
commands=[f"{self.bin_path} --batch --xlog.severity=debug reconcile"],
|
||||
)
|
||||
|
||||
+1
-1
@@ -4,5 +4,5 @@ After=network.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/bin/acmetool --batch reconcile
|
||||
ExecStart={{ bin_path }} --batch reconcile
|
||||
|
||||
+1
-1
@@ -3,7 +3,7 @@ Description=acmetool HTTP redirector
|
||||
|
||||
[Service]
|
||||
Type=notify
|
||||
ExecStart=/usr/bin/acmetool redirector --service.uid=daemon --bind=127.0.0.1:402
|
||||
ExecStart={{ bin_path }} redirector --service.uid=daemon --bind=127.0.0.1:402
|
||||
Restart=always
|
||||
RestartSec=30
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
"acme-enter-email": "{{ email }}"
|
||||
"acme-agreement:https://letsencrypt.org/documents/LE-SA-v1.7-June-04-2026.pdf": true
|
||||
"acme-agreement:https://letsencrypt.org/documents/LE-SA-v1.8-July-06-2026.pdf": true
|
||||
|
||||
@@ -51,7 +51,7 @@ def get_resource(arg, pkg=__package__):
|
||||
return importlib.resources.files(pkg).joinpath(arg)
|
||||
|
||||
|
||||
def configure_remote_units(deployer, mail_domain, units) -> None:
|
||||
def configure_remote_units(deployer, mail_domain, units, **kwargs) -> None:
|
||||
remote_base_dir = "/usr/local/lib/chatmaild"
|
||||
remote_venv_dir = f"{remote_base_dir}/venv"
|
||||
remote_chatmail_inipath = f"{remote_base_dir}/chatmail.ini"
|
||||
@@ -63,6 +63,7 @@ def configure_remote_units(deployer, mail_domain, units) -> None:
|
||||
config_path=remote_chatmail_inipath,
|
||||
remote_venv_dir=remote_venv_dir,
|
||||
mail_domain=mail_domain,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
basename = fn if "." in fn else f"{fn}.service"
|
||||
@@ -226,7 +227,7 @@ class Deployer:
|
||||
res = files.directory(name=name, path=path, present=False, **kwargs)
|
||||
return self._update_restart_signals(path, res)
|
||||
|
||||
def download_executable(self, url, dest, sha256sum, extract=None):
|
||||
def download_executable(self, url, dest, sha256sum, extract=None, mode="755"):
|
||||
existing = host.get_fact(Sha256File, dest)
|
||||
if existing == sha256sum:
|
||||
return
|
||||
@@ -243,7 +244,7 @@ class Deployer:
|
||||
f"({dl_cmd}"
|
||||
f" && echo '{sha256sum} {tmp}' | sha256sum -c"
|
||||
f" && mv {tmp} {dest})",
|
||||
f"chmod 755 {dest}",
|
||||
f"chmod {mode} {dest}",
|
||||
],
|
||||
)
|
||||
self.need_restart = True
|
||||
|
||||
@@ -33,6 +33,7 @@ from .filtermail.deployer import FiltermailDeployer
|
||||
from .mtail.deployer import MtailDeployer
|
||||
from .nginx.deployer import NginxDeployer
|
||||
from .opendkim.deployer import OpendkimDeployer
|
||||
from .pins import IROH_ARTIFACTS, TURN_ARTIFACTS
|
||||
from .postfix.deployer import PostfixDeployer
|
||||
from .selfsigned.deployer import SelfSignedTlsDeployer
|
||||
from .www import build_webpages, find_merge_conflict, get_paths
|
||||
@@ -138,9 +139,22 @@ class UnboundDeployer(Deployer):
|
||||
# On an IPv4-only system, if unbound is started but not configured,
|
||||
# it causes subsequent steps to fail to resolve hosts.
|
||||
with blocked_service_startup():
|
||||
# dns-root-data is an optional package
|
||||
# that contains /usr/share/dns/root.key
|
||||
#
|
||||
# This file is copied into /var/lib/unbound/root.key
|
||||
# at the start of "unbound" systemd unit
|
||||
# by /usr/libexec/unbound-helper shell script
|
||||
# from the "unbound" package as of version 1.17.1-2+deb12u4
|
||||
#
|
||||
# The same /var/lib/unbound/root.key can be retrieved directly
|
||||
# following the procedure from
|
||||
# <https://www.rfc-editor.org/info/rfc7958/#section-3.1>
|
||||
# with "unbound-anchor -a /var/lib/unbound/root.key"
|
||||
# We don't install and use "unbound-anchor".
|
||||
apt.packages(
|
||||
name="Install unbound",
|
||||
packages=["unbound", "unbound-anchor", "dnsutils"],
|
||||
packages=["unbound", "dns-root-data", "dnsutils"],
|
||||
)
|
||||
|
||||
def configure(self):
|
||||
@@ -166,12 +180,6 @@ class UnboundDeployer(Deployer):
|
||||
dest="/etc/resolv.conf",
|
||||
force=True,
|
||||
)
|
||||
server.shell(
|
||||
name="Generate root keys for validating DNSSEC",
|
||||
commands=[
|
||||
"unbound-anchor -a /var/lib/unbound/root.key || true",
|
||||
],
|
||||
)
|
||||
self.ensure_directory(
|
||||
path="/etc/unbound/unbound.conf.d",
|
||||
)
|
||||
@@ -246,6 +254,13 @@ class LegacyRemoveDeployer(Deployer):
|
||||
def install(self):
|
||||
apt.packages(name="Remove rspamd", packages="rspamd", present=False)
|
||||
|
||||
# unbound-anchor was used to download /var/lib/unbound/root.key
|
||||
# It is replaced by dns-root-data which contains /usr/share/dns/root.key.
|
||||
# unbound systemd unit copies /usr/share/dns/root.key
|
||||
# into /var/lib/unbound/root.key automatically on start
|
||||
# as long as /usr/share/dns/root.key is present.
|
||||
apt.packages(name="Remove unbound-anchor", packages="unbound-anchor", present=False)
|
||||
|
||||
# remove historic expunge script
|
||||
# which is now implemented through a systemd timer (chatmail-expire)
|
||||
self.remove_file("/etc/cron.d/expunge")
|
||||
@@ -287,55 +302,48 @@ def check_config(config):
|
||||
|
||||
|
||||
class TurnDeployer(Deployer):
|
||||
bin_path = "/usr/local/bin/chatmail-turn"
|
||||
|
||||
def __init__(self, mail_domain):
|
||||
self.mail_domain = mail_domain
|
||||
self.units = ["turnserver"]
|
||||
|
||||
def install(self):
|
||||
(url, sha256sum) = {
|
||||
"x86_64": (
|
||||
"https://github.com/chatmail/chatmail-turn/releases/download/v0.4/chatmail-turn-x86_64-linux",
|
||||
"1ec1f5c50122165e858a5a91bcba9037a28aa8cb8b64b8db570aa457c6141a8a",
|
||||
),
|
||||
"aarch64": (
|
||||
"https://github.com/chatmail/chatmail-turn/releases/download/v0.4/chatmail-turn-aarch64-linux",
|
||||
"0fb3e792419494e21ecad536464929dba706bb2c88884ed8f1788141d26fc756",
|
||||
),
|
||||
}[host.get_fact(facts.server.Arch)]
|
||||
self.download_executable(url, "/usr/local/bin/chatmail-turn", sha256sum)
|
||||
(url, sha256sum) = TURN_ARTIFACTS[host.get_fact(facts.server.Arch)]
|
||||
self.download_executable(url, self.bin_path, sha256sum)
|
||||
|
||||
def configure(self):
|
||||
configure_remote_units(self, self.mail_domain, self.units)
|
||||
configure_remote_units(
|
||||
self, self.mail_domain, self.units, bin_path=self.bin_path
|
||||
)
|
||||
|
||||
def activate(self):
|
||||
activate_remote_units(self, self.units)
|
||||
|
||||
|
||||
class IrohDeployer(Deployer):
|
||||
bin_path = "/usr/local/bin/iroh-relay"
|
||||
config_path = "/etc/iroh-relay.toml"
|
||||
|
||||
def __init__(self, enable_iroh_relay):
|
||||
self.enable_iroh_relay = enable_iroh_relay
|
||||
|
||||
def install(self):
|
||||
(url, sha256sum) = {
|
||||
"x86_64": (
|
||||
"https://github.com/n0-computer/iroh/releases/download/v0.35.0/iroh-relay-v0.35.0-x86_64-unknown-linux-musl.tar.gz",
|
||||
"45c81199dbd70f8c4c30fef7f3b9727ca6e3cea8f2831333eeaf8aa71bf0fac1",
|
||||
),
|
||||
"aarch64": (
|
||||
"https://github.com/n0-computer/iroh/releases/download/v0.35.0/iroh-relay-v0.35.0-aarch64-unknown-linux-musl.tar.gz",
|
||||
"f8ef27631fac213b3ef668d02acd5b3e215292746a3fc71d90c63115446008b1",
|
||||
),
|
||||
}[host.get_fact(facts.server.Arch)]
|
||||
(url, sha256sum) = IROH_ARTIFACTS[host.get_fact(facts.server.Arch)]
|
||||
self.download_executable(
|
||||
url,
|
||||
"/usr/local/bin/iroh-relay",
|
||||
self.bin_path,
|
||||
sha256sum,
|
||||
extract="gunzip | tar -xf - ./iroh-relay -O",
|
||||
)
|
||||
|
||||
def configure(self):
|
||||
self.ensure_systemd_unit("iroh-relay.service")
|
||||
self.put_file("iroh-relay.toml", "/etc/iroh-relay.toml")
|
||||
self.ensure_systemd_unit(
|
||||
"iroh-relay.service.j2",
|
||||
bin_path=self.bin_path,
|
||||
config_path=self.config_path,
|
||||
)
|
||||
self.put_file("iroh-relay.toml", self.config_path)
|
||||
|
||||
def activate(self):
|
||||
self.ensure_service(
|
||||
@@ -356,6 +364,8 @@ class ChatmailVenvDeployer(Deployer):
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
self.units = (
|
||||
# doveauth must restart when chatmaild/ini file changes
|
||||
"doveauth",
|
||||
"chatmail-metadata",
|
||||
"lastlogin",
|
||||
"chatmail-expire",
|
||||
@@ -504,10 +514,10 @@ def deploy_chatmail(config_path: Path, disable_mail: bool, website_only: bool) -
|
||||
("nginx", 443),
|
||||
(["master", "smtpd"], 465),
|
||||
(["master", "smtpd"], 587),
|
||||
(["imap-login", "dovecot"], 993),
|
||||
(["dovecot", "imap-login"], 993),
|
||||
("iroh-relay", 3340),
|
||||
("mtail", 3903),
|
||||
("stats", 3904),
|
||||
(["dovecot", "stats"], 3904),
|
||||
("nginx", 8443),
|
||||
(["master", "smtpd"], config.postfix_reinject_port),
|
||||
(["master", "smtpd"], config.postfix_reinject_port_incoming),
|
||||
|
||||
@@ -14,18 +14,9 @@ from cmdeploy.basedeploy import (
|
||||
configure_remote_units,
|
||||
is_in_container,
|
||||
)
|
||||
from cmdeploy.pins import DOVECOT_SHA256, DOVECOT_VERSION
|
||||
|
||||
DOVECOT_ARCHIVE_VERSION = "2.3.21+dfsg1-3"
|
||||
DOVECOT_PACKAGE_VERSION = f"1:{DOVECOT_ARCHIVE_VERSION}"
|
||||
|
||||
DOVECOT_SHA256 = {
|
||||
("core", "amd64"): "dd060706f52a306fa863d874717210b9fe10536c824afe1790eec247ded5b27d",
|
||||
("core", "arm64"): "e7548e8a82929722e973629ecc40fcfa886894cef3db88f23535149e7f730dc9",
|
||||
("imapd", "amd64"): "8d8dc6fc00bbb6cdb25d345844f41ce2f1c53f764b79a838eb2a03103eebfa86",
|
||||
("imapd", "arm64"): "178fa877ddd5df9930e8308b518f4b07df10e759050725f8217a0c1fb3fd707f",
|
||||
("lmtpd", "amd64"): "2f69ba5e35363de50962d42cccbfe4ed8495265044e244007d7ccddad77513ab",
|
||||
("lmtpd", "arm64"): "89f52fb36524f5877a177dff4a713ba771fd3f91f22ed0af7238d495e143b38f",
|
||||
}
|
||||
DOVECOT_PACKAGE_VERSION = f"1:{DOVECOT_VERSION}"
|
||||
|
||||
|
||||
class DovecotDeployer(Deployer):
|
||||
@@ -34,7 +25,7 @@ class DovecotDeployer(Deployer):
|
||||
def __init__(self, config, disable_mail):
|
||||
self.config = config
|
||||
self.disable_mail = disable_mail
|
||||
self.units = ["doveauth"]
|
||||
self.units = []
|
||||
|
||||
def install(self):
|
||||
arch = host.get_fact(Arch)
|
||||
@@ -117,7 +108,7 @@ def _download_dovecot_package(package: str, arch: str) -> tuple[str | None, bool
|
||||
if DOVECOT_PACKAGE_VERSION in installed_versions:
|
||||
return None, False
|
||||
|
||||
url_version = DOVECOT_ARCHIVE_VERSION.replace("+", "%2B")
|
||||
url_version = DOVECOT_VERSION.replace("+", "%2B")
|
||||
deb_base = f"{pkg_name}_{url_version}_{arch}.deb"
|
||||
primary_url = f"https://download.delta.chat/dovecot/{deb_base}"
|
||||
fallback_url = f"https://github.com/chatmail/dovecot/releases/download/upstream%2F{url_version}/{deb_base}"
|
||||
|
||||
@@ -37,11 +37,15 @@ default_client_limit = 20000
|
||||
# the following warning will be logged:
|
||||
# Warning: service(imap): process_limit (1024) reached, client connections are being dropped
|
||||
service imap {
|
||||
process_limit = 50000
|
||||
process_limit = {{ config.max_imap_connections }}
|
||||
}
|
||||
|
||||
mail_server_admin = mailto:root@{{ config.mail_domain }}
|
||||
mail_server_comment = Chatmail server
|
||||
{% if config.privacy_mail %}
|
||||
# Advertised to clients as IMAP METADATA /shared/admin (RFC 5464).
|
||||
# The privacy_mail contact from chatmail.ini is used because it is
|
||||
# the only address set by an operator.
|
||||
mail_server_admin = mailto:{{ config.privacy_mail }}
|
||||
{% endif %}
|
||||
|
||||
# `zlib` enables compressing messages stored in the maildir.
|
||||
# See
|
||||
|
||||
@@ -3,6 +3,7 @@ import os
|
||||
from pyinfra import facts, host
|
||||
|
||||
from cmdeploy.basedeploy import Deployer
|
||||
from cmdeploy.pins import FILTERMAIL_ARTIFACTS
|
||||
|
||||
|
||||
class FiltermailDeployer(Deployer):
|
||||
@@ -20,11 +21,7 @@ class FiltermailDeployer(Deployer):
|
||||
return
|
||||
|
||||
arch = host.get_fact(facts.server.Arch)
|
||||
url = f"https://github.com/chatmail/filtermail/releases/download/v0.7.1/filtermail-{arch}"
|
||||
sha256sum = {
|
||||
"x86_64": "fc2d8141166f8561b9711fb68c5327fc9421f814c46dc69671a4605a95b175c0",
|
||||
"aarch64": "37e52c5ddb373ef29b5ead89658407c53f48d10ce055a2dbd9c606fa1ebd5f7f",
|
||||
}[arch]
|
||||
url, sha256sum = FILTERMAIL_ARTIFACTS[arch]
|
||||
self.download_executable(url, self.bin_path, sha256sum)
|
||||
|
||||
def configure(self):
|
||||
|
||||
+1
-1
@@ -2,7 +2,7 @@
|
||||
Description=Iroh relay
|
||||
|
||||
[Service]
|
||||
ExecStart=/usr/local/bin/iroh-relay --config-path /etc/iroh-relay.toml
|
||||
ExecStart={{ bin_path }} --config-path {{ config_path }}
|
||||
Restart=on-failure
|
||||
RestartSec=5s
|
||||
User=iroh
|
||||
@@ -28,6 +28,13 @@ counter created_nonci_accounts
|
||||
}
|
||||
}
|
||||
|
||||
# doveauth refusing new addresses because a chatmail.ini
|
||||
# system resource limit is exceeded.
|
||||
counter rejected_registrations
|
||||
/registration rejected: / {
|
||||
rejected_registrations++
|
||||
}
|
||||
|
||||
counter postfix_timeouts
|
||||
/timeout after DATA/ {
|
||||
postfix_timeouts++
|
||||
@@ -73,6 +80,11 @@ counter incoming_unencrypted_mail_count
|
||||
filtered_incoming_mail_count++
|
||||
}
|
||||
|
||||
counter incoming_mailer_daemon_mail_count
|
||||
/Incoming: Filtering mailer-daemon message from/ {
|
||||
incoming_mailer_daemon_mail_count++
|
||||
filtered_incoming_mail_count++
|
||||
}
|
||||
|
||||
counter rejected_unencrypted_mail_count
|
||||
/Rejected unencrypted mail/ {
|
||||
|
||||
@@ -1,10 +1,14 @@
|
||||
from pyinfra import facts, host
|
||||
from pyinfra.operations import apt
|
||||
from pyinfra.operations import apt, server
|
||||
|
||||
from cmdeploy.basedeploy import Deployer
|
||||
from cmdeploy.pins import FILTERMAIL_ARTIFACTS, MTAIL_ARTIFACTS
|
||||
|
||||
|
||||
class MtailDeployer(Deployer):
|
||||
bin_path = "/usr/local/bin/mtail"
|
||||
progs_dir = "/etc/mtail"
|
||||
|
||||
def __init__(self, mtail_address):
|
||||
self.mtail_address = mtail_address
|
||||
|
||||
@@ -12,19 +16,10 @@ class MtailDeployer(Deployer):
|
||||
# Uninstall mtail package to install a static binary.
|
||||
apt.packages(name="Uninstall mtail", packages=["mtail"], present=False)
|
||||
|
||||
(url, sha256sum) = {
|
||||
"x86_64": (
|
||||
"https://github.com/google/mtail/releases/download/v3.0.8/mtail_3.0.8_linux_amd64.tar.gz",
|
||||
"d55cb601049c5e61eabab29998dbbcea95d480e5448544f9470337ba2eea882e",
|
||||
),
|
||||
"aarch64": (
|
||||
"https://github.com/google/mtail/releases/download/v3.0.8/mtail_3.0.8_linux_arm64.tar.gz",
|
||||
"f748db8ad2a1e0b63684d4c8868cf6a373a20f7e6922e5ece601fff0ee00eb1a",
|
||||
),
|
||||
}[host.get_fact(facts.server.Arch)]
|
||||
(url, sha256sum) = MTAIL_ARTIFACTS[host.get_fact(facts.server.Arch)]
|
||||
self.download_executable(
|
||||
url,
|
||||
"/usr/local/bin/mtail",
|
||||
self.bin_path,
|
||||
sha256sum,
|
||||
extract="gunzip | tar -xf - mtail -O",
|
||||
)
|
||||
@@ -36,8 +31,31 @@ class MtailDeployer(Deployer):
|
||||
"mtail/mtail.service.j2",
|
||||
address=self.mtail_address or "127.0.0.1",
|
||||
port=3903,
|
||||
bin_path=self.bin_path,
|
||||
progs_dir=self.progs_dir,
|
||||
)
|
||||
self.put_file("mtail/delivered_mail.mtail", "/etc/mtail/delivered_mail.mtail")
|
||||
if self.mtail_address:
|
||||
self.put_file(
|
||||
"mtail/delivered_mail.mtail", f"{self.progs_dir}/delivered_mail.mtail"
|
||||
)
|
||||
url, sha256sum = FILTERMAIL_ARTIFACTS['mtail']
|
||||
self.download_executable(
|
||||
url,
|
||||
f"{self.progs_dir}/filtermail.mtail",
|
||||
sha256sum,
|
||||
mode="644",
|
||||
)
|
||||
if self.need_restart:
|
||||
# Check if all installed mtail rules compile or fail early
|
||||
# --one_shot to exit, --port 0 to not clash with running mtail.
|
||||
server.shell(
|
||||
name="Validate mtail programs",
|
||||
commands=[
|
||||
f"timeout 30 {self.bin_path} --compile_only --one_shot"
|
||||
f" --progs {self.progs_dir} --logs /dev/null"
|
||||
" --address 127.0.0.1 --port 0"
|
||||
],
|
||||
)
|
||||
|
||||
def activate(self):
|
||||
active = bool(self.mtail_address)
|
||||
|
||||
@@ -5,7 +5,7 @@ Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/bin/sh -c "journalctl -f -o short-iso -n 0 | /usr/local/bin/mtail --address={{ address }} --port={{ port }} --progs /etc/mtail --logtostderr --logs -"
|
||||
ExecStart=/bin/sh -c "journalctl -f -o short-iso -n 0 | {{ bin_path }} --address={{ address }} --port={{ port }} --progs {{ progs_dir }} --logtostderr --logs -"
|
||||
Restart=on-failure
|
||||
RestartSec=2s
|
||||
|
||||
|
||||
@@ -128,7 +128,10 @@ http {
|
||||
proxy_set_header Connection "upgrade";
|
||||
}
|
||||
|
||||
location /relay/probe {
|
||||
# Endpoints Iroh uses for net_report probes and that clients
|
||||
# probe to tell whether this relay works. Both paths are served
|
||||
# by iroh-relay 0.35 and by the 1.0 line.
|
||||
location /ping {
|
||||
proxy_pass http://127.0.0.1:3340;
|
||||
proxy_http_version 1.1;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Versions, hashes, and download URLs for pre-built artifacts fetched during deploy."""
|
||||
|
||||
FILTERMAIL_VERSION = "v0.7.4"
|
||||
FILTERMAIL_ARTIFACTS = {
|
||||
"x86_64": (
|
||||
f"https://github.com/chatmail/filtermail/releases/download/{FILTERMAIL_VERSION}/filtermail-x86_64",
|
||||
"484cb8dff083134aefba9fce4a6b7ef4784a0f0e28e5108ecf8bb9e58a44fd2c",
|
||||
),
|
||||
"aarch64": (
|
||||
f"https://github.com/chatmail/filtermail/releases/download/{FILTERMAIL_VERSION}/filtermail-aarch64",
|
||||
"66aa0ca2ca9add7a12d92883d76f8786384092adfde24a3d3a1d0b1f30d23a9e",
|
||||
),
|
||||
"mtail": (
|
||||
f"https://raw.githubusercontent.com/chatmail/filtermail/{FILTERMAIL_VERSION}/contrib/filtermail.mtail",
|
||||
"948f688bb89ad47e6eb0fc8fa107e201a689f5adc264ff926be487a2a8562b51",
|
||||
),
|
||||
}
|
||||
|
||||
MTAIL_VERSION = "3.4.9"
|
||||
MTAIL_ARTIFACTS = {
|
||||
"x86_64": (
|
||||
f"https://github.com/jaqx0r/mtail/releases/download/v{MTAIL_VERSION}/mtail_{MTAIL_VERSION}_linux_amd64.tar.gz",
|
||||
"55f64a87f71955bb871c724b4aadf19fe9d854e6327196919c7fe44943427eab",
|
||||
),
|
||||
"aarch64": (
|
||||
f"https://github.com/jaqx0r/mtail/releases/download/v{MTAIL_VERSION}/mtail_{MTAIL_VERSION}_linux_arm64.tar.gz",
|
||||
"e0a2b66b372ca257d7daeb7ba10f9233a2192a1f9057618fccc6be5c854a2a3c",
|
||||
),
|
||||
}
|
||||
|
||||
DOVECOT_VERSION = "2.3.21+dfsg1-3"
|
||||
DOVECOT_SHA256 = {
|
||||
("core", "amd64"): "dd060706f52a306fa863d874717210b9fe10536c824afe1790eec247ded5b27d",
|
||||
("core", "arm64"): "e7548e8a82929722e973629ecc40fcfa886894cef3db88f23535149e7f730dc9",
|
||||
("imapd", "amd64"): "8d8dc6fc00bbb6cdb25d345844f41ce2f1c53f764b79a838eb2a03103eebfa86",
|
||||
("imapd", "arm64"): "178fa877ddd5df9930e8308b518f4b07df10e759050725f8217a0c1fb3fd707f",
|
||||
("lmtpd", "amd64"): "2f69ba5e35363de50962d42cccbfe4ed8495265044e244007d7ccddad77513ab",
|
||||
("lmtpd", "arm64"): "89f52fb36524f5877a177dff4a713ba771fd3f91f22ed0af7238d495e143b38f",
|
||||
}
|
||||
TURN_VERSION = "v0.4"
|
||||
TURN_ARTIFACTS = {
|
||||
"x86_64": (
|
||||
f"https://github.com/chatmail/chatmail-turn/releases/download/{TURN_VERSION}/chatmail-turn-x86_64-linux",
|
||||
"1ec1f5c50122165e858a5a91bcba9037a28aa8cb8b64b8db570aa457c6141a8a",
|
||||
),
|
||||
"aarch64": (
|
||||
f"https://github.com/chatmail/chatmail-turn/releases/download/{TURN_VERSION}/chatmail-turn-aarch64-linux",
|
||||
"0fb3e792419494e21ecad536464929dba706bb2c88884ed8f1788141d26fc756",
|
||||
),
|
||||
}
|
||||
|
||||
IROH_VERSION = "v0.35.0"
|
||||
IROH_ARTIFACTS = {
|
||||
"x86_64": (
|
||||
f"https://github.com/n0-computer/iroh/releases/download/{IROH_VERSION}/iroh-relay-{IROH_VERSION}-x86_64-unknown-linux-musl.tar.gz",
|
||||
"45c81199dbd70f8c4c30fef7f3b9727ca6e3cea8f2831333eeaf8aa71bf0fac1",
|
||||
),
|
||||
"aarch64": (
|
||||
f"https://github.com/n0-computer/iroh/releases/download/{IROH_VERSION}/iroh-relay-{IROH_VERSION}-aarch64-unknown-linux-musl.tar.gz",
|
||||
"f8ef27631fac213b3ef668d02acd5b3e215292746a3fc71d90c63115446008b1",
|
||||
),
|
||||
}
|
||||
@@ -85,6 +85,14 @@ inet_protocols = all
|
||||
|
||||
lmtp_header_checks = regexp:/etc/postfix/lmtp_header_cleanup
|
||||
|
||||
# Do not apply header checks to MIME headers
|
||||
# and other headers that are actually part of the message body.
|
||||
# Reference:
|
||||
# <https://www.postfix.org/postconf.5.html#disable_mime_input_processing>
|
||||
# <https://www.postfix.org/header_checks.5.html>
|
||||
# <https://stump.io/blog/2020/11/29/a-little-gotcha-with-postfixs-header_checks/>
|
||||
disable_mime_input_processing = yes
|
||||
|
||||
mua_client_restrictions = permit_sasl_authenticated, reject
|
||||
mua_sender_restrictions = reject_sender_login_mismatch, permit_sasl_authenticated, reject
|
||||
mua_helo_restrictions = permit_mynetworks, reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname, permit
|
||||
|
||||
@@ -18,7 +18,7 @@ smtp inet n - y - - smtpd
|
||||
-o smtpd_tls_mandatory_protocols=>=TLSv1.2
|
||||
-o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port_incoming }}
|
||||
-o smtpd_relay_restrictions=reject_unauth_destination
|
||||
submission inet n - y - 5000 smtpd
|
||||
submission inet n - y - {{ config.max_smtp_connections }} smtpd
|
||||
-o syslog_name=postfix/submission
|
||||
-o smtpd_tls_security_level=encrypt
|
||||
-o smtpd_tls_mandatory_protocols=>=TLSv1.3
|
||||
@@ -32,9 +32,9 @@ submission inet n - y - 5000 smtpd
|
||||
-o smtpd_sender_restrictions=$mua_sender_restrictions
|
||||
-o smtpd_recipient_restrictions=
|
||||
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
|
||||
-o smtpd_client_connection_count_limit=1000
|
||||
-o smtpd_client_connection_count_limit={{ config.max_smtp_connections // 5 }}
|
||||
-o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
|
||||
smtps inet n - y - 5000 smtpd
|
||||
smtps inet n - y - {{ config.max_smtp_connections }} smtpd
|
||||
-o syslog_name=postfix/smtps
|
||||
-o smtpd_tls_wrappermode=yes
|
||||
-o smtpd_tls_security_level=encrypt
|
||||
@@ -48,7 +48,7 @@ smtps inet n - y - 5000 smtpd
|
||||
-o smtpd_sender_restrictions=$mua_sender_restrictions
|
||||
-o smtpd_recipient_restrictions=
|
||||
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
|
||||
-o smtpd_client_connection_count_limit=1000
|
||||
-o smtpd_client_connection_count_limit={{ config.max_smtp_connections // 5 }}
|
||||
-o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
|
||||
#628 inet n - y - - qmqpd
|
||||
pickup unix n - y 60 1 pickup
|
||||
@@ -76,7 +76,6 @@ lmtp unix - - y - - lmtp
|
||||
anvil unix - - y - 1 anvil
|
||||
scache unix - - y - 1 scache
|
||||
postlog unix-dgram n - n - 1 postlogd
|
||||
filter unix - n n - - lmtp
|
||||
# Local SMTP server for reinjecting outgoing filtered mail.
|
||||
127.0.0.1:{{ config.postfix_reinject_port }} inet n - n - 100 smtpd
|
||||
-o syslog_name=postfix/reinject
|
||||
|
||||
@@ -5,5 +5,5 @@ After=network.target
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=vmail
|
||||
ExecStart=/usr/local/lib/chatmaild/venv/bin/chatmail-expire /usr/local/lib/chatmaild/chatmail.ini -v --remove
|
||||
ExecStart={execpath} {config_path} -v --remove
|
||||
|
||||
|
||||
@@ -5,5 +5,5 @@ After=network.target
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=vmail
|
||||
ExecStart=/usr/local/lib/chatmaild/venv/bin/chatmail-fsreport /usr/local/lib/chatmaild/chatmail.ini
|
||||
ExecStart={execpath} {config_path}
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ After=network.target
|
||||
[Service]
|
||||
Type=simple
|
||||
Restart=always
|
||||
ExecStart=/usr/local/bin/chatmail-turn --realm {mail_domain} --socket /run/chatmail-turn/turn.socket
|
||||
ExecStart={bin_path} --realm {mail_domain} --socket /run/chatmail-turn/turn.socket
|
||||
|
||||
# Create /run/chatmail-turn
|
||||
RuntimeDirectory=chatmail-turn
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
"""Run the lua scripts we ship under lupa, which bundles Lua 5.4 like dovecot."""
|
||||
|
||||
import pytest
|
||||
from lupa import lua54
|
||||
|
||||
from cmdeploy.basedeploy import get_resource
|
||||
|
||||
|
||||
class Lua:
|
||||
"""A Lua runtime to load shipped scripts and mocks into."""
|
||||
|
||||
def __init__(self):
|
||||
self.rt = lua54.LuaRuntime(unpack_returned_tuples=True)
|
||||
self.g = self.rt.globals()
|
||||
|
||||
def load(self, path):
|
||||
self.rt.execute(get_resource(path).read_text())
|
||||
|
||||
def table(self, **kwargs):
|
||||
return self.rt.table(**kwargs)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def lua():
|
||||
return Lua()
|
||||
@@ -1,10 +1,12 @@
|
||||
import ipaddress
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
|
||||
import imap_tools
|
||||
import pytest
|
||||
import requests
|
||||
from chatmaild.tests.test_appversions import check_appversions
|
||||
|
||||
from cmdeploy.cmdeploy import get_sshexec
|
||||
from cmdeploy.remote import rshell
|
||||
@@ -51,6 +53,17 @@ class TestMetadataTokens:
|
||||
assert res == b"1111 2222"
|
||||
assert b"Getmetadata completed" in client.readline()
|
||||
|
||||
def test_get_appversions(self, imap_mailbox):
|
||||
"get app version information shipped with the relay"
|
||||
client = imap_mailbox.client
|
||||
client.send(b'a01 GETMETADATA "" /shared/vendor/deltachat/appversions\n')
|
||||
res = client.readline()
|
||||
assert res[:1] == b"*"
|
||||
res = client.readline().strip().rstrip(b")")
|
||||
# the served value is a single line and passes the shipped file's schema
|
||||
check_appversions(json.loads(res))
|
||||
assert b"Getmetadata completed" in client.readline()
|
||||
|
||||
|
||||
class TestEndToEndDeltaChat:
|
||||
"Tests that use Delta Chat accounts on the chat mail instance."
|
||||
|
||||
@@ -211,7 +211,7 @@ class ImapConn:
|
||||
status, res = self.conn.select()
|
||||
if int(res[0]) == 0:
|
||||
raise ValueError("no messages in imap folder")
|
||||
status, results = self.conn.fetch("1:*", "(RFC822)")
|
||||
status, results = self.conn.fetch("1:*", "(BODY.PEEK[])")
|
||||
assert status == "OK"
|
||||
return results
|
||||
|
||||
|
||||
@@ -111,7 +111,7 @@ def test_download_dovecot_package_uses_archive_version_for_url_and_filename(
|
||||
|
||||
deb, changed = dovecot_deployer._download_dovecot_package("core", "amd64")
|
||||
|
||||
archive_version = dovecot_deployer.DOVECOT_ARCHIVE_VERSION.replace("+", "%2B")
|
||||
archive_version = dovecot_deployer.DOVECOT_VERSION.replace("+", "%2B")
|
||||
expected_deb = f"/root/dovecot-core_{archive_version}_amd64.deb"
|
||||
|
||||
# Verify the returned path uses archive version, not package version (with epoch)
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Test the push_notification.lua we ship, against a mocked dovecot mail API."""
|
||||
|
||||
import textwrap
|
||||
|
||||
import pytest
|
||||
|
||||
USER1 = "user12345@chat.example.org"
|
||||
USER2 = "user67890@chat.example.org"
|
||||
|
||||
DOVECOT_MOCKS = textwrap.dedent("""
|
||||
function make_user(username)
|
||||
local function mailbox(_, name)
|
||||
record("mailbox " .. name)
|
||||
return {
|
||||
sync = function() record("sync") end,
|
||||
metadata_set = function(_, k, v)
|
||||
record("metadata_set " .. k .. "=" .. v)
|
||||
end,
|
||||
free = function() record("free") end,
|
||||
}
|
||||
end
|
||||
return {username = username, mailbox = mailbox}
|
||||
end
|
||||
""")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def script(lua):
|
||||
lua.rt.execute(DOVECOT_MOCKS)
|
||||
lua.load("dovecot/push_notification.lua")
|
||||
return lua
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def deliver(script):
|
||||
def deliver(recipient, sender):
|
||||
calls = []
|
||||
script.g.record = calls.append
|
||||
user = script.g.make_user(recipient)
|
||||
ctx = script.g.dovecot_lua_notify_begin_txn(user)
|
||||
event = script.table(mailbox="INBOX", from_address=sender)
|
||||
script.g.dovecot_lua_notify_event_message_new(ctx, event)
|
||||
script.g.dovecot_lua_notify_end_txn(ctx, True)
|
||||
return calls
|
||||
|
||||
return deliver
|
||||
|
||||
|
||||
def test_entry_points_have_the_names_dovecot_calls(script):
|
||||
assert script.g.dovecot_lua_notify_begin_txn is not None
|
||||
assert script.g.dovecot_lua_notify_event_message_new is not None
|
||||
assert script.g.dovecot_lua_notify_end_txn is not None
|
||||
|
||||
|
||||
def test_begin_txn_returns_the_user_as_event_context(script):
|
||||
user = script.g.make_user(USER1)
|
||||
ctx = script.g.dovecot_lua_notify_begin_txn(user)
|
||||
ctx.marker = "seen"
|
||||
assert user.marker == "seen"
|
||||
|
||||
|
||||
def test_incoming_message_notifies_metadata_server(deliver):
|
||||
assert deliver(USER1, sender=USER2) == [
|
||||
"mailbox INBOX",
|
||||
"sync",
|
||||
"metadata_set /private/messagenew=",
|
||||
"free",
|
||||
]
|
||||
|
||||
|
||||
def test_own_message_does_not_wake_the_sending_device(deliver):
|
||||
assert deliver(USER1, sender=USER1) == [
|
||||
"mailbox INBOX",
|
||||
"sync",
|
||||
"free",
|
||||
]
|
||||
|
||||
|
||||
def test_message_without_from_address_is_notified(deliver):
|
||||
assert deliver(USER1, sender=None) == [
|
||||
"mailbox INBOX",
|
||||
"sync",
|
||||
"metadata_set /private/messagenew=",
|
||||
"free",
|
||||
]
|
||||
@@ -3,6 +3,8 @@
|
||||
# For the full list of built-in configuration values, see the documentation:
|
||||
# https://www.sphinx-doc.org/en/master/usage/configuration.html
|
||||
|
||||
import os
|
||||
|
||||
# -- Project information -----------------------------------------------------
|
||||
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
|
||||
|
||||
@@ -16,12 +18,24 @@ author = 'chatmail collective'
|
||||
extensions = [
|
||||
#'sphinx.ext.autodoc',
|
||||
#'sphinx.ext.viewdoc',
|
||||
'sphinx.ext.extlinks',
|
||||
'sphinxcontrib.mermaid',
|
||||
]
|
||||
|
||||
templates_path = ['_templates']
|
||||
exclude_patterns = []
|
||||
|
||||
# Repository links go through the roles below.
|
||||
# CI sets DOC_GITHUB_REF to the head commit of a pull request,
|
||||
gh_ref = os.environ.get("DOC_GITHUB_REF", "main")
|
||||
|
||||
extlinks = {
|
||||
"repofile": (f"https://github.com/chatmail/relay/blob/{gh_ref}/%s", "%s"),
|
||||
"repodir": (f"https://github.com/chatmail/relay/tree/{gh_ref}/%s", "%s"),
|
||||
}
|
||||
|
||||
# Warn about repository links spelled out in full instead of using the roles.
|
||||
extlinks_detect_hardcoded_links = True
|
||||
|
||||
|
||||
# -- Options for HTML output -------------------------------------------------
|
||||
|
||||
+1
-1
@@ -54,7 +54,7 @@ How can I upgrade my chatmail relay?
|
||||
------------------------------------
|
||||
|
||||
To upgrade to the latest ``main`` branch,
|
||||
``cd`` into your local checkout of `https://github.com/chatmail/relay/`_
|
||||
``cd`` into your local checkout of https://github.com/chatmail/relay/
|
||||
and run the following commands:
|
||||
|
||||
::
|
||||
|
||||
@@ -199,6 +199,93 @@ creating addresses, login with ssh to the deployment machine and run:
|
||||
Chatmail address creation will be denied while this file is present.
|
||||
|
||||
|
||||
.. _system-limits:
|
||||
|
||||
Configurable System Limits
|
||||
--------------------------
|
||||
|
||||
Limits for auto-rejecting address creation
|
||||
..........................................
|
||||
|
||||
A relay refuses creation of new addresses
|
||||
when the machine runs low on resources,
|
||||
but existing addresses keep working.
|
||||
|
||||
Three ``chatmail.ini`` settings control this,
|
||||
shown here with their defaults::
|
||||
|
||||
max_load_1m = 5
|
||||
min_available_memory = 200M
|
||||
min_free_disk_space = 1G
|
||||
|
||||
- ``max_load_1m`` is the maximum 1-minute load average,
|
||||
as reported by ``uptime``;
|
||||
it counts processes waiting for disk I/O as well as for CPU.
|
||||
It is deliberately not scaled by the number of CPUs
|
||||
because I/O rather than CPU is what typically limits a relay.
|
||||
|
||||
- ``min_available_memory`` is the minimum memory available without swapping.
|
||||
|
||||
- ``min_free_disk_space`` is the minimum free disk space
|
||||
on the file system holding the mailboxes.
|
||||
|
||||
The defaults suit the small machine described in
|
||||
`Minimal requirements and prerequisites`_.
|
||||
|
||||
.. note::
|
||||
|
||||
If you run a bigger machine,
|
||||
raise ``max_load_1m`` after watching ``uptime`` under typical load.
|
||||
|
||||
Rejections are logged by the ``doveauth`` service,
|
||||
so you can check whether a limit is set too tightly::
|
||||
|
||||
journalctl -u doveauth --grep 'registration rejected'
|
||||
|
||||
If ``mtail_address`` is set, rejections are also counted
|
||||
in the ``rejected_registrations`` metric.
|
||||
|
||||
|
||||
Overall IMAP and SMTP connection limits
|
||||
.......................................
|
||||
|
||||
Two further settings bound how many connections
|
||||
the relay accepts at all, again shown with their defaults::
|
||||
|
||||
max_imap_connections = 10000
|
||||
max_smtp_connections = 1000
|
||||
|
||||
``max_imap_connections`` becomes the Dovecot imap process limit,
|
||||
and ``max_smtp_connections`` the Postfix process limit
|
||||
on each of the submission and smtps ports.
|
||||
A single client IP may use up to a fifth of ``max_smtp_connections``.
|
||||
Each connection costs memory,
|
||||
so these limits defend the relay against running out of RAM.
|
||||
|
||||
Unless ``imap_compress`` is enabled,
|
||||
an IMAP connection that is idle for ``imap_hibernate_timeout``
|
||||
is handed over to the ``imap-hibernate`` process
|
||||
and does not count towards ``max_imap_connections``,
|
||||
which is why a relay can serve far more IMAP clients
|
||||
than this setting suggests.
|
||||
|
||||
If you run a large relay with 10k or 100k's of addresses,
|
||||
check current connection counts before upgrading
|
||||
and set the limits accordingly.
|
||||
|
||||
To see how close a running relay is to these two limits,
|
||||
copy ``scripts/check-connections.sh`` from the relay repository
|
||||
onto the relay and run it there::
|
||||
|
||||
imap 5 ports 143,993 (max_imap_connections)
|
||||
5 dovecot sessions, 0 of them in an active imap process
|
||||
submission 0 ports 465,587 (max_smtp_connections per port)
|
||||
incoming 0 port 25 (from other relays, no chatmail.ini limit)
|
||||
|
||||
It counts established sockets with ``ss``
|
||||
and cross-checks the IMAP number against ``doveadm who``.
|
||||
|
||||
|
||||
Running a relay with self-signed certificates
|
||||
----------------------------------------------
|
||||
|
||||
|
||||
+39
-14
@@ -6,13 +6,13 @@ Technical overview
|
||||
Directories of the relay repository
|
||||
-----------------------------------
|
||||
|
||||
The `chatmail relay repository <https://github.com/chatmail/relay/tree/main/>`_
|
||||
The `chatmail relay repository <https://github.com/chatmail/relay>`_
|
||||
has four main directories.
|
||||
|
||||
``scripts/``
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
`scripts <https://github.com/chatmail/relay/tree/main/scripts>`_
|
||||
:repodir:`scripts`
|
||||
offers two convenience tools for beginners:
|
||||
|
||||
- ``initenv.sh`` installs a local virtualenv Python environment and
|
||||
@@ -71,7 +71,7 @@ The deployed system components of a chatmail relay are:
|
||||
``chatmaild/``
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
`chatmaild <https://github.com/chatmail/relay/tree/main/chatmaild>`_
|
||||
:repodir:`chatmaild`
|
||||
is a Python package containing several small services which handle
|
||||
authentication, trigger push notifications on new messages, ensure
|
||||
that outbound mails are encrypted, delete inactive users, and some
|
||||
@@ -83,25 +83,24 @@ that integrate with Dovecot and Postfix to achieve instant-onboarding
|
||||
and only relaying OpenPGP end-to-end messages encrypted messages. A
|
||||
short overview of ``chatmaild`` services:
|
||||
|
||||
- `doveauth <https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/doveauth.py>`_
|
||||
- :repofile:`doveauth <chatmaild/src/chatmaild/doveauth.py>`
|
||||
implements create-on-login address semantics and is used by Dovecot
|
||||
during IMAP login and by Postfix during SMTP/SUBMISSION login which
|
||||
in turn uses `Dovecot SASL
|
||||
<https://doc.dovecot.org/2.3/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket>`_
|
||||
to authenticate logins.
|
||||
|
||||
- `chatmail-metadata <https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/metadata.py>`_
|
||||
is contacted by a `Dovecot lua
|
||||
script <https://github.com/chatmail/relay/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua>`_
|
||||
to store user-specific relay-side config. On new messages, it `passes
|
||||
the user’s push notification
|
||||
token <https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/notifier.py>`_
|
||||
- :repofile:`chatmail-metadata <chatmaild/src/chatmaild/metadata.py>`
|
||||
is contacted by a
|
||||
:repofile:`Dovecot lua script <cmdeploy/src/cmdeploy/dovecot/push_notification.lua>`
|
||||
to store user-specific relay-side config. On new messages, it
|
||||
:repofile:`passes the user’s push notification token <chatmaild/src/chatmaild/notifier.py>`
|
||||
to
|
||||
`notifications.delta.chat <https://delta.chat/en/help#instant-delivery>`_
|
||||
so the push notifications on the user’s phone can be triggered by
|
||||
Apple/Google/Huawei.
|
||||
|
||||
- `chatmail-expire <https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/expire.py>`_
|
||||
- :repofile:`chatmail-expire <chatmaild/src/chatmaild/expire.py>`
|
||||
deletes old messages, large messages, and entire mailboxes
|
||||
of users who have not logged in for longer than
|
||||
``delete_inactive_users_after`` days.
|
||||
@@ -109,15 +108,14 @@ short overview of ``chatmaild`` services:
|
||||
- ``chatmail-quota-expire`` is called by Dovecot's ``quota_warning`` mechanism
|
||||
and will automatically remove oldest messages to keep mailboxes well under ``max_mailbox_size``.
|
||||
|
||||
- `lastlogin <https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/lastlogin.py>`_
|
||||
- :repofile:`lastlogin <chatmaild/src/chatmaild/lastlogin.py>`
|
||||
is contacted by Dovecot when a user logs in and stores the date of
|
||||
the login.
|
||||
|
||||
``www/``
|
||||
~~~~~~~~~
|
||||
|
||||
`www <https://github.com/chatmail/relay/tree/main/www>`_ contains
|
||||
the html, css, and markdown files which make up a chatmail relay’s
|
||||
:repodir:`www` contains the html, css, and markdown files which make up a chatmail relay’s
|
||||
web page. Edit them before deploying to make your chatmail relay
|
||||
stand out.
|
||||
|
||||
@@ -249,6 +247,33 @@ Fresh chatmail addresses have a mailbox directory that contains:
|
||||
directories will typically be empty unless the user of that address
|
||||
hasn’t been online for a while.
|
||||
|
||||
App version information (experimental)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
A chatmail relay ships the
|
||||
:repofile:`appversions.json <chatmaild/src/chatmaild/defaults/appversions.json>`
|
||||
file of the ``chatmaild`` package
|
||||
and serves its content under the IMAP METADATA key
|
||||
``/shared/vendor/deltachat/appversions``.
|
||||
Chat apps installed outside of app stores read this key
|
||||
to learn about updates and where to download them.
|
||||
The mechanism is experimental and may change.
|
||||
|
||||
The file travels with the normal deploy:
|
||||
update the repository checkout and run ``cmdeploy run``.
|
||||
Local modifications of ``appversions.json`` are deployed as-is,
|
||||
so you can serve your own app version information,
|
||||
including links to app downloads.
|
||||
There is no automatic refresh:
|
||||
version information changes only when you deploy again.
|
||||
|
||||
.. note::
|
||||
|
||||
Note that as of August 2026, only Delta Chat Android Google Play version
|
||||
is beginning to support discovering app versions from relays.
|
||||
Generally, consumers of relay-provided app version information
|
||||
need to verify themselves that downloaded app files are valid.
|
||||
|
||||
Active ports
|
||||
~~~~~~~~~~~~
|
||||
|
||||
|
||||
@@ -14,10 +14,11 @@ We know of three work-in-progress alternative implementation efforts:
|
||||
it to support all of the features and configuration settings required
|
||||
to operate as a chatmail relay.
|
||||
|
||||
- `Madmail <https://github.com/themadorg/madmail>`_: an
|
||||
experimental fork of `Maddy Mail Server <https://maddy.email/>`_, modified
|
||||
for chatmail deployments. It provides a single binary solution
|
||||
for running a chatmail relay.
|
||||
- `Madmail <https://github.com/themadorg/madmail>`_: a Rust-based
|
||||
single-binary chatmail relay. Madmail v2 is a rewrite of an earlier
|
||||
experimental fork of `Maddy Mail Server <https://maddy.email/>`_.
|
||||
It includes SMTP, IMAP, encryption enforcement, and real-time
|
||||
services (TURN/Iroh), and runs on Linux and Windows.
|
||||
|
||||
- `Chatmail Cookbook <https://github.com/feld/chatmail-cookbook>`_:
|
||||
A Chef Cookbook implementing a relay server. The project follows the
|
||||
|
||||
Executable
+26
@@ -0,0 +1,26 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Show current IMAP and SMTP connections on a chatmail relay,
|
||||
# to compare against the max_imap_connections and max_smtp_connections
|
||||
# settings in chatmail.ini. Run this on the relay itself.
|
||||
set -e
|
||||
|
||||
# Count established TCP connections whose local port is one of the arguments.
|
||||
established() {
|
||||
filter=$(printf 'sport = :%s or ' "$@")
|
||||
ss -Htn state established "( ${filter% or } )" | wc -l
|
||||
}
|
||||
|
||||
# doveadm prints a header line and then one line per logged-in user,
|
||||
# with that user's number of connections in the second column.
|
||||
sessions=$(doveadm who | awk 'NR > 1 { n += $2 } END { print n + 0 }')
|
||||
|
||||
# Unless imap_compress is enabled, connections idle for
|
||||
# imap_hibernate_timeout are handed over to the imap-hibernate
|
||||
# process, so they cost no imap process while idle.
|
||||
active=$(pgrep -x imap | wc -l)
|
||||
|
||||
printf 'imap %6d ports 143,993 (max_imap_connections)\n' "$(established 143 993)"
|
||||
printf ' %6d dovecot sessions, %d of them in an active imap process\n' "$sessions" "$active"
|
||||
printf 'submission %6d ports 465,587 (max_smtp_connections per port)\n' "$(established 465 587)"
|
||||
printf 'incoming %6d port 25 (from other relays, no chatmail.ini limit)\n' "$(established 25)"
|
||||
Reference in New Issue
Block a user