Compare commits

...

10 Commits

Author SHA1 Message Date
holger krekel
f65ecc23fa fix: DNS check timeout with IPv6-broken authoritative NS
Add IPv4 fallback for authoritative NS queries in query_dns().
When dig @ns returns no useful answer (e.g. IPv6 transport to
the NS times out), retry with -4 to force IPv4. Also limits
timeout and tries (+timeout=10 +tries=2) to avoid hanging.

Fixes #851
2026-04-30 23:59:41 +02:00
missytake
16b00da373 chore: prepare 1.10.0 release (#943)
Co-authored-by: j4n <j4n@systemli.org>
2026-04-30 15:51:17 +02:00
j4n
75606f5eb8 fix(mtail): start after networking is fully up 2026-04-30 14:23:32 +02:00
holger krekel
d256538f81 testing: support custom filtermail binary through CHATMAIL_FILTERMAIL_BINARY env var 2026-04-29 20:27:12 +02:00
link2xt
fdf8e5e345 ci: setup zizmor
Zizmor is a linter for GitHub Actions
2026-04-29 16:58:19 +00:00
j4n
81a161d433 feat(ci): add repository_dispatch trigger to chatmail/docker
On push to main send a repository_dispatch event to chatmail/docker with
relay_ref, relay_sha, and relay_sha_short.

This triggers docker-ci.yaml to build a new Docker image from
the updated relay code, push to GHCR, and eventually run integration
tests via cmlxc's reusable lxc-test workflow.

Requires DOCKER_DISPATCH_TOKEN secret with repo scope on
chatmail/docker.

Also set workflow_dispatch to allow manual triggering of Docker builds
from any relay branch via the GitHub UI.
2026-04-29 15:43:19 +02:00
link2xt
454ac6248a docs: add documentation on reverse DNS (PTR) records 2026-04-27 16:43:29 +00:00
link2xt
85915652b3 feat: do not bind SMTP client sockets to public addresses
This change reverts 06560dd071

Main reason for using the same address for sending
as the one used in DNS is to pass FCrDNS
(forward-confirmed reverse DNS) checks:
IP address used by SMTP client should resolve
to the domain which in turn resolves to the same IP.
chatmail relays don't do check reverse DNS
for incoming connections,
but other email servers may do and reject email
if the check does not pass.

Most chatmail relays only have one IP address per address family,
so this configuration does not change anything.

For chatmail relays that have multiple addresses
and only publishing one IP to DNS,
source address used for outgoing SMTP connections
should be the public IP.

This can be ensured by configuring the source
address in the routing table,
e.g. with the `src` argument
to `ip route add/change/replace` command.

Solving this by binding SMTP client address
on the application level prevents chatmail relays
from configuring alternative routes.

Besides, some chatmail relays are NATed
and NAT is responsible for translating the address to the public one,
in which case using `smtp_bind_address_enforce`
will result in unnecessarily deferring all mails.
2026-04-27 16:43:29 +00:00
link2xt
1e8c56e08a docs(doc/README.md): scripts/initenv.sh should be used for building the docs
doc/README.md was outdated, it did not include sphinxcontrib-mermaid.
Better use scripts/initenv.sh which already installs all dependencies
and is used in CI.
2026-04-24 21:18:58 +00:00
holger krekel
a65f082817 feat: automatic oldest-first message removal from mailboxes to (almost) always stay under max_mailbox_size
Both dovecot-quota-threshold triggers and the daily expiry routine
will now expunge oldest messages from mailboxes automatically
when the mailbox reaches 75% of max_mailbox_size.
Delta Chat users should not see any warnings (at 80/95 percent) or bounce messages,
and existing over-quota mailboxes should start receiving mails again.
2026-04-24 23:17:31 +02:00
26 changed files with 551 additions and 38 deletions

View File

@@ -9,6 +9,8 @@ on:
pull_request:
branches: [ "main" ]
permissions: {}
# Newest push wins: Prevents multiple runs from clashing and wasting runner efforts
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
@@ -25,6 +27,7 @@ jobs:
# Otherwise `test_deployed_state` will be unhappy.
with:
ref: ${{ github.event.pull_request.head.sha }}
persist-credentials: false
- name: download filtermail
run: curl -L https://github.com/chatmail/filtermail/releases/download/v0.6.1/filtermail-x86_64 -o /usr/local/bin/filtermail && chmod +x /usr/local/bin/filtermail
- name: run chatmaild tests
@@ -38,6 +41,7 @@ jobs:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.pull_request.head.sha }}
persist-credentials: false
- name: initenv
run: scripts/initenv.sh

37
.github/workflows/docker-dispatch.yaml vendored Normal file
View File

@@ -0,0 +1,37 @@
# Notify the docker repo to build and test a new image after relay CI passes.
#
# Sends a repository_dispatch event to chatmail/docker with the relay ref
# and short SHA, which triggers docker-ci.yaml to build, push to GHCR,
# and run integration tests via cmlxc.
name: Trigger Docker build
on:
push:
branches: [main]
workflow_dispatch:
permissions: {}
jobs:
dispatch:
name: Dispatch build to chatmail/docker
runs-on: ubuntu-latest
if: github.repository == 'chatmail/relay'
steps:
- name: Compute short SHA
id: sha
run: echo "short=$(echo '${{ github.sha }}' | cut -c1-7)" >> "$GITHUB_OUTPUT"
- name: Send repository_dispatch
uses: peter-evans/repository-dispatch@ff45666b9427631e3450c54a1bcbee4d9ff4d7c0 # v3
with:
token: ${{ secrets.CHATMAIL_DOCKER_DISPATCH_TOKEN }}
repository: chatmail/docker
event-type: relay-updated
client-payload: >-
{
"relay_ref": "${{ github.ref_name }}",
"relay_sha": "${{ github.sha }}",
"relay_sha_short": "${{ steps.sha.outputs.short }}"
}

View File

@@ -7,6 +7,8 @@ on:
- 'scripts/build-docs.sh'
- '.github/workflows/docs-preview.yaml'
permissions: {}
jobs:
scripts:
name: build
@@ -16,6 +18,8 @@ jobs:
url: https://staging.chatmail.at/doc/relay/${{ steps.prepare.outputs.prid }}
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: initenv
run: scripts/initenv.sh
@@ -34,18 +38,22 @@ jobs:
- name: Get Pullrequest ID
id: prepare
run: |
export PULLREQUEST_ID=$(echo "${{ github.ref }}" | cut -d "/" -f3)
export PULLREQUEST_ID=$(echo "${GITHUB_REF}" | cut -d "/" -f3)
echo "prid=$PULLREQUEST_ID" >> $GITHUB_OUTPUT
if [ $(expr length "${{ secrets.USERNAME }}") -gt "1" ]; then echo "uploadtoserver=true" >> $GITHUB_OUTPUT; fi
- run: |
echo "baseurl: /${{ steps.prepare.outputs.prid }}" >> _config.yml
echo "baseurl: /${STEPS_PREPARE_OUTPUTS_PRID}" >> _config.yml
env:
STEPS_PREPARE_OUTPUTS_PRID: ${{ steps.prepare.outputs.prid }}
- name: Upload preview
run: |
mkdir -p "$HOME/.ssh"
echo "${{ secrets.CHATMAIL_STAGING_SSHKEY }}" > "$HOME/.ssh/key"
chmod 600 "$HOME/.ssh/key"
rsync -rILvh -e "ssh -i $HOME/.ssh/key -o StrictHostKeyChecking=no" $GITHUB_WORKSPACE/doc/build/ "${{ secrets.USERNAME }}@chatmail.at:/var/www/html/staging.chatmail.at/doc/relay/${{ steps.prepare.outputs.prid }}/"
rsync -rILvh -e "ssh -i $HOME/.ssh/key -o StrictHostKeyChecking=no" $GITHUB_WORKSPACE/doc/build/ "${{ secrets.USERNAME }}@chatmail.at:/var/www/html/staging.chatmail.at/doc/relay/${STEPS_PREPARE_OUTPUTS_PRID}/"
env:
STEPS_PREPARE_OUTPUTS_PRID: ${{ steps.prepare.outputs.prid }}
- name: check links
working-directory: doc

View File

@@ -10,6 +10,8 @@ on:
- 'scripts/build-docs.sh'
- '.github/workflows/docs.yaml'
permissions: {}
jobs:
scripts:
name: build
@@ -19,6 +21,8 @@ jobs:
url: https://chatmail.at/doc/relay/
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: initenv
run: scripts/initenv.sh

26
.github/workflows/zizmor-scan.yml vendored Normal file
View File

@@ -0,0 +1,26 @@
name: GitHub Actions Security Analysis with zizmor
on:
push:
branches: ["main"]
pull_request:
branches: ["**"]
permissions: {}
jobs:
zizmor:
name: Run zizmor
runs-on: ubuntu-latest
permissions:
security-events: write # Required for upload-sarif (used by zizmor-action) to upload SARIF files.
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Run zizmor
uses: zizmorcore/zizmor-action@b1d7e1fb5de872772f31590499237e7cce841e8e # v0.5.3

7
.github/zizmor.yml vendored Normal file
View File

@@ -0,0 +1,7 @@
rules:
unpinned-uses:
config:
policies:
actions/*: ref-pin
dependabot/*: ref-pin
chatmail/*: ref-pin

View File

@@ -1,5 +1,89 @@
# Changelog for chatmail deployment
## 1.10.0 2026-04-30
* start mtail after networking is fully up <https://github.com/chatmail/relay/pull/942>
* support specifying custom filtermail binary through environment variable <https://github.com/chatmail/relay/pull/941>
* add automated zizmor scanning of github workflows <https://github.com/chatmail/relay/pull/938>
* added dispatch for *automated builds of chatmail relay docker images* <https://github.com/chatmail/relay/pull/934>
* do not bind SMTP client sockets to public addresses <https://github.com/chatmail/relay/pull/932>
* underline in docs that scripts/initenv.sh should be used for building the docs <https://github.com/chatmail/relay/pull/933>
* automatic oldest-first message removal from mailboxes to always stay under max_mailbox_size <https://github.com/chatmail/relay/pull/929>
* remove --slow from cmdeploy test <https://github.com/chatmail/relay/pull/931>
* handle missing inotify sysctl keys in containers <https://github.com/chatmail/relay/pull/930>
* replace resolvconf with static resolv.conf <https://github.com/chatmail/relay/pull/928>
* disable fsync for LMTP and IMAP services <https://github.com/chatmail/relay/pull/925>
* re-use cmlxc workflow, replacing CI with hetzner staging servers with local lxc containers <https://github.com/chatmail/relay/pull/917>
* explicitly install resolvconf <https://github.com/chatmail/relay/pull/924>
* detect stale dovecot binary and force restart in activate() <https://github.com/chatmail/relay/pull/922>
* Rename filtermail_http_port to filtermail_http_port_incoming <https://github.com/chatmail/relay/pull/921>
* consolidated is_in_container() check https://github.com/chatmail/relay/pull/920>
* restart dovecot after package replacement (rebase, test condense) <https://github.com/chatmail/relay/pull/913>
* Set permissions on dovecot pin prefs <https://github.com/chatmail/relay/pull/915>
* Route `/mxdeliv/` to configurable port <https://github.com/chatmail/relay/pull/901>
* fix VM detection, automated testing fixes, use newer chatmail-turn and move to standard BIND DNS zone format <https://github.com/chatmail/relay/pull/912>
* Upgrade to filtermail 0.6.1 <https://github.com/chatmail/relay/pull/910>
* pin dovecot packages to prevent apt upgrades <https://github.com/chatmail/relay/pull/908>
* add rpc server to cmdeploy along with client <https://github.com/chatmail/relay/pull/906>
* remove unused deps from chatmaild <https://github.com/chatmail/relay/pull/905>
* set default smtp_tls_security_level to "verify" unconditionally <https://github.com/chatmail/relay/pull/902>
* featprefer IPv4 in SMTP client <https://github.com/chatmail/relay/pull/900>
* Install dovecot .deb packages atomically <https://github.com/chatmail/relay/pull/899>
* stop installing cron package <https://github.com/chatmail/relay/pull/898>
* Rewrite dovecot install logic, update <https://github.com/chatmail/relay/pull/862>
* fix a test and some linting fixes <https://github.com/chatmail/relay/pull/897>
* Disable IP verification on domain-literal addresses <https://github.com/chatmail/relay/pull/895>
* disable installing recommended packages globally on the relay <https://github.com/chatmail/relay/pull/887>
* multiple bug fixes across chatmaild and cmdeploy <https://github.com/chatmail/relay/pull/883>
* remove /metrics from the website <https://github.com/chatmail/relay/pull/703>
* add Prometheus textfile output to fsreport <https://github.com/chatmail/relay/pull/881>
* chown opendkim: private key <https://github.com/chatmail/relay/pull/879>
* make sure chatmail-metadata was started <https://github.com/chatmail/relay/pull/882>
* dovecot update url <https://github.com/chatmail/relay/pull/880>
* upgrade to filtermail v0.5.2 <https://github.com/chatmail/relay/pull/876>
* download dovecot packages from github release <https://github.com/chatmail/relay/pull/875>
* replace DKIM verification with filtermail v0.5 <https://github.com/chatmail/relay/pull/831>
* remove CFFI deltachat bindings usage, and consolidate test support with rpc-bindings <https://github.com/chatmail/relay/pull/872>
* prepare chatmaild/cmdeploy changes for Docker support <https://github.com/chatmail/relay/pull/857>
* stabilize online benchmark timing adding rate-limit-aware cooldown between iterations <https://github.com/chatmail/relay/pull/867>
* move rate-limit cooldown to benchmark fixture <https://github.com/chatmail/relay/pull/868>
* reconfigure acmetool from redirector to proxy mode <https://github.com/chatmail/relay/pull/861>
* make tests work with `--ssh-host localhost` <https://github.com/chatmail/relay/pull/856>
* mark f-string with f prefix in test_expunged <https://github.com/chatmail/relay/pull/863>
* install also if dovecot.service=False in SystemdEnabled Fact <https://github.com/chatmail/relay/pull/841>
* Introduce support for self-signed chatmail relays <https://github.com/chatmail/relay/pull/855>
* Strip Received headers before delivery <https://github.com/chatmail/relay/pull/849>
* upgrade to filtermail v0.3 <https://github.com/chatmail/relay/pull/850>
* fix link to Maddy and update madmail URL <https://github.com/chatmail/relay/pull/847>
* accept self-signed certificates for IP-only relays <https://github.com/chatmail/relay/pull/846>
* enforce sending from public IP addresses <https://github.com/chatmail/relay/pull/845>
* port check: check addresses, fix single services <https://github.com/chatmail/relay/pull/844>
* remediates issue with improper concat on resolver injection <https://github.com/chatmail/relay/pull/834>
* ipv6 boolean not being respected during operations <https://github.com/chatmail/relay/pull/832>
* upgrade to filtermail v0.2 by <https://github.com/chatmail/relay/pull/825>
* fix link to filtermail <https://github.com/chatmail/relay/pull/824>
* print timestamps when sending messages <https://github.com/chatmail/relay/pull/823>
* fix flaky test_exceed_rate_limit <https://github.com/chatmail/relay/pull/822>
* Replace filtermail with rust reimplementation <https://github.com/chatmail/relay/pull/808>
* Set default internal SMTP ports in Config <https://github.com/chatmail/relay/pull/819>
* separate metrics for incoming and outgoing messages <https://github.com/chatmail/relay/pull/820>
* disable appending the Received header <https://github.com/chatmail/relay/pull/815>
* fail on errors in postfix/dovecot config <https://github.com/chatmail/relay/pull/813>
* tweak idle/hibernate metrics some more <https://github.com/chatmail/relay/pull/811>
* add config flag to export statistics <https://github.com/chatmail/relay/pull/806>
* add --website-only option to run subcommand <https://github.com/chatmail/relay/pull/768>
* Strip DKIM-Signature header before LMTP <https://github.com/chatmail/relay/pull/803>
* properly make sure that postfix gets restarted on failure <https://github.com/chatmail/relay/pull/802>
* expire.py: use absolute path to maildirsize <https://github.com/chatmail/relay/pull/807>
* pin Dovecot documentation URLs to version 2.3 <https://github.com/chatmail/relay/pull/800>
* try to use "build machine" and "deployment server" consistently <https://github.com/chatmail/relay/pull/797>
* adds instructions for migrating control machines <https://github.com/chatmail/relay/pull/795>
* use consistent naming schema in getting started <https://github.com/chatmail/relay/pull/793>
* remove jsok/serialize-workflow-action dependency <https://github.com/chatmail/relay/pull/790>
* streamline migration guide wording, provide titled steps <https://github.com/chatmail/relay/pull/789>
* increases default max mailbox size <https://github.com/chatmail/relay/pull/792>
* use daemon_name for OpenDKIM sign-verify decision instead of IP <https://github.com/chatmail/relay/pull/784>
## 1.9.0 2025-12-18
### Documentation

View File

@@ -21,7 +21,8 @@ where = ['src']
[project.scripts]
doveauth = "chatmaild.doveauth:main"
chatmail-metadata = "chatmaild.metadata:main"
chatmail-expire = "chatmaild.expire:main"
chatmail-expire = "chatmaild.expire:daily_expire_main"
chatmail-quota-expire = "chatmaild.expire:quota_expire_main"
chatmail-fsreport = "chatmaild.fsreport:main"
lastlogin = "chatmaild.lastlogin:main"
turnserver = "chatmaild.turnserver:main"

View File

@@ -1,4 +1,3 @@
import os
from pathlib import Path
import iniconfig
@@ -47,8 +46,6 @@ class Config:
)
self.mtail_address = params.get("mtail_address")
self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
self.addr_v4 = os.environ.get("CHATMAIL_ADDR_V4", "")
self.addr_v6 = os.environ.get("CHATMAIL_ADDR_V6", "")
self.acme_email = params.get("acme_email", "")
self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
self.imap_compress = params.get("imap_compress", "false").lower() == "true"
@@ -95,6 +92,11 @@ class Config:
# old unused option (except for first migration from sqlite to maildir store)
self.passdb_path = Path(params.get("passdb_path", "/home/vmail/passdb.sqlite"))
@property
def max_mailbox_size_mb(self):
"""Return max_mailbox_size as an integer in megabytes."""
return parse_size_mb(self.max_mailbox_size)
def _getbytefile(self):
return open(self._inipath, "rb")
@@ -108,6 +110,16 @@ class Config:
return User(maildir, addr, password_path, uid="vmail", gid="vmail")
def parse_size_mb(limit):
"""Parse a size string like ``500M`` or ``2G`` and return megabytes."""
value = limit.strip().upper().removesuffix("B")
if value.endswith("G"):
return int(value[:-1]) * 1024
if value.endswith("M"):
return int(value[:-1])
return int(value)
def write_initial_config(inipath, mail_domain, overrides):
"""Write out default config file, using the specified config value overrides."""
content = get_default_config_content(mail_domain, **overrides)

View File

@@ -4,17 +4,26 @@ Expire old messages and addresses.
"""
import os
import re
import shutil
import sys
import time
from argparse import ArgumentParser
from collections import namedtuple
from datetime import datetime
from pathlib import Path
from stat import S_ISREG
from chatmaild.config import read_config
FileEntry = namedtuple("FileEntry", ("path", "mtime", "size"))
QuotaFileEntry = namedtuple("QuotaFileEntry", ("mtime", "quota_size", "path"))
# Quota cleanup factor of max_mailbox_size. The mailbox is reset to this size.
QUOTA_CLEANUP_FACTOR = 0.7
# e.g. "cur/1775324677.M448978P3029757.exam,S=3235,W=3305:2,S"
_dovecot_fn_rex = re.compile(r".+/(\d+)\..+,S=(\d+)")
def iter_mailboxes(basedir, maxnum):
@@ -74,6 +83,42 @@ class MailboxStat:
self.extrafiles.sort(key=lambda x: -x.size)
def parse_dovecot_filename(relpath):
m = _dovecot_fn_rex.match(relpath)
if not m:
return None
return QuotaFileEntry(int(m.group(1)), int(m.group(2)), relpath)
def scan_mailbox_messages(mbox):
messages = []
for sub in ("cur", "new"):
for name in os_listdir_if_exists(mbox / sub):
if entry := parse_dovecot_filename(f"{sub}/{name}"):
messages.append(entry)
return messages
def expire_to_target(mbox, target_bytes):
messages = scan_mailbox_messages(mbox)
total_size = sum(m.quota_size for m in messages)
# Keep recent 24 hours of messages protected from expiry because
# likely something is wrong with interactions on that address
# and quota-full signal can help the address owner's device to notice it
undeletable_messages_cutoff = time.time() - (3600 * 24)
removed = 0
for entry in sorted(messages):
if total_size <= target_bytes:
break
if entry.mtime > undeletable_messages_cutoff:
break
(mbox / entry.path).unlink(missing_ok=True)
total_size -= entry.quota_size
removed += 1
return removed
def print_info(msg):
print(msg, file=sys.stderr)
@@ -143,6 +188,19 @@ class Expiry:
else:
continue
changed = True
target_bytes = (
self.config.max_mailbox_size_mb * 1024 * 1024 * QUOTA_CLEANUP_FACTOR
)
removed = expire_to_target(Path(mbox.basedir), target_bytes)
if removed:
changed = True
self.del_files += removed
if self.verbose:
print_info(
f"quota-expire: removed {removed} message(s) from {mboxname}"
)
if changed:
self.remove_file(f"{mbox.basedir}/maildirsize")
@@ -154,9 +212,9 @@ class Expiry:
)
def main(args=None):
def daily_expire_main(args=None):
"""Expire mailboxes and messages according to chatmail config"""
parser = ArgumentParser(description=main.__doc__)
parser = ArgumentParser(description=daily_expire_main.__doc__)
ini = "/usr/local/lib/chatmaild/chatmail.ini"
parser.add_argument(
"chatmail_ini",
@@ -202,5 +260,33 @@ def main(args=None):
print(exp.get_summary())
if __name__ == "__main__":
main(sys.argv[1:])
def quota_expire_main(args=None):
"""Remove mailbox messages to stay within a megabyte target.
This entry point is called by dovecot when a quota threshold is passed.
"""
parser = ArgumentParser(description=quota_expire_main.__doc__)
parser.add_argument(
"target_mb",
type=int,
help="target mailbox size in megabytes",
)
parser.add_argument(
"mailbox_path",
type=Path,
help="path to a user mailbox",
)
args = parser.parse_args(args)
target_bytes = args.target_mb * 1024 * 1024
removed_count = expire_to_target(args.mailbox_path, target_bytes)
if removed_count:
(args.mailbox_path / "maildirsize").unlink(missing_ok=True)
print(
f"quota-expire: removed {removed_count} message(s)"
f" from {args.mailbox_path.name}",
file=sys.stderr,
)
return 0

View File

@@ -18,6 +18,7 @@ max_user_send_per_minute = 60
max_user_send_burst_size = 10
# maximum mailbox size of a chatmail address
# Oldest messages will be removed automatically, so mailboxes never run full.
max_mailbox_size = 500M
# maximum message size for an e-mail in bytes

View File

@@ -1,6 +1,6 @@
import pytest
from chatmaild.config import read_config
from chatmaild.config import parse_size_mb, read_config
def test_read_config_basic(example_config):
@@ -121,3 +121,17 @@ def test_config_tls_external_bad_format(make_config):
"tls_external_cert_and_key": "/only/one/path.pem",
},
)
def test_parse_size_mb():
assert parse_size_mb("500M") == 500
assert parse_size_mb("2G") == 2048
assert parse_size_mb(" 1g ") == 1024
assert parse_size_mb("100MB") == 100
assert parse_size_mb("256") == 256
def test_max_mailbox_size_mb(make_config):
config = make_config("chat.example.org")
assert config.max_mailbox_size == "500M"
assert config.max_mailbox_size_mb == 500

View File

@@ -1,7 +1,7 @@
import time
from chatmaild.doveauth import AuthDictProxy
from chatmaild.expire import main as main_expire
from chatmaild.expire import daily_expire_main as main_expire
def test_login_timestamps(example_config):

View File

@@ -1,5 +1,7 @@
import itertools
import os
import random
import time
from datetime import datetime
from fnmatch import fnmatch
from pathlib import Path
@@ -9,13 +11,19 @@ import pytest
from chatmaild.expire import (
FileEntry,
MailboxStat,
expire_to_target,
get_file_entry,
iter_mailboxes,
os_listdir_if_exists,
parse_dovecot_filename,
quota_expire_main,
scan_mailbox_messages,
)
from chatmaild.expire import main as expiry_main
from chatmaild.expire import daily_expire_main as expiry_main
from chatmaild.fsreport import main as report_main
MB = 1024 * 1024
def fill_mbox(folderdir):
password = folderdir.joinpath("password")
@@ -196,3 +204,51 @@ def test_os_listdir_if_exists(tmp_path):
tmp_path.joinpath("x").write_text("hello")
assert len(os_listdir_if_exists(str(tmp_path))) == 1
assert len(os_listdir_if_exists(str(tmp_path.joinpath("123123")))) == 0
# --- quota expire tests ---
_msg_counter = itertools.count(1)
def _create_message(basedir, sub, size, days_old=0, disk_size=None):
seq = next(_msg_counter)
mtime = int(time.time() - days_old * 86400)
name = f"{mtime}.M1P1Q{seq}.hostname,S={size},W={size}:2,S"
path = basedir / sub / name
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(b"x" * (disk_size if disk_size is not None else size))
os.utime(path, (mtime, mtime))
return path
def test_parse_dovecot_filename():
e = parse_dovecot_filename("cur/1775324677.M448978P3029757.exam,S=3235,W=3305:2,S")
assert e.path == "cur/1775324677.M448978P3029757.exam,S=3235,W=3305:2,S"
assert e.mtime == 1775324677
assert e.quota_size == 3235
assert parse_dovecot_filename("cur/msg_without_structure") is None
def test_expire_to_target(tmp_path):
_create_message(tmp_path, "cur", MB, days_old=10, disk_size=100)
_create_message(tmp_path, "new", MB, days_old=5)
_create_message(tmp_path, "cur", MB, days_old=0) # undeletable (<1 hour)
assert len(scan_mailbox_messages(tmp_path)) == 3
# removes oldest first, uses S= size not disk size
removed = expire_to_target(tmp_path, MB)
assert removed == 2
msgs = scan_mailbox_messages(tmp_path)
assert len(msgs) == 1
# the surviving message is the fresh undeletable one
assert msgs[0].mtime > time.time() - 3600
def test_quota_expire_main(tmp_path, capsys):
mbox = tmp_path / "user@example.org"
_create_message(mbox, "cur", 2 * MB, days_old=5)
(mbox / "maildirsize").write_text("x")
quota_expire_main([str(1), str(mbox)])
_, err = capsys.readouterr()
assert "quota-expire: removed 1 message(s) from user@example.org" in err
assert not (mbox / "maildirsize").exists()

View File

@@ -101,9 +101,6 @@ def run_cmd(args, out):
env["CHATMAIL_WEBSITE_ONLY"] = "True" if args.website_only else ""
env["CHATMAIL_DISABLE_MAIL"] = "True" if args.disable_mail else ""
env["CHATMAIL_REQUIRE_IROH"] = "True" if require_iroh else ""
if not args.dns_check_disabled:
env["CHATMAIL_ADDR_V4"] = remote_data.get("A") or ""
env["CHATMAIL_ADDR_V6"] = remote_data.get("AAAA") or ""
deploy_path = importlib.resources.files(__package__).joinpath("run.py").resolve()
pyinf = "pyinfra --dry" if args.dry_run else "pyinfra"

View File

@@ -149,12 +149,26 @@ plugin {
}
plugin {
# for now we define static quota-rules for all users
quota = maildir:User quota
quota_rule = *:storage={{ config.max_mailbox_size }}
quota_max_mail_size={{ config.max_message_size }}
quota_grace = 0
# quota_over_flag_value = TRUE
quota_rule = *:storage={{ config.max_mailbox_size_mb }}M
# Trigger at 75%% of quota, expire oldest messages down to 70%%.
# The percentages are chosen to prevent current Delta Chat users
# from seeing "quota warnings" which trigger at 80% and 95%.
quota_warning = storage=75%% quota-warning {{ config.max_mailbox_size_mb * 70 // 100 }} {{ config.mailboxes_dir }}/%u
}
service quota-warning {
executable = script /usr/local/lib/chatmaild/venv/bin/chatmail-quota-expire
user = vmail
unix_listener quota-warning {
user = vmail
mode = 0600
}
}
# push_notification configuration

View File

@@ -1,3 +1,5 @@
import os
from pyinfra import facts, host
from pyinfra.operations import files, systemd
@@ -13,6 +15,16 @@ class FiltermailDeployer(Deployer):
self.need_restart = False
def install(self):
local_bin = os.environ.get("CHATMAIL_FILTERMAIL_BINARY")
if local_bin:
self.need_restart |= files.put(
name="Upload locally built filtermail",
src=local_bin,
dest=self.bin_path,
mode="755",
).changed
return
arch = host.get_fact(facts.server.Arch)
url = f"https://github.com/chatmail/filtermail/releases/download/v0.6.1/filtermail-{arch}"
sha256sum = {

View File

@@ -78,3 +78,11 @@ counter rejected_unencrypted_mail_count
/Rejected unencrypted mail/ {
rejected_unencrypted_mail_count++
}
counter quota_expire_runs
counter quota_expire_removed_files
/quota-expire: removed (?P<count>\d+) message\(s\)/ {
quota_expire_runs++
quota_expire_removed_files += $count
}

View File

@@ -1,5 +1,6 @@
[Unit]
Description=mtail
After=multi-user.target
[Service]
Type=simple

View File

@@ -69,15 +69,6 @@ mynetworks = 127.0.0.0/8
{% else %}
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
{% endif %}
{% if config.addr_v4 %}
smtp_bind_address = {{ config.addr_v4 }}
{% endif %}
{% if config.addr_v6 %}
smtp_bind_address6 = {{ config.addr_v6 }}
{% endif %}
{% if config.addr_v4 or config.addr_v6 %}
smtp_bind_address_enforce = yes
{% endif %}
mailbox_size_limit = 0
message_size_limit = {{config.max_message_size}}
recipient_delimiter = +

View File

@@ -64,11 +64,13 @@ def get_dkim_entry(mail_domain, pre_command, dkim_selector):
)
def query_dns(typ, domain):
def query_dns(typ, domain, shell_exec=None):
if shell_exec is None:
shell_exec = shell
# Get autoritative nameserver from the SOA record.
soa_answers = [
x.split()
for x in shell(
for x in shell_exec(
f"dig -r -q {domain} -t SOA +noall +authority +answer", print=log_progress
).split("\n")
]
@@ -78,8 +80,27 @@ def query_dns(typ, domain):
ns = soa[0][4]
# Query authoritative nameserver directly to bypass DNS cache.
res = shell(f"dig @{ns} -r -q {domain} -t {typ} +short", print=log_progress)
return next((line for line in res.split("\n") if not line.startswith(";")), "")
return _dig_authoritative(ns, domain, typ, shell_exec=shell_exec)
def _parse_dig_result(output):
"""Return first non-comment, non-empty line from dig output, or empty string."""
lines = [line for line in output.split("\n") if not line.startswith(";")]
return next((line for line in lines if line.strip()), "")
def _dig_authoritative(ns, domain, typ, shell_exec=None):
"""Query authoritative NS, falling back to IPv4-only if default fails."""
if shell_exec is None:
shell_exec = shell
# limit timeout and tries to not hang on a broken default NS
cmd = f"dig @{ns} -r -q {domain} -t {typ} +short +timeout=10 +tries=2"
result = _parse_dig_result(shell_exec(cmd, print=log_progress))
if result:
return result
# Fallback: force IPv4 transport (handles broken IPv6 to NS)
return _parse_dig_result(shell_exec(cmd + " -4", print=log_progress))
def check_zonefile(zonefile, verbose=True):

View File

@@ -214,3 +214,59 @@ class TestZonefileChecks:
assert not mockout.captured_red
assert "correct" in mockout.captured_green[0]
assert not mockout.captured_red
class TestDigAuthoritative:
@pytest.fixture
def dig_auth(self):
"""Helper that calls _dig_authoritative with a recording shell function."""
calls = []
def run(first_result, ipv4_result=None):
def shell(cmd, print=print):
calls.append(cmd)
if "-4" in cmd:
return ipv4_result or ""
return first_result
result = remote.rdns._dig_authoritative(
"ns1.example.", "example.com", "A", shell_exec=shell
)
return result, calls
return run
def test_ipv4_fallback_on_error(self, dig_auth):
"""Fallback to -4 when first query returns only error lines."""
result, calls = dig_auth(
first_result=(
";; communications error to 2a01:4f8:10a:1044::80#53: timed out\n"
";; communications error to 2a01:4f8:10a:1044::80#53: timed out\n"
),
ipv4_result="1.2.3.4",
)
assert len(calls) == 2
assert "-4" not in calls[0]
assert "-4" in calls[1]
assert result == "1.2.3.4"
def test_no_fallback_on_success(self, dig_auth):
"""No -4 fallback when first query returns a valid answer."""
result, calls = dig_auth(first_result="1.2.3.4")
assert result == "1.2.3.4"
assert len(calls) == 1
def test_fallback_with_mixed_error_lines(self, dig_auth):
"""Fallback triggers when output has only error lines mixed with empty."""
result, calls = dig_auth(
first_result=(
";; communications error to 2a01:4f8::80#53: timed out\n"
"\n"
";; communications error to 2a01:4f8::80#53: timed out\n"
),
ipv4_result=";; some warning\n1.2.3.4",
)
assert len(calls) == 2
assert result == "1.2.3.4"

View File

@@ -4,12 +4,14 @@
You can use the `make` command and `make html` to build web pages.
You need a Python environment where the following install was excuted:
pip install furo sphinx-autobuild
You need a Python environment with `sphinx` and other
dependencies, you can create it by running `scripts/initenv.sh`
from the repository root.
To develop/change documentation, you can then do:
. venv/bin/activate
cd doc
make auto
A page will open at https://127.0.0.1:8000/ serving the docs and it will

View File

@@ -16,5 +16,6 @@ Contributions and feedback welcome through the https://github.com/chatmail/relay
proxy
migrate
overview
reverse_dns
related
faq

View File

@@ -102,8 +102,12 @@ short overview of ``chatmaild`` services:
Apple/Google/Huawei.
- `chatmail-expire <https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/expire.py>`_
deletes users if they have not logged in for a longer while.
The timeframe can be configured in ``chatmail.ini``.
deletes old messages, large messages, and entire mailboxes
of users who have not logged in for longer than
``delete_inactive_users_after`` days.
- ``chatmail-quota-expire`` is called by Dovecot's ``quota_warning`` mechanism
and will automatically remove oldest messages to keep mailboxes well under ``max_mailbox_size``.
- `lastlogin <https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/lastlogin.py>`_
is contacted by Dovecot when a user logs in and stores the date of
@@ -156,6 +160,8 @@ Chatmail relay dependency diagram
/home/vmail/.../user"];
dovecot --- |lastlogin.socket|lastlogin;
dovecot --- chatmail-metadata;
dovecot --- |quota-warning|chatmail-quota-expire;
chatmail-quota-expire --- maildir;
lastlogin --- maildir;
doveauth --- maildir;
chatmail-expire-daily --- maildir;

View File

@@ -0,0 +1,64 @@
Configuring reverse DNS
=======================
Some email servers reject the emails
if they don't pass `FCrDNS`_ check, also known as `iprev`_ check.
.. _FCrDNS: https://en.wikipedia.org/wiki/Forward-confirmed_reverse_DNS
.. _iprev: https://datatracker.ietf.org/doc/html/rfc8601#section-3
Passing the check requires that the IP address that email is sent from
should have a ``PTR`` record pointing to the domain name of the server,
and domain name record should have an ``A/AAAA`` record
pointing to the IP address.
Modern email relies on DKIM and SPF for authentication,
while iprev check exists for
`historical reasons <https://datatracker.ietf.org/doc/html/draft-ietf-dnsop-reverse-mapping-considerations-06#section-2.1>`_.
Chatmail relays don't resolve ``PTR`` records,
so you can ignore this section if configuring ``PTR`` records
is difficult and federation with legacy email servers that don't accept
valid DKIM signature for authentication is not important.
Multi-homed setups
------------------
If you have a server with multiple IP addresses,
also known as multi-homed setup,
and don't publish all IP addresses in DNS,
you need to make sure you are using
the published address when making outgoing connections.
For example, your server may have a static IP
address, and a so-called Floating IP or Virtual IP
that can be moved between servers in case of
migration or for failover.
By using Floating IP you can avoid downtime
and keep the IP address reputation
for destinatinons that rely on IP reputation and IP blocklists.
In this case you will only publish
the Floating IP to DNS and only use the static IP
to SSH into the server.
If you have such setup, make sure that
you not only set ``PTR`` records for the Floating IP,
but make outgoing connections using the Floating IP.
Otherwise reverse DNS check succeed,
but forward check making sure your domain name points
to the IP address will fail.
Such setup is indistinguishable from someone
setting IP address ``PTR`` with the domain they don't own
and as a result don't succeed.
On Linux you can configure source IP address with ``ip route`` command,
for example:
::
ip route change default via <default-gateway> dev eth0 src <source-address>
Make sure to persist the change after verifying it is working.
You can check what your outgoing IP address is
with ``curl icanhazip.com``.
Check both the IPv4 and IPv6 addresses.
For IPv4 address use ``curl ipv4.icanhazip.com`` or ``curl -4 icanhazip.com``
and similarly for IPv6 if you have it.