Commit Graph

485 Commits

Author SHA1 Message Date
holger krekel
010e9de08e lxc: poll until DNS answers before continuing; reset bridge on --destroy-all
After restarting pdns/pdns-recursor, wait up to 10 s for the recursor
to actually answer a query before proceeding.  Likewise, after
configure_dns(), poll from inside the relay container until the
configured DNS IP responds.

On --destroy-all, unset the incusbr0 dns.mode and raw.dnsmasq network
options so the next lxc-start starts from a clean bridge state.

DNSConfigurationError (caught in main()) is raised on timeout so the
CLI prints a clean error instead of failing later with a cryptic message.
2026-03-16 13:27:12 +01:00
holger krekel
75c42dc8c7 ignore sysctl permission problems (likely in containers) 2026-03-16 13:27:12 +01:00
holger krekel
cd16ef8c4a address link2xt review comments 2026-03-16 13:27:12 +01:00
holger krekel
ea7b875eb1 address link2xt comments (zone parsing and turn v0.4 release 2026-03-16 13:27:12 +01:00
holger krekel
3021e47866 make helpers testable and test them, also streamline intro of docs 2026-03-16 13:27:12 +01:00
holger krekel
b59417128c fix lxc-test to not re-run deploy when nothing changed + some other beautifications 2026-03-16 13:27:12 +01:00
holger krekel
49705863d3 don't use env vars but explicit pytest options to pass ssh info around. 2026-03-16 13:27:12 +01:00
holger krekel
861fdf7a50 feat: add LXC container support for local chatmail development
Add cmdeploy "lxc-test" command to run cmdeploy against local containers,
with supplementary lxc-start, lxc-stop and lxc-status subcommands.
See doc/source/lxc.rst for full documentation including prerequisites,
DNS setup, TLS handling, DNS-free testing, and known limitations.

Apart from adding lxc-specific docs, tests, and implementation files in the cmdeploy/lxc directory,
this PR adds the --ssh-config option to cmdeploy run/dns/status/test commands and pyinfra invocations,
and also to sshexec (Execnet) handling.  This allows for the host to need no DNS entries for a relay,
and route all resolution through ssh-config.  This is used by the "lxc-test" command, which performs
a completely local setup -- again, see docs for more details.

While working on DNS/SSH things i also unified all zone-file handling
to use actual BIND format as it is easy enough to parse back.
2026-03-16 13:27:12 +01:00
holger krekel
4fabfb31f8 fix test and some linting fixes 2026-03-16 13:25:57 +01:00
Jagoda Ślązak
36478dbfcf feat(filtermail): Disable IP verification on domain-literal addresses
Disables IP verification by upgrading filtermail to v0.6,
changelog: <https://github.com/chatmail/filtermail/releases/tag/v0.6.0>

Messages using domain-literal addresses no longer require
to match the origin SMTP connection IP anymore.

This allows for example a relay using IPv4 email addresses
to send messages to other relays over IPv6.

This is not considering a breaking change as IP-address-only
relays are not considered a stable feature.

Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
2026-03-13 20:47:10 +01:00
holger krekel
ff541b81ea chore: prevent installing recommended packages (e.g. installing cron leads to installing exim without it). 2026-03-08 23:40:16 +01:00
Alex V.
ed9b4092a8 test: add error-path tests for all bug fixes
- test_doveauth: invalid localpart chars rejected, concurrent same-account creation
- test_expire: --mdir filtering uses msg.path correctly
- test_metadata: TURN exception returns N\n, success returns credentials
- test_turnserver: socket timeout, connection refused, happy path
- test_dns: get_dkim_entry returns (None, None) on CalledProcessError
- test_rshell: dovecot_recalc_quota handles empty/malformed output
2026-03-05 16:27:15 +01:00
Alex V.
ae2ab52aa9 fix(security): remove deprecated TLS 1.0/1.1 from nginx config
TLS 1.0/1.1 deprecated by RFC 8996. Nginx default is TLSv1.2 TLSv1.3.
Aligns with postfix (>=TLSv1.2) and dovecot (TLSv1.3) in the same stack.
2026-03-05 16:27:15 +01:00
Alex V.
78a4e28408 fix: guard against IndexError in dovecot_recalc_quota
doveadm output ends with empty line, parts=[] causes parts[2] crash.
2026-03-05 16:27:15 +01:00
Alex V.
2432d4f498 fix: remove dead code and potential NameError in run_cmd
check_call always returns 0 or raises, making retcode!=0 branches
unreachable. Also remote_data was undefined with --skip-dns-check.
2026-03-05 16:27:15 +01:00
Alex V.
31301abb42 fix: handle build_webpages returning None in WebsiteDeployer
Exception in _build_webpages was silently caught, returning None.
rsync then received "None/" as source path, silently breaking deploy.
2026-03-05 16:27:15 +01:00
Alex V.
6b4edd8502 fix: return tuple from get_dkim_entry on CalledProcessError
Bare return yielded None, causing TypeError on tuple unpacking
in perform_initial_checks on fresh servers without DKIM keys.
2026-03-05 16:27:15 +01:00
Alex V.
9c467ab3e8 chore: fix ruff formatting in acmetool, dovecot, postfix deployers 2026-03-05 16:27:15 +01:00
link2xt
774350778b feat: remove /metrics from the website
Similar data is already generated by fsreport
available for the relay operator
and metrics for prometheus are generated by mtail.

Closes <https://github.com/chatmail/relay/issues/431>
2026-03-05 14:58:11 +01:00
j4n
06d53503e5 feat(chatmaild/fsreport): add Prometheus textfile output, count files
- Count files in report
- Extend size buckets to bigger messages (5, 10 MiB)
- Two textfile exporters:
  - Full, bucketed size statistics with --textfile option
  - Account count only matching metrics.py format with --legacy-metrics
    option (filename defaults to /var/www/html/metrics)
- Improve option help texts
2026-03-05 13:52:09 +01:00
missytake
2e38c61ca2 opendkim: chown opendkim: private key 2026-03-05 11:24:06 +01:00
missytake
9dd8ce8ce1 tests: make sure chatmail-metadata was started
fix a flaky test: https://github.com/chatmail/relay/pull/856#issuecomment-3919881473
since #856 chatmail-metadata is restarted every 5 second, if it didn't come up after that, the failure likely sits deeper.
2026-03-04 18:53:31 +01:00
j4n
0ae3f94ecc fix(cmdeploy): dovecot update url 2026-03-04 17:19:14 +01:00
Jagoda Ślązak
4481a12369 chore(deps): upgrade to filtermail v0.5.2
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
2026-03-04 15:53:50 +01:00
j4n
4e6ba7378d feat(cmdeploy): fall back to github url for dovecot 2026-03-02 10:29:03 +01:00
j4n
e428c646d1 fix(dovecot): download dovecot packages from github release 2026-02-26 21:06:55 +01:00
Jagoda Estera Ślązak
dbd5cd16f5 feat: replace DKIM verification with filtermail v0.5 (#831)
Upgrade to filtermail v0.5, which has a built-in DKIM verifier
and disable OpenDKIM on reinject_incoming.

Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
2026-02-25 12:39:33 +01:00
holger krekel
e21f2a0fa2 feat: support externally managed TLS via tls_external_cert_and_key option (#860)
Adds a new tls_external_cert_and_key config option for chatmail servers
that manage their own TLS certificates (e.g. via an external ACME client
or a load balancer).

A systemd path unit (tls-cert-reload.path) watches the certificate file
via inotify and automatically reloads dovecot and nginx when it changes.
Postfix reads certs per TLS handshake so needs no reload.

Also extracts openssl_selfsigned_args() so cert generation parameters
are shared between SelfSignedTlsDeployer and the e2e test.
2026-02-24 09:46:38 +01:00
holger krekel
8ca0909fa5 cleanup: remove CFFI deltachat bindings usage, and consolidate test support with rpc-bindings (#872)
* cleanup: remove CFFI deltachat bindings usage, and consolidate test support with rpc-bindings

major simplification: all chatmail fixtures used in the test are now created inside the cmdeploy plugin,
and do not inherit anything from other fixture machineries, let alone the legacy deltachat CFFI ones.
also fix that pytest report headers show correct chatmail domains under test
2026-02-24 08:27:56 +01:00
j4n
2c99cc84aa cmdeploy: prepare chatmaild/cmdeploy changes for Docker support
- chatmaild:
  - basedeploy.py: Add has_systemd() guard. During Docker image builds
    there's no running systemd, so deployers that query SystemdEnabled
    facts would crash; this change might also be helpful for non-systemd
    platforms.
- cmdeploy:
  - cmdeploy.py:
    - when deploying to @docker, auto-set CHATMAIL_NOPORTCHECK and
      CHATMAIL_NOSYSCTL since neither makes sense inside a container
    - --config default now reads CHATMAIL_INI env var, so Docker
      entrypoints can point to a mounted ini without CLI flags.
  - deployers.py:
    - skip port check / CHATMAIL_NOPORTCHECK
    - skip echobot systemd cleanup w/ has_systemd
  - dovecot/deployer.py:
    - Guard sysctl writes behind CHATMAIL_NOSYSCTL
    - invert dovecot install check so it works without systemd
  - sshexec.py: Add __call__ to LocalExec so cmdeploy status works with
    @local target. Without it, cmdeploy status tried to call the
    executor directly and got TypeError.

Consolidated from j4n/docker branch commits (selection):
- 8953fde feat(cmdeploy): read CHATMAIL_INI env var for default --config path
- 81d7782 fix(cmdeploy): add __call__ to LocalExec so status works with @local
- 8bba78e docker: disable port check if docker is running. fix #694
- 865b514 docker: replace config flags with env vars, drop docker param (instead of f26cb08)

Files: cmdeploy/src/cmdeploy/{basedeploy,cmdeploy,deployers,sshexec,dovecot/deployer}.py

Co-authored-by: Keonik1 <keonik.dev@gmail.com>
Co-authored-by: missytake <missytake@systemli.org>
2026-02-23 09:12:48 +01:00
holger krekel
7b5b180b4b refactor(benchmark): move rate-limit cooldown to benchmark fixture 2026-02-22 18:26:15 +01:00
373[Ø]™
193624e522 fix(benchmark): add rate-limit refill cooldown for send_10_receive_10 and avoid fixture signature mismatch 2026-02-22 15:58:21 +00:00
373[Ø]™
437287fadc feat(tests): add optional benchmark cooldown between iterations 2026-02-22 15:55:03 +00:00
link2xt
0ad679997a feat: reconfigure acmetool from redirector to proxy mode
This elimitates the problem of acmetool failing
to start when nginx is installed already and uses port 80.

This also makes nginx redirect HTTP requests to HTTPS
for setups that don't have acmetool.
2026-02-21 22:10:20 +00:00
missytake
38cc1c7cd6 fix(cmdeploy): make tests work with --ssh-host localhost (#856)
* tests: fix test_remote[imap]
* cmdeploy: call LocalExec directly, not .logged()
* tests: fix TestSSHExecutor.test_logged
* tests: fix test_status_cmd with --ssh-host @local
* tests: fix test_logged with --ssh-host localhost
* tests: fix TestSSHExecutor::test_exception with --ssh-host localhost
* ci: deploy with --ssh-host localhost on staging-ipv4
* metadata: lower RestartSec
2026-02-19 21:34:39 +01:00
link2xt
7a6ed8340e test: mark f-string with f prefix in test_expunged
This one was not marked accidentally.
2026-02-19 19:41:14 +00:00
missytake
2ce9e5fe78 dovecot: install also if dovecot.service=False in SystemdEnabled Fact 2026-02-19 16:00:25 +01:00
holger krekel
cf96be2cbb feat: support self-signed chatmail relays (#855)
feat: support self-signed TLS via underscore domain convention
Domains starting with "_" (e.g. _chat.example.org) automatically use
self-signed TLS certificates instead of ACME/Let's Encrypt. The TLS
mode is derived from the domain name — no separate config option needed.

Internally, when config.tls_cert_mode is "self" (underscore domain):
- Generate self-signed certificates via openssl
- Set Postfix smtp_tls_security_level to "encrypt" (opportunistic TLS)
- Add smtp_tls_policy_map entry for underscore domains
- Skip ACME, MTA-STS and www CNAME checks in `cmdeploy dns`
- Serve /new via GET (not redirect to dcaccount:) with rate-limiting
  (nginx limit_req, 2r/s burst=5)
- Return dclogin: URLs with ic=3 (AcceptInvalidCertificates) from /new
- Render QR codes client-side via JavaScript and qrcode-svg
- Use config.tls_cert_path/tls_key_path in Postfix, Dovecot and nginx
  templates instead of hardcoded ACME paths
2026-02-19 10:27:41 +01:00
Mark Felder
36eb63faa1 feat: Strip Received headers before delivery 2026-02-17 21:16:11 +01:00
Jagoda Estera Ślązak
91df11015e chore(deps): upgrade to filtermail v0.3 (#850)
## 0.3.0 - 2026-02-14

### Features

- Support legacy, pre-OpenPGP packet format

### Miscellaneous Tasks

- *(dist)* Switch to musl targets

### Refactor

- Remove unnecessary Arc
- Use a custom, minimal SMTP client instead of lettre

Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
2026-02-14 18:02:05 +01:00
missytake
0144fc3ea8 postfix: only look for square brackets, they are only allowed for address literals 2026-02-12 10:45:15 +01:00
missytake
e7ce6679b9 postfix: IPv6 literals have a prefix 2026-02-12 10:45:15 +01:00
missytake
d1adf52f89 postfix: also accept self-signed for IPv6-only 2026-02-12 10:45:15 +01:00
missytake
56d0e2ca27 postfix: be more exact with nauta.cu 2026-02-12 10:45:15 +01:00
missytake
2613558db6 postfix uses POSIX EREs, not PCRE, so some stuff doesn't work 2026-02-12 10:45:15 +01:00
missytake
6843fcb1a0 postfix: fix tls policy regexp map 2026-02-12 10:45:15 +01:00
missytake
ff54ad88d8 postfix: use regexp to match IPv4 addresses 2026-02-12 10:45:15 +01:00
missytake
cce2b27ae7 postfix: accept self-signed certificates for IP-only relays 2026-02-12 10:45:15 +01:00
j4n
87022e3681 fix(cmdeploy): check if dns_check_disabled before trying to warn about LE
If --skip-dns-check is used and retcode != 0, remote_data is undefined.
2026-02-11 12:13:24 +01:00
j4n
06560dd071 feat(postfix): bind to mail_domain's A/AAAA addresses for outbound mail
Carry forward A/AAAA address from the DNS check to the postfix deploy
stage and set accordingly in main.cf.
2026-02-11 12:13:24 +01:00