Commit Graph

1236 Commits

Author SHA1 Message Date
holger krekel
22a2b6a1c4 remove superflous sepchar argument 2026-03-16 13:27:12 +01:00
holger krekel
ef7e06d9f9 remove memory limits in containers 2026-03-16 13:27:12 +01:00
holger krekel
7450143c86 fix a potential hang because "journalctl -f" never finishes by itself, and handles might stay open if we don't close them during test teardown. 2026-03-16 13:27:12 +01:00
holger krekel
8a49489d54 try to discover and use host dns settings instead of 9.9.9.9 2026-03-16 13:27:12 +01:00
holger krekel
dcb9fbe73f make sure subprocesses by defaulit don't inherit stdin FD 2026-03-16 13:27:12 +01:00
holger krekel
da5c248562 try fix some test hangs potentially caused by filtermail's DNS cache 2026-03-16 13:27:12 +01:00
holger krekel
550498e936 force tcp connections for dns resolution 2026-03-16 13:27:12 +01:00
holger krekel
c4b018b7f2 change all timeouts to 60 2026-03-16 13:27:12 +01:00
holger krekel
7570c57d7e remove superflous """\ 2026-03-16 13:27:12 +01:00
holger krekel
68d1fa8f01 only use explicit server settings if the host resolves to ip address via ssh config 2026-03-16 13:27:12 +01:00
holger krekel
785efabe4c move --verbose option back to subcommands 2026-03-16 13:27:12 +01:00
holger krekel
3ba2703d04 cmdeploy: replace globals() subcommand scan with explicit SUBCOMMANDS list
The get_parser() loop that scanned globals() for *_cmd names was fragile
and forced # noqa: F401 on all lxc imports (ruff couldn't see they were
used dynamically).

Replace it with an explicit SUBCOMMANDS list of
(cmd_func, options_func, needs_config) tuples.  This makes the full set
of subcommands visible at a glance, their registration order defined,
and the imports unconditionally used (no more noqa suppressions).
2026-03-16 13:27:12 +01:00
holger krekel
6c95eeea80 lxc: dovecot sysctl: warn but skip when running in shared-kernel container
Replace the CHATMAIL_NOSYSCTL guard with an explicit systemd-detect-virt -c check.
2026-03-16 13:27:12 +01:00
holger krekel
783904f244 lxc: code cleanup and docs polish from review
Code:
- Fix check_ssh_config_include() to do a case-insensitive line match
  (read_text().splitlines() instead of filter(None, map(..., open()))).
- Drop the default help_text= from _add_name_args(); callers now must
  supply an explicit string.
- Expand the DNSContainer class docstring.
- Fix sysctl comment: note that incus provides net.* virtualization
  so the sysctl only affects the container's network namespace.

Docs (doc/source/lxc.rst):
- Remove double blank line after page title; fix missing comma.
- Replace the plain-text root-access note with a .. caution:: block.
- Tighten the Quick-start section and lxc-test CLI entry.
2026-03-16 13:27:12 +01:00
holger krekel
405dc2d1ee lxc: simplify to a single find_image(aliases) method
Replace the two-function find_relay_image / _find_relay_image pair
with Incus.find_image(aliases), which returns the first alias
that exists in the local image store, or None.

Container.launch() passes [RELAY_IMAGE_ALIAS, BASE_IMAGE_ALIAS]
and ensure_base_image() passes [BASE_IMAGE_ALIAS].
2026-03-16 13:27:12 +01:00
holger krekel
62fe113b59 lxc: extract blocked_service_startup() context manager into basedeploy
Move the policy-rc.d install/remove boilerplate into a shared context
manager in basedeploy.py so both UnboundDeployer and DovecotDeployer use
the same abstraction, and the DNS container's _install_powerdns() inline
shell uses the same pattern.

DovecotDeployer now wraps its three package installs in
blocked_service_startup() to prevent Dovecot from auto-starting on
initial install — avoiding bind conflicts on IPv4-only systems.
2026-03-16 13:27:12 +01:00
holger krekel
a102ed7d61 lxc: add Out class with --verbose, section timing, and coloured shell output
Move the Out output-printer class to cmdeploy/util.py so it is shared
across CLI modules.  All print/shell calls in lxc/cli.py, lxc/incus.py,
and dns.py now route through Out instead of bare print().

Key additions:
- Out.section() / Out.section_line(): coloured section headers scaled
  to the current terminal width (or $_CMDEPLOY_WIDTH for sub-processes).
- Out.shell(): merges stdout/stderr, prefixes each output line, and
  prints a red error line with the exit code on failure.
- Out.new_prefixed_out(): indented sub-printer that shares section_timings.
- 'cmdeploy -v / -vv' exposes the verbosity levels.
- Tests for Out added to test_util.py.
2026-03-16 13:27:12 +01:00
holger krekel
010e9de08e lxc: poll until DNS answers before continuing; reset bridge on --destroy-all
After restarting pdns/pdns-recursor, wait up to 10 s for the recursor
to actually answer a query before proceeding.  Likewise, after
configure_dns(), poll from inside the relay container until the
configured DNS IP responds.

On --destroy-all, unset the incusbr0 dns.mode and raw.dnsmasq network
options so the next lxc-start starts from a clean bridge state.

DNSConfigurationError (caught in main()) is raised on timeout so the
CLI prints a clean error instead of failing later with a cryptic message.
2026-03-16 13:27:12 +01:00
holger krekel
75c42dc8c7 ignore sysctl permission problems (likely in containers) 2026-03-16 13:27:12 +01:00
holger krekel
cd16ef8c4a address link2xt review comments 2026-03-16 13:27:12 +01:00
holger krekel
ea7b875eb1 address link2xt comments (zone parsing and turn v0.4 release 2026-03-16 13:27:12 +01:00
holger krekel
1a578ecc66 simplify start instructions 2026-03-16 13:27:12 +01:00
holger krekel
3021e47866 make helpers testable and test them, also streamline intro of docs 2026-03-16 13:27:12 +01:00
holger krekel
b59417128c fix lxc-test to not re-run deploy when nothing changed + some other beautifications 2026-03-16 13:27:12 +01:00
holger krekel
49705863d3 don't use env vars but explicit pytest options to pass ssh info around. 2026-03-16 13:27:12 +01:00
holger krekel
861fdf7a50 feat: add LXC container support for local chatmail development
Add cmdeploy "lxc-test" command to run cmdeploy against local containers,
with supplementary lxc-start, lxc-stop and lxc-status subcommands.
See doc/source/lxc.rst for full documentation including prerequisites,
DNS setup, TLS handling, DNS-free testing, and known limitations.

Apart from adding lxc-specific docs, tests, and implementation files in the cmdeploy/lxc directory,
this PR adds the --ssh-config option to cmdeploy run/dns/status/test commands and pyinfra invocations,
and also to sshexec (Execnet) handling.  This allows for the host to need no DNS entries for a relay,
and route all resolution through ssh-config.  This is used by the "lxc-test" command, which performs
a completely local setup -- again, see docs for more details.

While working on DNS/SSH things i also unified all zone-file handling
to use actual BIND format as it is easy enough to parse back.
2026-03-16 13:27:12 +01:00
holger krekel
4fabfb31f8 fix test and some linting fixes 2026-03-16 13:25:57 +01:00
Jagoda Ślązak
36478dbfcf feat(filtermail): Disable IP verification on domain-literal addresses
Disables IP verification by upgrading filtermail to v0.6,
changelog: <https://github.com/chatmail/filtermail/releases/tag/v0.6.0>

Messages using domain-literal addresses no longer require
to match the origin SMTP connection IP anymore.

This allows for example a relay using IPv4 email addresses
to send messages to other relays over IPv6.

This is not considering a breaking change as IP-address-only
relays are not considered a stable feature.

Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
2026-03-13 20:47:10 +01:00
holger krekel
ff541b81ea chore: prevent installing recommended packages (e.g. installing cron leads to installing exim without it). 2026-03-08 23:40:16 +01:00
Alex V.
ed9b4092a8 test: add error-path tests for all bug fixes
- test_doveauth: invalid localpart chars rejected, concurrent same-account creation
- test_expire: --mdir filtering uses msg.path correctly
- test_metadata: TURN exception returns N\n, success returns credentials
- test_turnserver: socket timeout, connection refused, happy path
- test_dns: get_dkim_entry returns (None, None) on CalledProcessError
- test_rshell: dovecot_recalc_quota handles empty/malformed output
2026-03-05 16:27:15 +01:00
Alex V.
1b8ad3ca12 fix: handle turn_credentials exceptions in metadata proxy
ConnectionRefusedError/FileNotFoundError/TimeoutError from
turn_credentials() would kill the dict proxy connection.
Return N (not found) response instead and log the error.
2026-03-05 16:27:15 +01:00
Alex V.
f85d304e65 fix: add 5s timeout to TURN credential socket
Hung TURN daemon would block dict proxy handler thread indefinitely.
Per Python docs, settimeout() raises TimeoutError on expiry.
2026-03-05 16:27:15 +01:00
Alex V.
4d1856d8f1 fix(security): validate localpart chars and fix account creation race
- Reject localparts with chars outside [a-z0-9._-] to prevent
  filesystem issues from crafted usernames via IMAP/SMTP auth
- Use filelock to serialize concurrent account creation for same
  address, preventing TOCTOU race where two threads both create
  an account and last writer wins
2026-03-05 16:27:15 +01:00
Alex V.
ae2ab52aa9 fix(security): remove deprecated TLS 1.0/1.1 from nginx config
TLS 1.0/1.1 deprecated by RFC 8996. Nginx default is TLSv1.2 TLSv1.3.
Aligns with postfix (>=TLSv1.2) and dovecot (TLSv1.3) in the same stack.
2026-03-05 16:27:15 +01:00
Alex V.
d0c396538b fix(security): use secrets.choice instead of random.choices for username
Per Python docs, secrets module should be used for security-sensitive
data. random.choices uses Mersenne Twister PRNG which is predictable.
secrets.choice was already used for password generation in the same file.
2026-03-05 16:27:15 +01:00
Alex V.
78a4e28408 fix: guard against IndexError in dovecot_recalc_quota
doveadm output ends with empty line, parts=[] causes parts[2] crash.
2026-03-05 16:27:15 +01:00
Alex V.
2432d4f498 fix: remove dead code and potential NameError in run_cmd
check_call always returns 0 or raises, making retcode!=0 branches
unreachable. Also remote_data was undefined with --skip-dns-check.
2026-03-05 16:27:15 +01:00
Alex V.
31301abb42 fix: handle build_webpages returning None in WebsiteDeployer
Exception in _build_webpages was silently caught, returning None.
rsync then received "None/" as source path, silently breaking deploy.
2026-03-05 16:27:15 +01:00
Alex V.
6b4edd8502 fix: return tuple from get_dkim_entry on CalledProcessError
Bare return yielded None, causing TypeError on tuple unpacking
in perform_initial_checks on fresh servers without DKIM keys.
2026-03-05 16:27:15 +01:00
Alex V.
9c467ab3e8 chore: fix ruff formatting in acmetool, dovecot, postfix deployers 2026-03-05 16:27:15 +01:00
link2xt
774350778b feat: remove /metrics from the website
Similar data is already generated by fsreport
available for the relay operator
and metrics for prometheus are generated by mtail.

Closes <https://github.com/chatmail/relay/issues/431>
2026-03-05 14:58:11 +01:00
j4n
06d53503e5 feat(chatmaild/fsreport): add Prometheus textfile output, count files
- Count files in report
- Extend size buckets to bigger messages (5, 10 MiB)
- Two textfile exporters:
  - Full, bucketed size statistics with --textfile option
  - Account count only matching metrics.py format with --legacy-metrics
    option (filename defaults to /var/www/html/metrics)
- Improve option help texts
2026-03-05 13:52:09 +01:00
Alex V.
b128935940 fix: use msg.path instead of nonexistent msg.relpath in fsreport
FileEntry namedtuple has (path, mtime, size), not relpath.
Crashes with AttributeError when --mdir flag is used.
2026-03-05 13:52:09 +01:00
missytake
2e38c61ca2 opendkim: chown opendkim: private key 2026-03-05 11:24:06 +01:00
missytake
9dd8ce8ce1 tests: make sure chatmail-metadata was started
fix a flaky test: https://github.com/chatmail/relay/pull/856#issuecomment-3919881473
since #856 chatmail-metadata is restarted every 5 second, if it didn't come up after that, the failure likely sits deeper.
2026-03-04 18:53:31 +01:00
j4n
0ae3f94ecc fix(cmdeploy): dovecot update url 2026-03-04 17:19:14 +01:00
Jagoda Ślązak
4481a12369 chore(deps): upgrade to filtermail v0.5.2
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
2026-03-04 15:53:50 +01:00
373[Ø]™
a47016e9f2 Merge pull request #875 from chatmail/dovecot-github
fix(dovecot): download dovecot packages from github release
2026-03-03 16:03:21 +00:00
j4n
4e6ba7378d feat(cmdeploy): fall back to github url for dovecot 2026-03-02 10:29:03 +01:00
j4n
e428c646d1 fix(dovecot): download dovecot packages from github release 2026-02-26 21:06:55 +01:00