Both dovecot-quota-threshold triggers and the daily expiry routine
will now expunge oldest messages from mailboxes automatically
when the mailbox reaches 75% of max_mailbox_size.
Delta Chat users should not see any warnings (at 80/95 percent) or bounce messages,
and existing over-quota mailboxes should start receiving mails again.
Similar data is already generated by fsreport
available for the relay operator
and metrics for prometheus are generated by mtail.
Closes <https://github.com/chatmail/relay/issues/431>
Adds a new tls_external_cert_and_key config option for chatmail servers
that manage their own TLS certificates (e.g. via an external ACME client
or a load balancer).
A systemd path unit (tls-cert-reload.path) watches the certificate file
via inotify and automatically reloads dovecot and nginx when it changes.
Postfix reads certs per TLS handshake so needs no reload.
Also extracts openssl_selfsigned_args() so cert generation parameters
are shared between SelfSignedTlsDeployer and the e2e test.
feat: support self-signed TLS via underscore domain convention
Domains starting with "_" (e.g. _chat.example.org) automatically use
self-signed TLS certificates instead of ACME/Let's Encrypt. The TLS
mode is derived from the domain name — no separate config option needed.
Internally, when config.tls_cert_mode is "self" (underscore domain):
- Generate self-signed certificates via openssl
- Set Postfix smtp_tls_security_level to "encrypt" (opportunistic TLS)
- Add smtp_tls_policy_map entry for underscore domains
- Skip ACME, MTA-STS and www CNAME checks in `cmdeploy dns`
- Serve /new via GET (not redirect to dcaccount:) with rate-limiting
(nginx limit_req, 2r/s burst=5)
- Return dclogin: URLs with ic=3 (AcceptInvalidCertificates) from /new
- Render QR codes client-side via JavaScript and qrcode-svg
- Use config.tls_cert_path/tls_key_path in Postfix, Dovecot and nginx
templates instead of hardcoded ACME paths
* docs: update index reference
* docs: adds control machine migration instructions
* docs: rename index ref
* docs: remove maddy-chatmail (404)
* docs: consistent underlining in header text
* docs: remove dedicated page reference
* docs: remove dedicated page for control machine migration
* docs: condense deployment machine migration into getting started per feedback
* docs: correct link to madmail
* docs: update verbiage based on feedback
* docs: update migration guide after nine migration
* use $OLD_IP4 and $NEW_IP4 to make docs more readable. Also streamline "set TTL to 5 minute" phrasing a bit.
* fix tar commands
* refactor: streamline and refactor the migration guide to provide more clarity and focus
* recommend a "higher TTL" concrete value
Co-authored-by: missytake <missytake@systemli.org>
* scriptify another location
---------
Co-authored-by: missytake <missytake@systemli.org>
The original https://github.com/chatmail/relay/pull/533 attempted to remove the header through postfix, but that is too early. Instead, remove the headers in the OpenDKIM `final.lua` script after the validation.
* refactor: Move all imports to top of cmdeploy/__init__.py
* refactor: Move addition of 9.9.9.9 resolver earlier
- Moved the "Add 9.9.9.9 to resolv.conf" step earlier, before the
creation of users or updates to any config files. This should not
affect any of those operations. Moving this step earlier makes it
easier to accommodate the restructuring of the deployment process
into separate components with separate stages for install,
configure, and activate.
- Added a Deployer class that defines the base for objects that will
handle installation of individual components, with install,
configure, and activate stages.
- The CMDEPLOY_STAGES environment variable is used to determine what
stages to run. If this is not defined, all stages run as usual.
- Added import of Deployer to cmdeploy/__init__.py. This is not yet
used, but the next series of commits will use it.
- In deploy_chatmail(), define an empty list of deployers, and call
the create_groups() and create_users() methods for the items in the
list. This list will get filled with Deployer objects in the next
series of commits.
* refactor: Add DovecotDeployer
* refactor: Add PostfixDeployer
- Removed now-unused 'debug' variable from deploy_chatmail().
* refactor: Add NginxDeployer
- Use policy-rc.d during nginx install. This is needed to keep nginx
from starting up and interfering with acmetool. For more information see:
- https://serverfault.com/questions/861583/how-to-stop-nginx-from-being-automatically-started-on-install
- https://major.io/p/install-debian-packages-without-starting-daemons/
- https://people.debian.org/~hmh/invokerc.d-policyrc.d-specification.txt
* refactor: Add OpendkimDeployer
- Note that this moves the installation of the opendkim package
earlier in the deployment sequence. Previously, it was installed
during the _configure_opendkim() routine.
* refactor: Add UnboundDeployer
* refactor: Add IrohDeployer
- This splits the existing deploy_iroh_relay() routine into methods
for the install, configure, and activate stages.
* refactor: Add JournaldDeployer
* refactor: Add AcmetoolDeployer
- This splits the existing deploy_acmetool() routine into methods for
the install, configure, and activate stages.
* refactor: Add MtailDeployer
- This splits the existing deploy_mtail() routine into methods for the
install, configure, and activate stages.
* refactor: Add MtastsDeployer
- This splits the existing _uninstall_mta_sts_daemon() routine into
methods for the configure and activate stages.
* refactor: Add RspamdDeployer
- This replaces the existing _remove_rspamd() routine with a method
for the install stage.
* refactor: Split _install_remote_venv_with_chatmaild into stages
- Split _install_remote_venv_with_chatmaild() into three routines, to
handle the install, configure, and activate stages.
- This moves the upload of chatmail.ini later in the deployment
process, because it is a configuration file specific to the
instance, not software installation that would be uniform across all
deployments.
* refactor: Add ChatmailVenvDeployer
* refactor: Add ChatmailDeployer
- This moves the installation of cron earlier in the deployment sequence.
* refactor: Add FcgiwrapDeployer
* refactor: Add EchobotDeployer
- This class is a special case because it has a dependency on the
Postfix and Dovecot deployers. When deciding whether to restart the
echobot service, it needs to know whether the Postfix and Dovecot
deployers restarted their services. To support this dependency, the
PostfixDeployer and DovecotDeployer objects are passed to the
EchobotDeployer object, so it can check their was_restarted
attributes.
* refactor: Add WebsiteDeployer
- This adds a step to create /var/www in the install stage, because
the directory needs to exist for the rsync in the configure stage to
work.
* refactor: Add TurnDeployer
- This splits the existing deploy_turn_server() routine into methods
for the install, configure, and activate stages.
* refactor: Move curl installation from IrohDeployer to ChatmailDeployer
- The 'curl' program is used in TurnDeployer and IrohDeployer, so it
makes more sense to install it at the beginning in ChatmailDeployer,
rather than have each thing that uses it install it separately.
* refactor: Reorder deploy_chatmail()
- The previous commits that added Deployer classes mostly kept
deployment operations in the same order that they were in before.
To organize the process into separate stages for install, configure,
and activate, we need to reorder the method calls. This is the
commit that does that, and thus this is the commit that has the
largest effect on the order of operations.
- The calls for the deployer objects are all reordered here so that
the methods are called in the same sequence for each stage. This
will allow us to collect the calls into loops in the next commit.
This commit provides a way to see a diff showing exactly how the
sequence changed.
- The sequence of deployers was largely based on preserving the order
of the "activate" stage, as this seems like the place order might be
the most likely to matter. Installation of packages and
configuration of files should generally be able to run in any order.
(ChatmailDeployer handles updating the apt data, and therefore needs
to be first, however.)
* refactor: Call install, configure, and activate methods in loops
- Revised deploy_chatmail() to use all_deployers to call the
install(), configure(), and activate() methods on all the deployers,
rather than listing them explicitly in the code.
* docs: Add architectural information about deployer classes
- Updated overview.rst to describe the Deployer class hierarchy and
the motivations behind it.
* fix: Block unbound from starting up on install
- On an IPv4-only system, if unbound is started but not configured, it
causes subsequent steps to fail to resolve hosts.
- Revised UnboundDeployer.install_impl() to use policy-rc.d to prevent
the service from starting when installed. This is the same
mechanism used to keep nginx from starting on install.
* feat: Remove obs-home-deltachat.gpg
- We don't install Dovecot from OBS anymore.
- Removed files.put() that creates
/etc/apt/keyrings/obs-home-deltachat.gpg; replaced this with a
files.file() that sets present=False to remove the file from any
existing installations where it already has been installed.
- Removed now-unused obs-home-deltachat.gpg file.
- Clarified description of sources.list operation.
- Suggested in review by missytake and hpk42.
* feat: Reorder deployers
- Moved fcgiwrap before nginx.
- Exchanged order of turn and unbound.
- Moved journald as early as possible.
- Suggested in review by missytake.
* chore: Add CHANGELOG.md entry for cmdeploy refactor
* refactor: Move unit list to ChatmailVenvDeployer
- Split _configure_remote_venv_with_chatmaild() into two functions.
_configure_remote_venv_with_chatmaild() handles details specific to
the "venv", while the new _configure_remote_units() is a more
general function that is applicable to several services.
- Renamed _activate_remote_venv_with_chatmaild() to
_activate_remote_units() because doesn't have anything
venv-specific.
- Removed list of units from helper functions (where it appeared
twice); moved it to ChatmailVenvDeployer, where its is passed as an
argument to _configure_remote_units() and _activate_remote_units().
* refactor: Move turnserver out of ChatmailVenvDeployer
- Revised TurnDeployer to use _configure_remote_units() and
_activate_remote_units(). This class no longer uses need_restart
and daemon_reload attributes to keep track of state. The activate
stage of ChatmailVenvDeployer was unconditionally restarting the
service every time, so we don't need to keep track of extra state in
an attempt to avoid restarting it; we can just handle the
unconditional restart in TurnDeployer.activate_impl().
- Removed turnserver from the unit list in ChatmailVenvDeployer.
* refactor: Move echobot out of ChatmailVenvDeployer
- Revised EchobotDeployer to use _configure_remote_units() and
_activate_remote_units(). The 'activate' stage of
ChatmailVenvDeployer was unconditionally restarting the service
every time, so EchobotDeployer no longer needs to depend on the
was_restarted attributes of the postfix and dovecot deployers in an
attempt to avoid restarting it; we can just handle the unconditional
restart in EchobotDeployer.activate_impl().
- Removed echobot from the unit list in ChatmailVenvDeployer.
- Removed now-unused was_restarted attribute from PostfixDeployer and
DovecotDeployer.
* refactor: Move doveauth out of ChatmailVenvDeployer
- Revised DovecotDeployer to use _configure_remote_units() and
_activate_remote_units() to deploy doveauth. This keeps the
Dovecot-related services in a single deployer class, leaving only
services that are part of the chatmail project in
ChatmailVenvDeployer.
- Removed doveauth from the unit list in ChatmailVenvDeployer.
* strike unnccessary deployer variables
* remove indirection with "stages"
* simplify required_users configuration (a method is not needed for now)
* further reduce indirections for staged install
* now that Deployer class is clean and not mixed with what is in Deployment, use the simpler "install", "configure" and "activate" namings instead of *_impl
* remove static method and Make Deployer instances not set any default state
* strike unneccessary *,** argument flexibility
* use a Deployer for setting the remote git hash
* refactor: Revise AcmetoolDeployer for new Deployer interface
* style: Formatting revisions
* refactor: Pass all constructor arguments by position
- The constructor arguments do not have default values; they are all
required. Revised deploy_chatmail() to pass them by position rather
than name, so that the caller is not coupled to the names of the
arguments inside the method definition.
* refactor: Simplify interface to Deployer.install()
- In the current code, the only class using the interface that sets
need_restart() from the return value of the install() method was
IrohDeployer. That interface was created when the install method
was a static method, but now it is an instance method with access to
'self'. Therefore, we don't need to pass anything up to the caller
to have them set the attribute, we can just set it.
- Revised IrohDeployer.install() to set self.need_restart directly,
rather than returning a value.
- Revised Deployment.install() to ignore the return value of the
deployers' install() methods.
- need_restart is still present in the base Deployer class to ensure
that it is always defined, even when classes do not set it in a
constructor. Apart from this initialization for convenience, there
is no longer any specific exposure of need_restart in the interface
of the Deployer class.
- In general, install() methods should use 'self' as little as
possible, preferably not at all. In particular, install() methods
should never depend on "config" data, such as the config dictionary
in self.config or specific values like self.mail_domain. This
ensures that these methods can be used to perform generic
installation operations that are applicable across multiple relay
deployments, and therefore can be called in the process of building
a general-purpose container image.
* docs: Update cmdeploy architecture details
- Revised cmdeploy documentation in doc/source/overview.rst to reflect
the recent revisions to the Deployer interface.
* docs: Remove section about use of objects
---------
Co-authored-by: holger krekel <holger@merlinux.eu>
refactor README.rst and architecture file into sphinx doc project, automatically deploying on main merges and PRs.
* add FAQs from https://chatmail.at/relays landing page
* fix links, and streamline postfix/dovecot mentioning
* add linkcheck to CI, fix several links and streamlihne DKIM section while at it
* some streamlining, rename to "overview"
* ci: upload documentation to chatmail.at/doc/relay
* ci: main should be uploaded when docs.yaml changes
* ci: fix typo
* Update .github/workflows/docs-preview.yaml
Co-authored-by: missytake <missytake@systemli.org>
