cqrenet/relay
3
0
Fork 0
mirror of https://github.com/chatmail/relay.git synced 2026-05-10 16:04:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
ae96b752a39f37493ce013203a24b1e0c6be3623
relay/CHANGELOG.md
cliffmccarthy 3df3c031d4 Organize cmdeploy into install, configure, and activate stages (#695) 
* refactor: Move all imports to top of cmdeploy/__init__.py

* refactor: Move addition of 9.9.9.9 resolver earlier

- Moved the "Add 9.9.9.9 to resolv.conf" step earlier, before the
  creation of users or updates to any config files.  This should not
  affect any of those operations.  Moving this step earlier makes it
  easier to accommodate the restructuring of the deployment process
  into separate components with separate stages for install,
  configure, and activate.

- Added a Deployer class that defines the base for objects that will
  handle installation of individual components, with install,
  configure, and activate stages.  
- The CMDEPLOY_STAGES environment variable is used to determine what
  stages to run.  If this is not defined, all stages run as usual.
- Added import of Deployer to cmdeploy/__init__.py.  This is not yet
  used, but the next series of commits will use it.
- In deploy_chatmail(), define an empty list of deployers, and call
  the create_groups() and create_users() methods for the items in the
  list.  This list will get filled with Deployer objects in the next
  series of commits.

* refactor: Add DovecotDeployer

* refactor: Add PostfixDeployer

- Removed now-unused 'debug' variable from deploy_chatmail().

* refactor: Add NginxDeployer

- Use policy-rc.d during nginx install.  This is needed to keep nginx
  from starting up and interfering with acmetool.  For more information see:
    - https://serverfault.com/questions/861583/how-to-stop-nginx-from-being-automatically-started-on-install
    - https://major.io/p/install-debian-packages-without-starting-daemons/
    - https://people.debian.org/~hmh/invokerc.d-policyrc.d-specification.txt

* refactor: Add OpendkimDeployer

- Note that this moves the installation of the opendkim package
  earlier in the deployment sequence.  Previously, it was installed
  during the _configure_opendkim() routine.

* refactor: Add UnboundDeployer

* refactor: Add IrohDeployer

- This splits the existing deploy_iroh_relay() routine into methods
  for the install, configure, and activate stages.

* refactor: Add JournaldDeployer

* refactor: Add AcmetoolDeployer

- This splits the existing deploy_acmetool() routine into methods for
  the install, configure, and activate stages.

* refactor: Add MtailDeployer

- This splits the existing deploy_mtail() routine into methods for the
  install, configure, and activate stages.

* refactor: Add MtastsDeployer

- This splits the existing _uninstall_mta_sts_daemon() routine into
  methods for the configure and activate stages.

* refactor: Add RspamdDeployer

- This replaces the existing _remove_rspamd() routine with a method
  for the install stage.

* refactor: Split _install_remote_venv_with_chatmaild into stages

- Split _install_remote_venv_with_chatmaild() into three routines, to
  handle the install, configure, and activate stages.
- This moves the upload of chatmail.ini later in the deployment
  process, because it is a configuration file specific to the
  instance, not software installation that would be uniform across all
  deployments.

* refactor: Add ChatmailVenvDeployer

* refactor: Add ChatmailDeployer

- This moves the installation of cron earlier in the deployment sequence.

* refactor: Add FcgiwrapDeployer

* refactor: Add EchobotDeployer

- This class is a special case because it has a dependency on the
  Postfix and Dovecot deployers.  When deciding whether to restart the
  echobot service, it needs to know whether the Postfix and Dovecot
  deployers restarted their services.  To support this dependency, the
  PostfixDeployer and DovecotDeployer objects are passed to the
  EchobotDeployer object, so it can check their was_restarted
  attributes.

* refactor: Add WebsiteDeployer

- This adds a step to create /var/www in the install stage, because
  the directory needs to exist for the rsync in the configure stage to
  work.

* refactor: Add TurnDeployer

- This splits the existing deploy_turn_server() routine into methods
  for the install, configure, and activate stages.

* refactor: Move curl installation from IrohDeployer to ChatmailDeployer

- The 'curl' program is used in TurnDeployer and IrohDeployer, so it
  makes more sense to install it at the beginning in ChatmailDeployer,
  rather than have each thing that uses it install it separately.

* refactor: Reorder deploy_chatmail()

- The previous commits that added Deployer classes mostly kept
  deployment operations in the same order that they were in before.
  To organize the process into separate stages for install, configure,
  and activate, we need to reorder the method calls.  This is the
  commit that does that, and thus this is the commit that has the
  largest effect on the order of operations.
- The calls for the deployer objects are all reordered here so that
  the methods are called in the same sequence for each stage.  This
  will allow us to collect the calls into loops in the next commit.
  This commit provides a way to see a diff showing exactly how the
  sequence changed.
- The sequence of deployers was largely based on preserving the order
  of the "activate" stage, as this seems like the place order might be
  the most likely to matter.  Installation of packages and
  configuration of files should generally be able to run in any order.
  (ChatmailDeployer handles updating the apt data, and therefore needs
  to be first, however.)

* refactor: Call install, configure, and activate methods in loops

- Revised deploy_chatmail() to use all_deployers to call the
  install(), configure(), and activate() methods on all the deployers,
  rather than listing them explicitly in the code.

* docs: Add architectural information about deployer classes

- Updated overview.rst to describe the Deployer class hierarchy and
  the motivations behind it.

* fix: Block unbound from starting up on install

- On an IPv4-only system, if unbound is started but not configured, it
  causes subsequent steps to fail to resolve hosts.
- Revised UnboundDeployer.install_impl() to use policy-rc.d to prevent
  the service from starting when installed.  This is the same
  mechanism used to keep nginx from starting on install.

* feat: Remove obs-home-deltachat.gpg

- We don't install Dovecot from OBS anymore.
- Removed files.put() that creates
  /etc/apt/keyrings/obs-home-deltachat.gpg; replaced this with a
  files.file() that sets present=False to remove the file from any
  existing installations where it already has been installed.
- Removed now-unused obs-home-deltachat.gpg file.
- Clarified description of sources.list operation.
- Suggested in review by missytake and hpk42.

* feat: Reorder deployers

- Moved fcgiwrap before nginx.
- Exchanged order of turn and unbound.
- Moved journald as early as possible.
- Suggested in review by missytake.

* chore: Add CHANGELOG.md entry for cmdeploy refactor

* refactor: Move unit list to ChatmailVenvDeployer

- Split _configure_remote_venv_with_chatmaild() into two functions.
  _configure_remote_venv_with_chatmaild() handles details specific to
  the "venv", while the new _configure_remote_units() is a more
  general function that is applicable to several services.
- Renamed _activate_remote_venv_with_chatmaild() to
  _activate_remote_units() because doesn't have anything
  venv-specific.
- Removed list of units from helper functions (where it appeared
  twice); moved it to ChatmailVenvDeployer, where its is passed as an
  argument to _configure_remote_units() and _activate_remote_units().

* refactor: Move turnserver out of ChatmailVenvDeployer

- Revised TurnDeployer to use _configure_remote_units() and
  _activate_remote_units().  This class no longer uses need_restart
  and daemon_reload attributes to keep track of state.  The activate
  stage of ChatmailVenvDeployer was unconditionally restarting the
  service every time, so we don't need to keep track of extra state in
  an attempt to avoid restarting it; we can just handle the
  unconditional restart in TurnDeployer.activate_impl().
- Removed turnserver from the unit list in ChatmailVenvDeployer.

* refactor: Move echobot out of ChatmailVenvDeployer

- Revised EchobotDeployer to use _configure_remote_units() and
  _activate_remote_units().  The 'activate' stage of
  ChatmailVenvDeployer was unconditionally restarting the service
  every time, so EchobotDeployer no longer needs to depend on the
  was_restarted attributes of the postfix and dovecot deployers in an
  attempt to avoid restarting it; we can just handle the unconditional
  restart in EchobotDeployer.activate_impl().
- Removed echobot from the unit list in ChatmailVenvDeployer.
- Removed now-unused was_restarted attribute from PostfixDeployer and
  DovecotDeployer.

* refactor: Move doveauth out of ChatmailVenvDeployer

- Revised DovecotDeployer to use _configure_remote_units() and
  _activate_remote_units() to deploy doveauth.  This keeps the
  Dovecot-related services in a single deployer class, leaving only
  services that are part of the chatmail project in
  ChatmailVenvDeployer.
- Removed doveauth from the unit list in ChatmailVenvDeployer.

* strike unnccessary deployer variables

* remove indirection with "stages"

* simplify required_users configuration (a method is not needed for now)

* further reduce indirections for staged install

* now that Deployer class is clean and not mixed with what is in Deployment, use the simpler "install", "configure" and "activate" namings instead of *_impl

* remove static method and Make Deployer instances not set any default state

* strike unneccessary *,** argument flexibility

* use a Deployer for setting the remote git hash

* refactor: Revise AcmetoolDeployer for new Deployer interface

* style: Formatting revisions

* refactor: Pass all constructor arguments by position

- The constructor arguments do not have default values; they are all
  required.  Revised deploy_chatmail() to pass them by position rather
  than name, so that the caller is not coupled to the names of the
  arguments inside the method definition.

* refactor: Simplify interface to Deployer.install()

- In the current code, the only class using the interface that sets
  need_restart() from the return value of the install() method was
  IrohDeployer.  That interface was created when the install method
  was a static method, but now it is an instance method with access to
  'self'.  Therefore, we don't need to pass anything up to the caller
  to have them set the attribute, we can just set it.
- Revised IrohDeployer.install() to set self.need_restart directly,
  rather than returning a value.
- Revised Deployment.install() to ignore the return value of the
  deployers' install() methods.
- need_restart is still present in the base Deployer class to ensure
  that it is always defined, even when classes do not set it in a
  constructor.  Apart from this initialization for convenience, there
  is no longer any specific exposure of need_restart in the interface
  of the Deployer class.
- In general, install() methods should use 'self' as little as
  possible, preferably not at all.  In particular, install() methods
  should never depend on "config" data, such as the config dictionary
  in self.config or specific values like self.mail_domain.  This
  ensures that these methods can be used to perform generic
  installation operations that are applicable across multiple relay
  deployments, and therefore can be called in the process of building
  a general-purpose container image.

* docs: Update cmdeploy architecture details

- Revised cmdeploy documentation in doc/source/overview.rst to reflect
  the recent revisions to the Deployer interface.

* docs: Remove section about use of objects

---------

Co-authored-by: holger krekel <holger@merlinux.eu>
2025-11-13 16:51:51 +01:00

18 KiB
Raw Blame History

Changelog for chatmail deployment

untagged

  • Organized cmdeploy into install, configure, and activate stages (#695)

  • docs: move readme.md docs to sphinx documentation rendered at https://chatmail.at/doc/relay (#711)

  • acmetool: replace cronjob with a systemd timer (#719)

  • remove xstore@testrun.org from default passthrough recipients (#722)

  • don't deploy the website if there are merge conflicts in the www folder (#714)

  • acmetool: use ECDSA keys instead of RSA (#689)

  • Require TLS 1.2 for outgoing SMTP connections (#685)

  • require STARTTLS for incoming port 25 connections (#684)

  • filtermail: run CPU-intensive handle_DATA in a thread pool executor (#676)

  • don't use the complicated logging module in filtermail to exclude a potential source of errors. (#674)

  • Specify nginx.conf to only handle mail_domain, www, and mta-sts domains (#636)

  • Setup TURN server (#621)

  • cmdeploy: make --ssh-host work with localhost (#659)

  • Update iroh-relay to 0.35.0 (#650)

  • filtermail: accept mails from Protonmail (#616)

  • Ignore all RCPT TO: parameters (#651)

  • Increase opendkim DNS Timeout from 5 to 60 seconds (#672)

  • Add config parameter for Let's Encrypt ACME email (#663)

  • Use max username length in newemail.py, not min (#648)

  • Add startup for fcgiwrap.service because sometimes it did not start automatically. (#657)

  • Add cmdeploy init --force command for recreating chatmail.ini (#656)

  • Increase maxproc for reinjecting ports from 10 to 100 (#646)

  • Allow ports 143 and 993 to be used by dovecot process (#639)

  • Add --skip-dns-check argument to cmdeploy run command, which disables DNS record checking before installation. (#661)

  • Rework expiry of message files and mailboxes in Python to only do a single iteration over sometimes millions of messages instead of doing "find" commands that iterate 9 times over the messages. Provide an "fsreport" CLI for more fine grained analysis of message files. (#637)

1.7.0 2025-09-11

  • Make www upload path configurable (#618)

  • Check whether GCC is installed in initenv.sh (#608)

  • Expire push notification tokens after 90 days (#583)

  • Use official mtail binary instead of mtail package (#581)

  • dovecot: install from download.delta.chat instead of openSUSE Build Service (#590)

  • Reconfigure Dovecot imap-login service to high-performance mode (#578)

  • Set timezone to improve dovecot performance (#584)

  • Increase nginx connection limits (#576)

  • If dns-utils needs to be installed before cmdeploy run, apt update to make sure it works (#560)

  • filtermail: respect config message size limit (#572)

  • Don't deploy if one of the ports used for chatmail relay services is occupied by an unexpected process (#568)

  • Add config value after how many days large files are deleted (#555)

  • cmdeploy: push relay version to /etc/chatmail-version (#573)

  • filtermail: allow partial body length in OpenPGP payloads (#570)

  • chatmaild: allow echobot to receive unencrypted messages by default (#556)

1.6.0 2025-04-11

  • Handle Port-25 connect errors more gracefully (common with VPNs) (#552)

  • Avoid "acmetool not found" during initial run (#550)

  • Fix timezone handling such that client/servers do not need to use same timezone. (#553)

  • Enforce end-to-end encryption for incoming messages. New user address mailboxes now get a enforceE2EEincoming file which prohibits incoming cleartext messages from other domains. An outside MTA trying to submit a cleartext message will get a "523 Encryption Needed" response, see RFC5248. If the file does not exist (as it the case for all existing accounts) incoming cleartext messages are accepted. (#538)

  • Enforce end-to-end encryption between local addresses (#535)

  • unbound: check that port 53 is not occupied by a different process (#537)

  • unbound: before unbound is there, use 9.9.9.9 for resolving (#518)

  • Limit the bind for the HTTPS server on 8443 to 127.0.0.1 (#522) (#532)

  • Send SNI when connecting to outside servers (#524)

  • postfix master.cf: use 127.0.0.1 for consistency (#544)

  • Pass through original_content instead of content in filtermail (#509)

  • Document TLS requirements in the readme (#514)

  • Remove cleanup service from submission ports (#512)

  • cmdeploy dovecot: delete big messages after 7 days (#504)

  • mtail: fix getting logs from STDIN (#502)

  • filtermail: don't require exactly 2 lines after openPGP payload (#497)

  • cmdeploy dns: offer alternative DKIM record format for some web interfaces (#470)

  • journald: remove old logs from disk (#490)

  • opendkim: restart once every day to mend RAM leaks (#498

  • migration guide: let opendkim own the DKIM keys directory (#468)

  • improve secure-join message detection (#473)

  • use old crypt lib in python < 3.11 (#483)

  • chatmaild: set umask to 0700 for doveauth + metadata (#490)

  • remove MTA-STS daemon (#488)

  • replace Subject with [...] for all outgoing mails. (#481)

  • opendkim: use su instead of sudo (#491)

1.5.0 2024-12-20

  • cmdeploy dns: always show recommended DNS records (#463)

  • add --all to cmdeploy dns (#462)

  • fix _mta-sts TXT DNS record (#461

  • deploy iroh-relay and also update "realtime relay services" in privacy policy. (#434) (#451)

  • add guide to migrate chatmail to a new server (#429)

  • disable anvil authentication penalty (#414

  • increase request_queue_size for UNIX sockets to 1000. (#437)

  • add argument to cmdeploy run for specifying a different SSH host than mail_domain (#439)

  • query autoritative nameserver to bypass DNS cache (#424)

  • add mtail support (new optional mtail_address ini value) This defines the address on which mtail exposes its metrics collected from the logs. If you want to collect the metrics with Prometheus, setup a private network (e.g. WireGuard interface) and assign an IP address from this network to the host. If you do not plan to collect metrics, keep this setting unset. (#388)

  • fix checking for required DNS records (#412)

  • add support for specifying whole domains for recipient passthrough list (#408)

  • add a paragraph about "account deletion" to info page (#405)

  • avoid nginx listening on ipv6 if v6 is dsiabled (#402)

  • refactor ssh-based execution to allow organizing remote functions in modules. (#396)

  • trigger "apt upgrade" during "cmdeploy run" (#398)

  • drop hispanilandia passthrough address (#401)

  • set CAA record flags to 0

  • add IMAP capabilities instead of overwriting them (#413)

  • fix OpenPGP payload check (#435)

  • fix Dovecot quota_max_mail_size to use max_message_size config value (#438)

1.4.1 2024-07-31

  • fix metadata dictproxy which would confuse transactions resulting in missed notifications and other issues. (#393) (#394)

  • add optional "imap_rawlog" config option. If true, .in/.out files are created in user home dirs containing the imap protocol messages. (#389)

1.4.0 2024-07-28

  • Add disable_ipv6 config option to chatmail.ini. Required if the server doesn't have IPv6 connectivity. (#312)

  • allow current K9/Thunderbird-mail releases to send encrypted messages outside by accepting their localized "encrypted subject" strings. (#370)

  • Migrate and remove sqlite database in favor of password/lastlogin tracking in a user's maildir.
    (#379)

  • Require pyinfra V3 installed on the client side, run ./scripts/initenv.sh to upgrade locally. (#378)

  • don't hardcode "/home/vmail" paths but rather set them once in the config object and use it everywhere else, thereby also improving testability.
    (#351) temporarily introduced obligatory "passdb_path" and "mailboxes_dir" settings but they were removed/obsoleted in (#380)

  • BREAKING: new required chatmail.ini value 'delete_inactive_users_after = 100' which removes users from database and mails after 100 days without any login. (#350)

  • Refine DNS checking to distinguish between "required" and "recommended" settings (#372)

  • reload nginx in the acmetool cronjob (#360)

  • remove checking of reverse-DNS PTR records. Chatmail-servers don't depend on it and even in the wider e-mail system it's not common anymore. If it's an issue, a chatmail operator can still care to properly set reverse DNS. (#348)

  • Make DNS-checking faster and more interactive, run it fully during "cmdeploy run", also introducing a generic mechanism for rapid remote ssh-based python function execution. (#346)

  • Don't fix file owner ship of /home/vmail (#345)

  • Support iterating over all users with doveadm commands (#344)

  • Test and fix for attempts to create inadmissible accounts (#333)

  • check that OpenPGP has only PKESK, SKESK and SEIPD packets (#323, #324)

  • improve filtermail checks for encrypted messages and drop support for unencrypted MDNs (#320)

  • replace bash with /bin/sh (#334)

  • Increase number of logged in IMAP sessions to 50000 (#335)

  • filtermail: do not allow ASCII armor without actual payload (#325)

  • Remove sieve to enable hardlink deduplication in LMTP (#343)

  • dovecot: enable gzip compression on disk (#341)

  • DKIM-sign Content-Type and oversign all signed headers (#296)

  • Add nonci_accounts metric (#347)

  • doveauth: log when a new account is created (#349)

  • Multiplex HTTPS, IMAP and SMTP on port 443 (#357)

1.3.0 - 2024-06-06

  • don't check necessary DNS records on cmdeploy init anymore (#316)

  • ensure cron and acl are installed (#293, #310)

  • change default for delete_mails_after from 40 to 20 days (#300)

  • save journald logs only to memory and save nginx logs to journald instead of file (#299)

  • fix writing of multiple obs repositories in /etc/apt/sources.list (#290)

  • metadata: add support for /shared/vendor/deltachat/irohrelay (#284)

  • Emit "XCHATMAIL" capability from IMAP server (#278)

  • Move echobot into /var/lib/echobot (#281)

  • Accept Let's Encrypt's new Terms of Services (#275)

  • Reload Dovecot and Postfix when TLS certificate updates (#271)

  • Use forked version of dovecot without hardcoded delays (#270)

1.2.0 - 2024-04-04

  • Install dig on the server to resolve DNS records (#267)

  • preserve notification order and exponentially backoff with retries for tokens where we didn't get a successful return (#265)

  • Run chatmail-metadata and doveauth as vmail (#261)

  • Apply systemd restrictions to echobot (#259)

  • re-enable running the CI in pull requests, but not concurrently (#258)

1.1.0 - 2024-03-28

The changelog starts to record changes from March 15th, 2024

  • Move systemd unit templates to cmdeploy package (#255)

  • Persist push tokens and support multiple device per address (#254)

  • Avoid warning for regular doveauth protocol's hello message. (#250)

  • Fix various tests to pass again with "cmdeploy test". (#245, #242

  • Ensure lets-encrypt certificates are reloaded after renewal ([#244]) https://github.com/chatmail/server/pull/244

  • Persist tokens to avoid iOS users loosing push-notifications when the chatmail metadata service is restarted (happens regularly during deploys) (#238

  • Fix failing sieve-script compile errors on incoming messages (#237

  • Fix quota reporting after expunging of old mails (#233

Reference in New Issue View Git Blame Copy Permalink