fix tar commands

I am running git-cliff 2.11.0.
Ran `git-cliff --init` to generate `cliff.toml`.
Removed emojis, replaced `doc` with `docs` to match chatmail core
convention.

CHANGELOG.md

@@ -1,6 +1,6 @@
 # Changelog for chatmail deployment 
 
-## untagged
+## 1.8.0 2025-12-12
 
 - Add imap_compress option to chatmail.ini
   ([#760](https://github.com/chatmail/relay/pull/760))

CONTRIBUTING.md

@@ -0,0 +1,7 @@
+# Contributing to the chatmail relay
+
+Commit messages follow the [Conventional Commits] notation.
+We use [git-cliff] to generate the changelog from commit messages before the release.
+
+[Conventional Commits]: https://www.conventionalcommits.org/
+[git-cliff]: https://git-cliff.org/

RELEASE.md

@@ -0,0 +1,15 @@
+# Releasing a new version of chatmail relay
+
+For example, to release version 1.9.0 of chatmail relay, do the following steps.
+
+1. Update the changelog: `git cliff --unreleased --tag 1.9.0 --prepend CHANGELOG.md` or `git cliff -u -t 1.9.0 -p CHANGELOG.md`.
+
+2. Open the changelog in the editor, edit it if required.
+
+3. Commit the changes to the changelog with a commit message `chore(release): prepare for 1.9.0`.
+
+3. Tag the release: `git tag --annotate 1.9.0`.
+
+4. Push the release tag: `git push origin 1.9.0`.
+
+5. Create a GitHub release: `gh release create 1.9.0`.

chatmaild/src/chatmaild/ini/chatmail.ini.f

@@ -43,7 +43,7 @@ passthrough_senders =
 
 # list of e-mail recipients for which to accept outbound un-encrypted mails
 # (space-separated, item may start with "@" to whitelist whole recipient domains)
-passthrough_recipients = echo@{mail_domain}
+passthrough_recipients =
 
 # path to www directory - documented here: https://chatmail.at/doc/relay/getting_started.html#custom-web-pages
 #www_folder = www

chatmaild/src/chatmaild/lastlogin.py

@@ -13,8 +13,6 @@ class LastLoginDictProxy(DictProxy):
         keyname = parts[1].split("/")
         value = parts[2] if len(parts) > 2 else ""
         if keyname[0] == "shared" and keyname[1] == "last-login":
-            if addr.startswith("echo@"):
-                return True
             addr = keyname[2]
             timestamp = int(value)
             user = self.config.get_user(addr)

chatmaild/src/chatmaild/user.py

@@ -19,7 +19,7 @@ class User:
 
     @property
     def can_track(self):
-        return "@" in self.addr and not self.addr.startswith("echo@")
+        return "@" in self.addr
 
     def get_userdb_dict(self):
         """Return a non-empty dovecot 'userdb' style dict
@@ -55,11 +55,9 @@ class User:
         try:
             write_bytes_atomic(self.password_path, password)
         except PermissionError:
-            if not self.addr.startswith("echo@"):
-                logging.error(f"could not write password for: {self.addr}")
-                raise
-        if not self.addr.startswith("echo@"):
-            self.enforce_E2EE_path.touch()
+            logging.error(f"could not write password for: {self.addr}")
+            raise
+        self.enforce_E2EE_path.touch()
 
     def set_last_login_timestamp(self, timestamp):
         """Track login time with daily granularity

cliff.toml

@@ -0,0 +1,94 @@
+# git-cliff ~ configuration file
+# https://git-cliff.org/docs/configuration
+
+
+[changelog]
+# A Tera template to be rendered for each release in the changelog.
+# See https://keats.github.io/tera/docs/#introduction
+body = """
+{% if version %}\
+    ## [{{ version | trim_start_matches(pat="v") }}] - {{ timestamp | date(format="%Y-%m-%d") }}
+{% else %}\
+    ## [unreleased]
+{% endif %}\
+{% for group, commits in commits | group_by(attribute="group") %}
+    ### {{ group | striptags | trim | upper_first }}
+    {% for commit in commits %}
+        - {% if commit.scope %}*({{ commit.scope }})* {% endif %}\
+            {% if commit.breaking %}[**breaking**] {% endif %}\
+            {{ commit.message | upper_first }}\
+    {% endfor %}
+{% endfor %}
+"""
+# Remove leading and trailing whitespaces from the changelog's body.
+trim = true
+# Render body even when there are no releases to process.
+render_always = true
+# An array of regex based postprocessors to modify the changelog.
+postprocessors = [
+    # Replace the placeholder <REPO> with a URL.
+    #{ pattern = '<REPO>', replace = "https://github.com/orhun/git-cliff" },
+]
+# render body even when there are no releases to process
+# render_always = true
+# output file path
+# output = "test.md"
+
+[git]
+# Parse commits according to the conventional commits specification.
+# See https://www.conventionalcommits.org
+conventional_commits = true
+# Exclude commits that do not match the conventional commits specification.
+filter_unconventional = true
+# Require all commits to be conventional.
+# Takes precedence over filter_unconventional.
+require_conventional = false
+# Split commits on newlines, treating each line as an individual commit.
+split_commits = false
+# An array of regex based parsers to modify commit messages prior to further processing.
+commit_preprocessors = [
+    # Replace issue numbers with link templates to be updated in `changelog.postprocessors`.
+    #{ pattern = '\((\w+\s)?#([0-9]+)\)', replace = "([#${2}](<REPO>/issues/${2}))"},
+    # Check spelling of the commit message using https://github.com/crate-ci/typos.
+    # If the spelling is incorrect, it will be fixed automatically.
+    #{ pattern = '.*', replace_command = 'typos --write-changes -' },
+]
+# Prevent commits that are breaking from being excluded by commit parsers.
+protect_breaking_commits = false
+# An array of regex based parsers for extracting data from the commit message.
+# Assigns commits to groups.
+# Optionally sets the commit's scope and can decide to exclude commits from further processing.
+commit_parsers = [
+    { message = "^feat", group = "Features" },
+    { message = "^fix", group = "Bug Fixes" },
+    { message = "^docs", group = "Documentation" },
+    { message = "^perf", group = "Performance" },
+    { message = "^refactor", group = "Refactor" },
+    { message = "^style", group = "Styling" },
+    { message = "^test", group = "Testing" },
+    { message = "^chore\\(release\\): prepare for", skip = true },
+    { message = "^chore\\(deps.*\\)", skip = true },
+    { message = "^chore\\(pr\\)", skip = true },
+    { message = "^chore\\(pull\\)", skip = true },
+    { message = "^chore|^ci", group = "Miscellaneous Tasks" },
+    { body = ".*security", group = "Security" },
+    { message = "^revert", group = "Revert" },
+    { message = ".*", group = "Other" },
+]
+# Exclude commits that are not matched by any commit parser.
+filter_commits = false
+# Fail on a commit that is not matched by any commit parser.
+fail_on_unmatched_commit = false
+# An array of link parsers for extracting external references, and turning them into URLs, using regex.
+link_parsers = []
+# Include only the tags that belong to the current branch.
+use_branch_tags = false
+# Order releases topologically instead of chronologically.
+topo_order = false
+# Order commits topologically instead of chronologically.
+topo_order_commits = true
+# Order of commits in each group/release within the changelog.
+# Allowed values: newest, oldest
+sort_commits = "oldest"
+# Process submodules commits
+recurse_submodules = false

cmdeploy/src/cmdeploy/acmetool/__init__.py

@@ -61,6 +61,19 @@ class AcmetoolDeployer(Deployer):
             mode="644",
         )
 
+        server.shell(
+            name=f"Remove old acmetool desired files for {self.domains[0]}",
+            commands=[f"rm -f /var/lib/acme/desired/{self.domains[0]}-*"],
+        )
+        files.template(
+            src=importlib.resources.files(__package__).joinpath("desired.yaml.j2"),
+            dest=f"/var/lib/acme/desired/{self.domains[0]}", # 0 is mailhost TLD
+            user="root",
+            group="root",
+            mode="644",
+            domains=self.domains,
+        )
+
         service_file = files.put(
             src=importlib.resources.files(__package__).joinpath(
                 "acmetool-redirector.service"
@@ -123,6 +136,6 @@ class AcmetoolDeployer(Deployer):
         self.need_restart_reconcile_timer = False
 
         server.shell(
-            name=f"Request certificate for: {', '.join(self.domains)}",
-            commands=[f"acmetool want --xlog.severity=debug {' '.join(self.domains)}"],
+            name=f"Reconcile certificates for: {', '.join(self.domains)}",
+            commands=["acmetool --batch --xlog.severity=debug reconcile"],
         )

cmdeploy/src/cmdeploy/acmetool/desired.yaml.j2

@@ -0,0 +1,6 @@
+satisfy:
+  names:
+{%- for domain in domains %}
+  - {{ domain }}
+{%- endfor %}
+

cmdeploy/src/cmdeploy/deployers.py

@@ -440,7 +440,6 @@ class ChatmailVenvDeployer(Deployer):
 class ChatmailDeployer(Deployer):
     required_users = [
         ("vmail", "vmail", None),
-        ("echobot", None, None),
         ("iroh", None, None),
     ]
 

cmdeploy/src/cmdeploy/dovecot/deployer.py

@@ -13,6 +13,8 @@ from cmdeploy.basedeploy import (
 
 
 class DovecotDeployer(Deployer):
+    daemon_reload = False
+
     def __init__(self, config, disable_mail):
         self.config = config
         self.disable_mail = disable_mail
@@ -27,7 +29,7 @@ class DovecotDeployer(Deployer):
 
     def configure(self):
         configure_remote_units(self.config.mail_domain, self.units)
-        self.need_restart = _configure_dovecot(self.config)
+        self.need_restart, self.daemon_reload = _configure_dovecot(self.config)
 
     def activate(self):
         activate_remote_units(self.units)
@@ -42,6 +44,7 @@ class DovecotDeployer(Deployer):
             running=False if self.disable_mail else True,
             enabled=False if self.disable_mail else True,
             restarted=restart,
+            daemon_reload=self.daemon_reload,
         )
         self.need_restart = False
 
@@ -80,9 +83,10 @@ def _install_dovecot_package(package: str, arch: str):
     apt.deb(name=f"Install dovecot-{package}", src=deb_filename)
 
 
-def _configure_dovecot(config: Config, debug: bool = False) -> bool:
+def _configure_dovecot(config: Config, debug: bool = False) -> (bool, bool):
     """Configures Dovecot IMAP server."""
     need_restart = False
+    daemon_reload = False
 
     main_config = files.template(
         src=get_resource("dovecot/dovecot.conf.j2"),
@@ -134,4 +138,11 @@ def _configure_dovecot(config: Config, debug: bool = False) -> bool:
     )
     need_restart |= timezone_env.changed
 
-    return need_restart
+    restart_conf = files.put(
+        name="dovecot: restart automatically on failure",
+        src=get_resource("service/10_restart.conf"),
+        dest="/etc/systemd/system/dovecot.service.d/10_restart.conf",
+    )
+    daemon_reload |= restart_conf.changed
+
+    return need_restart, daemon_reload

cmdeploy/src/cmdeploy/opendkim/final.lua

@@ -10,6 +10,7 @@ if nsigs == nil then
 end
 
 local valid = false
+local error_msg = "No valid DKIM signature found."
 for i = 1, nsigs do
 	sig = odkim.get_sighandle(ctx, i - 1)
 	sigres = odkim.sig_result(sig)
@@ -21,6 +22,8 @@ for i = 1, nsigs do
 	-- means the message is acceptable.
 	if sigres == 0 then
 		valid = true
+    else
+        error_msg = "DKIM signature is invalid, error code " .. tostring(sigres) .. ", search https://github.com/trusteddomainproject/OpenDKIM/blob/master/libopendkim/dkim.h#L108"
 	end
 end
 
@@ -31,7 +34,7 @@ if valid then
 		odkim.del_header(ctx, "DKIM-Signature", i)
 	end
 else
-	odkim.set_reply(ctx, "554", "5.7.1", "No valid DKIM signature found")
+	odkim.set_reply(ctx, "554", "5.7.1", error_msg)
 	odkim.set_result(ctx, SMFIS_REJECT)
 end
 

cmdeploy/src/cmdeploy/postfix/deployer.py

@@ -5,6 +5,7 @@ from cmdeploy.basedeploy import Deployer, get_resource
 
 class PostfixDeployer(Deployer):
     required_users = [("postfix", None, ["opendkim"])]
+    daemon_reload = False
 
     def __init__(self, config, disable_mail):
         self.config = config
@@ -60,6 +61,13 @@ class PostfixDeployer(Deployer):
             mode="644",
         )
         need_restart |= login_map.changed
+
+        restart_conf = files.put(
+            name="postfix: restart automatically on failure",
+            src=get_resource("service/10_restart.conf"),
+            dest="/etc/systemd/system/dovecot.service.d/10_restart.conf",
+        )
+        self.daemon_reload = restart_conf.changed
         self.need_restart = need_restart
 
     def activate(self):
@@ -73,5 +81,6 @@ class PostfixDeployer(Deployer):
             running=False if self.disable_mail else True,
             enabled=False if self.disable_mail else True,
             restarted=restart,
+            daemon_reload=self.daemon_reload,
         )
         self.need_restart = False

cmdeploy/src/cmdeploy/service/10_restart.conf

@@ -0,0 +1,3 @@
+[Service]
+Restart=always
+RestartSec=30

doc/README.md

@@ -6,7 +6,7 @@ You can use the `make` command and `make html` to build web pages.
 
 You need a Python environment where the following install was excuted: 
 
-    pip install sphinx-build furo sphinx-autobuild 
+    pip install furo sphinx-autobuild
 
 To develop/change documentation, you can then do: 
 

doc/source/migrate.rst

@@ -7,66 +7,88 @@ machine, you can use these steps. They were tested with a Linux laptop;
 you might need to adjust some of the steps to your environment.
 
 Let’s assume that your ``mail_domain`` is ``mail.example.org``, all
-involved machines run Debian 12, your old site’s IP address is
-``13.37.13.37``, and your new site’s IP address is ``13.12.23.42``.
+involved machines run Debian 12, your old site’s IP version 4 address is
+``$OLD_IP4``, and your new site’s IP4 address is ``$NEW_IP4``.
 
-Note, you should lower the TTLs of your DNS records to a value such as
-300 (5 minutes) so the migration happens as smoothly as possible.
+First of all, you should lower the Time To Live (TTL) of your DNS records
+to a value such as 300 (5 minutes).
+Short TTL values allow to change DNS records during the migration more timely.
 
 During the guide you might get a warning about changed SSH Host keys; in
 this case, just run ``ssh-keygen -R "mail.example.org"`` as recommended.
 
-1. First, disable mail services on the old site.
-
-   ::
-
-       cmdeploy run --disable-mail --ssh-host 13.37.13.37
-
-   Now your users will notice the migration and will not be able to send
-   or receive messages until the migration is completed.
-
-2. Now we want to copy ``/home/vmail``, ``/var/lib/acme``,
-   ``/etc/dkimkeys``, and ``/var/spool/postfix`` to
-   the new site. Login to the old site while forwarding your SSH agent
+1. First, to make the downtime during the migration shorter,
+   let's transfer the current state of the mailboxes.
+   Login to your old machine (while forwarding your ssh-agent with ``ssh -A``)
    so you can copy directly from the old to the new site with your SSH
    key:
 
    ::
 
-       ssh -A root@13.37.13.37
-       tar c - /home/vmail/mail /var/lib/acme /etc/dkimkeys /var/spool/postfix | ssh root@13.12.23.42 "tar x -C /"
+       ssh -A root@$OLD_IP4
+       tar c /home/vmail/mail | ssh root@$NEW_IP4 "tar x -C /"
 
-   This transfers all addresses, the TLS certificate,
+   This saves us time during the downtime,
+   at least the mailboxes are there already.
+   They contain user passwords, encrypted push notification tokens,
+   messages which might not have been fetched by all devices of the user yet,
+   and dovecot indexes which track the state of the mailbox.
+
+2. Then, from your local machine, install chatmail on the new machine, but don't activate it yet:
+
+   ::
+
+       CMDEPLOY_STAGES=install,configure cmdeploy run --ssh-host $NEW_IP4
+
+   The services are disabled for now; we will enable them later.
+   We first need to make the new site fully operational.
+
+3. Now it's getting serious: disable the mail services on the old site.
+
+   ::
+
+       cmdeploy run --disable-mail --ssh-host $OLD_IP4
+
+   Your users will start to notice the migration and will not be able to send
+   or receive messages until the migration is completed.
+   Other relays and mail servers will wait with delivering messages
+   until your relay is reachable again.
+
+4. Now we want to copy ``/home/vmail``, ``/var/lib/acme``,
+   ``/etc/dkimkeys``, and ``/var/spool/postfix`` to
+   the new site. Let's forward the SSH agent again to copy the files directly.
+   This time, we copy ``/home/vmail/mail`` with rsync to only copy the recent changes:
+
+   ::
+
+       ssh -A root@$OLD_IP4
+       tar c /var/lib/acme /etc/dkimkeys /var/spool/postfix | ssh root@$NEW_IP4 "tar x -C /"
+       rsync -azH /home/vmail/mail root@$NEW_IP4:/home/vmail/
+
+   This transfers all messages which have not been fetched yet, the TLS certificate,
    and DKIM keys (so DKIM DNS record remains valid).
    It also preserves the Postfix mail spool so any messages
    pending delivery will still be delivered.
 
-3. Install chatmail on the new machine:
-
-   ::
-
-       cmdeploy run --disable-mail --ssh-host 13.12.23.42
-
-   Postfix and Dovecot are disabled for now; we will enable them later.
-   We first need to make the new site fully operational.
-
-4. On the new site, run the following to ensure the ownership is correct
+5. Now login to the new site and run the following to ensure the ownership is correct
    in case UIDs/GIDs changed:
 
    ::
 
+       ssh root@$NEW_IP4
        chown root: -R /var/lib/acme
        chown opendkim: -R /etc/dkimkeys
        chown vmail: -R /home/vmail/mail
 
-5. Now, update DNS entries.
+6. Now, update the DNS entries.
+   You only need to change the ``A`` and ``AAAA`` records, for example:
 
-   If other MTAs try to deliver messages to your chatmail domain they
-   may fail intermittently, as DNS catches up with the new site settings
-   but normally will retry delivering messages for at least a week, so
-   messages will not be lost.
+   ::
 
-6. Finally, you can execute ``cmdeploy run --ssh-host 13.12.23.42`` to
+       mail.example.org.    IN A    $NEW_IP4
+       mail.example.org.    IN AAAA $NEW_IP6
+
+7. Finally, you can execute ``CMDEPLOY_STAGES=activate cmdeploy run --ssh-host $NEW_IP4`` to
    turn on chatmail on the new relay. Your users will be able to use the
    chatmail relay as soon as the DNS changes have propagated. Voilà!
 

doc/source/related.rst

@@ -7,7 +7,7 @@ Active development takes place in the `chatmail/relay github repository <https:/
 You can check out the `'chatmail' tag in the support.delta.chat forum <https://support.delta.chat/tag/chatmail>`_
 and ask to get added to a non-public support chat for debugging issues.
 
-We know of two work-in-progress alternative implementation efforts:
+We know of three work-in-progress alternative implementation efforts:
 
 -  `Mox <https://github.com/mjl-/mox>`_: A Golang email server. `Work
    is in progress <https://github.com/mjl-/mox/issues/251>`_ to modify
@@ -18,3 +18,10 @@ We know of two work-in-progress alternative implementation efforts:
    plugin for the `Maddy email server <https://maddy.email/>`_ which
    aims to implement the chatmail relay features and configuration
    options.
+
+-  `Chatmail Cookbook <https://github.com/feld/chatmail-cookbook>`_:
+   A Chef Cookbook implementing a relay server. The project follows the
+   official relay server software and configurations converted to a Chef
+   Cookbook with only minor differences. The cookbook uses DNS-01 for
+   certificate validation and additionally supports FreeBSD. It does not
+   require a Chef server to use.

www/src/index.md

@@ -23,7 +23,3 @@ you can also **scan this QR code** with Delta Chat:
 🐣 **Choose** your Avatar and Name
 
 💬 **Start** chatting with any Delta Chat contacts using [QR invite codes](https://delta.chat/en/help#howtoe2ee)
-
-{% if config.mail_domain != "nine.testrun.org" %}
-<div class="experimental">Note: this is only a temporary development chatmail service</div>
-{% endif %}