cmdeploy: print dots for every local DNS query

This reverts commit c18ef083bfbd6a5e38a7bdaaa3c8ef8cca61cf74.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -12,6 +12,7 @@ Please fill out as much of this form as you can (leaving out stuff that is not a
 
 - Server OS (Operating System) - preferably Debian 12: 
 - On which OS you run cmdeploy:
+- chatmail/relay version: `git rev-parse HEAD`
 
 ## Expected behavior
 

.github/workflows/ci.yaml

@@ -10,6 +10,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
+        # Checkout pull request HEAD commit instead of merge commit
+        # Otherwise `test_deployed_state` will be unhappy.
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
 
       - name: run chatmaild tests 
         working-directory: chatmaild

.gitignore

@@ -164,3 +164,10 @@ cython_debug/
 #.idea/
 
 chatmail.zone
+
+# docker
+/data/
+/custom/
+docker-compose.yaml
+.env
+/traefik/data/

CHANGELOG.md

@@ -2,9 +2,87 @@
 
 ## untagged
 
+- Add installation via docker compose (MVP 1). The instructions, known issues and limitations are located in `/docs` 
+  ([#614](https://github.com/chatmail/relay/pull/614))
+
+- Add markdown tabs blocks for rendering multilingual pages. Add russian language support to `index.md`, `privacy.md`, and `info.md`.
+  ([#614](https://github.com/chatmail/relay/pull/614))
+
+- Fix [Issue 604](https://github.com/chatmail/relay/issues/604), now the `--ssh_host` argument of the `cmdeploy run` command works correctly and does not depend on `config.mail_domain`.
+  ([#614](https://github.com/chatmail/relay/pull/614))
+
+- Add `--skip-dns-check` argument to `cmdeploy run` command, which disables DNS record checking before installation.
+  ([#614](https://github.com/chatmail/relay/pull/614))
+
+- Add `--force` argument to `cmdeploy init` command, which recreates the `chatmail.ini` file.
+  ([#614](https://github.com/chatmail/relay/pull/614))
+
+- Add startup for `fcgiwrap.service` because sometimes it did not start automatically.
+  ([#614](https://github.com/chatmail/relay/pull/614))
+
+- Add extended check when installing `unbound.service`. Now, if it is not shown who exactly is occupying port 53, but `unbound.service` is running, it is considered that the port is occupied by `unbound.service`.
+  ([#614](https://github.com/chatmail/relay/pull/614))
+
+- Add configuration parameters
+  ([#614](https://github.com/chatmail/relay/pull/614)):
+  - `is_development_instance` - Indicates that this instance is installed as a temporary/test one (default: `True`)
+  - `use_foreign_cert_manager` - Use a third-party certificate manager instead of acmetool (default: `False`)
+  - `acme_email` - Email address used by acmetool to obtain Let's Encrypt certificates (default: empty)
+  - `change_kernel_settings` - Whether to change kernel parameters during installation (default: `True`)
+  - `fs_inotify_max_user_instances_and_watchers` - Value for kernel parameters `fs.inotify.max_user_instances` and `fs.inotify.max_user_watches` (default: `65535`)
+
+- Check whether GCC is installed in initenv.sh
+  ([#608](https://github.com/chatmail/relay/pull/608))
+
+- Expire push notification tokens after 90 days
+  ([#583](https://github.com/chatmail/relay/pull/583))
+
+- Use official `mtail` binary instead of `mtail` package
+  ([#581](https://github.com/chatmail/relay/pull/581))
+
+- dovecot: install from download.delta.chat instead of openSUSE Build Service
+  ([#590](https://github.com/chatmail/relay/pull/590))
+
+- Reconfigure Dovecot imap-login service to high-performance mode
+  ([#578](https://github.com/chatmail/relay/pull/578))
+
+- Set timezone to improve dovecot performance
+  ([#584](https://github.com/chatmail/relay/pull/584))
+
+- Increase nginx connection limits
+  ([#576](https://github.com/chatmail/relay/pull/576))
+
+- If `dns-utils` needs to be installed before cmdeploy run, apt update to make sure it works
+  ([#560](https://github.com/chatmail/relay/pull/560))
+
+- filtermail: respect config message size limit
+  ([#572](https://github.com/chatmail/relay/pull/572))
+
+- Add config value after how many days large files are deleted
+  ([#555](https://github.com/chatmail/relay/pull/555))
+
+- cmdeploy: push relay version to /etc/chatmail-version
+  ([#573](https://github.com/chatmail/relay/pull/573))
+
+- filtermail: allow partial body length in OpenPGP payloads
+  ([#570](https://github.com/chatmail/relay/pull/570))
+
+- chatmaild: allow echobot to receive unencrypted messages by default
+  ([#556](https://github.com/chatmail/relay/pull/556))
+
+
+## 1.6.0 2025-04-11
+
+- Handle Port-25 connect errors more gracefully (common with VPNs)
+  ([#552](https://github.com/chatmail/relay/pull/552))
+
 - Avoid "acmetool not found" during initial run
   ([#550](https://github.com/chatmail/relay/pull/550))
 
+- Fix timezone handling such that client/servers do not need to use
+  same timezone. 
+  ([#553](https://github.com/chatmail/relay/pull/553))
+
 - Enforce end-to-end encryption for incoming messages. 
   New user address mailboxes now get a `enforceE2EEincoming` file 
   which prohibits incoming cleartext messages from other domains. 
@@ -17,6 +95,12 @@
 - Enforce end-to-end encryption between local addresses 
   ([#535](https://github.com/chatmail/server/pull/535))
 
+- unbound: check that port 53 is not occupied by a different process
+  ([#537](https://github.com/chatmail/server/pull/537))
+
+- unbound: before unbound is there, use 9.9.9.9 for resolving
+  ([#518](https://github.com/chatmail/relay/pull/518))
+
 - Limit the bind for the HTTPS server on 8443 to 127.0.0.1 
   ([#522](https://github.com/chatmail/server/pull/522))
   ([#532](https://github.com/chatmail/server/pull/532))

README.md

@@ -69,38 +69,41 @@ Please substitute it with your own domain.
     mta-sts.chat.example.com. 3600 IN CNAME chat.example.com.
    ```
 
-2. Clone the repository and bootstrap the Python virtualenv.
+2. On your local PC, clone the repository and bootstrap the Python virtualenv.
 
    ```
     git clone https://github.com/chatmail/relay
     cd relay
+   ```
+
+### Manual installation
+1. On your local PC, create chatmail configuration file `chatmail.ini`:
+
+   ```
     scripts/initenv.sh
-   ```
-
-3. Create chatmail configuration file `chatmail.ini`:
-
-   ```
     scripts/cmdeploy init chat.example.org  # <-- use your domain 
    ```
 
-4. Verify that SSH root login works:
+2. Verify that SSH root login to your remote server works:
 
    ```
-    ssh root@chat.example.org   # <-- use your domain 
+    ssh root@chat.example.org  # <-- use your domain 
    ```
 
-
-5. Deploy the remote chatmail relay server:
+3. From your local PC, deploy the remote chatmail relay server:
 
    ```
     scripts/cmdeploy run
    ```
-   This script will check that you have all necessary DNS records.
+   This script will also check that you have all necessary DNS records.
    If DNS records are missing, it will recommend
    which you should configure at your DNS provider
    (it can take some time until they are public).
 
-### Other helpful commands:
+### Docker installation
+Installation using docker compose is presented [here](./docs/DOCKER_INSTALLATION_EN.md)
+
+### Other helpful commands
 
 To check the status of your remotely running chatmail service:
 
@@ -159,7 +162,7 @@ This repository has four directories:
 The `cmdeploy/src/cmdeploy/cmdeploy.py` command line tool
 helps with setting up and managing the chatmail service.
 `cmdeploy init` creates the `chatmail.ini` config file.
-`cmdeploy run` uses a [pyinfra](https://pyinfra.com/)-based [script](`cmdeploy/src/cmdeploy/__init__.py`)
+`cmdeploy run` uses a [pyinfra](https://pyinfra.com/)-based [`script`](cmdeploy/src/cmdeploy/__init__.py)
 to automatically install or upgrade all chatmail components on a relay,
 according to the `chatmail.ini` config.
 
@@ -533,3 +536,15 @@ Then reboot the relay or do `sysctl -p` and `nft -f /etc/nftables.conf`.
 
 Once proxy relay is set up,
 you can add its IP address to the DNS.
+
+## Neighbors and Acquaintances
+
+Here are some related projects that you may be interested in:
+
+- [Mox](https://github.com/mjl-/mox): A Golang email server.  [Work is in
+  progress](https://github.com/mjl-/mox/issues/251) to modify it to support all
+  of the features and configuration settings required to operate as a chatmail
+  relay.
+- [Maddy-Chatmail](https://github.com/sadraiiali/maddy_chatmail): a plugin for the
+  [Maddy email server](https://maddy.email/) which aims to implement the
+  chatmail relay features and configuration options.

chatmaild/pyproject.toml

@@ -48,6 +48,9 @@ lint.select = [
   "PLE", # Pylint Error
   "PLW", # Pylint Warning
 ]
+lint.ignore = [
+  "PLC0415" # import-outside-top-level
+]
 
 [tool.tox]
 legacy_tox_ini = """

chatmaild/src/chatmaild/config.py

@@ -26,12 +26,16 @@ class Config:
         self.max_mailbox_size = params["max_mailbox_size"]
         self.max_message_size = int(params.get("max_message_size", "31457280"))
         self.delete_mails_after = params["delete_mails_after"]
+        self.delete_large_after = params["delete_large_after"]
         self.delete_inactive_users_after = int(params["delete_inactive_users_after"])
         self.username_min_length = int(params["username_min_length"])
         self.username_max_length = int(params["username_max_length"])
         self.password_min_length = int(params["password_min_length"])
         self.passthrough_senders = params["passthrough_senders"].split()
         self.passthrough_recipients = params["passthrough_recipients"].split()
+        self.is_development_instance = (
+            params.get("is_development_instance", "true").lower() == "true"
+        )
         self.filtermail_smtp_port = int(params["filtermail_smtp_port"])
         self.filtermail_smtp_port_incoming = int(
             params["filtermail_smtp_port_incoming"]
@@ -42,6 +46,16 @@ class Config:
         )
         self.mtail_address = params.get("mtail_address")
         self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
+        self.use_foreign_cert_manager = (
+            params.get("use_foreign_cert_manager", "false").lower() == "true"
+        )
+        self.acme_email = params["acme_email"]
+        self.change_kernel_settings = (
+            params.get("change_kernel_settings", "true").lower() == "true"
+        )
+        self.fs_inotify_max_user_instances_and_watchers = int(
+            params["fs_inotify_max_user_instances_and_watchers"]
+        )
         self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
         if "iroh_relay" not in params:
             self.iroh_relay = "https://" + params["mail_domain"]
@@ -64,7 +78,7 @@ class Config:
     def _getbytefile(self):
         return open(self._inipath, "rb")
 
-    def get_user(self, addr):
+    def get_user(self, addr) -> User:
         if not addr or "@" not in addr or "/" in addr:
             raise ValueError(f"invalid address {addr!r}")
 
@@ -115,7 +129,7 @@ def get_default_config_content(mail_domain, **overrides):
         lines = []
         for line in content.split("\n"):
             for key, value in privacy.items():
-                value_lines = value.strip().split("\n")
+                value_lines = value.format(mail_domain=mail_domain).strip().split("\n")
                 if not line.startswith(f"{key} =") or not value_lines:
                     continue
                 if len(value_lines) == 1:

chatmaild/src/chatmaild/filtermail.py

@@ -38,6 +38,12 @@ def check_openpgp_payload(payload: bytes):
 
         packet_type_id = payload[i] & 0x3F
         i += 1
+
+        while payload[i] >= 224 and payload[i] < 255:
+            # Partial body length.
+            partial_length = 1 << (payload[i] & 0x1F)
+            i += 1 + partial_length
+
         if payload[i] < 192:
             # One-octet length.
             body_len = payload[i]
@@ -56,7 +62,7 @@ def check_openpgp_payload(payload: bytes):
             )
             i += 5
         else:
-            # Partial body length is not allowed.
+            # Impossible, partial body length was processed above.
             return False
 
         i += body_len
@@ -167,7 +173,12 @@ async def asyncmain_beforequeue(config, mode):
     else:
         port = config.filtermail_smtp_port_incoming
         handler = IncomingBeforeQueueHandler(config)
-    HackedController(handler, hostname="127.0.0.1", port=port).start()
+    HackedController(
+        handler,
+        hostname="127.0.0.1",
+        port=port,
+        data_size_limit=config.max_message_size,
+    ).start()
 
 
 def recipient_matches_passthrough(recipient, passthrough_recipients):

chatmaild/src/chatmaild/ini/chatmail.ini.f

@@ -23,6 +23,9 @@ max_message_size = 31457280
 # days after which mails are unconditionally deleted
 delete_mails_after = 20
 
+# days after which large messages (>200k) are unconditionally deleted
+delete_large_after = 7
+
 # days after which users without a successful login are deleted (database and mails)
 delete_inactive_users_after = 90
 
@@ -40,12 +43,15 @@ passthrough_senders =
 
 # list of e-mail recipients for which to accept outbound un-encrypted mails
 # (space-separated, item may start with "@" to whitelist whole recipient domains)
-passthrough_recipients = xstore@testrun.org 
+passthrough_recipients = xstore@testrun.org echo@{mail_domain}
 
 #
 # Deployment Details
 #
 
+# set to "False" to remove the "development instance" banner on the main page.
+is_development_instance = True
+
 # SMTP outgoing filtermail and reinjection 
 filtermail_smtp_port = 10080
 postfix_reinject_port = 10025
@@ -57,6 +63,22 @@ postfix_reinject_port_incoming = 10026
 # if set to "True" IPv6 is disabled
 disable_ipv6 = False
 
+# if you set "True", acmetool will not be installed and you will have to manage certificates yourself.
+use_foreign_cert_manager = False
+
+# Your email adress, which will be used in acmetool to manage Let's Encrypt SSL certificates. Required if `use_foreign_cert_manager` param set as "False". 
+acme_email = 
+
+#
+# Kernel settings
+#
+
+# if you set "True", the kernel settings will be configured according to the values below
+change_kernel_settings  = True
+
+# change fs.inotify.max_user_instances and fs.inotify.max_user_watches kernel settings
+fs_inotify_max_user_instances_and_watchers = 65535
+
 # Defaults to https://iroh.{{mail_domain}} and running `iroh-relay` on the chatmail
 # service.
 # If you set it to anything else, the service will be disabled

chatmaild/src/chatmaild/ini/override-testrun.ini

@@ -1,7 +1,7 @@
 
 [privacy]
 
-passthrough_recipients = privacy@testrun.org xstore@testrun.org
+passthrough_recipients = privacy@testrun.org xstore@testrun.org echo@{mail_domain}
 
 privacy_postal =
     Merlinux GmbH, Represented by the managing director H. Krekel,

chatmaild/src/chatmaild/metadata.py

@@ -1,5 +1,7 @@
 import logging
 import sys
+import time
+from contextlib import contextmanager
 
 from .config import read_config
 from .dictproxy import DictProxy
@@ -7,8 +9,15 @@ from .filedict import FileDict
 from .notifier import Notifier
 
 
+def _is_valid_token_timestamp(timestamp, now):
+    # Token if invalid after 90 days
+    # or if the timestamp is in the future.
+    return timestamp > now - 3600 * 24 * 90 and timestamp < now + 60
+
+
 class Metadata:
-    # each SETMETADATA on this key appends to a list of unique device tokens
+    # each SETMETADATA on this key appends to dictionary
+    # mapping of unique device tokens
     # which only ever get removed if the upstream indicates the token is invalid
     DEVICETOKEN_KEY = "devicetoken"
 
@@ -18,21 +27,51 @@ class Metadata:
     def get_metadata_dict(self, addr):
         return FileDict(self.vmail_dir / addr / "metadata.json")
 
-    def add_token_to_addr(self, addr, token):
+    @contextmanager
+    def _modify_tokens(self, addr):
         with self.get_metadata_dict(addr).modify() as data:
-            tokens = data.setdefault(self.DEVICETOKEN_KEY, [])
-            if token not in tokens:
-                tokens.append(token)
+            tokens = data.setdefault(self.DEVICETOKEN_KEY, {})
+            now = int(time.time())
+            if isinstance(tokens, list):
+                data[self.DEVICETOKEN_KEY] = tokens = {t: now for t in tokens}
+
+            expired_tokens = [
+                token
+                for token, timestamp in tokens.items()
+                if not _is_valid_token_timestamp(tokens[token], now)
+            ]
+            for expired_token in expired_tokens:
+                del tokens[expired_token]
+
+            yield tokens
+
+    def add_token_to_addr(self, addr, token):
+        with self._modify_tokens(addr) as tokens:
+            tokens[token] = int(time.time())
 
     def remove_token_from_addr(self, addr, token):
-        with self.get_metadata_dict(addr).modify() as data:
-            tokens = data.get(self.DEVICETOKEN_KEY, [])
+        with self._modify_tokens(addr) as tokens:
             if token in tokens:
-                tokens.remove(token)
+                del tokens[token]
 
     def get_tokens_for_addr(self, addr):
         mdict = self.get_metadata_dict(addr).read()
-        return mdict.get(self.DEVICETOKEN_KEY, [])
+        tokens = mdict.get(self.DEVICETOKEN_KEY, {})
+
+        now = int(time.time())
+        if isinstance(tokens, dict):
+            token_list = [
+                token
+                for token, timestamp in tokens.items()
+                if _is_valid_token_timestamp(timestamp, now)
+            ]
+            if len(token_list) < len(tokens):
+                # Some tokens have expired, remove them.
+                with self._modify_tokens(addr) as _tokens:
+                    pass
+        else:
+            token_list = []
+        return token_list
 
 
 class MetadataDictProxy(DictProxy):

chatmaild/src/chatmaild/notifier.py

@@ -17,11 +17,11 @@ and which are scheduled for retry using exponential back-off timing.
 If a token notification would be scheduled more than DROP_DEADLINE seconds
 after its first attempt, it is dropped with a log error.
 
-Note that tokens are completely opaque to the notification machinery here
-and will in the future be encrypted foreclosing all ability to distinguish
+Note that tokens are opaque to the notification machinery here
+and are encrypted foreclosing all ability to distinguish
 which device token ultimately goes to which phone-provider notification service,
 or to understand the relation of "device tokens" and chatmail addresses.
-The meaning and format of tokens is basically a matter of Delta-Chat Core and
+The meaning and format of tokens is basically a matter of chatmail Core and
 the `notification.delta.chat` service.
 """
 
@@ -95,7 +95,12 @@ class Notifier:
                 logging.warning(f"removing spurious queue item: {queue_path!r}")
                 queue_path.unlink()
                 continue
-            queue_item = PersistentQueueItem.read_from_path(queue_path)
+            try:
+                queue_item = PersistentQueueItem.read_from_path(queue_path)
+            except ValueError:
+                logging.warning(f"removing spurious queue item: {queue_path!r}")
+                queue_path.unlink()
+                continue
             self.queue_for_retry(queue_item)
 
     def queue_for_retry(self, queue_item, retry_num=0):

chatmaild/src/chatmaild/tests/test_config.py

@@ -35,6 +35,7 @@ def test_read_config_testrun(make_config):
     assert config.max_user_send_per_minute == 60
     assert config.max_mailbox_size == "100M"
     assert config.delete_mails_after == "20"
+    assert config.delete_large_after == "7"
     assert config.username_min_length == 9
     assert config.username_max_length == 9
     assert config.password_min_length == 9

chatmaild/src/chatmaild/tests/test_filtermail.py

@@ -304,3 +304,45 @@ HELLOWORLD
 \r
 """
     assert check_armored_payload(payload) == False
+
+    # Test payload using partial body length
+    # as generated by GopenPGP.
+    payload = """-----BEGIN PGP MESSAGE-----\r
+\r
+wV4DdCVjRfOT3TQSAQdAY5+pjT6mlCxPGdR3be4w7oJJRUGIPI/Vnh+mJxGSm34w\r
+LNlVc89S1g22uQYFif2sUJsQWbpoHpNkuWpkSgOaHmNvrZiY/YU5iv+cZ3LbmtUG\r
+0uoBisSHh9O1c+5sYZSbrvYZ1NOwlD7Fv/U5/Mw4E5+CjxfdgNGp5o3DDddzPK78\r
+jseDhdSXxnaiIJC93hxNX6R1RPt3G2gukyzx69wciPQShcF8zf3W3o75Ed7B8etV\r
+QEeB16xzdFhKa9JxdjTu3osgCs21IO7wpcFkjc7nZzlW6jPnELJJaNmv4yOOCjMp\r
+6YAkaN/BkL+jHTznHDuDsT5ilnTXpwHDU1Cm9PIx/KFcNCQnIB+2DcdIHPHUH1ci\r
+jvqoeXAVWjKXEjS7PqPFuP/xGbrWG2ugs+toXJOKbgRkExvKs1dwPFKrgghvCVbW\r
+AcKejQKAPArLwpkA7aD875TZQShvGt74fNs45XBlGOYOnNOAJ1KAmzrXLIDViyyB\r
+kDsmTBk785xofuCkjBpXSe6vsMprPzCteDfaUibh8FHeJjucxPerwuOPEmnogNaf\r
+YyL4+iy8H8I9/p7pmUqILprxTG0jTOtlk0bTVzeiF56W1xbtSEMuOo4oFbQTyOM2\r
+bKXaYo774Jm+rRtKAnnI2dtf9RpK19cog6YNzfYjesLKbXDsPZbN5rmwyFiCvvxC\r
+kQ6JLob+B2fPdY2gzy7LypxktS8Zi1HJcWDHJGVmQodaDLqKUObb4M26bXDe6oxI\r
+NS8PJz5exVbM3KhZnUOEn6PJRBBf5a/ZqxlhZPcQo/oBuhKpBRpO5kSDwPIUByu3\r
+UlXLSkpMqe9pUarAOEuQjfl2RVY7U+RrQYp4YP5keMO+i8NCefAFbowTTufO1JIq\r
+2nVgCi/QVnxZyEc9OYt/8AE3g4cdojE+vsSDifZLSWYIetpfrohHv3dT3StD1QRG\r
+0QE6qq6oKpg/IL0cjvuX4c7a7bslv2fXp8t75y37RU6253qdIebhxc/cRhPbc/yu\r
+p0YLyD4SrvKTLP2ZV95jT4IPEpqm4AN3QmiOzdtqR2gLyb62L8QfqI/FdwsIiRiM\r
+hqydwoqt/lfSqG1WKPh+6EkMkH+TDiCC1BQdbN1MNcyUtcjb35PR2c8Ld2TF3guA\r
+jLIqMt/Vb7hBoMb2FcsOYY25ka9oV62OwgKWLXnFzk+modMR5fzb4kxVVAYEqP+D\r
+T5KO1Vs76v1fyPGOq6BbBCvLwTqe/e6IZInJles4v5jrhnLcGKmNGivCUDe6X6NY\r
+UKNt5RsZllwDQpaAb5dMNhyrk8SgIE7TBI7rvqIdUCE52Vy+0JDxFg5olRpFUfO6\r
+/MyTW3Yo/ekk/npHr7iYYqJTCc21bDGLWQcIo/XO7WPxrKNWGBNPFnkRdw0MaKr4\r
++cEM3V8NFnSEpC12xA+RX/CezuJtwXZK5MpG76eYqMO6qyC+c25YcFecEufDZDxx\r
+ZLqRszVRyxyWPtk/oIeQK2v9wOqY6N9/ff01gHz69vqYqN5bUw/QKZsmx1zW+gPw\r
+6x2tDK2BHeYl182gCbhlKISRFwCtbjqZSkiKWao/VtygHkw0fK34avJuyQ/X9YaN\r
+BRy+7Lf3VA53pnB5WJ1xwRXN8VDvmZeXzv2krHveCMemj0OjnRoCLu117xN0A5m9\r
+Fm/RoDix5PolDHtWTtr2m1n2hp2LHnj8at9lFEd0SKhAYHVL9KjzycwWODZRXt+x\r
+zGDDuooEeTvdY5NLyKcl4gETz1ZP4Ez5jGGjhPSwSpq1mU7UaJ9ZXXdr4KHyifW6\r
+ggNzNsGhXTap7IWZpTtqXABydfiBshmH2NjqtNDwBweJVSgP10+r0WhMWlaZs6xl\r
+V3o5yskJt6GlkwpJxZrTvN6Tiww/eW7HFV6NGf7IRSWY5tJc/iA7/92tOmkdvJ1q\r
+myLbG7cJB787QjplEyVe2P/JBO6xYvbkJLf9Q+HaviTO25rugRSrYsoKMDfO8VlQ\r
+1CcnTPVtApPZJEQzAWJEgVAM8uIlkqWJJMgyWT34sTkdBeCUFGloXQFs9Yxd0AGf\r
+/zHEkYZSTKpVSvAIGu4=\r
+=6iHb\r
+-----END PGP MESSAGE-----\r
+"""
+    assert check_armored_payload(payload) == True

chatmaild/src/chatmaild/tests/test_metadata.py

@@ -242,6 +242,22 @@ def test_requeue_removes_tmp_files(notifier, metadata, testaddr, caplog):
     assert queue_item.addr == testaddr
 
 
+def test_requeue_removes_invalid_files(notifier, metadata, testaddr, caplog):
+    metadata.add_token_to_addr(testaddr, "01234")
+    notifier.new_message_for_addr(testaddr, metadata)
+    # empty/invalid files should be ignored
+    p = notifier.queue_dir.joinpath("1203981203")
+    p.touch()
+    notifier2 = notifier.__class__(notifier.queue_dir)
+    notifier2.requeue_persistent_queue_items()
+    assert "spurious" in caplog.records[0].msg
+    assert not p.exists()
+    assert notifier2.retry_queues[0].qsize() == 1
+    when, queue_item = notifier2.retry_queues[0].get()
+    assert when <= int(time.time())
+    assert queue_item.addr == testaddr
+
+
 def test_start_and_stop_notification_threads(notifier, testaddr):
     threads = notifier.start_notification_threads(None)
     for retry_num, threadlist in threads.items():

chatmaild/src/chatmaild/user.py

@@ -58,7 +58,8 @@ class User:
             if not self.addr.startswith("echo@"):
                 logging.error(f"could not write password for: {self.addr}")
                 raise
-        self.enforce_E2EE_path.touch()
+        if not self.addr.startswith("echo@"):
+            self.enforce_E2EE_path.touch()
 
     def set_last_login_timestamp(self, timestamp):
         """Track login time with daily granularity

cmdeploy/pyproject.toml

@@ -20,6 +20,7 @@ dependencies = [
   "pytest-xdist", 
   "execnet",
   "imap_tools",
+  "pymdown-extensions",
 ]
 
 [project.scripts]
@@ -41,3 +42,6 @@ lint.select = [
   "PLE", # Pylint Error
   "PLW", # Pylint Warning
 ]
+lint.ignore = [
+  "PLC0415" # import-outside-top-level
+]

cmdeploy/src/cmdeploy/__init__.py

@@ -7,17 +7,35 @@ import io
 import shutil
 import subprocess
 import sys
+from io import StringIO
 from pathlib import Path
 
 from chatmaild.config import Config, read_config
 from pyinfra import facts, host
+from pyinfra.api import FactBase
 from pyinfra.facts.files import File
-from pyinfra.facts.systemd import SystemdEnabled
+from pyinfra.facts.server import Sysctl
+from pyinfra.facts.systemd import SystemdEnabled, SystemdStatus
 from pyinfra.operations import apt, files, pip, server, systemd
 
 from .acmetool import deploy_acmetool
 
 
+class Port(FactBase):
+    """
+    Returns the process occuping a port.
+    """
+
+    def command(self, port: int) -> str:
+        return (
+            "ss -lptn 'src :%d' | awk 'NR>1 {print $6,$7}' | sed 's/users:((\"//;s/\".*//'"
+            % (port,)
+        )
+
+    def process(self, output: [str]) -> str:
+        return output[0]
+
+
 def _build_chatmaild(dist_dir) -> None:
     dist_dir = Path(dist_dir).resolve()
     if dist_dir.exists():
@@ -230,7 +248,6 @@ def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
     )
     need_restart |= service_file.changed
 
-
     return need_restart
 
 
@@ -301,6 +318,40 @@ def _configure_postfix(config: Config, debug: bool = False) -> bool:
     return need_restart
 
 
+def _install_dovecot_package(package: str, arch: str):
+    arch = "amd64" if arch == "x86_64" else arch
+    arch = "arm64" if arch == "aarch64" else arch
+    url = f"https://download.delta.chat/dovecot/dovecot-{package}_2.3.21%2Bdfsg1-3_{arch}.deb"
+    deb_filename = "/root/" + url.split("/")[-1]
+
+    match (package, arch):
+        case ("core", "amd64"):
+            sha256 = "43f593332e22ac7701c62d58b575d2ca409e0f64857a2803be886c22860f5587"
+        case ("core", "arm64"):
+            sha256 = "4d21eba1a83f51c100f08f2e49f0c9f8f52f721ebc34f75018e043306da993a7"
+        case ("imapd", "amd64"):
+            sha256 = "8d8dc6fc00bbb6cdb25d345844f41ce2f1c53f764b79a838eb2a03103eebfa86"
+        case ("imapd", "arm64"):
+            sha256 = "178fa877ddd5df9930e8308b518f4b07df10e759050725f8217a0c1fb3fd707f"
+        case ("lmtpd", "amd64"):
+            sha256 = "2f69ba5e35363de50962d42cccbfe4ed8495265044e244007d7ccddad77513ab"
+        case ("lmtpd", "arm64"):
+            sha256 = "89f52fb36524f5877a177dff4a713ba771fd3f91f22ed0af7238d495e143b38f"
+        case _:
+            apt.packages(packages=[f"dovecot-{package}"])
+            return
+
+    files.download(
+        name=f"Download dovecot-{package}",
+        src=url,
+        dest=deb_filename,
+        sha256sum=sha256,
+        cache_time=60 * 60 * 24 * 365 * 10,  # never redownload the package
+    )
+
+    apt.deb(name=f"Install dovecot-{package}", src=deb_filename)
+
+
 def _configure_dovecot(config: Config, debug: bool = False) -> bool:
     """Configures Dovecot IMAP server."""
     need_restart = False
@@ -344,16 +395,28 @@ def _configure_dovecot(config: Config, debug: bool = False) -> bool:
         config=config,
     )
 
-    # as per https://doc.dovecot.org/configuration_manual/os/
+    # as per https://doc.dovecot.org/2.3/configuration_manual/os/
     # it is recommended to set the following inotify limits
-    for name in ("max_user_instances", "max_user_watches"):
-        key = f"fs.inotify.{name}"
-        server.sysctl(
-            name=f"Change {key}",
-            key=key,
-            value=65535,
-            persist=True,
-        )
+    if config.change_kernel_settings:
+        for name in ("max_user_instances", "max_user_watches"):
+            key = f"fs.inotify.{name}"
+            if host.get_fact(Sysctl)[key] == config.fs_inotify_max_user_instances_and_watchers:
+                # Skip updating limits if already sufficient
+                # (enables running in incus containers where sysctl readonly)
+                continue
+            server.sysctl(
+                name=f"Change {key}",
+                key=key,
+                value=config.fs_inotify_max_user_instances_and_watchers,
+                persist=True,
+            )
+
+    timezone_env = files.line(
+        name="Set TZ environment variable",
+        path="/etc/environment",
+        line="TZ=:/etc/localtime",
+    )
+    need_restart |= timezone_env.changed
 
     return need_restart
 
@@ -436,9 +499,26 @@ def check_config(config):
 
 
 def deploy_mtail(config):
-    apt.packages(
-        name="Install mtail",
-        packages=["mtail"],
+    # Uninstall mtail package, we are going to install a static binary.
+    apt.packages(name="Uninstall mtail", packages=["mtail"], present=False)
+
+    (url, sha256sum) = {
+        "x86_64": (
+            "https://github.com/google/mtail/releases/download/v3.0.8/mtail_3.0.8_linux_amd64.tar.gz",
+            "123c2ee5f48c3eff12ebccee38befd2233d715da736000ccde49e3d5607724e4",
+        ),
+        "aarch64": (
+            "https://github.com/google/mtail/releases/download/v3.0.8/mtail_3.0.8_linux_arm64.tar.gz",
+            "aa04811c0929b6754408676de520e050c45dddeb3401881888a092c9aea89cae",
+        ),
+    }[host.get_fact(facts.server.Arch)]
+
+    server.shell(
+        name="Download mtail",
+        commands=[
+            f"(echo '{sha256sum} /usr/local/bin/mtail' | sha256sum -c) || (curl -L {url} | gunzip | tar -x -f - mtail -O >/usr/local/bin/mtail.new && mv /usr/local/bin/mtail.new /usr/local/bin/mtail)",
+            "chmod 755 /usr/local/bin/mtail",
+        ],
     )
 
     # Using our own systemd unit instead of `/usr/lib/systemd/system/mtail.service`.
@@ -574,9 +654,15 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         path="/etc/apt/sources.list",
         line="deb [signed-by=/etc/apt/keyrings/obs-home-deltachat.gpg] https://download.opensuse.org/repositories/home:/deltachat/Debian_12/ ./",
         escape_regex_characters=True,
-        ensure_newline=True,
+        present=False,
     )
 
+    if host.get_fact(Port, port=53) != "unbound":
+        files.line(
+            name="Add 9.9.9.9 to resolv.conf",
+            path="/etc/resolv.conf",
+            line="nameserver 9.9.9.9",
+        )
     apt.update(name="apt update", cache_time=24 * 3600)
     apt.upgrade(name="upgrade apt packages", auto_remove=True)
 
@@ -588,6 +674,14 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
     # Run local DNS resolver `unbound`.
     # `resolvconf` takes care of setting up /etc/resolv.conf
     # to use 127.0.0.1 as the resolver.
+    from cmdeploy.cmdeploy import Out
+
+    process_on_53 = host.get_fact(Port, port=53)
+    if host.get_fact(SystemdStatus, services="unbound").get("unbound.service"):
+        process_on_53 = "unbound"
+    if process_on_53 not in (None, "unbound"):
+        Out().red(f"Can't install unbound: port 53 is occupied by: {process_on_53}")
+        exit(1)
     apt.packages(
         name="Install unbound",
         packages=["unbound", "unbound-anchor", "dnsutils"],
@@ -609,10 +703,12 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
     deploy_iroh_relay(config)
 
     # Deploy acmetool to have TLS certificates.
-    tls_domains = [mail_domain, f"mta-sts.{mail_domain}", f"www.{mail_domain}"]
-    deploy_acmetool(
-        domains=tls_domains,
-    )
+    if not config.use_foreign_cert_manager:
+        tls_domains = [mail_domain, f"mta-sts.{mail_domain}", f"www.{mail_domain}"]
+        deploy_acmetool(
+            email = config.acme_email,
+            domains=tls_domains,
+        )
 
     apt.packages(
         # required for setfacl for echobot
@@ -625,10 +721,10 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         packages="postfix",
     )
 
-    apt.packages(
-        name="Install Dovecot",
-        packages=["dovecot-imapd", "dovecot-lmtpd"],
-    )
+    if not "dovecot.service" in host.get_fact(SystemdEnabled):
+        _install_dovecot_package("core", host.get_fact(facts.server.Arch))
+        _install_dovecot_package("imapd", host.get_fact(facts.server.Arch))
+        _install_dovecot_package("lmtpd", host.get_fact(facts.server.Arch))
 
     apt.packages(
         name="Install nginx",
@@ -692,6 +788,13 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         enabled=True,
         restarted=nginx_need_restart,
     )
+    
+    systemd.service(
+        name="Start and enable fcgiwrap",
+        service="fcgiwrap.service",
+        running=True,
+        enabled=True,
+    )
 
     # This file is used by auth proxy.
     # https://wiki.debian.org/EtcMailName
@@ -725,5 +828,19 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         name="Ensure cron is installed",
         packages=["cron"],
     )
+    try:
+        git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode()
+    except Exception:
+        git_hash = "unknown\n"
+    try:
+        git_diff = subprocess.check_output(["git", "diff"]).decode()
+    except Exception:
+        git_diff = ""
+    files.put(
+        name="Upload chatmail relay git commiit hash",
+        src=StringIO(git_hash + git_diff),
+        dest="/etc/chatmail-version",
+        mode="700",
+    )
 
     deploy_mtail(config)

cmdeploy/src/cmdeploy/cmdeploy.py

@@ -32,17 +32,28 @@ def init_cmd_options(parser):
         action="store",
         help="fully qualified DNS domain name for your chatmail instance",
     )
+    parser.add_argument(
+        "--force",
+        dest="recreate_ini",
+        action="store_true",
+        help="force reacreate ini file",
+    )
 
 
 def init_cmd(args, out):
     """Initialize chatmail config file."""
     mail_domain = args.chatmail_domain
+    inipath = args.inipath
     if args.inipath.exists():
-        print(f"Path exists, not modifying: {args.inipath}")
-        return 1
-    else:
-        write_initial_config(args.inipath, mail_domain, overrides={})
-        out.green(f"created config file for {mail_domain} in {args.inipath}")
+        if not args.recreate_ini:
+            out.green(f"[WARNING] Path exists, not modifying: {inipath}")
+            return 0
+        else:
+            out.yellow(f"[WARNING] Force argument was provided, deleting config file: {inipath}")
+            inipath.unlink()
+    
+    write_initial_config(inipath, mail_domain, overrides={})
+    out.green(f"created config file for {mail_domain} in {inipath}")
 
 
 def run_cmd_options(parser):
@@ -61,7 +72,13 @@ def run_cmd_options(parser):
     parser.add_argument(
         "--ssh-host",
         dest="ssh_host",
-        help="specify an SSH host to deploy to; uses mail_domain from chatmail.ini by default",
+        help="Deploy to 'localhost', via 'docker', or to a specific SSH host",
+    )
+    parser.add_argument(
+        "--skip-dns-check",
+        dest="dns_check_disabled",
+        action="store_true",
+        help="disable checks nslookup for dns",
     )
 
 
@@ -70,9 +87,10 @@ def run_cmd(args, out):
 
     sshexec = args.get_sshexec()
     require_iroh = args.config.enable_iroh_relay
-    remote_data = dns.get_initial_remote_data(sshexec, args.config.mail_domain)
-    if not dns.check_initial_remote_data(remote_data, print=out.red):
-        return 1
+    if not args.dns_check_disabled:
+        remote_data = dns.get_initial_remote_data(sshexec, args.config.mail_domain)
+        if not dns.check_initial_remote_data(remote_data, print=out.red):
+            return 1
 
     env = os.environ.copy()
     env["CHATMAIL_INI"] = args.inipath
@@ -81,20 +99,31 @@ def run_cmd(args, out):
     deploy_path = importlib.resources.files(__package__).joinpath("deploy.py").resolve()
     pyinf = "pyinfra --dry" if args.dry_run else "pyinfra"
     ssh_host = args.config.mail_domain if not args.ssh_host else args.ssh_host
+
     cmd = f"{pyinf} --ssh-user root {ssh_host} {deploy_path} -y"
+    if sshexec in ["docker", "localhost"]:
+        cmd = f"{pyinf} @local {deploy_path} -y"
+
     if version.parse(pyinfra.__version__) < version.parse("3"):
         out.red("Please re-run scripts/initenv.sh to update pyinfra to version 3.")
         return 1
 
-    retcode = out.check_call(cmd, env=env)
-    if retcode == 0:
-        out.green("Deploy completed, call `cmdeploy dns` next.")
-    elif not remote_data["acme_account_url"]:
-        out.red("Deploy completed but letsencrypt not configured")
-        out.red("Run 'cmdeploy run' again")
-        retcode = 0
-    else:
+    try:
+        retcode = out.check_call(cmd, env=env)
+        if retcode == 0:
+            server_deployed_message = f"Chatmail server started: https://{args.config.mail_domain}/"
+            delimiter_line = "=" * len(server_deployed_message)
+            out.green(f"{delimiter_line}\n{server_deployed_message}\n{delimiter_line}")
+            out.green("Deploy completed, call `cmdeploy dns` next.")
+        elif not remote_data["acme_account_url"]:
+            out.red("Deploy completed but letsencrypt not configured")
+            out.red("Run 'cmdeploy run' again")
+            retcode = 0
+        else:
+            out.red("Deploy failed")
+    except subprocess.CalledProcessError:
         out.red("Deploy failed")
+        retcode = 1
     return retcode
 
 
@@ -106,6 +135,11 @@ def dns_cmd_options(parser):
         default=None,
         help="write out a zonefile",
     )
+    parser.add_argument(
+        "--ssh-host",
+        dest="ssh_host",
+        help="Run the DNS queries on 'localhost', in the chatmail 'docker' container, or on a specific SSH host",
+    )
 
 
 def dns_cmd(args, out):
@@ -247,8 +281,17 @@ class Out:
     def green(self, msg, file=sys.stderr):
         print(colored(msg, "green"), file=file)
 
-    def __call__(self, msg, red=False, green=False, file=sys.stdout):
-        color = "red" if red else ("green" if green else None)
+    def yellow(self, msg, file=sys.stderr):
+        print(colored(msg, "yellow"), file=file)
+
+    def __call__(self, msg, red=False, green=False, yellow=False, file=sys.stdout):
+        color = None
+        if red:
+            color = "red"
+        elif green:
+            color = "green"
+        elif yellow:
+            color = "yellow"
         print(colored(msg, color), file=file)
 
     def check_call(self, arg, env=None, quiet=False):
@@ -327,8 +370,14 @@ def main(args=None):
         return parser.parse_args(["-h"])
 
     def get_sshexec():
-        print(f"[ssh] login to {args.config.mail_domain}")
-        return SSHExec(args.config.mail_domain, verbose=args.verbose)
+        host = args.ssh_host if hasattr(args, "ssh_host") and args.ssh_host else args.config.mail_domain
+        if host in [ "@local", "localhost" ]:
+            return "localhost"
+        elif host == "docker":
+            return "docker"
+        
+        print(f"[ssh] login to {host}")
+        return SSHExec(host, verbose=args.verbose)
 
     args.get_sshexec = get_sshexec
 

cmdeploy/src/cmdeploy/dns.py

@@ -7,6 +7,10 @@ from . import remote
 
 
 def get_initial_remote_data(sshexec, mail_domain):
+    if sshexec == "docker":
+        return remote.rdns.perform_initial_checks(mail_domain, pre_command="docker exec chatmail ")
+    elif sshexec == "localhost":
+        return remote.rdns.perform_initial_checks(mail_domain, pre_command="")
     return sshexec.logged(
         call=remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=mail_domain)
     )
@@ -44,14 +48,17 @@ def check_full_zone(sshexec, remote_data, out, zonefile) -> int:
     """Check existing DNS records, optionally write them to zone file
     and return (exitcode, remote_data) tuple."""
 
-    required_diff, recommended_diff = sshexec.logged(
-        remote.rdns.check_zonefile,
-        kwargs=dict(zonefile=zonefile, mail_domain=remote_data["mail_domain"]),
-    )
+    if sshexec in ["docker", "localhost"]:
+        required_diff, recommended_diff = remote.rdns.check_zonefile(zonefile, remote_data["mail_domain"], verbose=False)
+    else:
+        required_diff, recommended_diff = sshexec.logged(
+            remote.rdns.check_zonefile,
+            kwargs=dict(zonefile=zonefile, mail_domain=remote_data["mail_domain"]),
+        )
 
     returncode = 0
     if required_diff:
-        out.red("Please set required DNS entries at your DNS provider:\n")
+        out.red("\nPlease set required DNS entries at your DNS provider:\n")
         for line in required_diff:
             out(line)
         out("")

cmdeploy/src/cmdeploy/dovecot/dovecot.conf.j2

@@ -177,20 +177,34 @@ service auth-worker {
 }
 
 service imap-login {
-    # High-security mode.
-    # Each process serves a single connection and exits afterwards.
-    # This is the default, but we set it explicitly to be sure.
-    # See <https://doc.dovecot.org/admin_manual/login_processes/#high-security-mode> for details.
-    service_count = 1
-
-    # Inrease the number of simultaneous connections.
+    # High-performance mode as described in
+    # <https://doc.dovecot.org/2.3/admin_manual/login_processes/#high-performance-mode>
     #
-    # As of Dovecot 2.3.19.1 the default is 100 processes.
-    # Combined with `service_count = 1` it means only 100 connections
-    # can be handled simultaneously.
-    process_limit = 10000
+    # So-called high-security mode described in
+    # <https://doc.dovecot.org/2.3/admin_manual/login_processes/#high-security-mode>
+    # and enabled by default with `service_count = 1` starts one process per connection
+    # and has problems logging in thousands of users after Dovecot restart.
+    service_count = 0
+
+    # Increase virtual memory size limit.
+    # Since imap-login processes handle TLS connections
+    # even after logging users in
+    # and many connections are handled by each process,
+    # memory size limit should be increased.
+    #
+    # Otherwise the whole process eventually dies
+    # with an error similar to
+    #   imap-login: Fatal: master: service(imap-login):
+    #   child 1422951 returned error 83
+    #   (Out of memory (service imap-login { vsz_limit=256 MB },
+    #    you may need to increase it)
+    # and takes down all its TLS connections at once.
+    vsz_limit = 1G
 
     # Avoid startup latency for new connections.
+    #
+    # Should be set to at least the number of CPU cores
+    # according to the documentation.
     process_min_avail = 10
 }
 

cmdeploy/src/cmdeploy/dovecot/expunge.cron.j2

@@ -1,5 +1,5 @@
 # delete already seen big mails after 7 days, in the INBOX
-2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +7 -size +200k -type f -delete
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +{{ config.delete_large_after }} -size +200k -type f -delete
 # delete all mails after {{ config.delete_mails_after }} days, in the Inbox
 2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
 # or in any IMAP subfolder

cmdeploy/src/cmdeploy/dovecot/push_notification.lua

@@ -2,15 +2,6 @@ function dovecot_lua_notify_begin_txn(user)
   return user
 end
 
-function contains(v, needle)
-  for _, keyword in ipairs(v) do
-    if keyword == needle then
-      return true
-    end
-  end
-  return false
-end
-
 function dovecot_lua_notify_event_message_new(user, event)
   local mbox = user:mailbox(event.mailbox)
   mbox:sync()

cmdeploy/src/cmdeploy/mtail/mtail.service.j2

@@ -3,7 +3,7 @@ Description=mtail
 
 [Service]
 Type=simple
-ExecStart=/bin/sh -c "journalctl -f -o short-iso -n 0 | /usr/bin/mtail --address={{ address }} --port={{ port }} --progs /etc/mtail --logtostderr --logs /dev/stdin"
+ExecStart=/bin/sh -c "journalctl -f -o short-iso -n 0 | /usr/local/bin/mtail --address={{ address }} --port={{ port }} --progs /etc/mtail --logtostderr --logs -"
 Restart=on-failure
 
 [Install]

cmdeploy/src/cmdeploy/nginx/nginx.conf.j2

@@ -2,11 +2,25 @@ load_module modules/ngx_stream_module.so;
 
 user www-data;
 worker_processes auto;
+
+# Increase the number of connections
+# that a worker process can open
+# to avoid errors such as
+#   accept4() failed (24: Too many open files)
+# and
+#   socket() failed (24: Too many open files) while connecting to upstream
+# in the logs.
+# <https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile>
+worker_rlimit_nofile 2048;
 pid /run/nginx.pid;
 error_log syslog:server=unix:/dev/log,facility=local3;
 
 events {
-	worker_connections 768;
+        # Increase to avoid errors such as
+        #   768 worker_connections are not enough while connecting to upstream
+        # in the logs.
+        # <https://nginx.org/en/docs/ngx_core_module.html#worker_connections>
+	worker_connections 2048;
 	# multi_accept on;
 }
 

cmdeploy/src/cmdeploy/remote/rdns.py

@@ -12,23 +12,23 @@ All functions of this module
 
 import re
 
-from .rshell import CalledProcessError, shell
+from .rshell import CalledProcessError, shell, log_progress
 
 
-def perform_initial_checks(mail_domain):
+def perform_initial_checks(mail_domain, pre_command=""):
     """Collecting initial DNS settings."""
     assert mail_domain
-    if not shell("dig", fail_ok=True):
-        shell("apt-get install -y dnsutils")
+    if not shell("dig", fail_ok=True, print=log_progress):
+        shell("apt-get update && apt-get install -y dnsutils", print=log_progress)
     A = query_dns("A", mail_domain)
     AAAA = query_dns("AAAA", mail_domain)
     MTA_STS = query_dns("CNAME", f"mta-sts.{mail_domain}")
     WWW = query_dns("CNAME", f"www.{mail_domain}")
 
     res = dict(mail_domain=mail_domain, A=A, AAAA=AAAA, MTA_STS=MTA_STS, WWW=WWW)
-    res["acme_account_url"] = shell("acmetool account-url", fail_ok=True)
+    res["acme_account_url"] = shell(pre_command + "acmetool account-url", fail_ok=True, print=log_progress)
     res["dkim_entry"], res["web_dkim_entry"] = get_dkim_entry(
-        mail_domain, dkim_selector="opendkim"
+        mail_domain, pre_command, dkim_selector="opendkim"
     )
 
     if not MTA_STS or not WWW or (not A and not AAAA):
@@ -40,11 +40,12 @@ def perform_initial_checks(mail_domain):
     return res
 
 
-def get_dkim_entry(mail_domain, dkim_selector):
+def get_dkim_entry(mail_domain, pre_command, dkim_selector):
     try:
         dkim_pubkey = shell(
-            f"openssl rsa -in /etc/dkimkeys/{dkim_selector}.private "
-            "-pubout 2>/dev/null | awk '/-/{next}{printf(\"%s\",$0)}'"
+            f"{pre_command} openssl rsa -in /etc/dkimkeys/{dkim_selector}.private "
+            "-pubout 2>/dev/null | awk '/-/{next}{printf(\"%s\",$0)}'",
+            print=log_progress
         )
     except CalledProcessError:
         return
@@ -61,7 +62,7 @@ def query_dns(typ, domain):
     # Get autoritative nameserver from the SOA record.
     soa_answers = [
         x.split()
-        for x in shell(f"dig -r -q {domain} -t SOA +noall +authority +answer").split(
+        for x in shell(f"dig -r -q {domain} -t SOA +noall +authority +answer", print=log_progress).split(
             "\n"
         )
     ]
@@ -71,13 +72,13 @@ def query_dns(typ, domain):
     ns = soa[0][4]
 
     # Query authoritative nameserver directly to bypass DNS cache.
-    res = shell(f"dig @{ns} -r -q {domain} -t {typ} +short")
+    res = shell(f"dig @{ns} -r -q {domain} -t {typ} +short", print=log_progress)
     if res:
         return res.split("\n")[0]
     return ""
 
 
-def check_zonefile(zonefile, mail_domain):
+def check_zonefile(zonefile, mail_domain, verbose=True):
     """Check expected zone file entries."""
     required = True
     required_diff = []
@@ -89,7 +90,7 @@ def check_zonefile(zonefile, mail_domain):
             continue
         if not zf_line.strip() or zf_line.startswith(";"):
             continue
-        print(f"dns-checking {zf_line!r}")
+        print(f"dns-checking {zf_line!r}") if verbose else log_progress("")
         zf_domain, zf_typ, zf_value = zf_line.split(maxsplit=2)
         zf_domain = zf_domain.rstrip(".")
         zf_value = zf_value.strip()

cmdeploy/src/cmdeploy/remote/rshell.py

@@ -1,7 +1,13 @@
 from subprocess import DEVNULL, CalledProcessError, check_output
+import sys
 
 
-def shell(command, fail_ok=False):
+def log_progress(data):
+    sys.stderr.write(".")
+    sys.stderr.flush()
+
+
+def shell(command, fail_ok=False, print=print):
     print(f"$ {command}")
     args = dict(shell=True)
     if fail_ok:

cmdeploy/src/cmdeploy/sshexec.py

@@ -70,10 +70,6 @@ class SSHExec:
                 raise self.FuncError(data)
 
     def logged(self, call, kwargs):
-        def log_progress(data):
-            sys.stderr.write(".")
-            sys.stderr.flush()
-
         title = call.__doc__
         if not title:
             title = call.__name__
@@ -82,6 +78,6 @@ class SSHExec:
             return self(call, kwargs, log_callback=print_stderr)
         else:
             print_stderr(title, end="")
-            res = self(call, kwargs, log_callback=log_progress)
+            res = self(call, kwargs, log_callback=remote.rshell.log_progress)
             print_stderr()
             return res

cmdeploy/src/cmdeploy/tests/online/test_0_login.py

@@ -90,8 +90,13 @@ def test_concurrent_logins_same_account(
 
 
 def test_no_vrfy(chatmail_config):
+    domain = chatmail_config.mail_domain
     sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    sock.connect((chatmail_config.mail_domain, 25))
+    sock.settimeout(10)
+    try:
+        sock.connect((domain, 25))
+    except socket.timeout:
+        pytest.skip(f"port 25 not reachable for {domain}")
     banner = sock.recv(1024)
     print(banner)
     sock.send(b"VRFY wrongaddress@%s\r\n" % (chatmail_config.mail_domain.encode(),))

cmdeploy/src/cmdeploy/tests/online/test_1_basic.py

@@ -1,5 +1,7 @@
 import datetime
 import smtplib
+import socket
+import subprocess
 
 import pytest
 
@@ -55,11 +57,20 @@ class TestSSHExecutor:
 
     def test_opendkim_restarted(self, sshexec):
         """check that opendkim is not running for longer than a day."""
-        out = sshexec(call=remote.rshell.shell, kwargs=dict(command="systemctl status opendkim"))
-        assert type(out) == str
-        since_date_str = out.split("since ")[1].split(";")[0]
-        since_date = datetime.datetime.strptime(since_date_str, "%a %Y-%m-%d %H:%M:%S %Z")
-        assert (datetime.datetime.now() - since_date).total_seconds() < 60 * 60 * 24
+        cmd = "systemctl show opendkim --timestamp=utc --property=ActiveEnterTimestamp"
+        out = sshexec(call=remote.rshell.shell, kwargs=dict(command=cmd))
+        datestring = out.split("=")[1]
+        since_date = datetime.datetime.strptime(datestring, "%a %Y-%m-%d %H:%M:%S %Z")
+        now = datetime.datetime.now(since_date.tzinfo)
+        assert (now - since_date).total_seconds() < 60 * 60 * 51
+
+
+def test_timezone_env(remote):
+    for line in remote.iter_output("env"):
+        print(line)
+        if line == "tz=:/etc/localtime":
+            return True
+    pytest.fail("TZ is not set")
 
 
 def test_remote(remote, imap_or_smtp):
@@ -116,9 +127,21 @@ def test_authenticated_from(cmsetup, maildata):
 
 @pytest.mark.parametrize("from_addr", ["fake@example.org", "fake@testrun.org"])
 def test_reject_missing_dkim(cmsetup, maildata, from_addr):
+    domain = cmsetup.maildomain
+    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    sock.settimeout(10)
+    try:
+        sock.connect((domain, 25))
+    except socket.timeout:
+        pytest.skip(f"port 25 not reachable for {domain}")
+
     recipient = cmsetup.gen_users(1)[0]
-    msg = maildata("encrypted.eml", from_addr=from_addr, to_addr=recipient.addr).as_string()
-    with smtplib.SMTP(cmsetup.maildomain, 25) as s:
+    msg = maildata(
+        "encrypted.eml", from_addr=from_addr, to_addr=recipient.addr
+    ).as_string()
+    conn = smtplib.SMTP(cmsetup.maildomain, 25, timeout=10)
+
+    with conn as s:
         with pytest.raises(smtplib.SMTPDataError, match="No valid DKIM signature"):
             s.sendmail(from_addr=from_addr, to_addrs=recipient.addr, msg=msg)
 
@@ -176,6 +199,25 @@ def test_expunged(remote, chatmail_config):
         f"find {chatmail_config.mailboxes_dir} -path '*/tmp/*' -mtime +{outdated_days} -type f",
         f"find {chatmail_config.mailboxes_dir} -path '*/.*/tmp/*' -mtime +{outdated_days} -type f",
     ]
+    outdated_days = int(chatmail_config.delete_large_after) + 1
+    find_cmds.append(
+        "find {chatmail_config.mailboxes_dir} -path '*/cur/*' -mtime +{outdated_days} -size +200k -type f"
+    )
     for cmd in find_cmds:
         for line in remote.iter_output(cmd):
             assert not line
+
+
+def test_deployed_state(remote):
+    git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode()
+    git_diff = subprocess.check_output(["git", "diff"]).decode()
+    git_status = [git_hash.strip()]
+    for line in git_diff.splitlines():
+        git_status.append(line.strip().lower())
+    remote_version = []
+    for line in remote.iter_output("cat /etc/chatmail-version"):
+        print(line)
+        remote_version.append(line)
+    # assert len(git_status) == len(remote_version)  # for some reason, we only get 11 lines from remote.iter_output()
+    for i in range(len(remote_version)):
+        assert git_status[i] == remote_version[i], "You have undeployed changes."

cmdeploy/src/cmdeploy/tests/plugin.py

@@ -307,6 +307,7 @@ def cmfactory(request, gencreds, tmpdir, maildomain):
     class Data:
         def read_path(self, path):
             return
+
     am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=Data())
 
     # nb. a bit hacky

cmdeploy/src/cmdeploy/www.py

@@ -25,7 +25,8 @@ def prepare_template(source):
     assert source.exists(), source
     render_vars = {}
     render_vars["pagename"] = "home" if source.stem == "index" else source.stem
-    render_vars["markdown_html"] = markdown.markdown(source.read_text())
+    # tabs usage for multiple languages https://facelessuser.github.io/pymdown-extensions/extensions/blocks/plugins/tab/
+    render_vars["markdown_html"] = markdown.markdown(source.read_text(), extensions=['pymdownx.blocks.tab'])
     page_layout = source.with_name("page-layout.html").read_text()
     return render_vars, page_layout
 

docker/chatmail_relay.dockerfile

@@ -0,0 +1,83 @@
+FROM jrei/systemd-debian:12 AS base
+
+ENV LANG=en_US.UTF-8
+
+RUN echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/01norecommend && \
+    echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf.d/01norecommend && \
+    apt-get update && \
+    apt-get install -y \
+        ca-certificates && \
+    DEBIAN_FRONTEND=noninteractive \
+    TZ=Europe/London \
+    apt-get install -y tzdata && \
+    apt-get install -y locales && \
+    sed -i -e "s/# $LANG.*/$LANG UTF-8/" /etc/locale.gen && \
+    dpkg-reconfigure --frontend=noninteractive locales && \
+    update-locale LANG=$LANG \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN apt-get update && \
+    apt-get install -y \
+        git \
+        python3 \
+        python3-venv \
+        python3-virtualenv \
+        gcc \
+        python3-dev \
+        opendkim \
+        opendkim-tools \
+        curl \
+        rsync \
+        unbound \
+        unbound-anchor \
+        dnsutils \
+        postfix \
+        acl \
+        nginx \
+        libnginx-mod-stream \
+        fcgiwrap \
+        cron \
+    && for pkg in core imapd lmtpd; do \
+      case "$pkg" in \
+        core) sha256="43f593332e22ac7701c62d58b575d2ca409e0f64857a2803be886c22860f5587" ;; \
+        imapd) sha256="8d8dc6fc00bbb6cdb25d345844f41ce2f1c53f764b79a838eb2a03103eebfa86" ;; \
+        lmtpd) sha256="2f69ba5e35363de50962d42cccbfe4ed8495265044e244007d7ccddad77513ab" ;; \
+      esac; \
+      url="https://download.delta.chat/dovecot/dovecot-${pkg}_2.3.21%2Bdfsg1-3_amd64.deb"; \
+      file="/tmp/$(basename "$url")"; \
+      curl -fsSL "$url" -o "$file"; \
+      echo "$sha256  $file" | sha256sum -c -; \
+      apt-get install -y "$file"; \
+      rm -f "$file"; \
+    done \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /opt/chatmail
+
+ARG SETUP_CHATMAIL_SERVICE_PATH=/lib/systemd/system/setup_chatmail.service
+COPY ./files/setup_chatmail.service "$SETUP_CHATMAIL_SERVICE_PATH"
+RUN ln -sf "$SETUP_CHATMAIL_SERVICE_PATH" "/etc/systemd/system/multi-user.target.wants/setup_chatmail.service"
+
+COPY --chmod=555 ./files/setup_chatmail_docker.sh /setup_chatmail_docker.sh
+COPY --chmod=555 ./files/update_ini.sh /update_ini.sh
+COPY --chmod=555 ./files/entrypoint.sh /entrypoint.sh
+
+## TODO: add git clone.
+## Problem: how correct save only required files inside container....
+# RUN git clone https://github.com/chatmail/relay.git -b master . \
+#  && ./scripts/initenv.sh
+
+# EXPOSE 443 25 587 143 993 
+
+VOLUME ["/sys/fs/cgroup", "/home"]
+
+STOPSIGNAL SIGRTMIN+3
+
+ENTRYPOINT ["/entrypoint.sh"]
+
+CMD [   "--default-standard-output=journal+console", \
+        "--default-standard-error=journal+console" ]
+
+## TODO: Add installation and configuration of chatmaild inside the Dockerfile.
+## This is required to ensure repeatable deployment.
+## In the current MVP, the chatmaild server is updated on every container restart.

docker/docker-compose-default.yaml

@@ -0,0 +1,59 @@
+services:
+  chatmail:
+    build: 
+      context: ./docker
+      dockerfile: chatmail_relay.dockerfile
+      tags:
+        - chatmail-relay:latest
+    image: chatmail-relay:latest
+    restart: unless-stopped
+    container_name: chatmail 
+    cgroup: host # required for systemd
+    tty: true # required for logs
+    tmpfs: # required for systemd
+      - /tmp
+      - /run
+      - /run/lock
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+    environment:
+      MAIL_DOMAIN: $MAIL_DOMAIN
+      CHANGE_KERNEL_SETTINGS: "False"
+      ACME_EMAIL: $ACME_EMAIL
+      # RECREATE_VENV: "false"
+      # MAX_MESSAGE_SIZE: "50M"
+      # DEBUG_COMMANDS_ENABLED: "true"
+      # FORCE_REINIT_INI_FILE: "true"
+      # USE_FOREIGN_CERT_MANAGER: "True"
+      # ENABLE_CERTS_MONITORING: "true"
+      # CERTS_MONITORING_TIMEOUT: 10
+      # IS_DEVELOPMENT_INSTANCE: "True"
+    ports:
+      - "80:80"
+      - "443:443"
+      - "25:25"
+      - "587:587"
+      - "143:143"
+      - "465:465"
+      - "993:993"
+    volumes:
+      ## system
+      - /sys/fs/cgroup:/sys/fs/cgroup:rw # required for systemd
+      - ./:/opt/chatmail
+      
+      ## data
+      - ./data/chatmail:/home
+      - ./data/chatmail-dkimkeys:/etc/dkimkeys
+      - ./data/chatmail-echobot:/run/echobot
+      - ./data/chatmail-acme:/var/lib/acme
+
+      ## custom resources
+      # - ./custom/www/src/index.md:/opt/chatmail/www/src/index.md
+
+      ## debug
+      # - ./docker/files/setup_chatmail_docker.sh:/setup_chatmail_docker.sh
+      # - ./docker/files/entrypoint.sh:/entrypoint.sh
+      # - ./docker/files/update_ini.sh:/update_ini.sh

docker/docker-compose-traefik.yaml

@@ -0,0 +1,136 @@
+services:
+  chatmail:
+    build: 
+      context: ./docker
+      dockerfile: chatmail_relay.dockerfile
+      tags:
+        - chatmail-relay:latest
+    image: chatmail-relay:latest
+    restart: unless-stopped
+    container_name: chatmail
+    depends_on:
+      - traefik-certs-dumper
+    cgroup: host # required for systemd
+    tty: true # required for logs
+    tmpfs: # required for systemd
+      - /tmp
+      - /run
+      - /run/lock
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+    environment: #all possible variables you can check inside README and /chatmaild/src/chatmaild/ini/chatmail.ini.f
+      MAIL_DOMAIN: $MAIL_DOMAIN
+      # MAX_MESSAGE_SIZE: "50M"
+      # DEBUG_COMMANDS_ENABLED: "true"
+      # FORCE_REINIT_INI_FILE: "true"
+      # RECREATE_VENV: "false"
+      USE_FOREIGN_CERT_MANAGER: "true"
+      CHANGE_KERNEL_SETTINGS: "false"
+      PATH_TO_SSL: "${CERTS_ROOT_DIR_CONTAINER}/${MAIL_DOMAIN}"
+      ENABLE_CERTS_MONITORING: "true"
+      # CERTS_MONITORING_TIMEOUT: 60
+      # IS_DEVELOPMENT_INSTANCE: "true"
+    ports:
+      - "25:25"
+      - "587:587"
+      - "143:143"
+      - "465:465"
+      - "993:993"
+    volumes:
+      ## system
+      - /sys/fs/cgroup:/sys/fs/cgroup:rw # required for systemd
+      - ./:/opt/chatmail
+      - ${CERTS_ROOT_DIR_HOST}:${CERTS_ROOT_DIR_CONTAINER}:ro
+      
+      ## data
+      - ./data/chatmail:/home
+      # - ./data/chatmail-dkimkeys:/etc/dkimkeys
+      # - ./data/chatmail-echobot:/run/echobot
+      # - ./data/chatmail-acme:/var/lib/acme
+
+      ## custom resources
+      # - ./custom/www/src/index.md:/opt/chatmail/www/src/index.md
+
+      ## debug
+      # - ./docker/files/setup_chatmail_docker.sh:/setup_chatmail_docker.sh
+      # - ./docker/files/entrypoint.sh:/entrypoint.sh
+      # - ./docker/files/update_ini.sh:/update_ini.sh
+
+    labels:
+      - traefik.enable=true
+      - traefik.http.services.chatmail-relay.loadbalancer.server.scheme=https
+      - traefik.http.services.chatmail-relay.loadbalancer.server.port=443
+      - traefik.http.services.chatmail-relay.loadbalancer.serverstransport=insecure@file
+      - traefik.http.routers.chatmail-relay.rule=Host(`${MAIL_DOMAIN}`) || Host(`mta-sts.${MAIL_DOMAIN}`) || Host(`www.${MAIL_DOMAIN}`)
+      - traefik.http.routers.chatmail-relay.service=chatmail-relay
+      - traefik.http.routers.chatmail-relay.tls=true
+      - traefik.http.routers.chatmail-relay.tls.certresolver=letsEncrypt
+
+  traefik_init:
+    image: alpine:latest
+    restart: on-failure
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+    working_dir: /app
+    entrypoint: sh -c '
+      touch acme.json &&
+      chown 0:0 ./acme.json &&
+      chmod 600 ./acme.json'
+    volumes:
+      - ./traefik/data:/app
+
+  traefik:
+    image: traefik:v3.3
+    container_name: traefik
+    restart: unless-stopped
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+    command:
+      - "--configFile=/config.yaml"
+      - "--certificatesresolvers.letsEncrypt.acme.email=${ACME_EMAIL}"
+    # ports:
+    #   - "80:80"
+    #   - "443:443"
+    network_mode: host
+    depends_on:
+      traefik_init:
+        condition: service_completed_successfully
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - ./traefik/config.yaml:/config.yaml
+      - ./traefik/data/acme.json:/acme.json
+      - ./traefik/dynamic-configs:/dynamic/conf
+    
+  traefik-certs-dumper:
+    image: ldez/traefik-certs-dumper:v2.10.0
+    restart: unless-stopped
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+    depends_on:
+      - traefik
+    entrypoint: sh -c '
+      apk add openssl &&
+      while ! [ -e /data/acme.json ]
+      || ! [ `jq ".[] | .Certificates | length" /data/acme.json | jq -s "add" ` != 0 ]; do
+      sleep 1
+      ; done
+      && traefik-certs-dumper file --version v3 --watch --domain-subdir=true
+      --source /data/acme.json --dest /data/letsencrypt/certs --post-hook "sh /post-hook.sh"'
+    environment:
+      CERTS_DIR: /data/letsencrypt/certs
+    volumes:
+      - ./traefik/data/letsencrypt:/data/letsencrypt
+      - ./traefik/data/acme.json:/data/acme.json
+      - ./traefik/post-hook.sh:/post-hook.sh

docker/example.env

@@ -0,0 +1,5 @@
+MAIL_DOMAIN="chat.example.com"
+ACME_EMAIL="my.email@gmail.com"
+
+CERTS_ROOT_DIR_HOST="./traefik/data/letsencrypt/certs"
+CERTS_ROOT_DIR_CONTAINER="/var/lib/acme/live"

docker/files/entrypoint.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+set -eo pipefail
+
+unlink /etc/nginx/sites-enabled/default || true
+
+if [ "${USE_FOREIGN_CERT_MANAGER,,}" == "true" ]; then
+    if [ ! -f "$PATH_TO_SSL/fullchain" ]; then
+        echo "Error: file '$PATH_TO_SSL/fullchain' does not exist. Exiting..." > /dev/stderr
+        sleep 2
+        exit 1
+    fi
+    if [ ! -f "$PATH_TO_SSL/privkey" ]; then
+        echo "Error: file '$PATH_TO_SSL/privkey' does not exist. Exiting..." > /dev/stderr
+        sleep 2
+        exit 1
+    fi
+fi
+
+SETUP_CHATMAIL_SERVICE_PATH="${SETUP_CHATMAIL_SERVICE_PATH:-/lib/systemd/system/setup_chatmail.service}"
+
+env_vars=$(printenv | cut -d= -f1 | xargs)
+sed -i "s|<envs_list>|$env_vars|g" $SETUP_CHATMAIL_SERVICE_PATH
+
+exec /lib/systemd/systemd $@

docker/files/setup_chatmail.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=Run container setup commands
+After=multi-user.target
+ConditionPathExists=/setup_chatmail_docker.sh
+
+[Service]
+Type=oneshot
+ExecStart=/bin/bash /setup_chatmail_docker.sh
+RemainAfterExit=true
+WorkingDirectory=/opt/chatmail
+PassEnvironment=<envs_list>
+
+[Install]
+WantedBy=multi-user.target

docker/files/setup_chatmail_docker.sh

@@ -0,0 +1,78 @@
+#!/bin/bash
+
+set -eo pipefail
+export INI_FILE="${INI_FILE:-chatmail.ini}"
+export ENABLE_CERTS_MONITORING="${ENABLE_CERTS_MONITORING:-true}"
+export CERTS_MONITORING_TIMEOUT="${CERTS_MONITORING_TIMEOUT:-60}"
+export PATH_TO_SSL="${PATH_TO_SSL:-/var/lib/acme/live/${MAIL_DOMAIN}}"
+export CHANGE_KERNEL_SETTINGS=${CHANGE_KERNEL_SETTINGS:-"False"}
+export RECREATE_VENV=${RECREATE_VENV:-"false"}
+
+if [ -z "$MAIL_DOMAIN" ]; then
+    echo "ERROR: Environment variable 'MAIL_DOMAIN' must be set!" >&2
+    exit 1
+fi
+
+debug_commands() {
+    echo "Executing debug commands"
+    # git config --global --add safe.directory /opt/chatmail
+    # ./scripts/initenv.sh
+}
+
+calculate_hash() {
+    find "$PATH_TO_SSL" -type f -exec sha1sum {} \; | sort | sha1sum | awk '{print $1}'
+}
+
+monitor_certificates() {
+    if [ "$ENABLE_CERTS_MONITORING" != "true" ]; then
+        echo "Certs monitoring disabled."
+        exit 0
+    fi
+
+    current_hash=$(calculate_hash)
+    previous_hash=$current_hash
+
+    while true; do
+        current_hash=$(calculate_hash)
+        if [[ "$current_hash" != "$previous_hash" ]]; then
+        # TODO: add an option to restart at a specific time interval 
+            echo "[INFO] Certificate's folder hash was changed, reloading nginx, dovecot and postfix services."
+            systemctl reload nginx.service
+            systemctl reload dovecot.service
+            systemctl reload postfix.service
+            previous_hash=$current_hash
+        fi
+        sleep $CERTS_MONITORING_TIMEOUT
+    done
+}
+
+### MAIN
+
+if [ "$DEBUG_COMMANDS_ENABLED" == "true" ]; then
+    debug_commands
+fi
+
+if [ "$FORCE_REINIT_INI_FILE" == "true" ]; then 
+    INI_CMD_ARGS=--force
+fi
+
+/usr/sbin/opendkim-genkey -D /etc/dkimkeys -d $MAIL_DOMAIN -s opendkim
+chown opendkim:opendkim /etc/dkimkeys/opendkim.private
+chown opendkim:opendkim /etc/dkimkeys/opendkim.txt
+
+# TODO: Move to debug_commands after git clone is moved to dockerfile. 
+git config --global --add safe.directory /opt/chatmail
+if [ "$RECREATE_VENV" == "true" ]; then
+    rm -rf venv
+fi
+./scripts/initenv.sh
+
+./scripts/cmdeploy init --config "${INI_FILE}" $INI_CMD_ARGS $MAIL_DOMAIN
+bash /update_ini.sh
+
+./scripts/cmdeploy run --ssh-host docker
+
+echo "ForwardToConsole=yes" >> /etc/systemd/journald.conf
+systemctl restart systemd-journald
+
+monitor_certificates &

docker/files/update_ini.sh

@@ -0,0 +1,79 @@
+#!/bin/bash
+set -eo pipefail
+
+INI_FILE="${INI_FILE:-chatmail.ini}"
+
+if [ ! -f "$INI_FILE" ]; then
+    echo "Error: file $INI_FILE not found." >&2
+    exit 1
+fi
+
+TMP_FILE="$(mktemp)"
+
+convert_to_bytes() {
+    local value="$1"
+    if [[ "$value" =~ ^([0-9]+)([KkMmGgTt])$ ]]; then
+        local num="${BASH_REMATCH[1]}"
+        local unit="${BASH_REMATCH[2]}"
+        case "$unit" in
+            [Kk]) echo $((num * 1024)) ;;
+            [Mm]) echo $((num * 1024 * 1024)) ;;
+            [Gg]) echo $((num * 1024 * 1024 * 1024)) ;;
+            [Tt]) echo $((num * 1024 * 1024 * 1024 * 1024)) ;;
+        esac
+    elif [[ "$value" =~ ^[0-9]+$ ]]; then
+        echo "$value"
+    else
+        echo "Error: incorrect size format: $value." >&2
+        return 1
+    fi
+}
+
+process_specific_params() {
+    local key=$1
+    local value=$2
+    local destination_file=$3
+
+    if [[ "$key" == "max_message_size" ]]; then
+        converted=$(convert_to_bytes "$value") || exit 1
+        if grep -q -e "## .* = .* bytes" "$destination_file"; then 
+            sed "s|## .* = .* bytes|## $value = $converted bytes|g" "$destination_file";
+        else
+            echo "## $value = $converted bytes"  >> "$destination_file"
+        fi
+        echo "$key = $converted" >> "$destination_file"
+    else
+        echo "$key = $value" >> "$destination_file"
+    fi
+}
+
+while IFS= read -r line; do
+    if [[ "$line" =~ ^[[:space:]]*#.* || "$line" =~ ^[[:space:]]*$ ]]; then
+        echo "$line" >> "$TMP_FILE"
+        continue
+    fi
+
+    if [[ "$line" =~ ^([a-z0-9_]+)[[:space:]]*=[[:space:]]*(.*)$ ]]; then
+        key="${BASH_REMATCH[1]}"
+        current_value="${BASH_REMATCH[2]}"
+        env_var_name=$(echo "$key" | tr 'a-z' 'A-Z')
+        env_value="${!env_var_name}"
+
+        if [[ -n "$env_value" ]]; then
+            process_specific_params "$key" "$env_value" "$TMP_FILE"
+        else
+            echo "$line" >> "$TMP_FILE"
+        fi
+    else
+        echo "$line" >> "$TMP_FILE"
+    fi
+done < "$INI_FILE"
+
+PERMS=$(stat -c %a "$INI_FILE")
+OWNER=$(stat -c %u "$INI_FILE")
+GROUP=$(stat -c %g "$INI_FILE")
+
+chmod "$PERMS" "$TMP_FILE"
+chown "$OWNER":"$GROUP" "$TMP_FILE"
+
+mv "$TMP_FILE" "$INI_FILE"

docs/DOCKER_INSTALLATION_EN.md

@@ -0,0 +1,216 @@
+# Known issues and limitations
+
+- Chatmail will be reinstalled every time the container is started (longer the first time, faster on subsequent starts). This is how the original installer works because it wasn’t designed for Docker. At the end of the documentation, there’s a [proposed solution](#locking-the-chatmail-version).
+- Requires cgroups v2 configured in the system. Operation with cgroups v1 has not been tested.
+- Yes, of course, using systemd inside a container is a hack, and it would be better to split it into several services, but since this is an MVP, it turned out to be easier to do it this way initially than to rewrite the entire deployment system.
+- The Docker image is only suitable for amd64. If you need to run it on a different architecture, try modifying the Dockerfile (specifically the part responsible for installing dovecot).
+
+# Docker installation
+This section provides instructions for installing Chatmail using docker-compose.
+
+## Preliminary setup
+We use `chat.example.org` as the Chatmail domain in the following steps.
+Please substitute it with your own domain.
+
+1. Setup the initial DNS records.
+   The following is an example in the familiar BIND zone file format with
+   a TTL of 1 hour (3600 seconds).
+   Please substitute your domain and IP addresses.
+
+   ```
+    chat.example.com. 3600 IN A 198.51.100.5
+    chat.example.com. 3600 IN AAAA 2001:db8::5
+    www.chat.example.com. 3600 IN CNAME chat.example.com.
+    mta-sts.chat.example.com. 3600 IN CNAME chat.example.com.
+   ```
+
+2. clone the repository on your server.
+
+   ```shell
+    git clone https://github.com/chatmail/relay
+    cd relay
+   ```
+
+## Installation
+When installing via Docker, there are several options:
+
+- Use the built-in nginx and acmetool in Chatmail container to host the chat and manage certificates.
+- Use third-party tools for certificate management.
+
+For the third-party certificate manager example, traefik will be used, but you can use whatever is more convenient for you.
+
+1. Copy the file `./docker/docker-compose-default.yaml` or `./docker/docker-compose-traefik.yaml` and rename it to `docker-compose.yaml`. This is necessary because `docker-compose.yaml` is in `.gitignore` and won’t cause conflicts when updating the git repository.
+
+```shell
+cp ./docker/docker-compose-default.yaml docker-compose.yaml
+## or
+# cp ./docker/docker-compose-traefik.yaml docker-compose.yaml
+```
+
+2. Copy `./docker/example.env` and rename it to `.env`. This file stores variables used in `docker-compose.yaml`.
+
+```shell
+cp ./docker/example.env .env
+```
+
+3. Configure environment variables in the `.env` file. These variables are used in the `docker-compose.yaml` file to pass repeated values.
+
+4. Configure kernel parameters because they cannot be changed inside the container, specifically `fs.inotify.max_user_instances` and `fs.inotify.max_user_watches`. Run the following:
+
+```shell
+echo "fs.inotify.max_user_instances=65536" | sudo tee -a /etc/sysctl.d/99-inotify.conf
+echo "fs.inotify.max_user_watches=65536" | sudo tee -a /etc/sysctl.d/99-inotify.conf
+sudo sysctl --system
+```
+
+5. Configure container environment variables. Below is the list of variables used during deployment:
+
+- `MAIL_DOMAIN` – The domain name of the future server. (required)
+- `DEBUG_COMMANDS_ENABLED` – Run debug commands before installation. (default: `false`)
+- `FORCE_REINIT_INI_FILE` – Recreate the ini configuration file on startup. (default: `false`)
+- `USE_FOREIGN_CERT_MANAGER` – Use a third-party certificate manager. (default: `false`)
+- `RECREATE_VENV` - Recreate the virtual environment (venv). If set to `true`, the environment will be recreated when the container starts, which will increase the startup time of the service but can help avoid certain errors. (default: `false`)
+- `INI_FILE` – Path to the ini configuration file. (default: `./chatmail.ini`)
+- `PATH_TO_SSL` – Path to where the certificates are stored. (default: `/var/lib/acme/live/${MAIL_DOMAIN}`)
+- `ENABLE_CERTS_MONITORING` – Enable certificate monitoring if `USE_FOREIGN_CERT_MANAGER=true`. If certificates change, services will be automatically restarted. (default: `false`)
+- `CERTS_MONITORING_TIMEOUT` – Interval in seconds to check if certificates have changed. (default: `'60'`)
+
+You can also use any variables from the [ini configuration file](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/ini/chatmail.ini.f); they must be in uppercase.
+
+Mandatory variables for deployment via Docker:
+
+- `CHANGE_KERNEL_SETTINGS` – Change kernel settings (`fs.inotify.max_user_instances` and `fs.inotify.max_user_watches`) on startup. Changing kernel settings inside the container is not possible! (default: `False`)
+
+6. Build the Docker image:
+
+```shell
+docker compose build
+```
+
+7. Start docker compose and wait for the installation to finish:
+
+```shell
+docker compose up -d # start service
+docker compose logs -f chatmail # view container logs, press CTRL+C to exit
+```
+
+8. After installation is complete, you can open `https://<your_domain_name>` in your browser.
+
+9. To send messages to other chatmail relays,
+   you need to set additional DNS records.
+   Run `docker exec chatmail scripts/cmdeploy.sh dns --ssh-host localhost`
+   to see recommended DNS records and check whether they are correct.
+
+## Using custom files
+
+When using Docker, you can apply modified configuration files to make the installation more personalized. This is usually needed for the `www/src` section so that the Chatmail landing page is customized to your taste, but it can be used for any other cases as well.
+
+To replace files correctly:
+
+1. Create the `./custom` directory. It is in `.gitignore`, so it won’t cause conflicts when updating.
+
+```shell
+mkdir -p ./custom
+```
+
+2. Modify the required file. For example, `index.md`:
+
+```shell
+mkdir -p ./custom/www/src
+nano ./custom/www/src/index.md
+```
+
+3. In `docker-compose.yaml`, add the file mount in the `volumes` section:
+
+```yaml
+services:
+  chatmail:
+    volumes:
+      ...
+      ## custom resources
+      - ./custom/www/src/index.md:/opt/chatmail/www/src/index.md
+```
+
+4. Restart the service:
+
+```shell
+docker compose down
+docker compose up -d
+```
+
+## Locking the Chatmail version
+
+> [!note]
+> These steps are optional and should only be done if you are not satisfied that the service is installed each time the container starts.
+
+Since the current Docker version installs the Chatmail service every time the container starts, you can lock the container version after installation as follows:
+
+1. Commit the current state of the configured container:
+
+```shell
+docker container commit chatmail configured-chatmail:$(date +'%Y-%m-%d')
+docker image ls | grep configured-chatmail
+```
+
+2. Change the entrypoint for the container in `docker-compose.yaml` to:
+
+```yaml
+services:
+  chatmail:
+    image: <image name from step 1>
+    volumes:
+      ...
+      ## custom resources
+      - ./custom/setup_chatmail_docker.sh:/setup_chatmail_docker.sh
+```
+
+3. Create the file `./custom/setup_chatmail_docker.sh` with the new configuration:
+
+```shell
+mkdir -p ./custom
+cat > ./custom/setup_chatmail_docker.sh << 'EOF'
+#!/bin/bash
+
+set -eo pipefail
+
+export ENABLE_CERTS_MONITORING="${ENABLE_CERTS_MONITORING:-true}"
+export CERTS_MONITORING_TIMEOUT="${CERTS_MONITORING_TIMEOUT:-60}"
+export PATH_TO_SSL="${PATH_TO_SSL:-/var/lib/acme/live/${MAIL_DOMAIN}}"
+
+calculate_hash() {
+    find "$PATH_TO_SSL" -type f -exec sha1sum {} \; | sort | sha1sum | awk '{print $1}'
+}
+
+monitor_certificates() {
+    if [ "$ENABLE_CERTS_MONITORING" != "true" ]; then
+        echo "Certs monitoring disabled."
+        exit 0
+    fi
+
+    current_hash=$(calculate_hash)
+    previous_hash=$current_hash
+
+    while true; do
+        current_hash=$(calculate_hash)
+        if [[ "$current_hash" != "$previous_hash" ]]; then
+        # TODO: add an option to restart at a specific time interval 
+            echo "[INFO] Certificate's folder hash was changed, reloading nginx, dovecot and postfix services."
+            systemctl reload nginx.service
+            systemctl reload dovecot.service
+            systemctl reload postfix.service
+            previous_hash=$current_hash
+        fi
+        sleep $CERTS_MONITORING_TIMEOUT
+    done
+}
+
+monitor_certificates &
+EOF
+```
+
+4. Restart the service:
+
+```shell
+docker compose down
+docker compose up -d
+```

docs/DOCKER_INSTALLATION_RU.md

@@ -0,0 +1,189 @@
+# Известные проблемы и ограничения
+- Chatmail будет переустановлен при каждом запуске контейнера (при первом - долго, при последующих быстрее). Так устроен изначальный установщик, потому что он не был заточен под docker. В конце документации [представлено](#фиксирование-версии-chatmail) возможное решение
+- Требуется настроенный в системе cgroups v2. Работа с cgroups v1 не тестировалась.
+- Да, понятно дело что systemd использовать в контейнере костыль и надо это всё разнести на несколько сервисов, но это MVP и в первом приближении оказалось сделать проще так, чем переписывать всю систему развертывания.
+- docker образ подходит только для amd64, если нужно запустить на другой архитектуре, попробуйте изменить dockerfile (конкретно ту часть что ответсвенна за установку dovecot)
+
+# Docker installation
+Здесь представлена инструкция по установке chatmail с помощью docker-compose.
+
+## Предварительная настройка
+We use `chat.example.org` as the chatmail domain in the following steps.
+Please substitute it with your own domain. 
+
+1. Настройте начальные записи DNS.Ниже приведен пример в привычном формате файла зоны BIND сTTL 1 час (3600 секунд).
+   Замените домен и IP-адреса на свои.
+
+   ```
+    chat.example.com. 3600 IN A 198.51.100.5
+    chat.example.com. 3600 IN AAAA 2001:db8::5
+    www.chat.example.com. 3600 IN CNAME chat.example.com.
+    mta-sts.chat.example.com. 3600 IN CNAME chat.example.com.
+   ```
+
+2. Склонируйте репозиторий на свой сервер.
+
+   ```shell
+    git clone https://github.com/chatmail/relay
+    cd relay
+   ```
+
+## Installation
+При установке через docker есть несколько вариантов:
+- использовать встроенный в chatmail контейнер nginx и acmetool для хостинга чата и управления сертификатами.
+- использовать сторонние инструменты для менеджмента сертификатов
+
+В качестве примера для стороннего менеджера сертификатов будет использоваться traefik, но вы можете использовать то что удобнее вам.
+
+1. Скопировать файл `./docker/docker-compose-default.yaml` или `./docker/docker-compose-traefik.yaml` и переименовать в `docker-compose.yaml`. Это нужно потому что `docker-compose.yaml` находится в `.gitignore` и не будет создавать конфликты при обновлении гит репозитория.
+```shell
+cp ./docker/docker-compose-default.yaml docker-compose.yaml
+## or
+# cp ./docker/docker-compose-traefik.yaml docker-compose.yaml
+```
+
+2. Скопировать `./docker/example.env` и переименовать в `.env`. Здесь хранятся переменные, которые используятся в `docker-compose.yaml`.
+```shell
+cp ./docker/example.env .env
+```
+
+3. Настроить переменные окружения в `.env` файле. Эти переменные используются в `docker-compose.yaml` файле, чтобы передавать повторяющиеся значения.
+
+4. Настроить параметры ядра, потому что внутри контейнера их нельзя изменить, а конкретно `fs.inotify.max_user_instances` и `fs.inotify.max_user_watches`. Для этого выполнить следующее:
+```shell
+echo "fs.inotify.max_user_instances=65536" | sudo tee -a /etc/sysctl.d/99-inotify.conf
+echo "fs.inotify.max_user_watches=65536" | sudo tee -a /etc/sysctl.d/99-inotify.conf
+sudo sysctl --system
+```
+
+5. Настроить переменные окружения контейнера. Ниже перечислен список переменных учавствующих при развертывании.
+- `MAIL_DOMAIN` - Доменное имя будущего сервера. (required)
+- `DEBUG_COMMANDS_ENABLED` - Выполнить debug команды перед установкой. (default: `false`)
+- `FORCE_REINIT_INI_FILE` - Пересоздавать ini файл конфигурации при запуске. (default: `false`)
+- `USE_FOREIGN_CERT_MANAGER` - Использовать сторонний менеджер сертификатов. (default: `false`)
+- `RECREATE_VENV` - Пересоздать виртуальное окружение (venv). Если выставлено `true`, то окружение будет пересоздано при запуске контейнера, из-за чего включение сервиса займет больше времени, но поможет избежать ряда ошибок. (default: `false`)
+- `INI_FILE` - путь к ini файлу конфигурации. (default: `./chatmail.ini`)
+- `PATH_TO_SSL` - Путь где располагаются сертификаты. (default: `/var/lib/acme/live/${MAIL_DOMAIN}`)
+- `ENABLE_CERTS_MONITORING` - Включить мониторинг сертификатов, если `USE_FOREIGN_CERT_MANAGER=true`.  Если сертфикаты изменятся сервисы будут автоматически перезапущены. (default: `false`)
+- `CERTS_MONITORING_TIMEOUT` - Раз во сколько секунд проверять что изменились сертификаты. (default: `'60'`)
+
+Также могут быть использованы все переменные из [ini файла конфигурации](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/ini/chatmail.ini.f), они обязаны быть в uppercase формате. 
+
+Ниже перечислены переменные, которые обязательны быть выставлены при развертывании через docker:
+- `CHANGE_KERNEL_SETTINGS` - Менять настройки ядра (`fs.inotify.max_user_instances` и `fs.inotify.max_user_watches`) при запуске. При запуске в контейнере смена настроек ядра не может быть выполнена! (default: `False`)
+
+6. Собрать docker образ
+```shell
+docker compose build
+```
+
+7. Запустить docker compose и дождаться завершения установки
+```shell
+docker compose up -d # запуск сервиса
+docker compose logs -f chatmail # просмотр логов контейнера. Для выхода нажать CTRL+C
+```
+
+8. По окончанию установки можно открыть в браузер `https://<your_domain_name>`
+
+## Использование кастомных файлов
+При использовании docker есть возможность использовать измененые файлы конфигурации, чтобы сделать установку более персонализированной. Обычно это требуется для секции `www/src`, чтобы ознакомительная страница Chatmail была сделана на ваш вкус. Но также это можно использовать и для любых других случаев.
+
+Для того чтобы корректно выполнить подмену файлов необходимо 
+1. создать каталог `./custom`, он находится в `.gitignore`, поэтому при обновлении не вызовет конфликтов.
+```shell
+mkdir -p ./custom
+```
+
+2. Изменить нужный файл. Для примера возьмем `index.md`
+```shell
+mkdir -p ./custom/www/src
+nano ./custom/www/src/index.md
+```
+
+3. В `docker-compose.yaml` добавить монтирование файла с помощью секции `volumes`
+```yaml
+services:
+  chatmail:
+    volumes:
+      ...
+      ## custom resources
+      - ./custom/www/src/index.md:/opt/chatmail/www/src/index.md
+```
+
+4. Перезапустить сервис
+```shell
+docker compose down
+docker compose up -d
+```
+
+## Фиксирование версии Chatmail
+> [!note]
+> Это опциональные шаги, их делать требуется только если вас не устраивает что сервис устанавливается каждый раз при запуске
+
+Поскольку в текущей версии docker chatmail сервис устанавливается каждый раз запуске контейнера, чтобы этого не происходило можно зафиксировать версию контейнера после установки. Делается это следующим образом:
+
+1. Зафиксировать текущее состояние сконфигурированного контейнера
+```shell
+docker container commit chatmail configured-chatmail:$(date +'%Y-%m-%d')
+docker image ls | grep configured-chatmail
+```
+
+2. Изменить entrypoint для контейнера в `docker-compose.yaml` на
+```yaml
+services:
+  chatmail:
+    image: <image name from step 1>
+    volumes:
+      ...
+      ## custom resources
+      - ./custom/setup_chatmail_docker.sh:/setup_chatmail_docker.sh
+```
+
+3. Создать файл `./custom/setup_chatmail_docker.sh` с новым файлом конфигурации
+```shell
+mkdir -p ./custom
+cat > ./custom/setup_chatmail_docker.sh << 'EOF'
+#!/bin/bash
+
+set -eo pipefail
+
+export ENABLE_CERTS_MONITORING="${ENABLE_CERTS_MONITORING:-true}"
+export CERTS_MONITORING_TIMEOUT="${CERTS_MONITORING_TIMEOUT:-60}"
+export PATH_TO_SSL="${PATH_TO_SSL:-/var/lib/acme/live/${MAIL_DOMAIN}}"
+
+calculate_hash() {
+    find "$PATH_TO_SSL" -type f -exec sha1sum {} \; | sort | sha1sum | awk '{print $1}'
+}
+
+monitor_certificates() {
+    if [ "$ENABLE_CERTS_MONITORING" != "true" ]; then
+        echo "Certs monitoring disabled."
+        exit 0
+    fi
+
+    current_hash=$(calculate_hash)
+    previous_hash=$current_hash
+
+    while true; do
+        current_hash=$(calculate_hash)
+        if [[ "$current_hash" != "$previous_hash" ]]; then
+        # TODO: add an option to restart at a specific time interval 
+            echo "[INFO] Certificate's folder hash was changed, reloading nginx, dovecot and postfix services."
+            systemctl reload nginx.service
+            systemctl reload dovecot.service
+            systemctl reload postfix.service
+            previous_hash=$current_hash
+        fi
+        sleep $CERTS_MONITORING_TIMEOUT
+    done
+}
+
+monitor_certificates &
+EOF
+```
+
+4. Перезапустить сервис
+```shell
+docker compose down
+docker compose up -d
+```

scripts/initenv.sh

@@ -1,5 +1,23 @@
 #!/bin/sh
 set -e
+
+if command -v lsb_release 2>&1 >/dev/null; then
+  case "$(lsb_release -is)" in
+    Ubuntu | Debian )
+      if ! dpkg -l | grep python3-dev 2>&1 >/dev/null
+      then
+        echo "You need to install python3-dev for installing the other dependencies."
+        exit 1
+      fi
+      if ! gcc --version 2>&1 >/dev/null
+      then
+        echo "You need to install gcc for building Python dependencies."
+        exit 1
+      fi
+      ;;
+  esac
+fi
+
 python3 -m venv --upgrade-deps venv
 
 venv/bin/pip install -e chatmaild 

traefik/config.yaml

@@ -0,0 +1,33 @@
+log:
+  level: TRACE
+
+entryPoints:
+  web:
+    address: ":80"
+    http:
+      redirections:
+        entryPoint:
+          to: websecure
+          permanent: true
+  websecure:
+    address: ":443"
+
+providers:
+  docker:
+    endpoint: "unix:///var/run/docker.sock"
+    exposedByDefault: false
+  file:
+    directory: /dynamic/conf
+    watch: true
+
+serverstransport:
+  insecureskipverify: true
+
+certificatesResolvers:
+  letsEncrypt:
+    acme:
+      storage: /acme.json
+      caServer: "https://acme-v02.api.letsencrypt.org/directory"
+      tlschallenge: true
+      httpChallenge:
+        entryPoint: web

traefik/dynamic-configs/insecure.yaml

@@ -0,0 +1,4 @@
+http:
+  serversTransports:
+    insecure:
+      insecureSkipVerify: true

traefik/post-hook.sh

@@ -0,0 +1,15 @@
+CERTS_DIR=${CERTS_DIR:-"/data/letsencrypt/certs"}
+
+echo "CERTS_DIR: $CERTS_DIR"
+
+for dir in "$CERTS_DIR"/*/; do
+    echo "Processing: $dir"
+    cd "$dir"
+    if [ -f "certificate.crt" ]; then
+        ln -sf certificate.crt fullchain
+    fi
+    if [ -f "privatekey.key" ]; then
+        ln -sf privatekey.key privkey
+    fi
+    cd -
+done

www/src/index.md

@@ -1,7 +1,8 @@
-
 <img class="banner" src="collage-top.png"/>
 
-## Dear [Delta Chat](https://get.delta.chat) users and newcomers ... 
+/// tab | 🇬🇧 English
+
+## Dear [Delta Chat](https://get.delta.chat) users and newcomers ...
 
 {% if config.mail_domain != "nine.testrun.org" %}
 Welcome to instant, interoperable and [privacy-preserving](privacy.html) messaging :) 
@@ -23,7 +24,34 @@ you can also **scan this QR code** with Delta Chat:
 🐣 **Choose** your Avatar and Name
 
 💬 **Start** chatting with any Delta Chat contacts using [QR invite codes](https://delta.chat/en/help#howtoe2ee)
+///
+
+/// tab | 🇷🇺 Русский
+
+## Уважаемые пользователи и новички [Delta Chat](https://get.delta.chat)...
 
 {% if config.mail_domain != "nine.testrun.org" %}
+Добро пожаловать в мир мгновенного, совместимого и [конфиденциального](privacy.html) обмена сообщениями :) 
+{% else %}
+Вы находитесь на сервере по умолчанию ({{ config.mail_domain }}) 
+для пользователей Delta Chat. Подробную информацию о том, как он избегает хранения личной информации,
+см. в нашей [политике конфиденциальности](privacy.html). 
+{% endif %}
+
+<a class="cta-button" href="DCACCOUNT:https://{{ config.mail_domain }}/new">Создать чат-профиль на {{config.mail_domain}}</a>
+
+Если вы открыли эту страницу на устройстве,
+где нет приложения Delta Chat, вы можете
+**отсканировать этот QR-код** с помощью Delta Chat:
+
+<a href="DCACCOUNT:https://{{ config.mail_domain }}/new">
+    <img width=300 style="float: none;" src="qr-chatmail-invite-{{config.mail_domain}}.png" /></a>
+
+🐣 **Выберите** аватар и имя
+
+💬 **Начните** чат с любыми контактами Delta Chat через [QR-приглашения](https://delta.chat/ru/help#howtoe2ee)
+///
+
+{% if config.is_development_instance == True %}
 <div class="experimental">Note: this is only a temporary development chatmail service</div>
 {% endif %}

www/src/info.md

@@ -1,3 +1,6 @@
+<img class="banner" src="collage-info.png"/>
+
+/// tab | 🇬🇧 English
 
 ## More information 
 
@@ -41,3 +44,47 @@ This chatmail provider is run by a small voluntary group of devs and sysadmins,
 who [publically develop chatmail provider setups](https://github.com/deltachat/chatmail).
 Chatmail setups aim to be very low-maintenance, resource efficient and 
 interoperable with any other standards-compliant e-mail service. 
+///
+
+/// tab | 🇷🇺 Русский
+
+## Дополнительная информация
+
+{{ config.mail_domain }} предоставляет малозатратный, ресурсосберегающий и совместимый с другими системами почтовый сервис для всех. За `chatmail` фактически скрывается 
+обычный почтовый адрес, как и любой другой, но оптимизированный 
+для использования в чатах, особенно DeltaChat.
+
+### Ограничения по скорости и хранению
+
+* Незашифрованные сообщения блокируются для получателей вне
+  {{config.mail_domain}}, но добавление контакта через [QR-коды приглашения](https://delta.chat/en/help#howtoe2ee)
+  позволяет свободно обмениваться сообщениями между с ним.
+
+* Вы можете отправлять до {{ config.max_user_send_per_minute }} сообщений в минуту.
+
+- Вы можете хранить до [{{ config.max_mailbox_size }} сообщений на сервере](https://delta.chat/en/help#what-happens-if-i-turn-on-delete-old-messages-from-server).
+
+* Сообщения в любом случае будут удалены с сервера через {{ config.delete_mails_after }} дней после поступления на сервер.
+  Или раньше, если хранилище превышает допустимый объем.
+
+### <a name="account-deletion"></a> Удаление аккаунта
+
+Если вы удалите профиль {{ config.mail_domain }} через приложение Delta Chat,
+соответствующая учетная запись на сервере и все связанные с ней данные
+будут автоматически удалены через {{ config.delete_inactive_users_after }} дней.
+
+Если вы используете несколько устройств,
+вам необходимо удалить профиль чата на каждом из них,
+чтобы все данные аккаунта были удалены с сервера.
+
+Если у вас есть дополнительные вопросы или запросы по поводу удаления аккаунта,
+пожалуйста, отправьте сообщение со своей учетной записи на {{ config.privacy_mail }}.
+
+### Кто операторы? Какое ПО используется?
+
+Этот chatmail провайдер управляется небольшой группой добровольцев — разработчиков и системных администраторов,
+которые [публично разрабатывают инфраструктуру chatmail провайдеров](https://github.com/deltachat/chatmail).
+Chatmail стремится быть максимально простыми в обслуживании, ресурсосберегающими и
+совместимыми с любым другим почтовым сервисом, соответствующим стандартам.
+
+///

www/src/main.css

@@ -84,3 +84,57 @@ code {
     color: white !important;
     font-weight: bold;
 }
+
+.tabbed-set {
+  position: relative;
+  display: flex;
+  flex-wrap: wrap;
+  margin: 1em 0;
+  border-radius: 0.1rem;
+}
+
+.tabbed-set > input {
+  display: none;
+}
+
+.tabbed-set label {
+  width: auto;
+  padding: 0.9375em 1.25em 0.78125em;
+  font-weight: 700;
+  font-size: 0.84em;
+  white-space: nowrap;
+  border-bottom: 0.15rem solid transparent;
+  border-top-left-radius: 0.1rem;
+  border-top-right-radius: 0.1rem;
+  cursor: pointer;
+  transition: background-color 250ms, color 250ms;
+}
+
+.tabbed-set .tabbed-content {
+  width: 100%;
+  display: none;
+  box-shadow: 0 -.05rem #ddd;
+}
+
+.tabbed-set input {
+  position: absolute;
+  opacity: 0;
+}
+
+.tabbed-set input:checked:nth-child(n+1) + label {
+  color: red;
+  border-color: red;
+}
+
+@media screen {
+  .tabbed-set input:nth-child(n+1):checked + label + .tabbed-content {
+    order: 99;
+    display: block;
+  }
+}
+
+@media print {
+  .tabbed-content {
+    display: contents;
+  }
+}

www/src/privacy.md

@@ -1,3 +1,6 @@
+<img class="banner" src="collage-privacy.png"/>
+
+/// tab | 🇬🇧 English
 
 # Privacy Policy for {{ config.mail_domain }} 
 
@@ -267,5 +270,199 @@ as of *October 2024*.
 Due to the further development of our service and offers
 or due to changed legal or official requirements,
 it may become necessary to revise this data protection declaration from time to time.
+///
 
+/// tab | 🇷🇺 Русский
 
+# Политика конфиденциальности для {{ config.mail_domain }}
+
+{% if config.mail_domain == "nine.testrun.org" %}
+Добро пожаловать на `{{config.mail_domain}}` — это основной сервер Chatmail для новых пользователей Delta Chat.  
+Он поддерживается небольшой командой системных администраторов на добровольной основе.  
+Альтернативные сервера вы можете найти [здесь](https://delta.chat/en/chatmail).  
+{% endif %}
+
+## Кратко: Личные данные не запрашиваются и не собираются
+
+Этот сервер Chatmail не запрашивает и не сохраняет личную информацию.  
+Серверы Chatmail существуют исключительно для надёжной передачи (временного хранения и доставки) зашифрованных сообщений между устройствами пользователей, использующих мессенджер Delta Chat.  
+
+Технически, Chatmail-сервер можно представить как «маршрутизатор сообщений» с поддержкой сквозного шифрования в масштабе интернета.  
+
+В отличие от классических почтовых сервисов (например, Gmail),  
+Chatmail-серверы не запрашивают личные данные и не хранят письма постоянно.  
+Они ближе по устройству к серверам Signal,  
+однако не используют номера телефонов и могут безопасно и автоматически взаимодействовать как с другими Chatmail-серверами, так и с обычной электронной почтой.  
+
+Отличия от традиционных почтовых серверов:
+
+- безусловное удаление сообщений через {{ config.delete_mails_after }} дней;
+- невозможность отправки незашифрованных сообщений;
+- отсутствие хранения IP-адресов;
+- IP-адреса не обрабатываются в связке с адресами электронной почты.
+
+Из-за отсутствия обработки персональных данных  
+данный сервер, возможно, формально не обязан иметь политику конфиденциальности.
+
+Тем не менее, ниже приведена юридическая информация  
+для удобства специалистов по защите данных и юристов, изучающих работу Chatmail.
+
+---
+
+## 1. Название и контактная информация
+
+Ответственный за обработку ваших персональных данных:
+
+```
+{{ config.privacy_postal }}
+```
+
+Эл. почта: {{ config.privacy_mail }}
+
+Назначен ответственный по защите данных:
+
+```
+{{ config.privacy_pdo }}
+```
+
+---
+
+## 2. Обработка при использовании чата и электронной почты
+
+Мы предоставляем сервисы, оптимизированные для работы с приложением [Delta Chat](https://delta.chat),  
+и обрабатываем только те данные, которые необходимы для настройки и технической реализации доставки сообщений.  
+Цель обработки — дать пользователям возможность читать, писать, управлять, удалять, отправлять и получать сообщения.  
+
+Для этого мы используем серверное ПО, обеспечивающее передачу сообщений.
+
+Обрабатываются следующие данные:
+
+- Исходящие и входящие сообщения (SMTP) временно хранятся до их доставки получателю;
+- Сообщения доступны получателю через IMAP до их удаления пользователем или по истечении установленного срока  
+  (*обычно 4–8 недель*);
+- Протоколы IMAP и SMTP защищены паролем, уникальным для каждого аккаунта;
+- Пользователи могут самостоятельно просматривать или удалять сообщения через любой стандартный IMAP-клиент;
+- Также возможно подключение к «службе передачи в реальном времени»,  
+  которая устанавливает P2P-соединение между устройствами и позволяет отправлять временные сообщения,  
+  которые *никогда* не сохраняются на сервере — даже в зашифрованном виде.
+
+### 2.1 Создание аккаунта
+
+Аккаунт создаётся одним из двух способов:
+
+- с помощью QR-кода приглашения,  
+  отсканированного через приложение Delta Chat;
+
+- автоматически, при создании и регистрации аккаунта в {{ config.mail_domain }} через приложение Delta Chat.
+
+В любом случае, обрабатывается только созданный адрес электронной почты.  
+Номера телефонов, другие адреса электронной почты или любые другие идентификаторы не требуются.  
+Правовое основание для обработки —  
+статья 6 (1) пункт b Общего регламента по защите данных (GDPR),  
+так как вы заключаете пользовательский договор, пользуясь нашим сервисом.
+
+### 2.2 Обработка почтовых сообщений
+
+Кроме того, мы обрабатываем данные,  
+необходимые для обеспечения стабильной работы инфраструктуры сервера,  
+доставки сообщений и предотвращения злоупотреблений.
+
+- Поэтому может потребоваться обработка содержимого и/или метаданных  
+  (например, заголовков писем и технической информации SMTP) во время передачи;
+
+- Мы храним логи передаваемых сообщений ограниченное время —  
+  они используются для устранения проблем с доставкой и ошибок ПО.
+
+Также мы вводим ограничения для защиты системы от перегрузок:
+
+- ограничения скорости (rate limits),
+- лимиты на объём хранения,
+- ограничения на размер сообщений,
+- любые другие меры, необходимые для стабильной работы сервера и предотвращения злоупотреблений.
+
+Обработка вышеуказанных данных необходима для предоставления сервиса.  
+Правовое основание — статья 6 (1) пункт b GDPR.  
+Обработка данных в целях безопасности и предотвращения злоупотреблений основана на статье 6 (1) пункт f GDPR,  
+и соответствует нашим законным интересам.
+
+Мы не используем собранные данные для определения вашей личности.
+
+---
+
+## 3. Обработка при посещении сайта
+
+При посещении нашего сайта браузер вашего устройства  
+автоматически передаёт определённую информацию на сервер,  
+где она временно сохраняется в так называемых лог-файлах.  
+Эти данные автоматически удаляются (обычно через *7 дней*).
+
+Среди собираемых данных:
+
+- тип используемого браузера,
+- операционная система,
+- дата и время доступа,
+- страна и IP-адрес,
+- запрашиваемый файл или ресурс,
+- объём переданных данных,
+- статус доступа (успешно, ошибка и т.п.),
+- страница, с которой был сделан запрос.
+
+Хостинг нашего сайта осуществляется внешним провайдером.  
+Личные данные, собираемые на сайте, хранятся на его серверах.  
+Провайдер обрабатывает данные строго по нашим инструкциям,  
+в пределах заключённого договора на обработку данных (ст. 28 GDPR).
+
+Цели обработки:
+
+- обеспечение стабильного подключения к сайту;
+- удобство использования сайта;
+- контроль безопасности и стабильности системы;
+- административные цели.
+
+Правовое основание — статья 6 (1) пункт f GDPR.  
+Собранные данные не используются для установления вашей личности.
+
+---
+
+## 4. Передача данных
+
+Мы не сохраняем личные данные,  
+но письма, ожидающие доставки, могут содержать личную информацию.  
+Такие данные не передаются третьим лицам, за исключением следующих случаев:
+
+a) при наличии вашего явного согласия (ст. 6 п.1 п. a GDPR);  
+
+b) если передача необходима для защиты прав, интересов или правовой позиции (ст. 6 п.1 п. f GDPR);  
+
+c) если это требуется по закону (ст. 6 п.1 п. c GDPR);  
+
+d) если это необходимо для исполнения договора с вами (ст. 6 п.1 п. b GDPR);  
+
+e) если обработка осуществляется сервис-провайдером по нашему поручению,  
+   с которым заключён договор (ст. 28 GDPR),  
+   предусматривающий меры безопасности и контроль с нашей стороны.
+
+---
+
+## 5. Права субъектов данных
+
+Ваши права закреплены в статьях 12–23 GDPR.  
+Так как сервер не хранит персональные данные — даже в зашифрованном виде —  
+предоставление информации или подача возражений не требуются.  
+Удаление данных можно выполнить напрямую через приложение Delta Chat.
+
+Если у вас есть вопросы или жалобы, напишите нам:  
+{{ config.privacy_mail }}
+
+Также вы можете обратиться в надзорный орган по месту вашего проживания,  
+работы или к органу, ответственному за нашу деятельность:  
+`{{ config.privacy_supervisor }}`.
+
+---
+
+## 6. Актуальность политики конфиденциальности
+
+Настоящая политика действует с *октября 2024 года*.  
+В случае изменений в услугах или законодательства  
+она может быть обновлена.
+///