increase number of login connections

Is this function even doing anything? If so reject PR. I'm still
trying to understand the code.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -12,6 +12,7 @@ Please fill out as much of this form as you can (leaving out stuff that is not a
 
 - Server OS (Operating System) - preferably Debian 12: 
 - On which OS you run cmdeploy:
+- chatmail/relay version: `git rev-parse HEAD`
 
 ## Expected behavior
 

CHANGELOG.md

@@ -2,6 +2,43 @@
 
 ## untagged
 
+- Increase dovecot imap-login limits
+  ([#577](https://github.com/chatmail/relay/pull/577))
+
+- Increase nginx connection limits
+  ([#576](https://github.com/chatmail/relay/pull/576))
+
+- If `dns-utils` needs to be installed before cmdeploy run, apt update to make sure it works
+  ([#560](https://github.com/chatmail/relay/pull/560))
+
+- filtermail: respect config message size limit
+  ([#572](https://github.com/chatmail/relay/pull/572))
+
+- Add config value after how many days large files are deleted
+  ([#555](https://github.com/chatmail/relay/pull/555))
+
+- cmdeploy: push relay version to /etc/chatmail-version
+  ([#573](https://github.com/chatmail/relay/pull/573))
+
+- filtermail: allow partial body length in OpenPGP payloads
+  ([#570](https://github.com/chatmail/relay/pull/570))
+
+- chatmaild: allow echobot to receive unencrypted messages by default
+  ([#556](https://github.com/chatmail/relay/pull/556))
+
+
+## 1.6.0 2025-04-11
+
+- Handle Port-25 connect errors more gracefully (common with VPNs)
+  ([#552](https://github.com/chatmail/relay/pull/552))
+
+- Avoid "acmetool not found" during initial run
+  ([#550](https://github.com/chatmail/relay/pull/550))
+
+- Fix timezone handling such that client/servers do not need to use
+  same timezone. 
+  ([#553](https://github.com/chatmail/relay/pull/553))
+
 - Enforce end-to-end encryption for incoming messages. 
   New user address mailboxes now get a `enforceE2EEincoming` file 
   which prohibits incoming cleartext messages from other domains. 
@@ -14,6 +51,12 @@
 - Enforce end-to-end encryption between local addresses 
   ([#535](https://github.com/chatmail/server/pull/535))
 
+- unbound: check that port 53 is not occupied by a different process
+  ([#537](https://github.com/chatmail/server/pull/537))
+
+- unbound: before unbound is there, use 9.9.9.9 for resolving
+  ([#518](https://github.com/chatmail/relay/pull/518))
+
 - Limit the bind for the HTTPS server on 8443 to 127.0.0.1 
   ([#522](https://github.com/chatmail/server/pull/522))
   ([#532](https://github.com/chatmail/server/pull/532))
@@ -21,6 +64,9 @@
 - Send SNI when connecting to outside servers
   ([#524](https://github.com/chatmail/server/pull/524))
 
+- postfix master.cf: use 127.0.0.1 for consistency
+  ([#544](https://github.com/chatmail/relay/pull/544))
+
 - Pass through `original_content` instead of `content` in filtermail
   ([#509](https://github.com/chatmail/server/pull/509))
 

README.md

@@ -159,7 +159,7 @@ This repository has four directories:
 The `cmdeploy/src/cmdeploy/cmdeploy.py` command line tool
 helps with setting up and managing the chatmail service.
 `cmdeploy init` creates the `chatmail.ini` config file.
-`cmdeploy run` uses a [pyinfra](https://pyinfra.com/)-based [script](`cmdeploy/src/cmdeploy/__init__.py`)
+`cmdeploy run` uses a [pyinfra](https://pyinfra.com/)-based [`script`](cmdeploy/src/cmdeploy/__init__.py)
 to automatically install or upgrade all chatmail components on a relay,
 according to the `chatmail.ini` config.
 
@@ -533,3 +533,12 @@ Then reboot the relay or do `sysctl -p` and `nft -f /etc/nftables.conf`.
 
 Once proxy relay is set up,
 you can add its IP address to the DNS.
+
+## Neighbors and Acquaintances
+
+Here are some related projects that you may be interested in:
+
+- [Mox](https://github.com/mjl-/mox): A Golang email server.  [Work is in
+  progress](https://github.com/mjl-/mox/issues/251) to modify it to support all
+  of the features and configuration settings required to operate as a chatmail
+  relay.

chatmaild/src/chatmaild/config.py

@@ -26,6 +26,7 @@ class Config:
         self.max_mailbox_size = params["max_mailbox_size"]
         self.max_message_size = int(params.get("max_message_size", "31457280"))
         self.delete_mails_after = params["delete_mails_after"]
+        self.delete_large_after = params["delete_large_after"]
         self.delete_inactive_users_after = int(params["delete_inactive_users_after"])
         self.username_min_length = int(params["username_min_length"])
         self.username_max_length = int(params["username_max_length"])
@@ -64,7 +65,7 @@ class Config:
     def _getbytefile(self):
         return open(self._inipath, "rb")
 
-    def get_user(self, addr):
+    def get_user(self, addr) -> User:
         if not addr or "@" not in addr or "/" in addr:
             raise ValueError(f"invalid address {addr!r}")
 
@@ -115,7 +116,7 @@ def get_default_config_content(mail_domain, **overrides):
         lines = []
         for line in content.split("\n"):
             for key, value in privacy.items():
-                value_lines = value.strip().split("\n")
+                value_lines = value.format(mail_domain=mail_domain).strip().split("\n")
                 if not line.startswith(f"{key} =") or not value_lines:
                     continue
                 if len(value_lines) == 1:

chatmaild/src/chatmaild/filtermail.py

@@ -38,6 +38,12 @@ def check_openpgp_payload(payload: bytes):
 
         packet_type_id = payload[i] & 0x3F
         i += 1
+
+        while payload[i] >= 224 and payload[i] < 255:
+            # Partial body length.
+            partial_length = 1 << (payload[i] & 0x1F)
+            i += 1 + partial_length
+
         if payload[i] < 192:
             # One-octet length.
             body_len = payload[i]
@@ -56,7 +62,7 @@ def check_openpgp_payload(payload: bytes):
             )
             i += 5
         else:
-            # Partial body length is not allowed.
+            # Impossible, partial body length was processed above.
             return False
 
         i += body_len
@@ -167,7 +173,12 @@ async def asyncmain_beforequeue(config, mode):
     else:
         port = config.filtermail_smtp_port_incoming
         handler = IncomingBeforeQueueHandler(config)
-    HackedController(handler, hostname="127.0.0.1", port=port).start()
+    HackedController(
+        handler,
+        hostname="127.0.0.1",
+        port=port,
+        data_size_limit=config.max_message_size,
+    ).start()
 
 
 def recipient_matches_passthrough(recipient, passthrough_recipients):

chatmaild/src/chatmaild/ini/chatmail.ini.f

@@ -23,6 +23,9 @@ max_message_size = 31457280
 # days after which mails are unconditionally deleted
 delete_mails_after = 20
 
+# days after which large messages (>200k) are unconditionally deleted
+delete_large_after = 7
+
 # days after which users without a successful login are deleted (database and mails)
 delete_inactive_users_after = 90
 
@@ -40,7 +43,7 @@ passthrough_senders =
 
 # list of e-mail recipients for which to accept outbound un-encrypted mails
 # (space-separated, item may start with "@" to whitelist whole recipient domains)
-passthrough_recipients = xstore@testrun.org 
+passthrough_recipients = xstore@testrun.org echo@{mail_domain}
 
 #
 # Deployment Details

chatmaild/src/chatmaild/ini/override-testrun.ini

@@ -1,7 +1,7 @@
 
 [privacy]
 
-passthrough_recipients = privacy@testrun.org xstore@testrun.org
+passthrough_recipients = privacy@testrun.org xstore@testrun.org echo@{mail_domain}
 
 privacy_postal =
     Merlinux GmbH, Represented by the managing director H. Krekel,

chatmaild/src/chatmaild/notifier.py

@@ -95,7 +95,12 @@ class Notifier:
                 logging.warning(f"removing spurious queue item: {queue_path!r}")
                 queue_path.unlink()
                 continue
-            queue_item = PersistentQueueItem.read_from_path(queue_path)
+            try:
+                queue_item = PersistentQueueItem.read_from_path(queue_path)
+            except ValueError:
+                logging.warning(f"removing spurious queue item: {queue_path!r}")
+                queue_path.unlink()
+                continue
             self.queue_for_retry(queue_item)
 
     def queue_for_retry(self, queue_item, retry_num=0):

chatmaild/src/chatmaild/tests/test_config.py

@@ -35,6 +35,7 @@ def test_read_config_testrun(make_config):
     assert config.max_user_send_per_minute == 60
     assert config.max_mailbox_size == "100M"
     assert config.delete_mails_after == "20"
+    assert config.delete_large_after == "7"
     assert config.username_min_length == 9
     assert config.username_max_length == 9
     assert config.password_min_length == 9

chatmaild/src/chatmaild/tests/test_filtermail.py

@@ -304,3 +304,45 @@ HELLOWORLD
 \r
 """
     assert check_armored_payload(payload) == False
+
+    # Test payload using partial body length
+    # as generated by GopenPGP.
+    payload = """-----BEGIN PGP MESSAGE-----\r
+\r
+wV4DdCVjRfOT3TQSAQdAY5+pjT6mlCxPGdR3be4w7oJJRUGIPI/Vnh+mJxGSm34w\r
+LNlVc89S1g22uQYFif2sUJsQWbpoHpNkuWpkSgOaHmNvrZiY/YU5iv+cZ3LbmtUG\r
+0uoBisSHh9O1c+5sYZSbrvYZ1NOwlD7Fv/U5/Mw4E5+CjxfdgNGp5o3DDddzPK78\r
+jseDhdSXxnaiIJC93hxNX6R1RPt3G2gukyzx69wciPQShcF8zf3W3o75Ed7B8etV\r
+QEeB16xzdFhKa9JxdjTu3osgCs21IO7wpcFkjc7nZzlW6jPnELJJaNmv4yOOCjMp\r
+6YAkaN/BkL+jHTznHDuDsT5ilnTXpwHDU1Cm9PIx/KFcNCQnIB+2DcdIHPHUH1ci\r
+jvqoeXAVWjKXEjS7PqPFuP/xGbrWG2ugs+toXJOKbgRkExvKs1dwPFKrgghvCVbW\r
+AcKejQKAPArLwpkA7aD875TZQShvGt74fNs45XBlGOYOnNOAJ1KAmzrXLIDViyyB\r
+kDsmTBk785xofuCkjBpXSe6vsMprPzCteDfaUibh8FHeJjucxPerwuOPEmnogNaf\r
+YyL4+iy8H8I9/p7pmUqILprxTG0jTOtlk0bTVzeiF56W1xbtSEMuOo4oFbQTyOM2\r
+bKXaYo774Jm+rRtKAnnI2dtf9RpK19cog6YNzfYjesLKbXDsPZbN5rmwyFiCvvxC\r
+kQ6JLob+B2fPdY2gzy7LypxktS8Zi1HJcWDHJGVmQodaDLqKUObb4M26bXDe6oxI\r
+NS8PJz5exVbM3KhZnUOEn6PJRBBf5a/ZqxlhZPcQo/oBuhKpBRpO5kSDwPIUByu3\r
+UlXLSkpMqe9pUarAOEuQjfl2RVY7U+RrQYp4YP5keMO+i8NCefAFbowTTufO1JIq\r
+2nVgCi/QVnxZyEc9OYt/8AE3g4cdojE+vsSDifZLSWYIetpfrohHv3dT3StD1QRG\r
+0QE6qq6oKpg/IL0cjvuX4c7a7bslv2fXp8t75y37RU6253qdIebhxc/cRhPbc/yu\r
+p0YLyD4SrvKTLP2ZV95jT4IPEpqm4AN3QmiOzdtqR2gLyb62L8QfqI/FdwsIiRiM\r
+hqydwoqt/lfSqG1WKPh+6EkMkH+TDiCC1BQdbN1MNcyUtcjb35PR2c8Ld2TF3guA\r
+jLIqMt/Vb7hBoMb2FcsOYY25ka9oV62OwgKWLXnFzk+modMR5fzb4kxVVAYEqP+D\r
+T5KO1Vs76v1fyPGOq6BbBCvLwTqe/e6IZInJles4v5jrhnLcGKmNGivCUDe6X6NY\r
+UKNt5RsZllwDQpaAb5dMNhyrk8SgIE7TBI7rvqIdUCE52Vy+0JDxFg5olRpFUfO6\r
+/MyTW3Yo/ekk/npHr7iYYqJTCc21bDGLWQcIo/XO7WPxrKNWGBNPFnkRdw0MaKr4\r
++cEM3V8NFnSEpC12xA+RX/CezuJtwXZK5MpG76eYqMO6qyC+c25YcFecEufDZDxx\r
+ZLqRszVRyxyWPtk/oIeQK2v9wOqY6N9/ff01gHz69vqYqN5bUw/QKZsmx1zW+gPw\r
+6x2tDK2BHeYl182gCbhlKISRFwCtbjqZSkiKWao/VtygHkw0fK34avJuyQ/X9YaN\r
+BRy+7Lf3VA53pnB5WJ1xwRXN8VDvmZeXzv2krHveCMemj0OjnRoCLu117xN0A5m9\r
+Fm/RoDix5PolDHtWTtr2m1n2hp2LHnj8at9lFEd0SKhAYHVL9KjzycwWODZRXt+x\r
+zGDDuooEeTvdY5NLyKcl4gETz1ZP4Ez5jGGjhPSwSpq1mU7UaJ9ZXXdr4KHyifW6\r
+ggNzNsGhXTap7IWZpTtqXABydfiBshmH2NjqtNDwBweJVSgP10+r0WhMWlaZs6xl\r
+V3o5yskJt6GlkwpJxZrTvN6Tiww/eW7HFV6NGf7IRSWY5tJc/iA7/92tOmkdvJ1q\r
+myLbG7cJB787QjplEyVe2P/JBO6xYvbkJLf9Q+HaviTO25rugRSrYsoKMDfO8VlQ\r
+1CcnTPVtApPZJEQzAWJEgVAM8uIlkqWJJMgyWT34sTkdBeCUFGloXQFs9Yxd0AGf\r
+/zHEkYZSTKpVSvAIGu4=\r
+=6iHb\r
+-----END PGP MESSAGE-----\r
+"""
+    assert check_armored_payload(payload) == True

chatmaild/src/chatmaild/tests/test_metadata.py

@@ -242,6 +242,22 @@ def test_requeue_removes_tmp_files(notifier, metadata, testaddr, caplog):
     assert queue_item.addr == testaddr
 
 
+def test_requeue_removes_invalid_files(notifier, metadata, testaddr, caplog):
+    metadata.add_token_to_addr(testaddr, "01234")
+    notifier.new_message_for_addr(testaddr, metadata)
+    # empty/invalid files should be ignored
+    p = notifier.queue_dir.joinpath("1203981203")
+    p.touch()
+    notifier2 = notifier.__class__(notifier.queue_dir)
+    notifier2.requeue_persistent_queue_items()
+    assert "spurious" in caplog.records[0].msg
+    assert not p.exists()
+    assert notifier2.retry_queues[0].qsize() == 1
+    when, queue_item = notifier2.retry_queues[0].get()
+    assert when <= int(time.time())
+    assert queue_item.addr == testaddr
+
+
 def test_start_and_stop_notification_threads(notifier, testaddr):
     threads = notifier.start_notification_threads(None)
     for retry_num, threadlist in threads.items():

chatmaild/src/chatmaild/user.py

@@ -58,7 +58,8 @@ class User:
             if not self.addr.startswith("echo@"):
                 logging.error(f"could not write password for: {self.addr}")
                 raise
-        self.enforce_E2EE_path.touch()
+        if not self.addr.startswith("echo@"):
+            self.enforce_E2EE_path.touch()
 
     def set_last_login_timestamp(self, timestamp):
         """Track login time with daily granularity

cmdeploy/src/cmdeploy/__init__.py

@@ -7,17 +7,35 @@ import io
 import shutil
 import subprocess
 import sys
+from io import StringIO
 from pathlib import Path
 
 from chatmaild.config import Config, read_config
 from pyinfra import facts, host
+from pyinfra.api import FactBase
 from pyinfra.facts.files import File
+from pyinfra.facts.server import Sysctl
 from pyinfra.facts.systemd import SystemdEnabled
 from pyinfra.operations import apt, files, pip, server, systemd
 
 from .acmetool import deploy_acmetool
 
 
+class Port(FactBase):
+    """
+    Returns the process occuping a port.
+    """
+
+    def command(self, port: int) -> str:
+        return (
+            "ss -lptn 'src :%d' | awk 'NR>1 {print $6,$7}' | sed 's/users:((\"//;s/\".*//'"
+            % (port,)
+        )
+
+    def process(self, output: [str]) -> str:
+        return output[0]
+
+
 def _build_chatmaild(dist_dir) -> None:
     dist_dir = Path(dist_dir).resolve()
     if dist_dir.exists():
@@ -230,7 +248,6 @@ def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
     )
     need_restart |= service_file.changed
 
-
     return need_restart
 
 
@@ -348,6 +365,10 @@ def _configure_dovecot(config: Config, debug: bool = False) -> bool:
     # it is recommended to set the following inotify limits
     for name in ("max_user_instances", "max_user_watches"):
         key = f"fs.inotify.{name}"
+        if host.get_fact(Sysctl)[key] > 65535:
+            # Skip updating limits if already sufficient
+            # (enables running in incus containers where sysctl readonly)
+            continue
         server.sysctl(
             name=f"Change {key}",
             key=key,
@@ -577,6 +598,12 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         ensure_newline=True,
     )
 
+    if host.get_fact(Port, port=53) != "unbound":
+        files.line(
+            name="Add 9.9.9.9 to resolv.conf",
+            path="/etc/resolv.conf",
+            line="nameserver 9.9.9.9",
+        )
     apt.update(name="apt update", cache_time=24 * 3600)
     apt.upgrade(name="upgrade apt packages", auto_remove=True)
 
@@ -588,6 +615,12 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
     # Run local DNS resolver `unbound`.
     # `resolvconf` takes care of setting up /etc/resolv.conf
     # to use 127.0.0.1 as the resolver.
+    from cmdeploy.cmdeploy import Out
+
+    process_on_53 = host.get_fact(Port, port=53)
+    if process_on_53 not in (None, "unbound"):
+        Out().red(f"Can't install unbound: port 53 is occupied by: {process_on_53}")
+        exit(1)
     apt.packages(
         name="Install unbound",
         packages=["unbound", "unbound-anchor", "dnsutils"],
@@ -725,5 +758,13 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         name="Ensure cron is installed",
         packages=["cron"],
     )
+    git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode()
+    git_diff = subprocess.check_output(["git", "diff"]).decode()
+    files.put(
+        name="Upload chatmail relay git commiit hash",
+        src=StringIO(git_hash + git_diff),
+        dest="/etc/chatmail-version",
+        mode="700",
+    )
 
     deploy_mtail(config)

cmdeploy/src/cmdeploy/cmdeploy.py

@@ -86,15 +86,19 @@ def run_cmd(args, out):
         out.red("Please re-run scripts/initenv.sh to update pyinfra to version 3.")
         return 1
 
-    retcode = out.check_call(cmd, env=env)
-    if retcode == 0:
-        out.green("Deploy completed, call `cmdeploy dns` next.")
-    elif not remote_data["acme_account_url"]:
-        out.red("Deploy completed but letsencrypt not configured")
-        out.red("Run 'cmdeploy run' again")
-        retcode = 0
-    else:
+    try:
+        retcode = out.check_call(cmd, env=env)
+        if retcode == 0:
+            out.green("Deploy completed, call `cmdeploy dns` next.")
+        elif not remote_data["acme_account_url"]:
+            out.red("Deploy completed but letsencrypt not configured")
+            out.red("Run 'cmdeploy run' again")
+            retcode = 0
+        else:
+            out.red("Deploy failed")
+    except subprocess.CalledProcessError:
         out.red("Deploy failed")
+        retcode = 1
     return retcode
 
 

cmdeploy/src/cmdeploy/dovecot/dovecot.conf.j2

@@ -177,17 +177,20 @@ service auth-worker {
 }
 
 service imap-login {
-    # High-security mode.
-    # Each process serves a single connection and exits afterwards.
-    # This is the default, but we set it explicitly to be sure.
+    # We are not using "high-security-mode" because even if dovecot
+    # login would be compromised, there are only E2EE messages to be
+    # found or sent, and an attacker doesn't have the key used with this address. 
     # See <https://doc.dovecot.org/admin_manual/login_processes/#high-security-mode> for details.
-    service_count = 1
+    
+    # Performance Mode. Each process serves up to 100 connections and exits afterwards. 
+    service_count = 100
 
     # Inrease the number of simultaneous connections.
     #
     # As of Dovecot 2.3.19.1 the default is 100 processes.
     # Combined with `service_count = 1` it means only 100 connections
     # can be handled simultaneously.
+    # We allow up to 5000 * 100 = 500K connections 
     process_limit = 10000
 
     # Avoid startup latency for new connections.

cmdeploy/src/cmdeploy/dovecot/expunge.cron.j2

@@ -1,5 +1,5 @@
 # delete already seen big mails after 7 days, in the INBOX
-2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +7 -size +200k -type f -delete
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +{{ config.delete_large_after }} -size +200k -type f -delete
 # delete all mails after {{ config.delete_mails_after }} days, in the Inbox
 2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
 # or in any IMAP subfolder

cmdeploy/src/cmdeploy/dovecot/push_notification.lua

@@ -2,15 +2,6 @@ function dovecot_lua_notify_begin_txn(user)
   return user
 end
 
-function contains(v, needle)
-  for _, keyword in ipairs(v) do
-    if keyword == needle then
-      return true
-    end
-  end
-  return false
-end
-
 function dovecot_lua_notify_event_message_new(user, event)
   local mbox = user:mailbox(event.mailbox)
   mbox:sync()

cmdeploy/src/cmdeploy/nginx/nginx.conf.j2

@@ -2,11 +2,25 @@ load_module modules/ngx_stream_module.so;
 
 user www-data;
 worker_processes auto;
+
+# Increase the number of connections
+# that a worker process can open
+# to avoid errors such as
+#   accept4() failed (24: Too many open files)
+# and
+#   socket() failed (24: Too many open files) while connecting to upstream
+# in the logs.
+# <https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile>
+worker_rlimit_nofile 2048;
 pid /run/nginx.pid;
 error_log syslog:server=unix:/dev/log,facility=local3;
 
 events {
-	worker_connections 768;
+        # Increase to avoid errors such as
+        #   768 worker_connections are not enough while connecting to upstream
+        # in the logs.
+        # <https://nginx.org/en/docs/ngx_core_module.html#worker_connections>
+	worker_connections 2048;
 	# multi_accept on;
 }
 

cmdeploy/src/cmdeploy/postfix/master.cf.j2

@@ -77,13 +77,13 @@ scache    unix  -       -       y       -       1       scache
 postlog   unix-dgram n  -       n       -       1       postlogd
 filter    unix -        n       n       -       -       lmtp
 # Local SMTP server for reinjecting outgoing filtered mail.
-localhost:{{ config.postfix_reinject_port }} inet  n       -       n       -       10      smtpd
+127.0.0.1:{{ config.postfix_reinject_port }} inet  n       -       n       -       10      smtpd
   -o syslog_name=postfix/reinject
   -o smtpd_milters=unix:opendkim/opendkim.sock
   -o cleanup_service_name=authclean
 
 # Local SMTP server for reinjecting incoming filtered mail 
-localhost:{{ config.postfix_reinject_port_incoming }} inet  n       -       n       -       10      smtpd
+127.0.0.1:{{ config.postfix_reinject_port_incoming }} inet  n       -       n       -       10      smtpd
   -o syslog_name=postfix/reinject_incoming
   -o smtpd_milters=unix:opendkim/opendkim.sock
 

cmdeploy/src/cmdeploy/remote/rdns.py

@@ -19,7 +19,7 @@ def perform_initial_checks(mail_domain):
     """Collecting initial DNS settings."""
     assert mail_domain
     if not shell("dig", fail_ok=True):
-        shell("apt-get install -y dnsutils")
+        shell("apt-get update && apt-get install -y dnsutils")
     A = query_dns("A", mail_domain)
     AAAA = query_dns("AAAA", mail_domain)
     MTA_STS = query_dns("CNAME", f"mta-sts.{mail_domain}")

cmdeploy/src/cmdeploy/remote/rshell.py

@@ -1,10 +1,13 @@
-from subprocess import CalledProcessError, check_output
+from subprocess import DEVNULL, CalledProcessError, check_output
 
 
 def shell(command, fail_ok=False):
     print(f"$ {command}")
+    args = dict(shell=True)
+    if fail_ok:
+        args["stderr"] = DEVNULL
     try:
-        return check_output(command, shell=True).decode().rstrip()
+        return check_output(command, **args).decode().rstrip()
     except CalledProcessError:
         if not fail_ok:
             raise

cmdeploy/src/cmdeploy/tests/online/test_0_login.py

@@ -90,8 +90,13 @@ def test_concurrent_logins_same_account(
 
 
 def test_no_vrfy(chatmail_config):
+    domain = chatmail_config.mail_domain
     sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    sock.connect((chatmail_config.mail_domain, 25))
+    sock.settimeout(10)
+    try:
+        sock.connect((domain, 25))
+    except socket.timeout:
+        pytest.skip(f"port 25 not reachable for {domain}")
     banner = sock.recv(1024)
     print(banner)
     sock.send(b"VRFY wrongaddress@%s\r\n" % (chatmail_config.mail_domain.encode(),))

cmdeploy/src/cmdeploy/tests/online/test_1_basic.py

@@ -1,5 +1,6 @@
 import datetime
 import smtplib
+import subprocess
 
 import pytest
 
@@ -55,11 +56,12 @@ class TestSSHExecutor:
 
     def test_opendkim_restarted(self, sshexec):
         """check that opendkim is not running for longer than a day."""
-        out = sshexec(call=remote.rshell.shell, kwargs=dict(command="systemctl status opendkim"))
-        assert type(out) == str
-        since_date_str = out.split("since ")[1].split(";")[0]
-        since_date = datetime.datetime.strptime(since_date_str, "%a %Y-%m-%d %H:%M:%S %Z")
-        assert (datetime.datetime.now() - since_date).total_seconds() < 60 * 60 * 24
+        cmd = "systemctl show opendkim --timestamp=utc --property=ActiveEnterTimestamp"
+        out = sshexec(call=remote.rshell.shell, kwargs=dict(command=cmd))
+        datestring = out.split("=")[1]
+        since_date = datetime.datetime.strptime(datestring, "%a %Y-%m-%d %H:%M:%S %Z")
+        now = datetime.datetime.now(since_date.tzinfo)
+        assert (now - since_date).total_seconds() < 60 * 60 * 51
 
 
 def test_remote(remote, imap_or_smtp):
@@ -117,8 +119,15 @@ def test_authenticated_from(cmsetup, maildata):
 @pytest.mark.parametrize("from_addr", ["fake@example.org", "fake@testrun.org"])
 def test_reject_missing_dkim(cmsetup, maildata, from_addr):
     recipient = cmsetup.gen_users(1)[0]
-    msg = maildata("encrypted.eml", from_addr=from_addr, to_addr=recipient.addr).as_string()
-    with smtplib.SMTP(cmsetup.maildomain, 25) as s:
+    msg = maildata(
+        "encrypted.eml", from_addr=from_addr, to_addr=recipient.addr
+    ).as_string()
+    try:
+        conn = smtplib.SMTP(cmsetup.maildomain, 25, timeout=10)
+    except TimeoutError:
+        pytest.skip(f"port 25 not reachable for {cmsetup.maildomain}")
+
+    with conn as s:
         with pytest.raises(smtplib.SMTPDataError, match="No valid DKIM signature"):
             s.sendmail(from_addr=from_addr, to_addrs=recipient.addr, msg=msg)
 
@@ -176,6 +185,25 @@ def test_expunged(remote, chatmail_config):
         f"find {chatmail_config.mailboxes_dir} -path '*/tmp/*' -mtime +{outdated_days} -type f",
         f"find {chatmail_config.mailboxes_dir} -path '*/.*/tmp/*' -mtime +{outdated_days} -type f",
     ]
+    outdated_days = int(chatmail_config.delete_large_after) + 1
+    find_cmds.append(
+        "find {chatmail_config.mailboxes_dir} -path '*/cur/*' -mtime +{outdated_days} -size +200k -type f"
+    )
     for cmd in find_cmds:
         for line in remote.iter_output(cmd):
             assert not line
+
+
+def test_deployed_state(remote):
+    git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode()
+    git_diff = subprocess.check_output(["git", "diff"]).decode()
+    git_status = [git_hash.strip()]
+    for line in git_diff.splitlines():
+        git_status.append(line.strip().lower())
+    remote_version = []
+    for line in remote.iter_output("cat /etc/chatmail-version"):
+        print(line)
+        remote_version.append(line)
+    # assert len(git_status) == len(remote_version)  # for some reason, we only get 11 lines from remote.iter_output()
+    for i in range(len(remote_version)):
+        assert git_status[i] == remote_version[i], "You have undeployed changes."

cmdeploy/src/cmdeploy/tests/plugin.py

@@ -307,6 +307,7 @@ def cmfactory(request, gencreds, tmpdir, maildomain):
     class Data:
         def read_path(self, path):
             return
+
     am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=Data())
 
     # nb. a bit hacky

scripts/initenv.sh

@@ -1,5 +1,18 @@
 #!/bin/sh
 set -e
+
+if command -v lsb_release 2>&1 >/dev/null; then
+  case "$(lsb_release -is)" in
+    Ubuntu | Debian )
+      if ! dpkg -l | grep python3-dev 2>&1 >/dev/null
+      then
+        echo "You need to install python3-dev for installing the other dependencies."
+        exit 1
+      fi
+      ;;
+  esac
+fi
+
 python3 -m venv --upgrade-deps venv
 
 venv/bin/pip install -e chatmaild 

www/src/info.md

@@ -6,29 +6,6 @@ interoperable e-mail service for everyone. What's behind a `chatmail` is
 effectively a normal e-mail address just like any other but optimized 
 for the usage in chats, especially DeltaChat.
 
-### Choosing a chatmail address instead of using a random one
-
-In the Delta Chat account setup you may tap `Create a profile` then `Use other server` and choose `Classic e-mail login`. Here fill the two fields like this: 
-
-- `E-Mail Address`: invent a word with
-{% if username_min_length == username_max_length %}
-  *exactly* {{ username_min_length }}
-{% else %}
-  {{ username_min_length}}
-  {% if username_max_length == "more" %}
-    or more
-  {% else %}
-    to {{ username_max_length }}
-  {% endif %}
-{% endif %}
-  characters
-  and append `@{{config.mail_domain}}` to it.
-
-- `Existing Password`: invent at least {{ password_min_length }} characters.
-
-If the e-mail address is not yet taken, you'll get that account. 
-The first login sets your password. 
-
 
 ### Rate and storage limits 
 
@@ -38,10 +15,11 @@ The first login sets your password.
 
 - You may send up to {{ config.max_user_send_per_minute }} messages per minute.
 
-- Messages are unconditionally removed {{ config.delete_mails_after }} days after arriving on the server.
-
 - You can store up to [{{ config.max_mailbox_size }} messages on the server](https://delta.chat/en/help#what-happens-if-i-turn-on-delete-old-messages-from-server).
 
+- Messages are unconditionally removed latest {{ config.delete_mails_after }} days after arriving on the server.
+  Earlier, if storage may exceed otherwise.
+
 
 ### <a name="account-deletion"></a> Account deletion