Test dig output with dns comments

We already require that outgoing connections
use STARTTLS so other servers need a valid TLS
certificate to accept messages from us.
It is then very unlikely that they cannot use TLS
to send messages to us.

Conversely, if they only can send messages to use without TLS,
it likely does not have STARTLS on its port 25
and then we don't want to accept messages from them
because we will likely not be able to reply.

.github/ISSUE_TEMPLATE/config.yml

@@ -1,5 +1 @@
 blank_issues_enabled: true
-contact_links:
-  - name: Mutual Help Chat Group
-    url: https://i.delta.chat/#6CBFF8FFD505C0FDEA20A66674F2916EA8FBEE99&a=invitebot%40nine.testrun.org&g=Chatmail%20Mutual%20Help&x=7sFF7Ik50pWv6J1z7RVC5527&i=X69wTFfvCfs3d-JzqP0kVA3i&s=ibp-447dU-wUq-52QanwAtWc
-    about: If you have troubles setting up the relay server, feel free to ask here.

CHANGELOG.md

@@ -2,9 +2,15 @@
 
 ## untagged
 
+- acmetool: use ECDSA keys instead of RSA
+  ([#689](https://github.com/chatmail/relay/pull/689))
+
 - Require TLS 1.2 for outgoing SMTP connections
   ([#685](https://github.com/chatmail/relay/pull/685))
 
+- require STARTTLS for incoming port 25 connections
+  ([#684](https://github.com/chatmail/relay/pull/684))
+
 - filtermail: run CPU-intensive handle_DATA in a thread pool executor
   ([#676](https://github.com/chatmail/relay/pull/676))
 
@@ -24,7 +30,7 @@
   ([#650](https://github.com/chatmail/relay/pull/650))
 
 - filtermail: accept mails from Protonmail
-  ([#616](https://github.com/chatmail/relay/pull/655))
+  ([#616](https://github.com/chatmail/relay/pull/616))
 
 - Ignore all RCPT TO: parameters
   ([#651](https://github.com/chatmail/relay/pull/651))
@@ -57,7 +63,7 @@
   to only do a single iteration over sometimes millions of messages
   instead of doing "find" commands that iterate 9 times over the messages. 
   Provide an "fsreport" CLI for more fine grained analysis of message files. 
-  ([#637](https://github.com/chatmail/relay/pull/632))
+  ([#637](https://github.com/chatmail/relay/pull/637))
 
 
 ## 1.7.0 2025-09-11

README.md

@@ -180,6 +180,10 @@ The components of chatmail are:
 - [Iroh relay](https://www.iroh.computer/docs/concepts/relay) 
   which helps client devices to establish Peer-to-Peer connections 
 
+- [TURN](https://github.com/chatmail/chatmail-turn)
+  to enable relay users to start webRTC calls
+  even if a p2p connection can't be established
+
 - and the chatmaild services, explained in the next section:
 
 ### chatmaild
@@ -304,6 +308,8 @@ Chatmail address creation will be denied while this file is present.
 [Nginx](https://www.nginx.com/) listens on port 8443 (HTTPS-ALT) and 443 (HTTPS).
 Port 443 multiplexes HTTPS, IMAP and SMTP using ALPN to redirect connections to ports 8443, 465 or 993.
 [acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (HTTP).
+[chatmail-turn](https://github.com/chatmail/chatmail-turn) listens on UDP port 3478 (STUN/TURN),
+and temporarily opens UDP ports when users request them. UDP port range is not restricted, any free port may be allocated.
 
 chatmail-core based apps will, however, discover all ports and configurations
 automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail relay server.

chatmaild/src/chatmaild/turnserver.py

@@ -6,4 +6,4 @@ def turn_credentials() -> str:
     with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client_socket:
         client_socket.connect("/run/chatmail-turn/turn.socket")
         with client_socket.makefile("rb") as file:
-            return file.readline().decode("utf-8")
+            return file.readline().decode("utf-8").strip()

cmdeploy/src/cmdeploy/__init__.py

@@ -338,9 +338,9 @@ def _install_dovecot_package(package: str, arch: str):
 
     match (package, arch):
         case ("core", "amd64"):
-            sha256 = "43f593332e22ac7701c62d58b575d2ca409e0f64857a2803be886c22860f5587"
+            sha256 = "dd060706f52a306fa863d874717210b9fe10536c824afe1790eec247ded5b27d"
         case ("core", "arm64"):
-            sha256 = "4d21eba1a83f51c100f08f2e49f0c9f8f52f721ebc34f75018e043306da993a7"
+            sha256 = "e7548e8a82929722e973629ecc40fcfa886894cef3db88f23535149e7f730dc9"
         case ("imapd", "amd64"):
             sha256 = "8d8dc6fc00bbb6cdb25d345844f41ce2f1c53f764b79a838eb2a03103eebfa86"
         case ("imapd", "arm64"):

cmdeploy/src/cmdeploy/acmetool/target.yaml.j2

@@ -1,7 +1,8 @@
 request:
   provider: https://acme-v02.api.letsencrypt.org/directory
   key:
-    type: rsa
+    type: ecdsa
+    ecdsa-curve: nistp256
   challenge:
     webroot-paths:
     - /var/www/html/.well-known/acme-challenge

cmdeploy/src/cmdeploy/dovecot/dovecot.conf.j2

@@ -70,6 +70,12 @@ userdb {
 # Mailboxes are stored in the "mail" directory of the vmail user home.
 mail_location = maildir:{{ config.mailboxes_dir }}/%u
 
+# index/cache files are not very useful for chatmail relay operations 
+# but it's not clear how to disable them completely. 
+# According to https://doc.dovecot.org/2.3/settings/advanced/#core_setting-mail_cache_max_size
+# if the cache file becomes larger than the specified size, it is truncated by dovecot 
+mail_cache_max_size = 500K
+
 namespace inbox {
   inbox = yes
 

cmdeploy/src/cmdeploy/postfix/master.cf.j2

@@ -14,6 +14,7 @@ smtp      inet  n       -       y       -       -       smtpd -v
 {%- else %}
 smtp      inet  n       -       y       -       -       smtpd 
 {%- endif %}
+  -o smtpd_tls_security_level=encrypt
   -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port_incoming }}
 submission inet n       -       y       -       5000    smtpd
   -o syslog_name=postfix/submission

cmdeploy/src/cmdeploy/remote/rdns.py

@@ -73,9 +73,7 @@ def query_dns(typ, domain):
 
     # Query authoritative nameserver directly to bypass DNS cache.
     res = shell(f"dig @{ns} -r -q {domain} -t {typ} +short", print=log_progress)
-    if res:
-        return res.split("\n")[0]
-    return ""
+    return next((line for line in res.split("\n") if not line.startswith(';')), '')
 
 
 def check_zonefile(zonefile, verbose=True):

cmdeploy/src/cmdeploy/tests/online/benchmark.py

@@ -37,7 +37,7 @@ class TestDC:
 
     def test_ping_pong(self, benchmark, cmfactory):
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
 
         def dc_ping_pong():
             chat.send_text("ping")
@@ -49,7 +49,7 @@ class TestDC:
 
     def test_send_10_receive_10(self, benchmark, cmfactory, lp):
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
 
         def dc_send_10_receive_10():
             for i in range(10):

cmdeploy/src/cmdeploy/tests/online/test_0_login.py

@@ -1,5 +1,5 @@
 import queue
-import socket
+import smtplib
 import threading
 
 import pytest
@@ -91,25 +91,23 @@ def test_concurrent_logins_same_account(
 
 def test_no_vrfy(chatmail_config):
     domain = chatmail_config.mail_domain
-    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    sock.settimeout(10)
-    try:
-        sock.connect((domain, 25))
-    except socket.timeout:
-        pytest.skip(f"port 25 not reachable for {domain}")
-    banner = sock.recv(1024)
-    print(banner)
-    sock.send(b"VRFY wrongaddress@%s\r\n" % (chatmail_config.mail_domain.encode(),))
-    result = sock.recv(1024)
+
+    s = smtplib.SMTP(domain)
+    s.starttls()
+
+    s.putcmd("vrfy", f"wrongaddress@{chatmail_config.mail_domain}")
+    result = s.getreply()
     print(result)
-    sock.send(b"VRFY echo@%s\r\n" % (chatmail_config.mail_domain.encode(),))
-    result2 = sock.recv(1024)
+    s.putcmd("vrfy", f"echo@{chatmail_config.mail_domain}")
+    result2 = s.getreply()
     print(result2)
-    assert result[0:10] == result2[0:10]
-    sock.send(b"VRFY wrongaddress\r\n")
-    result = sock.recv(1024)
+    assert result[0] == result2[0] == 252
+    assert result[1][0:6] == result2[1][0:6] == b"2.0.0 "
+    s.putcmd("vrfy", "wrongaddress")
+    result = s.getreply()
     print(result)
-    sock.send(b"VRFY echo\r\n")
-    result2 = sock.recv(1024)
+    s.putcmd("vrfy", "echo")
+    result2 = s.getreply()
     print(result2)
-    assert result[0:10] == result2[0:10] == b"252 2.0.0 "
+    assert result[0] == result2[0] == 252
+    assert result[1][0:6] == result2[1][0:6] == b"2.0.0 "

cmdeploy/src/cmdeploy/tests/online/test_1_basic.py

@@ -143,6 +143,7 @@ def test_reject_missing_dkim(cmsetup, maildata, from_addr):
         "encrypted.eml", from_addr=from_addr, to_addr=recipient.addr
     ).as_string()
     conn = smtplib.SMTP(cmsetup.maildomain, 25, timeout=10)
+    conn.starttls()
 
     with conn as s:
         with pytest.raises(smtplib.SMTPDataError, match="No valid DKIM signature"):

cmdeploy/src/cmdeploy/tests/online/test_2_deltachat.py

@@ -56,7 +56,7 @@ class TestEndToEndDeltaChat:
         """Test that a DC account can send a message to a second DC account
         on the same chat-mail instance."""
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
         chat.send_text("message0")
 
         lp.sec("wait for ac2 to receive message")
@@ -70,7 +70,7 @@ class TestEndToEndDeltaChat:
         before quota is exceeded, and thus depends on the speed of the upload.
         """
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
 
         user = ac2.get_config("configured_addr")
 
@@ -153,7 +153,7 @@ def test_hide_senders_ip_address(cmfactory):
     assert ipaddress.ip_address(public_ip)
 
     user1, user2 = cmfactory.get_online_accounts(2)
-    chat = cmfactory.get_protected_chat(user1, user2)
+    chat = cmfactory.get_accepted_chat(user1, user2)
 
     chat.send_text("testing submission header cleanup")
     user2._evtracker.wait_next_incoming_message()

cmdeploy/src/cmdeploy/tests/test_dns.py

@@ -1,3 +1,5 @@
+from copy import deepcopy
+
 import pytest
 
 from cmdeploy import remote
@@ -8,38 +10,63 @@ from cmdeploy.dns import check_full_zone, check_initial_remote_data
 def mockdns_base(monkeypatch):
     qdict = {}
 
-    def query_dns(typ, domain):
-        try:
-            return qdict[typ][domain]
-        except KeyError:
-            return ""
+    def shell(command, fail_ok=False, print=print):
+        if command.startswith("dig"):
+            if command == "dig":
+                return "."
+            if "SOA" in command:
+                return (
+                    "delta.chat. 21600 IN SOA ns1.first-ns.de. dns.hetzner.com."
+                    " 2025102800 14400 1800 604800 3600"
+                )
+            command_chunks = command.split()
+            domain, typ = command_chunks[4], command_chunks[6]
+            try:
+                return qdict[typ][domain]
+            except KeyError:
+                return ""
+        return remote.rshell.shell(command=command, fail_ok=fail_ok, print=print)
 
-    monkeypatch.setattr(remote.rdns, query_dns.__name__, query_dns)
+    monkeypatch.setattr(remote.rdns, shell.__name__, shell)
     return qdict
 
 
 @pytest.fixture
-def mockdns(mockdns_base):
-    mockdns_base.update(
-        {
-            "A": {"some.domain": "1.1.1.1"},
-            "AAAA": {"some.domain": "fde5:cd7a:9e1c:3240:5a99:936f:cdac:53ae"},
-            "CNAME": {
-                "mta-sts.some.domain": "some.domain.",
-                "www.some.domain": "some.domain.",
-            },
-        }
-    )
+def mockdns_expected():
+    return {
+        "A": {"some.domain": "1.1.1.1"},
+        "AAAA": {"some.domain": "fde5:cd7a:9e1c:3240:5a99:936f:cdac:53ae"},
+        "CNAME": {
+            "mta-sts.some.domain": "some.domain.",
+            "www.some.domain": "some.domain.",
+        },
+    }
+
+
+@pytest.fixture(params=["plain", "with-dns-comments"])
+def mockdns(request, mockdns_base, mockdns_expected):
+    mockdns_base.update(deepcopy(mockdns_expected))
+    match request.param:
+        case "plain":
+            pass
+        case "with-dns-comments":
+            for typ, data in mockdns_base.items():
+                for host, result in data.items():
+                    mockdns_base[typ][host] = (
+                        ";; some unsuccessful attempt result\n"
+                        "; and another with a single semicolon\n"
+                        f"{result}"
+                    )
     return mockdns_base
 
 
 class TestPerformInitialChecks:
-    def test_perform_initial_checks_ok1(self, mockdns):
+    def test_perform_initial_checks_ok1(self, mockdns, mockdns_expected):
         remote_data = remote.rdns.perform_initial_checks("some.domain")
-        assert remote_data["A"] == mockdns["A"]["some.domain"]
-        assert remote_data["AAAA"] == mockdns["AAAA"]["some.domain"]
-        assert remote_data["MTA_STS"] == mockdns["CNAME"]["mta-sts.some.domain"]
-        assert remote_data["WWW"] == mockdns["CNAME"]["www.some.domain"]
+        assert remote_data["A"] == mockdns_expected["A"]["some.domain"]
+        assert remote_data["AAAA"] == mockdns_expected["AAAA"]["some.domain"]
+        assert remote_data["MTA_STS"] == mockdns_expected["CNAME"]["mta-sts.some.domain"]
+        assert remote_data["WWW"] == mockdns_expected["CNAME"]["www.some.domain"]
 
     @pytest.mark.parametrize("drop", ["A", "AAAA"])
     def test_perform_initial_checks_with_one_of_A_AAAA(self, mockdns, drop):