count ci accounts correctly

close #126

README.md

@@ -18,14 +18,7 @@ after which the initially specified password is required for using them.
 We subsequently use `CHATMAIL_DOMAIN` as a placeholder for your fully qualified 
 DNS domain name (FQDN), for example `chat.example.org`.
 
-1. Setup DNS `A` and `AAAA` records for your `CHATMAIL_DOMAIN`. 
+1. Install the `cmdeploy` command in a virtualenv
-   Verify that DNS is set and SSH root login works:
-
-   ```
-        ssh root@CHATMAIL_DOMAIN 
-   ```
-
-2. Install the `cmdeploy` command in a virtualenv
 
    ```
     git clone https://github.com/deltachat/chatmail
@@ -33,12 +26,20 @@ DNS domain name (FQDN), for example `chat.example.org`.
     scripts/initenv.sh
    ```
   
-3. Create chatmail configuration file `chatmail.ini`:
+2. Create chatmail configuration file `chatmail.ini`:
 
    ```
     scripts/cmdeploy init CHATMAIL_DOMAIN
    ```
 
+3. Setup first DNS records for your `CHATMAIL_DOMAIN`,
+   according to the hints provided by `cmdeploy init`.
+   Verify that SSH root login works:
+
+   ```
+    ssh root@CHATMAIL_DOMAIN
+   ```
+
 4. Deploy to the remote chatmail server:
 
    ```
@@ -52,24 +53,70 @@ DNS domain name (FQDN), for example `chat.example.org`.
     scripts/cmdeploy dns
    ```
 
-6. To check status of your remotely running chatmail service: 
+### Other helpful commands:
+
+To check the status of your remotely running chatmail service:
 
 ```
 scripts/cmdeploy status
 ```
 
-7. To test your chatmail service: 
+To test whether your chatmail service is working correctly:
 
 ```
 scripts/cmdeploy test
 ```
 
-8. To benchmark your chatmail service: 
+To measure the performance of your chatmail service:
 
 ```
 scripts/cmdeploy bench
 ```
 
+## Overview of this repository
+
+This repository drives the development of "chatmail instances",
+comprised of minimal setups of
+
+- [postfix smtp server](https://www.postfix.org)
+- [dovecot imap server](https://www.dovecot.org)
+
+as well as custom services that are integrated with these two:
+
+- `chatmaild/src/chatmaild/doveauth.py` implements
+  create-on-login account creation semantics and is used
+  by Dovecot during login authentication and by Postfix
+  which in turn uses [Dovecot SASL](https://doc.dovecot.org/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket)
+  to authenticate users
+  to send mails for them.
+
+- `chatmaild/src/chatmaild/filtermail.py` prevents
+  unencrypted e-mail from leaving the chatmail instance
+  and is integrated into postfix's outbound mail pipelines.
+
+There is also the `cmdeploy/src/cmdeploy/cmdeploy.py` command line tool
+which helps with setting up and managing the chatmail service.
+`cmdeploy run` uses [pyinfra-based scripting](https://pyinfra.com/)
+in `cmdeploy/src/cmdeploy/__init__.py`
+to automatically install all chatmail components on a server.
+
+
+### Home page and getting started for users
+
+`cmdeploy run` also creates default static Web pages and deploys them
+to an nginx web server under `https://CHATMAIL_DOMAIN`.
+
+- a default `index.html` along with a QR code that users can click to
+  create accounts on your chatmail provider,
+
+- a default `info.html` that is linked from the home page,
+
+- a default `policy.html` that is linked from the home page.
+
+All `.html` files are generated
+by the according markdown `.md` file in the `www/src` directory.
+
+
 ### Refining the web pages
 
 
@@ -88,75 +135,24 @@ This starts a local live development cycle for chatmail Web pages:
 - Starts a browser window automatically where you can "refresh" as needed.
 
 
-### Home page and getting started for users 
+## Emergency Commands to disable automatic account creation
 
-`cmdeploy run` sets up mail services,
+If you need to stop account creation,
-and also creates default static Web pages and deploys them: 
+e.g. because some script is wildly creating accounts,
+login to the server with ssh and run:
 
-- a default `index.html` along with a QR code that users can click to 
+```
-  create accounts on your chatmail provider,
+    touch /etc/chatmail-nocreate
-
+```
-- a default `info.html` that is linked from the home page,
-
-- a default `policy.html` that is linked from the home page. 
-
-All `.html` files are generated 
-by the according markdown `.md` file in the `www/src` directory.
 
+While this file is present, account creation will be blocked.
 
 ### Ports
 
 Postfix listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
 Dovecot listens on ports 143(imap) and 993 (imaps).
 
-Delta Chat will, however, discover all ports and configurations 
+Delta Chat apps will, however, discover all ports and configurations
-automatically by reading the `autoconfig.xml` file from the chatmail instance. 
+automatically by reading the `autoconfig.xml` file from the chatmail service.
-
-
-## Emergency Commands to disable automatic account creation 
-
-If you need to stop account creation,
-e.g. because some script is wildly creating accounts, run:
-
-    touch /etc/chatmail-nocreate
-
-While this file is present, account creation will be blocked. 
-
-
-## Running tests and benchmarks (offline and online) 
-
-1. Set `CHATMAIL_SSH` so that `ssh root@$CHATMAIL_SSH` allows 
-   to login to the chatmail instance server. 
-
-2. To run local and online tests: 
-
-    scripts/test.sh 
-
-3. To run benchmarks against your chatmail instance: 
-
-    scripts/bench.sh 
-
-
-## Development Background for chatmail instances 
-
-This repository drives the development of "chatmail instances", 
-comprised of minimal setups of 
-
-- [postfix smtp server](https://www.postfix.org) 
-- [dovecot imap server](https://www.dovecot.org) 
-
-as well as two custom services that are integrated with these two: 
-
-- `chatmaild/src/chatmaild/doveauth.py` implements
-  create-on-login account creation semantics and is used
-  by Dovecot during login authentication and by Postfix
-  which in turn uses [Dovecot SASL](https://doc.dovecot.org/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket)
-  to authenticate users
-  to send mails for them. 
-
-- `chatmaild/src/chatmaild/filtermail.py` prevents 
-  unencrypted e-mail from leaving the chatmail instance
-  and is integrated into postfix's outbound mail pipelines. 
-
 
 

chatmaild/pyproject.toml

@@ -8,6 +8,8 @@ version = "0.2"
 dependencies = [
   "aiosmtpd",
   "iniconfig",
+  "deltachat-rpc-server",
+  "deltachat-rpc-client",
 ]
 
 [tool.setuptools]
@@ -19,6 +21,8 @@ where = ['src']
 [project.scripts]
 doveauth = "chatmaild.doveauth:main"
 filtermail = "chatmaild.filtermail:main"
+echobot = "chatmaild.echo:main"
+chatmail-metrics = "chatmaild.metrics:main"
 
 [project.entry-points.pytest11]
 "chatmaild.testplugin" = "chatmaild.tests.plugin"

chatmaild/src/chatmaild/doveauth.py

@@ -46,6 +46,7 @@ def is_allowed_to_create(config: Config, user, cleartext_password) -> bool:
         len(localpart) > config.username_max_length
         or len(localpart) < config.username_min_length
     ):
+        if localpart != "echo":
             logging.warning(
                 "localpart %s has to be between %s and %s chars long",
                 localpart,

chatmaild/src/chatmaild/echo.py

@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+"""Advanced echo bot example.
+
+it will echo back any message that has non-empty text and also supports the /help command.
+"""
+import logging
+import os
+import sys
+from threading import Thread
+
+from deltachat_rpc_client import Bot, DeltaChat, EventType, Rpc, events
+
+from chatmaild.newemail import create_newemail_dict
+from chatmaild.config import read_config
+
+hooks = events.HookCollection()
+
+
+@hooks.on(events.RawEvent)
+def log_event(event):
+    if event.kind == EventType.INFO:
+        logging.info(event.msg)
+    elif event.kind == EventType.WARNING:
+        logging.warning(event.msg)
+
+
+@hooks.on(events.RawEvent(EventType.ERROR))
+def log_error(event):
+    logging.error(event.msg)
+
+
+@hooks.on(events.MemberListChanged)
+def on_memberlist_changed(event):
+    logging.info(
+        "member %s was %s", event.member, "added" if event.member_added else "removed"
+    )
+
+
+@hooks.on(events.GroupImageChanged)
+def on_group_image_changed(event):
+    logging.info("group image %s", "deleted" if event.image_deleted else "changed")
+
+
+@hooks.on(events.GroupNameChanged)
+def on_group_name_changed(event):
+    logging.info("group name changed, old name: %s", event.old_name)
+
+
+@hooks.on(events.NewMessage(func=lambda e: not e.command))
+def echo(event):
+    snapshot = event.message_snapshot
+    if snapshot.text or snapshot.file:
+        snapshot.chat.send_message(text=snapshot.text, file=snapshot.file)
+
+
+@hooks.on(events.NewMessage(command="/help"))
+def help_command(event):
+    snapshot = event.message_snapshot
+    snapshot.chat.send_text("Send me any message and I will echo it back")
+
+
+def main():
+    path = os.environ.get("PATH")
+    venv_path = sys.argv[0].strip("echobot")
+    os.environ["PATH"] = path + ":" + venv_path
+    with Rpc() as rpc:
+        deltachat = DeltaChat(rpc)
+        system_info = deltachat.get_system_info()
+        logging.info("Running deltachat core %s", system_info.deltachat_core_version)
+
+        accounts = deltachat.get_all_accounts()
+        account = accounts[0] if accounts else deltachat.add_account()
+
+        bot = Bot(account, hooks)
+        if not bot.is_configured():
+            config = read_config(sys.argv[1])
+            password = create_newemail_dict(config).get("password")
+            email = "echo@" + config.mail_domain
+            configure_thread = Thread(
+                target=bot.configure, kwargs={"email": email, "password": password}
+            )
+            configure_thread.start()
+        bot.run_forever()
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.INFO)
+    main()

chatmaild/src/chatmaild/echobot.service.f

@@ -0,0 +1,11 @@
+[Unit]
+Description=Chatmail echo bot for testing it works
+
+[Service]
+ExecStart={execpath} {config_path}
+Environment="PATH={remote_venv_dir}:$PATH"
+Restart=always
+RestartSec=30
+
+[Install]
+WantedBy=multi-user.target

chatmaild/src/chatmaild/ini/chatmail.ini.f

@@ -17,8 +17,8 @@ max_user_send_per_minute = 60
 # maximum mailbox size of a chatmail account
 max_mailbox_size = 100M
 
-# time after which seen mails are deleted
+# days after which mails are unconditionally deleted
-delete_mails_after = 40d
+delete_mails_after = 40
 
 # minimum length a username must have
 username_min_length = 9

chatmaild/src/chatmaild/metrics.py

@@ -0,0 +1,25 @@
+#!/usr/bin/env python3
+from pathlib import Path
+import time
+import sys
+
+
+def main(vmail_dir=None):
+    if vmail_dir is None:
+        vmail_dir = sys.argv[1]
+
+    accounts = 0
+    ci_accounts = 0
+
+    for path in Path(vmail_dir).iterdir():
+        accounts += 1
+        if path.name[:3] in ("ci-", "ac_"):
+            ci_accounts += 1
+
+    timestamp = int(time.time() * 1000)
+    print(f"accounts {accounts} {timestamp}")
+    print(f"ci_accounts {ci_accounts} {timestamp}")
+
+
+if __name__ == "__main__":
+    main()

chatmaild/src/chatmaild/newemail.py

@@ -4,16 +4,22 @@
 
 import json
 import random
+import secrets
+import string
 
 from chatmaild.config import read_config, Config
 
 CONFIG_PATH = "/usr/local/lib/chatmaild/chatmail.ini"
+ALPHANUMERIC = string.ascii_lowercase + string.digits
+ALPHANUMERIC_PUNCT = string.ascii_letters + string.digits + string.punctuation
 
 
 def create_newemail_dict(config: Config):
-    alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
+    user = "".join(random.choices(ALPHANUMERIC, k=config.username_min_length))
-    user = "".join(random.choices(alphanumeric, k=config.username_min_length))
+    password = "".join(
-    password = "".join(random.choices(alphanumeric, k=config.password_min_length + 3))
+        secrets.choice(ALPHANUMERIC_PUNCT)
+        for _ in range(config.password_min_length + 3)
+    )
     return dict(email=f"{user}@{config.mail_domain}", password=f"{password}")
 
 

chatmaild/src/chatmaild/tests/test_config.py

@@ -24,7 +24,7 @@ def test_read_config_testrun(make_config):
     assert config.postfix_reinject_port == 10025
     assert config.max_user_send_per_minute == 60
     assert config.max_mailbox_size == "100M"
-    assert config.delete_mails_after == "40d"
+    assert config.delete_mails_after == "40"
     assert config.username_min_length == 9
     assert config.username_max_length == 9
     assert config.password_min_length == 9

chatmaild/src/chatmaild/tests/test_metrics.py

@@ -0,0 +1,16 @@
+from chatmaild.metrics import main
+
+
+def test_main(tmp_path, capsys):
+    for x in ("ci-asllkj", "ac_12l3kj", "qweqwe", "ci-l1k2j31l2k3"):
+        tmp_path.joinpath(x).mkdir()
+    main(tmp_path)
+    out, _ = capsys.readouterr()
+    d = {}
+    for line in out.split("\n"):
+        if line.strip():
+            name, num, _ = line.split()
+            d[name] = int(num)
+
+    assert d["accounts"] == 4
+    assert d["ci_accounts"] == 3

cmdeploy/src/cmdeploy/__init__.py

@@ -84,14 +84,28 @@ def _install_remote_venv_with_chatmaild(config) -> None:
         ],
     )
 
+    files.template(
+        src=importlib.resources.files(__package__).joinpath("metrics.cron.j2"),
+        dest="/etc/cron.d/chatmail-metrics",
+        user="root",
+        group="root",
+        mode="644",
+        config={
+            "mail_domain": config.mail_domain,
+            "execpath": f"{remote_venv_dir}/bin/chatmail-metrics",
+        },
+    )
+
     # install systemd units
     for fn in (
         "doveauth",
         "filtermail",
+        "echobot",
     ):
         params = dict(
             execpath=f"{remote_venv_dir}/bin/{fn}",
             config_path=remote_chatmail_inipath,
+            remote_venv_dir=remote_venv_dir,
         )
         source_path = importlib.resources.files("chatmaild").joinpath(f"{fn}.service.f")
         content = source_path.read_text().format(**params).encode()
@@ -112,7 +126,7 @@ def _install_remote_venv_with_chatmaild(config) -> None:
         )
 
 
-def _configure_opendkim(domain: str, dkim_selector: str) -> bool:
+def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
     """Configures OpenDKIM"""
     need_restart = False
 
@@ -350,20 +364,22 @@ def check_config(config):
     if mail_domain != "testrun.org" and not mail_domain.endswith(".testrun.org"):
         blocked_words = "merlinux schmieder testrun.org".split()
         for value in config.__dict__.values():
-            if any(x in value for x in blocked_words):
+            if any(x in str(value) for x in blocked_words):
                 raise ValueError(
                     f"please set your own privacy contacts/addresses in {config._inipath}"
                 )
     return config
 
 
-def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> None:
+def deploy_chatmail(config_path: Path) -> None:
     """Deploy a chat-mail instance.
 
-    :param mail_domain: domain part of your future email addresses
+    :param config_path: path to chatmail.ini
-    :param mail_server: the DNS name under which your mail server is reachable
-    :param dkim_selector:
     """
+    config = read_config(config_path)
+    check_config(config)
+    mail_domain = config.mail_domain
+
     from .www import build_webpages
 
     apt.update(name="apt update", cache_time=24 * 3600)
@@ -393,7 +409,7 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
     )
 
     # Deploy acmetool to have TLS certificates.
-    deploy_acmetool(nginx_hook=True, domains=[mail_server, f"mta-sts.{mail_server}"])
+    deploy_acmetool(nginx_hook=True, domains=[mail_domain, f"mta-sts.{mail_domain}"])
 
     apt.packages(
         name="Install Postfix",
@@ -423,11 +439,7 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
         packages=["fcgiwrap"],
     )
 
-    pkg_root = importlib.resources.files(__package__)
+    www_path = importlib.resources.files(__package__).joinpath("../../../www").resolve()
-    chatmail_ini = pkg_root.joinpath("../../../chatmail.ini").resolve()
-    config = read_config(chatmail_ini)
-    check_config(config)
-    www_path = pkg_root.joinpath("../../../www").resolve()
 
     build_dir = www_path.joinpath("build")
     src_dir = www_path.joinpath("src")
@@ -438,7 +450,7 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
     debug = False
     dovecot_need_restart = _configure_dovecot(config, debug=debug)
     postfix_need_restart = _configure_postfix(config, debug=debug)
-    opendkim_need_restart = _configure_opendkim(mail_domain, dkim_selector)
+    opendkim_need_restart = _configure_opendkim(mail_domain)
     mta_sts_need_restart = _install_mta_sts_daemon()
     nginx_need_restart = _configure_nginx(mail_domain)
 

cmdeploy/src/cmdeploy/cmdeploy.py

@@ -52,8 +52,8 @@ def run_cmd(args, out):
     """Deploy chatmail services on the remote server."""
 
     env = os.environ.copy()
-    env["CHATMAIL_DOMAIN"] = args.config.mail_domain
+    env["CHATMAIL_INI"] = args.inipath
-    deploy_path = "cmdeploy/src/cmdeploy/deploy.py"
+    deploy_path = importlib.resources.files(__package__).joinpath("deploy.py").resolve()
     pyinf = "pyinfra --dry" if args.dry_run else "pyinfra"
     cmd = f"{pyinf} --ssh-user root {args.config.mail_domain} {deploy_path}"
     out.check_call(cmd, env=env)

cmdeploy/src/cmdeploy/deploy.py

@@ -1,18 +1,16 @@
 import os
+import importlib.resources
 import pyinfra
 from cmdeploy import deploy_chatmail
 
 
 def main():
-    mail_domain = os.getenv("CHATMAIL_DOMAIN")
+    config_path = os.getenv(
-    mail_server = os.getenv("CHATMAIL_SERVER", mail_domain)
+        "CHATMAIL_INI",
-    dkim_selector = os.getenv("CHATMAIL_DKIM_SELECTOR", "dkim")
+        importlib.resources.files("cmdeploy").joinpath("../../../chatmail.ini"),
+    )
 
-    assert mail_domain
+    deploy_chatmail(config_path)
-    assert mail_server
-    assert dkim_selector
-
-    deploy_chatmail(mail_domain, mail_server, dkim_selector)
 
 
 if pyinfra.is_cli:

cmdeploy/src/cmdeploy/dovecot/expunge.cron.j2

@@ -1,4 +1,10 @@
-2 0 * * * dovecot doveadm expunge -A SEEN BEFORE {{ config.delete_mails_after }} INBOX
+# delete all mails after {{ config.delete_mails_after }} days, in the Inbox
-2 0 * * * dovecot doveadm expunge -A SEEN BEFORE {{ config.delete_mails_after }} Deltachat
+2 0 * * * dovecot find /home/vmail/mail/{{ config.mail_domain }}/*/cur -mtime +{{ config.delete_mails_after }} -type f -delete
-2 0 * * * dovecot doveadm expunge -A SEEN BEFORE {{ config.delete_mails_after }} Trash
+# or in any IMAP subfolder
-2 30 * * * dovecot doveadm purge -A
+2 0 * * * dovecot find /home/vmail/mail/{{ config.mail_domain }}/*/.*/cur -mtime +{{ config.delete_mails_after }} -type f -delete
+# even if they are unseen
+2 0 * * * dovecot find /home/vmail/mail/{{ config.mail_domain }}/*/new -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * dovecot find /home/vmail/mail/{{ config.mail_domain }}/*/.*/new -mtime +{{ config.delete_mails_after }} -type f -delete
+# or only temporary (but then they shouldn't be around after {{ config.delete_mails_after }} days anyway).
+2 0 * * * dovecot find /home/vmail/mail/{{ config.mail_domain }}/*/tmp -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * dovecot find /home/vmail/mail/{{ config.mail_domain }}/*/.*/tmp -mtime +{{ config.delete_mails_after }} -type f -delete

cmdeploy/src/cmdeploy/metrics.cron.j2

@@ -0,0 +1 @@
+*/5 * * * * root {{ config.execpath }} /home/vmail/mail/{{ config.mail_domain }} >/var/www/html/metrics

cmdeploy/src/cmdeploy/nginx/nginx.conf.j2

@@ -41,6 +41,10 @@ http {
 			try_files $uri $uri/ =404;
 		}
 
+                location /metrics {
+                        default_type text/plain;
+                }
+
         # add cgi-bin support
         include /usr/share/doc/fcgiwrap/examples/nginx.conf;
 	}

cmdeploy/src/cmdeploy/www.py

@@ -36,29 +36,6 @@ def build_webpages(src_dir, build_dir, config):
         print(traceback.format_exc())
 
 
-def timespan_to_english(timespan):
-    val = int(timespan[:-1])
-    c = timespan[-1].lower()
-    match c:
-        case "y":
-            return f"{val} years"
-        case "m":
-            return f"{val} months"
-        case "w":
-            return f"{val} weeks"
-        case "d":
-            return f"{val} days"
-        case "h":
-            return f"{val} hours"
-        case "c":
-            return f"{val} seconds"
-        case _:
-            raise ValueError(
-                c
-                + " is not a valid time unit. Try [y]ears, [w]eeks, [d]ays, or [h]ours"
-            )
-
-
 def int_to_english(number):
     if number >= 0 and number <= 12:
         a = [
@@ -104,9 +81,6 @@ def _build_webpages(src_dir, build_dir, config):
             render_vars["password_min_length"] = int_to_english(
                 config.password_min_length
             )
-            render_vars["delete_mails_after"] = timespan_to_english(
-                config.delete_mails_after
-            )
             target = build_dir.joinpath(path.stem + ".html")
 
             # recursive jinja2 rendering

www/src/index.md

@@ -14,7 +14,6 @@ Welcome to instant, interoperable and [privacy-preserving](privacy.html) messagi
 
 💬 **Start** chatting with any Delta Chat contacts using [QR invite codes](https://delta.chat/en/help#howtoe2ee)
 
-<div class="experimental">Note: this is an experimental service</div>
+{% if config.mail_domain != "nine.testrun.org" %}
-
+<div class="experimental">Note: this is only a temporary development chatmail service</div>
-
+{% endif %}
-

www/src/info.md

@@ -3,6 +3,11 @@
 
 ## More information 
 
+`nine.testrun.org` provides a low-maintenance, resource efficient and 
+interoperable e-mail service for everyone. What's behind a `chatmail` is 
+effectively a normal e-mail address just like any other but optimized 
+for the usage in chats, especially DeltaChat.
+
 ### Choosing a chatmail address instead of using a random one
 
 In the Delta Chat account setup 
@@ -37,7 +42,7 @@ The first login sets your password.
 
 - You may send up to {{ config.max_user_send_per_minute }} messages per minute.
 
-- Seen messages are removed {{ delete_mails_after }} after arriving on the server.
+- Messages are unconditionally removed {{ config.delete_mails_after }} days after arriving on the server.
 
 - You can store up to [{{ config.max_mailbox_size }} messages on the server](https://delta.chat/en/help#what-happens-if-i-turn-on-delete-old-messages-from-server).