Compare commits

...

16 Commits

Author SHA1 Message Date
holger krekel
a994a1b950 count ci accounts correctly 2023-12-16 16:47:46 +01:00
missytake
a1355c10ca fix: check config failed for non-testrun domains 2023-12-15 20:25:58 +01:00
link2xt
92ca3283fd Add metrics 2023-12-14 22:22:10 +00:00
missytake
cea1f3f5f7 dovecot: remove -depth from expunge find commands 2023-12-14 19:11:43 +01:00
missytake
39550d3096 small fixes 2023-12-14 19:11:43 +01:00
missytake
070003b983 dovecot: deleting mails with find instead of doveadm expunge 2023-12-14 19:11:43 +01:00
missytake
049ed79e59 dovecot: unconditionally delete all mails after 40 days 2023-12-14 19:11:43 +01:00
missytake
a9e55e3b25 cmdeploy: get cmdeploy run --config working 2023-12-14 18:50:14 +01:00
Septias
5a178ed235 feat: one more paragraph to explain chatmail
close #126
2023-12-14 16:39:41 +01:00
Floris Bruynooghe
8a338f1320 Use more characters for passwords (#124)
This expands the character set used for passwords generated for new
accounts.  The set it taken from the set used by the pass tool.  The
special characters is the full GNU grep [:punct:] set.
2023-12-14 11:51:22 +01:00
Sebastian Klähn
d437b8a943 Merge pull request #125 from deltachat/sk/fix_typo
fix: align spelling of Delta Chat
2023-12-14 11:35:41 +01:00
Septias
a4d520a9ad fix: align spelling of Delta Chat 2023-12-14 11:00:48 +01:00
missytake
9c7dfdf2ff echobot: add echo bot for trying out sending 2023-12-13 22:04:30 +01:00
holger krekel
ffd15e4a9f refine warnings for experimental service,
only show for non-nine domains.
2023-12-13 19:59:52 +01:00
missytake
7f8e0620ca README: formatting 2023-12-13 19:50:45 +01:00
missytake
dc9aebcb55 README: rework and reorder 2023-12-13 19:43:39 +01:00
20 changed files with 313 additions and 167 deletions

186
README.md
View File

@@ -18,14 +18,7 @@ after which the initially specified password is required for using them.
We subsequently use `CHATMAIL_DOMAIN` as a placeholder for your fully qualified
DNS domain name (FQDN), for example `chat.example.org`.
1. Setup DNS `A` and `AAAA` records for your `CHATMAIL_DOMAIN`.
Verify that DNS is set and SSH root login works:
```
ssh root@CHATMAIL_DOMAIN
```
2. Install the `cmdeploy` command in a virtualenv
1. Install the `cmdeploy` command in a virtualenv
```
git clone https://github.com/deltachat/chatmail
@@ -33,12 +26,20 @@ DNS domain name (FQDN), for example `chat.example.org`.
scripts/initenv.sh
```
3. Create chatmail configuration file `chatmail.ini`:
2. Create chatmail configuration file `chatmail.ini`:
```
scripts/cmdeploy init CHATMAIL_DOMAIN
```
3. Setup first DNS records for your `CHATMAIL_DOMAIN`,
according to the hints provided by `cmdeploy init`.
Verify that SSH root login works:
```
ssh root@CHATMAIL_DOMAIN
```
4. Deploy to the remote chatmail server:
```
@@ -52,111 +53,106 @@ DNS domain name (FQDN), for example `chat.example.org`.
scripts/cmdeploy dns
```
6. To check status of your remotely running chatmail service:
### Other helpful commands:
```
scripts/cmdeploy status
```
To check the status of your remotely running chatmail service:
7. To test your chatmail service:
```
scripts/cmdeploy test
```
8. To benchmark your chatmail service:
```
scripts/cmdeploy bench
```
### Refining the web pages
```
scripts/cmdeploy webdev
```
scripts/cmdeploy status
```
This starts a local live development cycle for chatmail Web pages:
To test whether your chatmail service is working correctly:
- uses the `www/src/page-layout.html` file for producing static
HTML pages from `www/src/*.md` files
```
scripts/cmdeploy test
```
- continously builds the web presence reading files from `www/src` directory
and generating html files and copying assets to the `www/build` directory.
To measure the performance of your chatmail service:
- Starts a browser window automatically where you can "refresh" as needed.
```
scripts/cmdeploy bench
```
## Overview of this repository
### Home page and getting started for users
This repository drives the development of "chatmail instances",
comprised of minimal setups of
`cmdeploy run` sets up mail services,
and also creates default static Web pages and deploys them:
- [postfix smtp server](https://www.postfix.org)
- [dovecot imap server](https://www.dovecot.org)
- a default `index.html` along with a QR code that users can click to
create accounts on your chatmail provider,
- a default `info.html` that is linked from the home page,
- a default `policy.html` that is linked from the home page.
All `.html` files are generated
by the according markdown `.md` file in the `www/src` directory.
### Ports
Postfix listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
Dovecot listens on ports 143(imap) and 993 (imaps).
Delta Chat will, however, discover all ports and configurations
automatically by reading the `autoconfig.xml` file from the chatmail instance.
## Emergency Commands to disable automatic account creation
If you need to stop account creation,
e.g. because some script is wildly creating accounts, run:
touch /etc/chatmail-nocreate
While this file is present, account creation will be blocked.
## Running tests and benchmarks (offline and online)
1. Set `CHATMAIL_SSH` so that `ssh root@$CHATMAIL_SSH` allows
to login to the chatmail instance server.
2. To run local and online tests:
scripts/test.sh
3. To run benchmarks against your chatmail instance:
scripts/bench.sh
## Development Background for chatmail instances
This repository drives the development of "chatmail instances",
comprised of minimal setups of
- [postfix smtp server](https://www.postfix.org)
- [dovecot imap server](https://www.dovecot.org)
as well as two custom services that are integrated with these two:
as well as custom services that are integrated with these two:
- `chatmaild/src/chatmaild/doveauth.py` implements
create-on-login account creation semantics and is used
by Dovecot during login authentication and by Postfix
which in turn uses [Dovecot SASL](https://doc.dovecot.org/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket)
to authenticate users
to send mails for them.
to send mails for them.
- `chatmaild/src/chatmaild/filtermail.py` prevents
- `chatmaild/src/chatmaild/filtermail.py` prevents
unencrypted e-mail from leaving the chatmail instance
and is integrated into postfix's outbound mail pipelines.
and is integrated into postfix's outbound mail pipelines.
There is also the `cmdeploy/src/cmdeploy/cmdeploy.py` command line tool
which helps with setting up and managing the chatmail service.
`cmdeploy run` uses [pyinfra-based scripting](https://pyinfra.com/)
in `cmdeploy/src/cmdeploy/__init__.py`
to automatically install all chatmail components on a server.
### Home page and getting started for users
`cmdeploy run` also creates default static Web pages and deploys them
to an nginx web server under `https://CHATMAIL_DOMAIN`.
- a default `index.html` along with a QR code that users can click to
create accounts on your chatmail provider,
- a default `info.html` that is linked from the home page,
- a default `policy.html` that is linked from the home page.
All `.html` files are generated
by the according markdown `.md` file in the `www/src` directory.
### Refining the web pages
```
scripts/cmdeploy webdev
```
This starts a local live development cycle for chatmail Web pages:
- uses the `www/src/page-layout.html` file for producing static
HTML pages from `www/src/*.md` files
- continously builds the web presence reading files from `www/src` directory
and generating html files and copying assets to the `www/build` directory.
- Starts a browser window automatically where you can "refresh" as needed.
## Emergency Commands to disable automatic account creation
If you need to stop account creation,
e.g. because some script is wildly creating accounts,
login to the server with ssh and run:
```
touch /etc/chatmail-nocreate
```
While this file is present, account creation will be blocked.
### Ports
Postfix listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
Dovecot listens on ports 143(imap) and 993 (imaps).
Delta Chat apps will, however, discover all ports and configurations
automatically by reading the `autoconfig.xml` file from the chatmail service.

View File

@@ -8,6 +8,8 @@ version = "0.2"
dependencies = [
"aiosmtpd",
"iniconfig",
"deltachat-rpc-server",
"deltachat-rpc-client",
]
[tool.setuptools]
@@ -19,6 +21,8 @@ where = ['src']
[project.scripts]
doveauth = "chatmaild.doveauth:main"
filtermail = "chatmaild.filtermail:main"
echobot = "chatmaild.echo:main"
chatmail-metrics = "chatmaild.metrics:main"
[project.entry-points.pytest11]
"chatmaild.testplugin" = "chatmaild.tests.plugin"

View File

@@ -46,13 +46,14 @@ def is_allowed_to_create(config: Config, user, cleartext_password) -> bool:
len(localpart) > config.username_max_length
or len(localpart) < config.username_min_length
):
logging.warning(
"localpart %s has to be between %s and %s chars long",
localpart,
config.username_min_length,
config.username_max_length,
)
return False
if localpart != "echo":
logging.warning(
"localpart %s has to be between %s and %s chars long",
localpart,
config.username_min_length,
config.username_max_length,
)
return False
return True

View File

@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""Advanced echo bot example.
it will echo back any message that has non-empty text and also supports the /help command.
"""
import logging
import os
import sys
from threading import Thread
from deltachat_rpc_client import Bot, DeltaChat, EventType, Rpc, events
from chatmaild.newemail import create_newemail_dict
from chatmaild.config import read_config
hooks = events.HookCollection()
@hooks.on(events.RawEvent)
def log_event(event):
if event.kind == EventType.INFO:
logging.info(event.msg)
elif event.kind == EventType.WARNING:
logging.warning(event.msg)
@hooks.on(events.RawEvent(EventType.ERROR))
def log_error(event):
logging.error(event.msg)
@hooks.on(events.MemberListChanged)
def on_memberlist_changed(event):
logging.info(
"member %s was %s", event.member, "added" if event.member_added else "removed"
)
@hooks.on(events.GroupImageChanged)
def on_group_image_changed(event):
logging.info("group image %s", "deleted" if event.image_deleted else "changed")
@hooks.on(events.GroupNameChanged)
def on_group_name_changed(event):
logging.info("group name changed, old name: %s", event.old_name)
@hooks.on(events.NewMessage(func=lambda e: not e.command))
def echo(event):
snapshot = event.message_snapshot
if snapshot.text or snapshot.file:
snapshot.chat.send_message(text=snapshot.text, file=snapshot.file)
@hooks.on(events.NewMessage(command="/help"))
def help_command(event):
snapshot = event.message_snapshot
snapshot.chat.send_text("Send me any message and I will echo it back")
def main():
path = os.environ.get("PATH")
venv_path = sys.argv[0].strip("echobot")
os.environ["PATH"] = path + ":" + venv_path
with Rpc() as rpc:
deltachat = DeltaChat(rpc)
system_info = deltachat.get_system_info()
logging.info("Running deltachat core %s", system_info.deltachat_core_version)
accounts = deltachat.get_all_accounts()
account = accounts[0] if accounts else deltachat.add_account()
bot = Bot(account, hooks)
if not bot.is_configured():
config = read_config(sys.argv[1])
password = create_newemail_dict(config).get("password")
email = "echo@" + config.mail_domain
configure_thread = Thread(
target=bot.configure, kwargs={"email": email, "password": password}
)
configure_thread.start()
bot.run_forever()
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
main()

View File

@@ -0,0 +1,11 @@
[Unit]
Description=Chatmail echo bot for testing it works
[Service]
ExecStart={execpath} {config_path}
Environment="PATH={remote_venv_dir}:$PATH"
Restart=always
RestartSec=30
[Install]
WantedBy=multi-user.target

View File

@@ -17,8 +17,8 @@ max_user_send_per_minute = 60
# maximum mailbox size of a chatmail account
max_mailbox_size = 100M
# time after which seen mails are deleted
delete_mails_after = 40d
# days after which mails are unconditionally deleted
delete_mails_after = 40
# minimum length a username must have
username_min_length = 9

View File

@@ -0,0 +1,25 @@
#!/usr/bin/env python3
from pathlib import Path
import time
import sys
def main(vmail_dir=None):
if vmail_dir is None:
vmail_dir = sys.argv[1]
accounts = 0
ci_accounts = 0
for path in Path(vmail_dir).iterdir():
accounts += 1
if path.name[:3] in ("ci-", "ac_"):
ci_accounts += 1
timestamp = int(time.time() * 1000)
print(f"accounts {accounts} {timestamp}")
print(f"ci_accounts {ci_accounts} {timestamp}")
if __name__ == "__main__":
main()

View File

@@ -4,16 +4,22 @@
import json
import random
import secrets
import string
from chatmaild.config import read_config, Config
CONFIG_PATH = "/usr/local/lib/chatmaild/chatmail.ini"
ALPHANUMERIC = string.ascii_lowercase + string.digits
ALPHANUMERIC_PUNCT = string.ascii_letters + string.digits + string.punctuation
def create_newemail_dict(config: Config):
alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
user = "".join(random.choices(alphanumeric, k=config.username_min_length))
password = "".join(random.choices(alphanumeric, k=config.password_min_length + 3))
user = "".join(random.choices(ALPHANUMERIC, k=config.username_min_length))
password = "".join(
secrets.choice(ALPHANUMERIC_PUNCT)
for _ in range(config.password_min_length + 3)
)
return dict(email=f"{user}@{config.mail_domain}", password=f"{password}")

View File

@@ -24,7 +24,7 @@ def test_read_config_testrun(make_config):
assert config.postfix_reinject_port == 10025
assert config.max_user_send_per_minute == 60
assert config.max_mailbox_size == "100M"
assert config.delete_mails_after == "40d"
assert config.delete_mails_after == "40"
assert config.username_min_length == 9
assert config.username_max_length == 9
assert config.password_min_length == 9

View File

@@ -0,0 +1,16 @@
from chatmaild.metrics import main
def test_main(tmp_path, capsys):
for x in ("ci-asllkj", "ac_12l3kj", "qweqwe", "ci-l1k2j31l2k3"):
tmp_path.joinpath(x).mkdir()
main(tmp_path)
out, _ = capsys.readouterr()
d = {}
for line in out.split("\n"):
if line.strip():
name, num, _ = line.split()
d[name] = int(num)
assert d["accounts"] == 4
assert d["ci_accounts"] == 3

View File

@@ -84,14 +84,28 @@ def _install_remote_venv_with_chatmaild(config) -> None:
],
)
files.template(
src=importlib.resources.files(__package__).joinpath("metrics.cron.j2"),
dest="/etc/cron.d/chatmail-metrics",
user="root",
group="root",
mode="644",
config={
"mail_domain": config.mail_domain,
"execpath": f"{remote_venv_dir}/bin/chatmail-metrics",
},
)
# install systemd units
for fn in (
"doveauth",
"filtermail",
"echobot",
):
params = dict(
execpath=f"{remote_venv_dir}/bin/{fn}",
config_path=remote_chatmail_inipath,
remote_venv_dir=remote_venv_dir,
)
source_path = importlib.resources.files("chatmaild").joinpath(f"{fn}.service.f")
content = source_path.read_text().format(**params).encode()
@@ -112,7 +126,7 @@ def _install_remote_venv_with_chatmaild(config) -> None:
)
def _configure_opendkim(domain: str, dkim_selector: str) -> bool:
def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
"""Configures OpenDKIM"""
need_restart = False
@@ -350,20 +364,22 @@ def check_config(config):
if mail_domain != "testrun.org" and not mail_domain.endswith(".testrun.org"):
blocked_words = "merlinux schmieder testrun.org".split()
for value in config.__dict__.values():
if any(x in value for x in blocked_words):
if any(x in str(value) for x in blocked_words):
raise ValueError(
f"please set your own privacy contacts/addresses in {config._inipath}"
)
return config
def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> None:
def deploy_chatmail(config_path: Path) -> None:
"""Deploy a chat-mail instance.
:param mail_domain: domain part of your future email addresses
:param mail_server: the DNS name under which your mail server is reachable
:param dkim_selector:
:param config_path: path to chatmail.ini
"""
config = read_config(config_path)
check_config(config)
mail_domain = config.mail_domain
from .www import build_webpages
apt.update(name="apt update", cache_time=24 * 3600)
@@ -393,7 +409,7 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
)
# Deploy acmetool to have TLS certificates.
deploy_acmetool(nginx_hook=True, domains=[mail_server, f"mta-sts.{mail_server}"])
deploy_acmetool(nginx_hook=True, domains=[mail_domain, f"mta-sts.{mail_domain}"])
apt.packages(
name="Install Postfix",
@@ -423,11 +439,7 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
packages=["fcgiwrap"],
)
pkg_root = importlib.resources.files(__package__)
chatmail_ini = pkg_root.joinpath("../../../chatmail.ini").resolve()
config = read_config(chatmail_ini)
check_config(config)
www_path = pkg_root.joinpath("../../../www").resolve()
www_path = importlib.resources.files(__package__).joinpath("../../../www").resolve()
build_dir = www_path.joinpath("build")
src_dir = www_path.joinpath("src")
@@ -438,7 +450,7 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
debug = False
dovecot_need_restart = _configure_dovecot(config, debug=debug)
postfix_need_restart = _configure_postfix(config, debug=debug)
opendkim_need_restart = _configure_opendkim(mail_domain, dkim_selector)
opendkim_need_restart = _configure_opendkim(mail_domain)
mta_sts_need_restart = _install_mta_sts_daemon()
nginx_need_restart = _configure_nginx(mail_domain)

View File

@@ -52,8 +52,8 @@ def run_cmd(args, out):
"""Deploy chatmail services on the remote server."""
env = os.environ.copy()
env["CHATMAIL_DOMAIN"] = args.config.mail_domain
deploy_path = "cmdeploy/src/cmdeploy/deploy.py"
env["CHATMAIL_INI"] = args.inipath
deploy_path = importlib.resources.files(__package__).joinpath("deploy.py").resolve()
pyinf = "pyinfra --dry" if args.dry_run else "pyinfra"
cmd = f"{pyinf} --ssh-user root {args.config.mail_domain} {deploy_path}"
out.check_call(cmd, env=env)

View File

@@ -1,18 +1,16 @@
import os
import importlib.resources
import pyinfra
from cmdeploy import deploy_chatmail
def main():
mail_domain = os.getenv("CHATMAIL_DOMAIN")
mail_server = os.getenv("CHATMAIL_SERVER", mail_domain)
dkim_selector = os.getenv("CHATMAIL_DKIM_SELECTOR", "dkim")
config_path = os.getenv(
"CHATMAIL_INI",
importlib.resources.files("cmdeploy").joinpath("../../../chatmail.ini"),
)
assert mail_domain
assert mail_server
assert dkim_selector
deploy_chatmail(mail_domain, mail_server, dkim_selector)
deploy_chatmail(config_path)
if pyinfra.is_cli:

View File

@@ -1,4 +1,10 @@
2 0 * * * dovecot doveadm expunge -A SEEN BEFORE {{ config.delete_mails_after }} INBOX
2 0 * * * dovecot doveadm expunge -A SEEN BEFORE {{ config.delete_mails_after }} Deltachat
2 0 * * * dovecot doveadm expunge -A SEEN BEFORE {{ config.delete_mails_after }} Trash
2 30 * * * dovecot doveadm purge -A
# delete all mails after {{ config.delete_mails_after }} days, in the Inbox
2 0 * * * dovecot find /home/vmail/mail/{{ config.mail_domain }}/*/cur -mtime +{{ config.delete_mails_after }} -type f -delete
# or in any IMAP subfolder
2 0 * * * dovecot find /home/vmail/mail/{{ config.mail_domain }}/*/.*/cur -mtime +{{ config.delete_mails_after }} -type f -delete
# even if they are unseen
2 0 * * * dovecot find /home/vmail/mail/{{ config.mail_domain }}/*/new -mtime +{{ config.delete_mails_after }} -type f -delete
2 0 * * * dovecot find /home/vmail/mail/{{ config.mail_domain }}/*/.*/new -mtime +{{ config.delete_mails_after }} -type f -delete
# or only temporary (but then they shouldn't be around after {{ config.delete_mails_after }} days anyway).
2 0 * * * dovecot find /home/vmail/mail/{{ config.mail_domain }}/*/tmp -mtime +{{ config.delete_mails_after }} -type f -delete
2 0 * * * dovecot find /home/vmail/mail/{{ config.mail_domain }}/*/.*/tmp -mtime +{{ config.delete_mails_after }} -type f -delete

View File

@@ -0,0 +1 @@
*/5 * * * * root {{ config.execpath }} /home/vmail/mail/{{ config.mail_domain }} >/var/www/html/metrics

View File

@@ -41,6 +41,10 @@ http {
try_files $uri $uri/ =404;
}
location /metrics {
default_type text/plain;
}
# add cgi-bin support
include /usr/share/doc/fcgiwrap/examples/nginx.conf;
}

View File

@@ -36,29 +36,6 @@ def build_webpages(src_dir, build_dir, config):
print(traceback.format_exc())
def timespan_to_english(timespan):
val = int(timespan[:-1])
c = timespan[-1].lower()
match c:
case "y":
return f"{val} years"
case "m":
return f"{val} months"
case "w":
return f"{val} weeks"
case "d":
return f"{val} days"
case "h":
return f"{val} hours"
case "c":
return f"{val} seconds"
case _:
raise ValueError(
c
+ " is not a valid time unit. Try [y]ears, [w]eeks, [d]ays, or [h]ours"
)
def int_to_english(number):
if number >= 0 and number <= 12:
a = [
@@ -104,9 +81,6 @@ def _build_webpages(src_dir, build_dir, config):
render_vars["password_min_length"] = int_to_english(
config.password_min_length
)
render_vars["delete_mails_after"] = timespan_to_english(
config.delete_mails_after
)
target = build_dir.joinpath(path.stem + ".html")
# recursive jinja2 rendering

View File

@@ -14,7 +14,6 @@ Welcome to instant, interoperable and [privacy-preserving](privacy.html) messagi
💬 **Start** chatting with any Delta Chat contacts using [QR invite codes](https://delta.chat/en/help#howtoe2ee)
<div class="experimental">Note: this is an experimental service</div>
{% if config.mail_domain != "nine.testrun.org" %}
<div class="experimental">Note: this is only a temporary development chatmail service</div>
{% endif %}

View File

@@ -3,6 +3,11 @@
## More information
`nine.testrun.org` provides a low-maintenance, resource efficient and
interoperable e-mail service for everyone. What's behind a `chatmail` is
effectively a normal e-mail address just like any other but optimized
for the usage in chats, especially DeltaChat.
### Choosing a chatmail address instead of using a random one
In the Delta Chat account setup
@@ -37,7 +42,7 @@ The first login sets your password.
- You may send up to {{ config.max_user_send_per_minute }} messages per minute.
- Seen messages are removed {{ delete_mails_after }} after arriving on the server.
- Messages are unconditionally removed {{ config.delete_mails_after }} days after arriving on the server.
- You can store up to [{{ config.max_mailbox_size }} messages on the server](https://delta.chat/en/help#what-happens-if-i-turn-on-delete-old-messages-from-server).

View File

@@ -62,7 +62,7 @@ we process the following data and details:
Creating an account happens in one of two ways on our mail servers:
- with a QR invitation token
which is scanned using the DeltaChat app
which is scanned using the Delta Chat app
and then the account is created.
- by letting Delta Chat otherwise create an account