Compare commits

..

4 Commits

Author SHA1 Message Date
holger krekel
009f549619 document some attributes in chatmail.ini 2023-12-09 01:20:17 +01:00
holger krekel
99d36235fe get passthrough_recipients list from config 2023-12-09 01:07:37 +01:00
holger krekel
b52a8c969f various fixes 2023-12-09 00:22:58 +01:00
holger krekel
8520a9d8f2 introduce basic config file 2023-12-08 21:56:15 +01:00
41 changed files with 500 additions and 707 deletions

View File

@@ -6,35 +6,26 @@ on:
jobs:
tox:
name: isolated chatmaild tests
name: chatmail tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: run chatmaild tests
working-directory: chatmaild
run: pipx run tox
- name: run deploy-chatmail offline tests
working-directory: deploy-chatmail
run: pipx run tox
- name: run deploy-chatmail offline tests
working-directory: deploy-chatmail
run: pipx run tox
scripts:
name: deploy-chatmail tests
name: chatmail script invocations
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: initenv
run: scripts/initenv.sh
- name: append venv/bin to PATH
run: echo venv/bin >>$GITHUB_PATH
- name: run formatting checks
run: cmdeploy fmt -v
- name: run deploy-chatmail offline tests
run: pytest tests
- name: initialize with chatmail domain
run: cmdeploy init chat.example.org
# all other cmdeploy commands require a staging server
# see https://github.com/deltachat/chatmail/issues/100
- name: run init.sh
run: ./scripts/init.sh
- name: run test.sh
run: ./scripts/test.sh

4
.gitignore vendored
View File

@@ -3,8 +3,10 @@ __pycache__/
*.py[cod]
*$py.class
*.swp
www/privacy.html*
www/index.html*
www/info.html*
*qr-*.png
chatmail.ini
# C extensions

105
README.md
View File

@@ -1,9 +1,9 @@
<img width="800px" src="www/src/collage-top.png"/>
# Chatmail services optimized for Delta Chat apps
# Chatmail instances optimized for Delta Chat apps
This repository helps to setup a ready-to-use chatmail server
This repository helps to setup a ready-to-use chatmail instance
comprised of a minimal setup of the battle-tested
[postfix smtp](https://www.postfix.org) and [dovecot imap](https://www.dovecot.org) services.
@@ -13,83 +13,33 @@ for use by [Delta Chat apps](https://delta.chat).
Chatmail accounts are automatically created by a first login,
after which the initially specified password is required for using them.
## Deploying your own chatmail server
## Getting Started deploying your own chatmail instance
We subsequently use `CHATMAIL_DOMAIN` as a placeholder for your fully qualified
DNS domain name (FQDN), for example `chat.example.org`.
1. Prepare your local (presumably Linux) system:
1. Setup DNS `A` and `AAAA` records for your `CHATMAIL_DOMAIN`.
Verify that DNS is set and SSH root login works:
scripts/init.sh
```
ssh root@CHATMAIL_DOMAIN
```
2. Setup a domain with `A` and `AAAA` records for your chatmail server.
2. Install the `cmdeploy` command in a virtualenv
3. Set environment variable to the chatmail domain you want to setup:
```
source scripts/initenv.sh
```
3. Create chatmail configuration file `chatmail.ini`:
export CHATMAIL_DOMAIN=c1.testrun.org # replace with your host
```
cmdeploy init CHATMAIL_DOMAIN
```
4. Fill in privacy contact data into the `chatmail.ini` file
4. Deploy to the remote chatmail server:
5. Deploy the chat mail instance to your chatmail server:
```
cmdeploy run
```
scripts/deploy.sh
5. To output a DNS zone file from which you can transfer DNS records
to your DNS provider:
This script remotely sets up packages and configures the chatmail provider.
```
cmdeploy dns
```
6. To check status of your remotely running chatmail service:
```
cmdeploy status
```
7. To test your chatmail service:
```
cmdeploy test
```
8. To benchmark your chatmail service:
```
cmdeploy bench
```
### Refining the web pages
```
cmdeploy webdev
```
This starts a local live development cycle for chatmail Web pages:
- uses the `www/src/page-layout.html` file for producing static
HTML pages from `www/src/*.md` files
- continously builds the web presence reading files from `www/src` directory
and generating html files and copying assets to the `www/build` directory.
- Starts a browser window automatically where you can "refresh" as needed.
6. Run `scripts/generate-dns-zone.sh` and
transfer the generated DNS records at your DNS provider
### Home page and getting started for users
`cmdeploy run` sets up mail services,
and also creates default static Web pages and deploys them:
The `deploy.sh` script deploys
- a default `index.html` along with a QR code that users can click to
create accounts on your chatmail provider,
@@ -98,10 +48,31 @@ and also creates default static Web pages and deploys them:
- a default `policy.html` that is linked from the home page.
All `.html` files are generated
by the according markdown `.md` file in the `www/src` directory.
All files are generated by the according markdown `.md` file in the `www` directory.
### Refining the web pages
The `scripts/webdev.sh` script supports live development of the chatmail web presence:
```
scripts/init.sh # to locally initialize python virtual environments etc.
scripts/webdev.sh
```
- uses the `www/src/page-layout.html` file for producing html documents
from `www/src/*.md` files.
- continously builds the web presence reading files from `www/src` directory
and generating html files and copying assets to the `www/build` directory.
- Starts a browser window automatically where you can "refresh" as needed.
Note that this script is not needed for running `scripts/deploy.sh"
which deploys the whole chatmail setup remotely.
The code that generates the web pages is identical
which means that `webdev.sh` gives a pretty good preview.
### Ports
Postfix listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).

32
chatmail.ini Normal file
View File

@@ -0,0 +1,32 @@
[params]
# how many mails a user can send out per minute
max_user_send_per_minute = 60
# list of e-mail recipients for which to accept outbound un-encrypted mails
passthrough_recipients = privacy@testrun.org xstore@testrun.org
# where the filtermail SMTP service listens
filtermail_smtp_port = 10080
# to which port to re-inject messages after they passed filtermail
postfix_reinject_port = 10025
[privacy:testrun]
# the settings in this section are only applied
# if the instantiated mail domain shell-matches the 'domain' setting
domain = *.testrun.org
privacy_postal =
Merlinux GmbH, Represented by the managing director H. Krekel,
Reichgrafen Str. 20, 79102 Freiburg, Germany
privacy_mail = delta-privacy@merlinux.eu
privacy_pdo =
Prof. Dr. Fabian Schmieder, lexICT UG (limited), Ostfeldstr. 49, 30559 Hannover.
You can contact him at *delta-privacy@merlinux.eu* (Keyword: DPO)
privacy_supervisor =
State Commissioner for Data Protection and Freedom of Information of
Baden-Württemberg in 70173 Stuttgart, Germany.

View File

@@ -1,4 +0,0 @@
include src/chatmaild/*.f
include src/chatmaild/ini/*.ini.f
include src/chatmaild/ini/*.ini
include src/chatmaild/tests/mail-data/*

View File

@@ -1,5 +1,5 @@
[build-system]
requires = ["setuptools>=61"]
requires = ["setuptools>=45"]
build-backend = "setuptools.build_meta"
[project]
@@ -10,19 +10,10 @@ dependencies = [
"iniconfig",
]
[tool.setuptools]
include-package-data = true
[tool.setuptools.packages.find]
where = ['src']
[project.scripts]
doveauth = "chatmaild.doveauth:main"
filtermail = "chatmaild.filtermail:main"
[project.entry-points.pytest11]
"chatmaild.testplugin" = "chatmaild.tests.plugin"
[tool.pytest.ini_options]
addopts = "-v -ra --strict-markers"
log_format = "%(asctime)s %(levelname)s %(message)s"
@@ -46,7 +37,8 @@ commands =
ruff src/
[testenv]
passenv = CHATMAIL_DOMAIN
deps = pytest
pdbpp
commands = pytest -v -rsXx {posargs}
commands = pytest -v -rsXx {posargs: ../tests/chatmaild}
"""

View File

@@ -1,51 +1,41 @@
from pathlib import Path
from fnmatch import fnmatch
import iniconfig
system_mailname_path = Path("/etc/mailname")
def read_config(inipath):
cfg = iniconfig.IniConfig(inipath)
return Config(inipath, params=cfg.sections["params"])
def read_config(inipath, mailname=None):
if mailname is None:
with open(system_mailname_path) as f:
mailname = f.read().strip()
ini = iniconfig.IniConfig(inipath)
privacy = {}
for section in ini:
if section.name.startswith("privacy:"):
domain = section["domain"]
if fnmatch(mailname, domain):
privacy = section
break
return Config(inipath, mailname, privacy, params=ini.sections["params"])
class Config:
def __init__(self, inipath, params):
def __init__(self, inipath, mailname, privacy, params):
self._inipath = inipath
self.mailname = self.mail_domain = params["mailname"]
self.mailname = mailname
self.privacy_postal = privacy.get("privacy_postal")
self.privacy_mail = privacy.get("privacy_mail")
self.privacy_pdo = privacy.get("privacy_pdo")
self.privacy_supervisor = privacy.get("privacy_supervisor")
self.max_user_send_per_minute = int(params["max_user_send_per_minute"])
self.filtermail_smtp_port = int(params["filtermail_smtp_port"])
self.postfix_reinject_port = int(params["postfix_reinject_port"])
self.passthrough_recipients = params["passthrough_recipients"].split()
self.privacy_postal = params.get("privacy_postal")
self.privacy_mail = params.get("privacy_mail")
self.privacy_pdo = params.get("privacy_pdo")
self.privacy_supervisor = params.get("privacy_supervisor")
def _getbytefile(self):
return open(self._inipath, "rb")
def write_initial_config(inipath, mailname):
from importlib.resources import files
inidir = files(__package__).joinpath("ini")
content = inidir.joinpath("chatmail.ini.f").read_text().format(mailname=mailname)
if mailname.endswith(".testrun.org"):
override_inipath = inidir.joinpath("override-testrun.ini")
privacy = iniconfig.IniConfig(override_inipath)["privacy"]
lines = []
for line in content.split("\n"):
for key, value in privacy.items():
value_lines = value.strip().split("\n")
if not line.startswith(f"{key} =") or not value_lines:
continue
if len(value_lines) == 1:
lines.append(f"{key} = {value}")
else:
lines.append(f"{key} =")
for vl in value_lines:
lines.append(f" {vl}")
break
else:
lines.append(line)
content = "\n".join(lines)
inipath.write_text(content)

View File

@@ -1,5 +1,5 @@
[Unit]
Description=Chatmail dict authentication proxy for dovecot
Description=Dict authentication proxy for dovecot
[Service]
ExecStart={execpath} /run/dovecot/doveauth.socket vmail /home/vmail/passdb.sqlite

View File

@@ -7,6 +7,7 @@ from email.parser import BytesParser
from email import policy
from email.utils import parseaddr
from aiosmtpd.smtp import SMTP
from aiosmtpd.controller import Controller
from smtplib import SMTP as SMTPClient
@@ -126,10 +127,7 @@ class BeforeQueueHandler:
is_outgoing = recipient_domain != envelope_from_domain
if is_outgoing and not mail_encrypted:
is_securejoin = message.get("secure-join") in [
"vc-request",
"vg-request",
]
is_securejoin = message.get("secure-join") in ["vc-request", "vg-request"]
if not is_securejoin:
return f"500 Invalid unencrypted mail to <{recipient}>"

View File

@@ -1,33 +0,0 @@
[params]
# mail domain (MUST be set to fully qualified chat mail domain)
mailname = {mailname}
#
# If you only do private test deploys, you don't need to modify any settings below
#
# how many mails a user can send out per minute
max_user_send_per_minute = 60
# list of e-mail recipients for which to accept outbound un-encrypted mails
passthrough_recipients =
# where the filtermail SMTP service listens
filtermail_smtp_port = 10080
# postfix accepts on the localhost reinject SMTP port
postfix_reinject_port = 10025
# postal address of privacy contact
privacy_postal =
# email address of privacy contact
privacy_mail =
# postal address of the privacy data officer
privacy_pdo =
# postal address of the privacy supervisor
privacy_supervisor =

View File

@@ -1,16 +0,0 @@
[privacy]
passthrough_recipients = privacy@testrun.org
privacy_postal =
Merlinux GmbH, Represented by the managing director H. Krekel,
Reichgrafen Str. 20, 79102 Freiburg, Germany
privacy_mail = privacy@testrun.org
privacy_pdo =
Prof. Dr. Fabian Schmieder, lexICT UG (limited), Ostfeldstr. 49, 30559 Hannover.
You can contact him at *delta-privacy@merlinux.eu* (Keyword: DPO)
privacy_supervisor =
State Commissioner for Data Protection and Freedom of Information of
Baden-Württemberg in 70173 Stuttgart, Germany.

View File

@@ -1,68 +0,0 @@
import random
import importlib.resources
import itertools
from email.parser import BytesParser
from email import policy
import pytest
from chatmaild.database import Database
from chatmaild.config import read_config, write_initial_config
@pytest.fixture
def make_config(tmp_path):
inipath = tmp_path.joinpath("chatmail.ini")
def make_conf(mailname):
write_initial_config(inipath, mailname=mailname)
return read_config(inipath)
return make_conf
@pytest.fixture
def example_config(make_config):
return make_config("chat.example.org")
@pytest.fixture
def maildomain(example_config):
return example_config.mailname
@pytest.fixture
def gencreds(maildomain):
count = itertools.count()
next(count)
def gen(domain=None):
domain = domain if domain else maildomain
while 1:
num = next(count)
alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
user = "".join(random.choices(alphanumeric, k=10))
user = f"ac{num}_{user}"[:9]
password = "".join(random.choices(alphanumeric, k=12))
yield f"{user}@{domain}", f"{password}"
return lambda domain=None: next(gen(domain))
@pytest.fixture()
def db(tmpdir):
db_path = tmpdir / "passdb.sqlite"
print("database path:", db_path)
return Database(db_path)
@pytest.fixture
def maildata(request):
datadir = importlib.resources.files(__package__).joinpath("mail-data")
assert datadir.exists(), datadir
def maildata(name, from_addr, to_addr):
data = datadir.joinpath(name).read_text()
text = data.format(from_addr=from_addr, to_addr=to_addr)
return BytesParser(policy=policy.default).parsebytes(text.encode())
return maildata

View File

@@ -1,27 +0,0 @@
from chatmaild.config import read_config
def test_read_config_basic(make_config):
config = make_config("chat.example.org")
assert config.mailname == "chat.example.org"
assert not config.privacy_supervisor and not config.privacy_mail
assert not config.privacy_pdo and not config.privacy_postal
inipath = config._inipath
inipath.write_text(inipath.read_text().replace("60", "37"))
config = read_config(inipath)
assert config.max_user_send_per_minute == 37
assert config.mailname == "chat.example.org"
def test_read_config_testrun(make_config):
config = make_config("something.testrun.org")
assert config.mailname == "something.testrun.org"
assert len(config.privacy_postal.split("\n")) > 1
assert len(config.privacy_supervisor.split("\n")) > 1
assert len(config.privacy_pdo.split("\n")) > 1
assert config.privacy_mail == "privacy@testrun.org"
assert config.filtermail_smtp_port == 10080
assert config.postfix_reinject_port == 10025
assert config.max_user_send_per_minute == 60
assert config.passthrough_recipients

View File

@@ -1,5 +1,5 @@
[build-system]
requires = ["setuptools>=68"]
requires = ["setuptools>=45"]
build-backend = "setuptools.build_meta"
[project]
@@ -10,22 +10,24 @@ dependencies = [
"pillow",
"qrcode",
"markdown",
"pytest",
"setuptools>=68",
"termcolor",
"build",
"tox",
"ruff",
"black",
"pytest",
"pytest-xdist",
]
[project.scripts]
cmdeploy = "deploy_chatmail.cmdeploy:main"
[project.entry-points.pytest11]
"chatmaild.testplugin" = "chatmaild.tests.plugin"
[tool.pytest.ini_options]
addopts = "-v -ra --strict-markers"
[tool.tox]
legacy_tox_ini = """
[tox]
isolated_build = true
envlist = lint
[testenv:lint]
skipdist = True
skip_install = True
deps =
ruff
black
commands =
black --quiet --check --diff src/
ruff src/
"""

View File

@@ -14,7 +14,8 @@ from pyinfra.facts.files import File
from pyinfra.facts.systemd import SystemdEnabled
from .acmetool import deploy_acmetool
from chatmaild.config import read_config, Config
import chatmaild.filtermail
from chatmaild.config import read_config
def _build_chatmaild(dist_dir) -> None:
@@ -195,7 +196,7 @@ def _install_mta_sts_daemon() -> bool:
server.shell(
name="install postfix-mta-sts-resolver with pip",
commands=[
"python3 -m virtualenv /usr/local/lib/postfix-mta-sts-resolver",
"python3 -m venv /usr/local/lib/postfix-mta-sts-resolver",
"/usr/local/lib/postfix-mta-sts-resolver/bin/pip install postfix-mta-sts-resolver",
],
)
@@ -215,7 +216,7 @@ def _install_mta_sts_daemon() -> bool:
return need_restart
def _configure_postfix(config: Config, debug: bool = False) -> bool:
def _configure_postfix(config: chatmaild.config.Config, debug: bool = False) -> bool:
"""Configures Postfix SMTP server."""
need_restart = False
@@ -412,7 +413,7 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
pkg_root = importlib.resources.files(__package__)
chatmail_ini = pkg_root.joinpath("../../../chatmail.ini").resolve()
config = read_config(chatmail_ini)
config = read_config(chatmail_ini, mailname=mail_domain)
check_config(config)
www_path = pkg_root.joinpath("../../../www").resolve()

View File

@@ -1,12 +0,0 @@
{chatmail_domain}. MX 10 {chatmail_domain}.
_submission._tcp.{chatmail_domain}. SRV 0 1 587 {chatmail_domain}.
_submissions._tcp.{chatmail_domain}. SRV 0 1 465 {chatmail_domain}.
_imap._tcp.{chatmail_domain}. SRV 0 1 143 {chatmail_domain}.
_imaps._tcp.{chatmail_domain}. SRV 0 1 993 {chatmail_domain}.
{chatmail_domain}. IN CAA 128 issue "letsencrypt.org;accounturi={acme_account_url}"
{chatmail_domain}. TXT "v=spf1 a:{chatmail_domain} -all"
_dmarc.{chatmail_domain}. TXT "v=DMARC1;p=reject;rua=mailto:{email};ruf=mailto:{email};fo=1;adkim=r;aspf=r"
_mta-sts.{chatmail_domain}. IN TXT "v=STSv1; id={sts_id}"
mta-sts.{chatmail_domain}. IN CNAME {chatmail_domain}.
_smtp._tls.{chatmail_domain}. IN TXT "v=TLSRPTv1;rua=mailto:{email}"
{dkim_entry}

View File

@@ -1,301 +0,0 @@
"""
Provides the `cmdeploy` entry point function,
along with command line option and subcommand parsing.
"""
import argparse
import datetime
import shutil
import subprocess
import importlib.resources
import importlib.util
import os
import sys
from pathlib import Path
from termcolor import colored
from chatmaild.config import read_config, write_initial_config
#
# cmdeploy sub commands and options
#
def init_cmd_options(parser):
parser.add_argument(
"chatmail_domain",
action="store",
help="fully qualified DNS domain name for your chatmail instance",
)
def init_cmd(args, out):
"""Initialize chatmail config file."""
if args.inipath.exists():
out.red(f"Path exists, not modifying: {args.inipath}")
raise SystemExit(1)
write_initial_config(args.inipath, args.chatmail_domain)
out.green(f"created config file for {args.chatmail_domain} in {args.inipath}")
def run_cmd_options(parser):
parser.add_argument(
"--dry-run",
dest="dry_run",
action="store_true",
help="don't actually modify the server",
)
def run_cmd(args, out):
"""Deploy chatmail services on the remote server."""
env = os.environ.copy()
env["CHATMAIL_DOMAIN"] = args.config.mailname
deploypy = "deploy-chatmail/src/deploy_chatmail/deploy.py"
pyinf = "pyinfra --dry" if args.dry_run else "pyinfra"
cmd = f"{pyinf} --ssh-user root {args.config.mailname} {deploypy}"
out.check_call(cmd, env=env)
def dns_cmd(args, out):
"""Generate dns zone file."""
template = importlib.resources.files(__package__).joinpath("chatmail.zone.f")
ssh = f"ssh root@{args.config.mailname}"
def read_dkim_entries(entry):
lines = []
for line in entry.split("\n"):
if line.startswith(";") or not line.strip():
continue
line = line.replace("\t", " ")
lines.append(line)
return "\n".join(lines)
acme_account_url = out.shell_output(f"{ssh} -- acmetool account-url")
dkim_entry = read_dkim_entries(out.shell_output(f"{ssh} -- opendkim-genzone -F"))
out(
f"[writing {args.config.mailname} zone data (using space as separator) to stdout output]",
green=True,
)
print(
template.read_text()
.format(
acme_account_url=acme_account_url,
email=f"root@{args.config.mailname}",
sts_id=datetime.datetime.now().strftime("%Y%m%d%H%M"),
chatmail_domain=args.config.mailname,
dkim_entry=dkim_entry,
)
.strip()
)
def status_cmd(args, out):
"""Display status for online chatmail instance."""
ssh = f"ssh root@{args.config.mailname}"
out.green(f"chatmail domain: {args.config.mailname}")
if args.config.privacy_mail:
out.green("privacy settings: present")
else:
out.red("no privacy settings")
s1 = "systemctl --type=service --state=running"
for line in out.shell_output(f"{ssh} -- {s1}").split("\n"):
if line.startswith(" "):
print(line)
def test_cmd(args, out):
"""Run local and online tests for chatmail deployment.
This will automatically pip-install 'deltachat' if it's not available.
"""
x = importlib.util.find_spec("deltachat")
if x is None:
out.check_call(f"{sys.executable} -m pip install deltachat")
pytest_path = shutil.which("pytest")
ret = out.run_ret(
[pytest_path, "tests/", "-n4", "-rs", "-x", "-vrx", "--durations=5"]
)
return ret
def fmt_cmd_options(parser):
parser.add_argument(
"--verbose",
"-v",
dest="verbose",
action="store_true",
help="provide information on invocations",
)
parser.add_argument(
"--check",
"-c",
action="store_true",
help="only check but don't fix problems",
)
def fmt_cmd(args, out):
"""Run formattting fixes (fuff and black) on all chatmail source code."""
chatmaild = importlib.resources.files("chatmaild")
deploy_chatmail = importlib.resources.files("deploy_chatmail")
tests = deploy_chatmail.joinpath("../../../tests")
sources = list(str(x) for x in [chatmaild, deploy_chatmail, tests])
black_args = [shutil.which("black")]
ruff_args = [shutil.which("ruff")]
if args.check:
black_args.append("--check")
else:
ruff_args.append("--fix")
if not args.verbose:
black_args.append("-q")
ruff_args.append("-q")
black_args.extend(sources)
ruff_args.extend(sources)
out.check_call(" ".join(black_args), quiet=not args.verbose)
out.check_call(" ".join(ruff_args), quiet=not args.verbose)
return 0
def bench_cmd(args, out):
"""Run benchmarks against an online chatmail instance."""
pytest_path = shutil.which("pytest")
benchmark = "tests/online/benchmark.py"
subprocess.check_call([pytest_path, benchmark, "-vrx"])
def webdev_cmd(args, out):
"""Run local web development loop for static web pages."""
from .www import main
main()
#
# Parsing command line options and starting commands
#
class Out:
"""Convenience output printer providing coloring."""
def red(self, msg, file=sys.stderr):
print(colored(msg, "red"), file=file)
def green(self, msg, file=sys.stderr):
print(colored(msg, "green"), file=file)
def __call__(self, msg, red=False, green=False, file=sys.stdout):
color = "red" if red else ("green" if green else None)
print(colored(msg, color), file=file)
def shell_output(self, arg):
self(f"[$ {arg}]", file=sys.stderr)
return subprocess.check_output(arg, shell=True).decode()
def check_call(self, arg, env=None, quiet=False):
if not quiet:
self(f"[$ {arg}]", file=sys.stderr)
return subprocess.check_call(arg, shell=True, env=env)
def run_ret(self, args, env=None, quiet=False):
if not quiet:
cmdstring = " ".join(args)
self(f"[$ {cmdstring}]", file=sys.stderr)
proc = subprocess.run(args, env=env)
return proc.returncode
def add_config_option(parser):
parser.add_argument(
"--config",
dest="inipath",
action="store",
default=Path("chatmail.ini"),
type=Path,
help="path to the chatmail.ini file",
)
def add_subcommand(subparsers, func):
name = func.__name__
assert name.endswith("_cmd")
name = name[:-4]
doc = func.__doc__.strip()
help = doc.split("\n")[0].strip(".")
p = subparsers.add_parser(name, description=doc, help=help)
p.set_defaults(func=func)
add_config_option(p)
return p
description = """
Setup your chatmail server configuration and
deploy it via SSH to your remote location.
"""
def get_parser():
"""Return an ArgumentParser for the 'cmdeploy' CLI"""
parser = argparse.ArgumentParser(description=description.strip())
subparsers = parser.add_subparsers(title="subcommands")
# find all subcommands in the module namespace
glob = globals()
for name, func in glob.items():
if name.endswith("_cmd"):
subparser = add_subcommand(subparsers, func)
addopts = glob.get(name + "_options")
if addopts is not None:
addopts(subparser)
return parser
def main(args=None):
"""Provide main entry point for 'xdcget' CLI invocation."""
parser = get_parser()
args = parser.parse_args(args=args)
if not hasattr(args, "func"):
return parser.parse_args(["-h"])
out = Out()
kwargs = {}
if args.func.__name__ not in ("init_cmd", "fmt_cmd"):
if not args.inipath.exists():
out.red(f"expecting {args.inipath} to exist, run init first?")
raise SystemExit(1)
try:
args.config = read_config(args.inipath)
except Exception as ex:
out.red(ex)
raise SystemExit(1)
try:
res = args.func(args, out, **kwargs)
if res is None:
res = 0
return res
except KeyboardInterrupt:
out.red("KeyboardInterrupt")
sys.exit(130)
if __name__ == "__main__":
main()

View File

@@ -66,12 +66,12 @@ def _build_webpages(src_dir, build_dir, config):
def main():
chatmail_domain = "example.testrun.org"
path = importlib.resources.files(__package__)
reporoot = path.joinpath("../../../").resolve()
inipath = reporoot.joinpath("chatmail.ini")
config = read_config(inipath)
config.webdev = True
assert config.mailname
config = read_config(inipath, mailname=chatmail_domain)
config["webdev"] = True
www_path = reporoot.joinpath("www")
src_path = www_path.joinpath("src")
stats = None

65
plan.txt Normal file
View File

@@ -0,0 +1,65 @@
# Chat-mail server development (up until Oct 18th)
## Dovecot goals/steps
- automatic expiry of messages older than M days
- also expunge unread messages
- limit: configure max-connections per account
## nami: send out rate limit / rspamd
- basic outgoing send rate/limits (depending on "account-rating")
use rspamd in a minimal way, check support dkim-signing
(including an online test exceeding rate limit)
## doveauth questions/futures
- bcrypt-password scheme is slow: require long passwords, use faster hashing
- define user-name and password policies, and implement them
(be very restrictive at the beginning, we can relax later)
- password is part of the dictproxy-lookup key, is it safe to use auth-caching?
## How to limit creation of accounts?
attack: a 3-line bash script to fill the chatmail db with millions of unused accouts
- make it computationally expensive (somehow try to except our tests from it)
1st pass instant onboarding: create userid + cheap password -- if it fails then
2nd pass instant onboarding: create userdid + comput. expensive password
- probably also do firewall: limit number of new tcp-connections per IP address per duration
## Open/deferred questions
- automatic expiry of users that haven't logged in for N days
Is it neccessary? If all messages are gone, does the existence of
an e-mail address bother anybody?
## web page for chat-mail servers?
- documentation for users, privacy policy etc.
(probably also with provider-messages ...)
## online tests (first with plain python/pytest)
- write tests for dovecot login (exists)
- write tests for postfix logins (exists)
- write A<>B send/receive tests (exists)
## Delta Chat
1. qr code that defines access to a chatmail instance (like mailadm but without http etc.)
2. support for creating username/password and verifying login works

4
scripts/bench.sh Executable file
View File

@@ -0,0 +1,4 @@
#!/bin/bash
set -e
venv/bin/pytest tests/online/benchmark.py -vrx

8
scripts/deploy.sh Executable file
View File

@@ -0,0 +1,8 @@
#!/usr/bin/env bash
echo -----------------------------------------
echo deploying to $CHATMAIL_DOMAIN
echo -----------------------------------------
venv/bin/pyinfra --ssh-user root "$CHATMAIL_DOMAIN" \
deploy-chatmail/src/deploy_chatmail/deploy.py

23
scripts/generate-dns-zone.sh Executable file
View File

@@ -0,0 +1,23 @@
#!/bin/sh
: ${CHATMAIL_DOMAIN:=c1.testrun.org}
: ${CHATMAIL_SSH:=$CHATMAIL_DOMAIN}
set -e
SSH="ssh root@$CHATMAIL_SSH"
EMAIL="root@$CHATMAIL_DOMAIN"
ACME_ACCOUNT_URL="$($SSH -- acmetool account-url)"
cat <<EOF
$CHATMAIL_DOMAIN. MX 10 $CHATMAIL_DOMAIN.
$CHATMAIL_DOMAIN. TXT "v=spf1 a:$CHATMAIL_DOMAIN -all"
_dmarc.$CHATMAIL_DOMAIN. TXT "v=DMARC1;p=reject;rua=mailto:$EMAIL;ruf=mailto:$EMAIL;fo=1;adkim=r;aspf=r"
_submission._tcp.$CHATMAIL_DOMAIN. SRV 0 1 587 $CHATMAIL_DOMAIN.
_submissions._tcp.$CHATMAIL_DOMAIN. SRV 0 1 465 $CHATMAIL_DOMAIN.
_imap._tcp.$CHATMAIL_DOMAIN. SRV 0 1 143 $CHATMAIL_DOMAIN.
_imaps._tcp.$CHATMAIL_DOMAIN. SRV 0 1 993 $CHATMAIL_DOMAIN.
$CHATMAIL_DOMAIN. IN CAA 128 issue "letsencrypt.org;accounturi=$ACME_ACCOUNT_URL"
_mta-sts.$CHATMAIL_DOMAIN. IN TXT "v=STSv1; id=$(date -u '+%Y%m%d%H%M')"
mta-sts.$CHATMAIL_DOMAIN. IN CNAME $CHATMAIL_DOMAIN.
_smtp._tls.$CHATMAIL_DOMAIN. IN TXT "v=TLSRPTv1;rua=mailto:$EMAIL"
EOF
$SSH opendkim-genzone -F | sed 's/^;.*$//;/^$/d'

View File

@@ -0,0 +1,14 @@
import os
import time
import imaplib
domain = os.environ.get("CHATMAIL_DOMAIN", "c3.testrun.org")
print("connecting")
conn = imaplib.IMAP4_SSL(domain)
print("logging in")
conn.login(f"imapcapa", "pass")
status, res = conn.capability()
for capa in sorted(res[0].decode().split()):
print(capa)

8
scripts/init.sh Executable file
View File

@@ -0,0 +1,8 @@
#!/bin/sh
set -e
python3 -m venv venv
pip=venv/bin/pip
$pip install pyinfra pytest build 'setuptools>=68' tox
$pip install -e deploy-chatmail
$pip install -e chatmaild

View File

@@ -1,9 +0,0 @@
#!/bin/bash
set -e
python3 -m venv venv
venv/bin/pip install -e deploy-chatmail
venv/bin/pip install -e chatmaild
source venv/bin/activate
echo activated 'venv' python virtualenv environment containing "cmdeploy" tool

View File

@@ -0,0 +1,28 @@
#!/usr/bin/env python3
import os
import time
import imaplib
domain = os.environ.get("CHATMAIL_DOMAIN", "c3.testrun.org")
NUM_CONNECTIONS=10
conns = []
start = time.time()
for i in range(NUM_CONNECTIONS):
print(f"opening connection {i} to {domain}")
conn = imaplib.IMAP4_SSL(domain)
conns.append(conn)
tlsdone = time.time()
duration = tlsdone-start
print(f"{duration}: TLS connections opening TLS connections")
for i, conn in enumerate(conns):
print(f"logging into connection {i}")
conn.login(f"measure{i}", "pass")
logindone = time.time()
duration = logindone - tlsdone
print(f"{duration}: LOGINS done")

8
scripts/remote-deploy.sh Executable file
View File

@@ -0,0 +1,8 @@
#!/bin/sh
set -e
: ${CHATMAIL_DOMAIN:=c1.testrun.org}
: ${CHATMAIL_SSH:=$CHATMAIL_DOMAIN}
rsync -avz . "root@$CHATMAIL_SSH:/root/chatmail" --exclude='/.git' --filter="dir-merge,- .gitignore"
ssh "root@$CHATMAIL_SSH" "cd /root/chatmail; apt install -y python3-venv; python3 -m venv venv; venv/bin/pip install pyinfra build; venv/bin/python3 -m build -n --sdist chatmaild --outdir dist; venv/bin/pip install -e ./deploy-chatmail -e ./chatmaild; export CHATMAIL_DOMAIN=$CHATMAIL_DOMAIN; venv/bin/pyinfra @local deploy.py"

4
scripts/test.sh Executable file
View File

@@ -0,0 +1,4 @@
#!/bin/bash
venv/bin/tox -c chatmaild
venv/bin/tox -c deploy-chatmail
venv/bin/pytest tests/online -rs -vrx --durations=5 $@

9
scripts/webdev.sh Executable file
View File

@@ -0,0 +1,9 @@
#!/usr/bin/env bash
echo -----------------------------------------
echo starting local webdev
echo -----------------------------------------
venv/bin/python3 -m deploy_chatmail.www

View File

@@ -0,0 +1,88 @@
from chatmaild.config import read_config
import chatmaild.config
def test_read_config_without_mailname(tmp_path, create_ini, monkeypatch):
mailname_path = tmp_path.joinpath("mailname")
mailname_path.write_text("something.example.org")
monkeypatch.setattr(chatmaild.config, "system_mailname_path", mailname_path)
inipath = create_ini(
"""
[params]
max_user_send_per_minute = 40
filtermail_smtp_port = 9875
postfix_reinject_port = 9999
passthrough_recipients =
"""
)
config = read_config(inipath)
assert config.mailname == "something.example.org"
def test_read_config_without_privacy_policy(tmp_path, create_ini):
inipath = create_ini(
"""
[params]
max_user_send_per_minute = 40
filtermail_smtp_port = 9875
postfix_reinject_port = 9999
passthrough_recipients =
[privacy:testrun]
domain = *.example.org
"""
)
config = read_config(inipath, "something.example.org")
assert config.mailname == "something.example.org"
assert config.max_user_send_per_minute == 40
assert config.filtermail_smtp_port == 9875
assert config.postfix_reinject_port == 9999
assert config.passthrough_recipients == []
assert not config.privacy_postal
assert not config.privacy_mail
assert not config.privacy_pdo
assert not config.privacy_supervisor
def test_read_config(create_ini):
inipath = create_ini(
"""
[params]
max_user_send_per_minute = 40
filtermail_smtp_port = 10080
postfix_reinject_port = 10025
passthrough_recipients = x@example.org y@example.org
[privacy:testrun]
domain = *.testrun.org
privacy_postal =
Postal Ltd
privacy_mail = privacy@merlinux.eu
privacy_pdo =
Postal PDO
You can contact him at *delta-privacy@merlinux.eu* (Keyword: DPO)
privacy_supervisor =
line1
line2 with space
"""
)
config = read_config(inipath, "something.testrun.org")
assert config.mailname == "something.testrun.org"
assert config.filtermail_smtp_port == 10080
assert config.postfix_reinject_port == 10025
assert config.passthrough_recipients == ["x@example.org", "y@example.org"]
assert config.privacy_postal == "Postal Ltd"
assert config.privacy_mail == "privacy@merlinux.eu"
lines = config.privacy_pdo.split("\n")
assert lines[0] == "Postal PDO"
assert lines[1].startswith("You can ")
lines = config.privacy_supervisor.split("\n")
assert lines[0] == "line1"
assert lines[1] == "line2 with space"

View File

@@ -59,8 +59,8 @@ def test_handle_dovecot_request(db):
assert userdata["password"].startswith("{SHA512-CRYPT}")
def test_50_concurrent_lookups_different_accounts(db, gencreds):
num_threads = 50
def test_100_concurrent_lookups_different_accounts(db, gencreds):
num_threads = 100
req_per_thread = 5
results = queue.Queue()

View File

@@ -5,6 +5,8 @@ from chatmaild.filtermail import (
check_mdn,
)
from chatmaild.config import read_config
import pytest
@@ -15,8 +17,8 @@ def maildomain():
@pytest.fixture
def handler(make_config, maildomain):
config = make_config(maildomain)
def handler(create_ini, maildomain):
config = read_config(create_ini(), maildomain)
return BeforeQueueHandler(config)
@@ -26,36 +28,27 @@ def test_reject_forged_from(maildata, gencreds, handler):
rcpt_tos = [gencreds()[0]]
# test that the filter lets good mail through
to_addr = gencreds()[0]
env.content = maildata(
"plain.eml", from_addr=env.mail_from, to_addr=to_addr
).as_bytes()
env.content = maildata("plain.eml", from_addr=env.mail_from).as_bytes()
assert not handler.check_DATA(envelope=env)
# test that the filter rejects forged mail
env.content = maildata(
"plain.eml", from_addr="forged@c3.testrun.org", to_addr=to_addr
).as_bytes()
env.content = maildata("plain.eml", from_addr="forged@c3.testrun.org").as_bytes()
error = handler.check_DATA(envelope=env)
assert "500" in error
def test_filtermail_no_encryption_detection(maildata):
msg = maildata(
"plain.eml", from_addr="some@example.org", to_addr="other@example.org"
)
msg = maildata("plain.eml")
assert not check_encrypted(msg)
# https://xkcd.com/1181/
msg = maildata(
"fake-encrypted.eml", from_addr="some@example.org", to_addr="other@example.org"
)
msg = maildata("fake-encrypted.eml")
assert not check_encrypted(msg)
def test_filtermail_encryption_detection(maildata):
msg = maildata("encrypted.eml", from_addr="1@example.org", to_addr="2@example.org")
msg = maildata("encrypted.eml")
assert check_encrypted(msg)
# if the subject is not "..." it is not considered ac-encrypted
@@ -108,8 +101,9 @@ def test_send_rate_limiter():
def test_excempt_privacy(maildata, gencreds, handler):
from_addr = gencreds()[0]
to_addr = "privacy@testrun.org"
handler.config.passthrough_recipients = [to_addr]
false_to = "privacy@something.org"
false_to = "privacy@tstrn.org"
false_to2 = "prvcy@testrun.org"
assert to_addr in handler.config.passthrough_recipients
msg = maildata("plain.eml", from_addr, to_addr)
@@ -123,7 +117,7 @@ def test_excempt_privacy(maildata, gencreds, handler):
class env2:
mail_from = from_addr
rcpt_tos = [to_addr, false_to]
rcpt_tos = [to_addr, false_to, false_to2]
content = msg.as_bytes()
assert "500" in handler.check_DATA(envelope=env2)

View File

@@ -3,14 +3,17 @@ import io
import time
import random
import subprocess
import textwrap
import imaplib
import smtplib
import importlib.resources
import itertools
from email.parser import BytesParser
from email import policy
from pathlib import Path
import pytest
from chatmaild.database import Database
from chatmaild.config import read_config
conftestdir = Path(__file__).parent
@@ -37,22 +40,19 @@ def pytest_runtest_setup(item):
@pytest.fixture
def chatmail_config(pytestconfig):
current = basedir = Path()
while 1:
path = current.joinpath("chatmail.ini").resolve()
if path.exists():
return read_config(path)
if current == current.parent:
break
current = current.parent
pytest.skip(f"no chatmail.ini file found in {basedir} or parent dirs")
def inipath():
dpath = importlib.resources.files("chatmaild")
inipath = dpath.joinpath("../../../chatmail.ini").resolve()
assert inipath.exists()
return inipath
@pytest.fixture
def maildomain(chatmail_config):
return chatmail_config.mailname
def maildomain():
domain = os.environ.get("CHATMAIL_DOMAIN")
if not domain:
pytest.skip("set CHATMAIL_DOMAIN to a ssh-reachable chatmail instance")
return domain
@pytest.fixture
@@ -350,6 +350,22 @@ def lp(request):
return LP()
@pytest.fixture
def maildata(request, gencreds):
datadir = conftestdir.joinpath("mail-data")
def maildata(name, from_addr=None, to_addr=None):
if from_addr is None:
from_addr = gencreds()[0]
if to_addr is None:
to_addr = gencreds()[0]
data = datadir.joinpath(name).read_text()
text = data.format(from_addr=from_addr, to_addr=to_addr)
return BytesParser(policy=policy.default).parsebytes(text.encode())
return maildata
@pytest.fixture
def cmsetup(maildomain, gencreds):
return CMSetup(maildomain, gencreds)
@@ -396,3 +412,16 @@ class CMUser:
imap.login(self.addr, self.password)
self._imap = imap
return self._imap
@pytest.fixture
def create_ini(tmp_path, inipath):
def create_ini_func(source=None):
if source is None:
source = inipath.read_text()
p = tmp_path.joinpath("chatmail.ini")
assert not p.exists(), p
p.write_text(textwrap.dedent(source))
return p
return create_ini_func

View File

@@ -20,7 +20,7 @@ def test_use_two_chatmailservers(cmfactory, maildomain2):
@pytest.mark.parametrize("forgeaddr", ["internal", "someone@example.org"])
def test_reject_forged_from(cmsetup, maildata, gencreds, lp, forgeaddr):
def test_reject_forged_from(cmsetup, maildata, lp, forgeaddr):
user1, user3 = cmsetup.gen_users(2)
lp.sec("send encrypted message with forged from")

View File

@@ -1,33 +0,0 @@
import os
import pytest
from deploy_chatmail.cmdeploy import get_parser, main
from chatmaild.config import read_config
@pytest.fixture(autouse=True)
def _chdir(tmp_path):
old = os.getcwd()
os.chdir(tmp_path)
yield
os.chdir(old)
class TestCmdline:
def test_parser(self, capsys):
parser = get_parser()
parser.parse_args([])
init = parser.parse_args(["init", "chat.example.org"])
run = parser.parse_args(["run"])
assert init and run
def test_init(self, tmp_path):
main(["init", "chat.example.org"])
inipath = tmp_path.joinpath("chatmail.ini")
config = read_config(inipath)
assert config.mailname == "chat.example.org"
def test_init_not_overwrite(self):
main(["init", "chat.example.org"])
with pytest.raises(SystemExit):
main(["init", "chat.example.org"])

View File

@@ -1,13 +1,48 @@
import textwrap
import importlib.resources
from deploy_chatmail.www import build_webpages
from chatmaild.config import read_config
def test_build_webpages(tmp_path, make_config):
def make_config(create_ini, domain="example.org"):
inipath = create_ini(
textwrap.dedent(
f"""\
[params]
max_user_send_per_minute = 60
filtermail_smtp_port = 10080
postfix_reinject_port = 10025
passthrough_recipients =
[privacy:{domain}]
domain = example.org
privacy_postal =
address-line1
address-line2
privacy_mail = privacy@{domain}
privacy_pdo =
address-line3
"""
)
)
return read_config(inipath, domain)
def test_build_webpages(tmp_path, create_ini):
pkgroot = importlib.resources.files("deploy_chatmail")
src_dir = pkgroot.joinpath("../../../www/src").resolve()
assert src_dir.exists(), src_dir
config = make_config("chat.example.org")
config = make_config(create_ini, "example.org")
build_dir = tmp_path.joinpath("build")
build_webpages(src_dir, build_dir, config)
assert len([x for x in build_dir.iterdir() if x.suffix == ".html"]) >= 3
def test_get_settings(tmp_path, create_ini):
config = make_config(create_ini, "example.org")
assert config.privacy_postal == "address-line1\naddress-line2"
assert config.privacy_mail == "privacy@example.org"
assert config.privacy_pdo == "address-line3"
assert config.mailname == "example.org"