Compare commits

..

3 Commits

Author SHA1 Message Date
holger krekel
a0c49d54bd follow link2xt advise and don't check subject/body at all -- turns out there were no tests anyway. 2023-10-22 14:52:54 +02:00
holger krekel
30dae0c09c passes the test 2023-10-22 14:48:54 +02:00
holger krekel
d94521e610 add a failing test for read receipts between two instances 2023-10-22 14:48:54 +02:00
41 changed files with 166 additions and 841 deletions

2
.gitignore vendored
View File

@@ -159,5 +159,3 @@ cython_debug/
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
chatmail.zone

View File

@@ -1,68 +1,23 @@
# Chatmail instances optimized for Delta Chat apps
# Chat Mail server configuration
This repository helps to setup a ready-to-use chatmail instance
This repository setups a ready-to-go chatmail instance
comprised of a minimal setup of the battle-tested
[postfix smtp](https://www.postfix.org) and [dovecot imap](https://www.dovecot.org) services.
[postfix smtp server](https://www.postfix.org) and [dovecot imap server](https://www.dovecot.org).
The setup is designed and optimized for providing chatmail accounts
for use by [Delta Chat apps](https://delta.chat).
## Getting started
Chatmail accounts are automatically created by a first login,
after which the initially specified password is required for using them.
## Getting Started deploying your own chatmail instance
1. Prepare your local (presumably Linux) system:
1. prepare your local system:
scripts/init.sh
2. Setup a domain with `A` and `AAAA` records for your chatmail server.
3. Set environment variable to the chatmail domain you want to setup:
2. set environment variable to the chatmail domain you want to setup:
export CHATMAIL_DOMAIN=c1.testrun.org # replace with your host
4. Deploy the chat mail instance to your chatmail server:
3. run the deploy of the chat mail instance:
scripts/deploy.sh
This script uses `pyinfra` and `ssh` to setup packages and configure
the chatmail instance on your remote server.
5. Run `scripts/generate-dns-zone.sh` and
transfer the generated DNS records at your DNS provider
### Home page and getting started for users
- The `deploy.sh` script deploys a default `index.html`
along with a QR code that users can click to
create accounts on the chatmail provider.
- Start a Delta Chat app and create a new account
by typing an e-mail address with an arbitrary username
and `@<your-chatmail-domain>` appended.
Use an at least 10-character random password.
### Ports
Postfix listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
Dovecot listens on ports 143(imap) and 993 (imaps).
Delta Chat will, however, discover all ports and configurations
automatically by reading the `autoconfig.xml` file from the chatmail instance.
## Emergency Commands to disable automatic account creation
If you need to stop account creation,
e.g. because some script is wildly creating accounts, run:
touch /etc/chatmail-nocreate
While this file is present, account creation will be blocked.
## Running tests and benchmarks (offline and online)
@@ -77,27 +32,28 @@ While this file is present, account creation will be blocked.
scripts/bench.sh
## Running tests (offline and online)
## Development Background for chatmail instances
```
## Dovecot/Postfix configuration
This repository drives the development of "chatmail instances",
comprised of minimal setups of
### Ports
- [postfix smtp server](https://www.postfix.org)
- [dovecot imap server](https://www.dovecot.org)
Postfix listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
Dovecot listens on ports 143(imap) and 993 (imaps).
as well as two custom services that are integrated with these two:
## DNS
- `chatmaild/src/chatmaild/doveauth.py` implements
create-on-login account creation semantics and is used
by Dovecot during login authentication and by Postfix
which in turn uses [Dovecot SASL](https://doc.dovecot.org/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket)
to authenticate users
to send mails for them.
- `chatmaild/src/chatmaild/filtermail.py` prevents
unencrypted e-mail from leaving the chatmail instance
and is integrated into postfix's outbound mail pipelines.
For DKIM you must add a DNS entry as found in /etc/opendkim/selector.txt on your chatmail instance.
The above `scripts/deploy.sh` prints out the DKIM selector and DNS entry you
need to setup with your DNS provider.
## Emergency Commands
If you need to stop account creation,
e.g. because some script is wildly creating accounts,
just run `touch /tmp/nocreate`.
You can remove the file
as soon as the attacker was banned
by different means.

View File

@@ -6,18 +6,15 @@ build-backend = "setuptools.build_meta"
name = "chatmaild"
version = "0.1"
dependencies = [
"aiosmtpd",
"aiosmtpd"
]
[project.scripts]
doveauth = "chatmaild.doveauth:main"
doveauth-dictproxy = "chatmaild.dictproxy:main"
filtermail = "chatmaild.filtermail:main"
[tool.pytest.ini_options]
addopts = "-v -ra --strict-markers"
log_format = "%(asctime)s %(levelname)s %(message)s"
log_date_format = "%Y-%m-%d %H:%M:%S"
log_level = "INFO"
[tool.tox]
legacy_tox_ini = """

View File

@@ -33,6 +33,13 @@ class Connection:
def cursor(self):
return self._sqlconn.cursor()
def create_user(self, addr: str, password: str):
"""Create a row in the users table."""
self.execute("PRAGMA foreign_keys=on")
q = """INSERT INTO users (addr, password, last_login)
VALUES (?, ?, ?)"""
self.execute(q, (addr, password, int(time.time())))
def get_user(self, addr: str) -> {}:
"""Get a row from the users table."""
q = "SELECT addr, password, last_login from users WHERE addr = ?"

View File

@@ -1,6 +1,5 @@
import logging
import os
import time
import sys
import json
import crypt
@@ -22,29 +21,15 @@ def encrypt_password(password: str):
return "{SHA512-CRYPT}" + passhash
def is_allowed_to_create(user, cleartext_password) -> bool:
"""Return True if user and password are admissable."""
def create_user(db, user, password):
if os.path.exists(NOCREATE_FILE):
logging.warning(f"blocked account creation because {NOCREATE_FILE!r} exists.")
return False
if len(cleartext_password) < 10:
logging.warning("Password needs to be at least 10 characters long")
return False
parts = user.split("@")
if len(parts) != 2:
logging.warning(f"user {user!r} is not a proper e-mail address")
return False
localpart, domain = parts
if domain == "nine.testrun.org":
# nine.testrun.org policy, username has to be exactly nine chars
if len(localpart) != 9:
logging.warning(f"localpart {localpart!r} has not exactly nine chars")
return False
return True
logging.warning(
f"Didn't create account: {NOCREATE_FILE} exists. Delete the file to enable account creation."
)
return
with db.write_transaction() as conn:
conn.create_user(user, password)
return dict(home=f"/home/vmail/{user}", uid="vmail", gid="vmail", password=password)
def get_user_data(db, user):
@@ -60,34 +45,16 @@ def lookup_userdb(db, user):
return get_user_data(db, user)
def lookup_passdb(db, user, cleartext_password):
with db.write_transaction() as conn:
userdata = conn.get_user(user)
if userdata:
# Update last login time.
conn.execute(
"UPDATE users SET last_login=? WHERE addr=?", (int(time.time()), user)
)
userdata["uid"] = "vmail"
userdata["gid"] = "vmail"
return userdata
if not is_allowed_to_create(user, cleartext_password):
return
encrypted_password = encrypt_password(cleartext_password)
q = """INSERT INTO users (addr, password, last_login)
VALUES (?, ?, ?)"""
conn.execute(q, (user, encrypted_password, int(time.time())))
return dict(
home=f"/home/vmail/{user}",
uid="vmail",
gid="vmail",
password=encrypted_password,
)
def lookup_passdb(db, user, password):
userdata = get_user_data(db, user)
if not userdata:
return create_user(db, user, encrypt_password(password))
userdata["password"] = userdata["password"].strip()
return userdata
def handle_dovecot_request(msg, db, mail_domain):
print(f"received msg: {msg!r}", file=sys.stderr)
short_command = msg[0]
if short_command == "L": # LOOKUP
parts = msg[1:].split("\t")
@@ -105,18 +72,19 @@ def handle_dovecot_request(msg, db, mail_domain):
reply_command = "N"
elif type == "passdb":
if user.endswith(f"@{mail_domain}"):
res = lookup_passdb(db, user, cleartext_password=args[0])
res = lookup_passdb(db, user, password=args[0])
if res:
reply_command = "O"
else:
reply_command = "N"
print(f"res: {res!r}", file=sys.stderr)
json_res = json.dumps(res) if res else ""
return f"{reply_command}{json_res}\n"
return None
class ThreadedUnixStreamServer(ThreadingMixIn, UnixStreamServer):
request_queue_size = 100
pass
def main():
@@ -128,20 +96,15 @@ def main():
class Handler(StreamRequestHandler):
def handle(self):
try:
while True:
msg = self.rfile.readline().strip().decode()
if not msg:
break
res = handle_dovecot_request(msg, db, mail_domain)
if res:
self.wfile.write(res.encode("ascii"))
self.wfile.flush()
else:
logging.warn("request had no answer: %r", msg)
except Exception:
logging.exception("Exception in the handler")
raise
while True:
msg = self.rfile.readline().strip().decode()
if not msg:
break
res = handle_dovecot_request(msg, db, mail_domain)
if res:
print(f"sending result: {res!r}", file=sys.stderr)
self.wfile.write(res.encode("ascii"))
self.wfile.flush()
try:
os.unlink(socket)

View File

@@ -2,7 +2,7 @@
Description=Dict authentication proxy for dovecot
[Service]
ExecStart=/usr/local/bin/doveauth /run/dovecot/doveauth.socket vmail /home/vmail/passdb.sqlite
ExecStart=/usr/local/bin/doveauth-dictproxy /run/dovecot/doveauth.socket vmail /home/vmail/passdb.sqlite
Restart=always
RestartSec=30

View File

@@ -149,7 +149,7 @@ class SendRateLimiter:
def main():
args = sys.argv[1:]
assert len(args) == 1
logging.basicConfig(level=logging.WARN)
logging.basicConfig(level=logging.INFO)
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
task = asyncmain_beforequeue(port=int(args[0]))

View File

@@ -1,28 +0,0 @@
#!/usr/bin/python3
""" CGI script for creating new accounts. """
import json
import random
mailname_path = "/etc/mailname"
def create_newemail_dict(domain):
alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
user = "".join(random.choices(alphanumeric, k=9))
password = "".join(random.choices(alphanumeric, k=12))
return dict(email=f"{user}@{domain}", password=f"{password}")
def print_new_account():
domain = open(mailname_path).read().strip()
creds = create_newemail_dict(domain=domain)
print("Content-Type: application/json")
print("")
print(json.dumps(creds))
if __name__ == "__main__":
print_new_account()

View File

@@ -7,7 +7,6 @@ name = "deploy-chatmail"
version = "0.1"
dependencies = [
"pyinfra",
"qrcode",
]
[tool.pytest.ini_options]

View File

@@ -4,14 +4,11 @@ Chat Mail pyinfra deploy.
import importlib.resources
from pathlib import Path
from pyinfra import host
from pyinfra.operations import apt, files, server, systemd
from pyinfra import host, logger
from pyinfra.operations import apt, files, server, systemd, python
from pyinfra.facts.files import File
from pyinfra.facts.systemd import SystemdEnabled
from .acmetool import deploy_acmetool
from .genqr import gen_qr_png_data
def _install_chatmaild() -> None:
chatmaild_filename = "chatmaild-0.1.tar.gz"
@@ -27,8 +24,8 @@ def _install_chatmaild() -> None:
)
apt.packages(
name="apt install python3-aiosmtpd python3-pip python3-venv",
packages=["python3-aiosmtpd", "python3-pip", "python3-venv"],
name="apt install python3-aiosmtpd",
packages=["python3-aiosmtpd", "python3-pip"],
)
# --no-deps because aiosmtplib is installed with `apt`.
@@ -37,19 +34,8 @@ def _install_chatmaild() -> None:
commands=[f"pip install --break-system-packages {remote_path}"],
)
# disable legacy doveauth-dictproxy.service
if host.get_fact(SystemdEnabled).get("doveauth-dictproxy.service"):
systemd.service(
name="Disable legacy doveauth-dictproxy.service",
service="doveauth-dictproxy.service",
running=False,
enabled=False,
)
# install systemd units
for fn in (
"doveauth",
"doveauth-dictproxy",
"filtermail",
):
files.put(
@@ -84,36 +70,6 @@ def _configure_opendkim(domain: str, dkim_selector: str) -> bool:
mode="644",
config={"domain_name": domain, "opendkim_selector": dkim_selector},
)
need_restart |= main_config.changed
files.directory(
name="Add opendkim directory to /etc",
path="/etc/opendkim",
user="opendkim",
group="opendkim",
mode="750",
present=True,
)
keytable = files.template(
src=importlib.resources.files(__package__).joinpath("opendkim/KeyTable"),
dest="/etc/dkimkeys/KeyTable",
user="opendkim",
group="opendkim",
mode="644",
config={"domain_name": domain, "opendkim_selector": dkim_selector},
)
need_restart |= keytable.changed
signing_table = files.template(
src=importlib.resources.files(__package__).joinpath("opendkim/SigningTable"),
dest="/etc/dkimkeys/SigningTable",
user="opendkim",
group="opendkim",
mode="644",
config={"domain_name": domain, "opendkim_selector": dkim_selector},
)
need_restart |= signing_table.changed
files.directory(
name="Add opendkim socket directory to /var/spool/postfix",
@@ -134,43 +90,7 @@ def _configure_opendkim(domain: str, dkim_selector: str) -> bool:
_sudo_user="opendkim",
)
return need_restart
def _install_mta_sts_daemon() -> bool:
need_restart = False
config = files.put(
name="upload postfix-mta-sts-resolver config",
src=importlib.resources.files(__package__).joinpath(
"postfix/mta-sts-daemon.yml"
),
dest="/etc/mta-sts-daemon.yml",
user="root",
group="root",
mode="644",
)
need_restart |= config.changed
server.shell(
name="install postfix-mta-sts-resolver with pip",
commands=[
"python3 -m venv /usr/local/lib/postfix-mta-sts-resolver",
"/usr/local/lib/postfix-mta-sts-resolver/bin/pip install postfix-mta-sts-resolver",
],
)
systemd_unit = files.put(
name="upload mta-sts-daemon systemd unit",
src=importlib.resources.files(__package__).joinpath(
"postfix/mta-sts-daemon.service"
),
dest="/etc/systemd/system/mta-sts-daemon.service",
user="root",
group="root",
mode="644",
)
need_restart |= systemd_unit.changed
need_restart |= main_config.changed
return need_restart
@@ -235,17 +155,6 @@ def _configure_dovecot(mail_server: str, debug: bool = False) -> bool:
mode="644",
)
# as per https://doc.dovecot.org/configuration_manual/os/
# it is recommended to set the following inotify limits
for name in ("max_user_instances", "max_user_watches"):
key = f"fs.inotify.{name}"
server.sysctl(
name=f"Change {key}",
key=key,
value=65535,
persist=True,
)
return need_restart
@@ -254,7 +163,7 @@ def _configure_nginx(domain: str, debug: bool = False) -> bool:
need_restart = False
main_config = files.template(
src=importlib.resources.files(__package__).joinpath("nginx/nginx.conf.j2"),
src=importlib.resources.files(__package__).joinpath("nginx.conf.j2"),
dest="/etc/nginx/nginx.conf",
user="root",
group="root",
@@ -264,7 +173,7 @@ def _configure_nginx(domain: str, debug: bool = False) -> bool:
need_restart |= main_config.changed
autoconfig = files.template(
src=importlib.resources.files(__package__).joinpath("nginx/autoconfig.xml.j2"),
src=importlib.resources.files(__package__).joinpath("autoconfig.xml.j2"),
dest="/var/www/html/.well-known/autoconfig/mail/config-v1.1.xml",
user="root",
group="root",
@@ -273,46 +182,6 @@ def _configure_nginx(domain: str, debug: bool = False) -> bool:
)
need_restart |= autoconfig.changed
mta_sts_config = files.template(
src=importlib.resources.files(__package__).joinpath("nginx/mta-sts.txt.j2"),
dest="/var/www/html/.well-known/mta-sts.txt",
user="root",
group="root",
mode="644",
config={"domain_name": domain},
)
need_restart |= mta_sts_config.changed
# install CGI newemail script
#
cgi_dir = "/usr/lib/cgi-bin"
files.directory(
name=f"Ensure {cgi_dir} exists",
path=cgi_dir,
user="root",
group="root",
)
files.put(
name="Upload cgi newemail.py script",
src=importlib.resources.files("chatmaild").joinpath("newemail.py").open("rb"),
dest=f"{cgi_dir}/newemail.py",
user="root",
group="root",
mode="755",
)
qr_data = gen_qr_png_data(domain)
files.put(
name="Upload QR code for account creation",
src=qr_data,
dest="/var/www/html/qrcode.png",
user="root",
group="root",
mode="644",
)
return need_restart
@@ -337,7 +206,7 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
)
# Deploy acmetool to have TLS certificates.
deploy_acmetool(nginx_hook=True, domains=[mail_server, f"mta-sts.{mail_server}"])
deploy_acmetool(nginx_hook=True, domains=[mail_server])
apt.packages(
name="Install Postfix",
@@ -362,35 +231,13 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
packages=["nginx"],
)
apt.packages(
name="Install fcgiwrap",
packages=["fcgiwrap"],
)
_install_chatmaild()
debug = False
dovecot_need_restart = _configure_dovecot(mail_server, debug=debug)
postfix_need_restart = _configure_postfix(mail_domain, debug=debug)
opendkim_need_restart = _configure_opendkim(mail_domain, dkim_selector)
mta_sts_need_restart = _install_mta_sts_daemon()
nginx_need_restart = _configure_nginx(mail_domain)
# deploy web pages and info if we have them
pkg_root = importlib.resources.files(__package__)
www_path = pkg_root.joinpath(f"../../../www/{mail_domain}").resolve()
if www_path.is_dir():
files.rsync(f"{www_path}/", "/var/www/html", flags=["-avz"])
else:
index_path = www_path.parent.joinpath("default/index.html.j2")
files.template(
src=index_path,
dest="/var/www/html/index.html",
user="root",
group="root",
mode="644",
config={"mail_domain": mail_domain},
)
systemd.service(
name="Start and enable OpenDKIM",
service="opendkim.service",
@@ -399,15 +246,6 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
restarted=opendkim_need_restart,
)
systemd.service(
name="Start and enable MTA-STS daemon",
service="mta-sts-daemon.service",
daemon_reload=True,
running=True,
enabled=True,
restarted=mta_sts_need_restart,
)
systemd.service(
name="Start and enable Postfix",
service="postfix.service",
@@ -454,3 +292,14 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
enabled=True,
restarted=journald_conf,
)
def callback():
result = server.shell(
commands=[
f"""sed 's/\tIN/ 600 IN/;s/\t(//;s/\"$//;s/^\t \"//g; s/ ).*//' """
f"""/etc/dkimkeys/{dkim_selector}.txt | tr --delete '\n'"""
]
)
logger.info(f"Add this TXT entry into DNS zone: {result.stdout}")
python.call(name="Print TXT entry for DKIM", function=callback)

View File

@@ -46,7 +46,8 @@ def deploy_acmetool(nginx_hook=False, email="", domains=[]):
mode="644",
)
server.shell(
name=f"Request certificate for: { ', '.join(domains) }",
commands=[f"acmetool want { ' '.join(domains)}"],
)
for domain in domains:
server.shell(
name=f"Request certificate for {domain}",
commands=[f"acmetool want {domain}"],
)

Binary file not shown.

Before

Width:  |  Height:  |  Size: 13 KiB

View File

@@ -6,7 +6,7 @@ from deploy_chatmail import deploy_chatmail
def main():
mail_domain = os.getenv("CHATMAIL_DOMAIN")
mail_server = os.getenv("CHATMAIL_SERVER", mail_domain)
dkim_selector = os.getenv("CHATMAIL_DKIM_SELECTOR", "dkim")
dkim_selector = os.getenv("CHATMAIL_DKIM_SELECTOR", "2023")
assert mail_domain
assert mail_server

View File

@@ -19,7 +19,7 @@ mail_plugins = quota
# these are the capabilities Delta Chat cares about actually
# so let's keep the network overhead per login small
# https://github.com/deltachat/deltachat-core-rust/blob/master/src/imap/capabilities.rs
imap_capability = IMAP4rev1 IDLE MOVE QUOTA CONDSTORE NOTIFY
imap_capability = IMAP4rev1 IDLE MOVE QUOTA CONDSTORE
# Authentication for system users.
@@ -118,24 +118,6 @@ service auth-worker {
user = vmail
}
service imap-login {
# High-security mode.
# Each process serves a single connection and exits afterwards.
# This is the default, but we set it explicitly to be sure.
# See <https://doc.dovecot.org/admin_manual/login_processes/#high-security-mode> for details.
service_count = 1
# Inrease the number of simultaneous connections.
#
# As of Dovecot 2.3.19.1 the default is 100 processes.
# Combined with `service_count = 1` it means only 100 connections
# can be handled simultaneously.
process_limit = 10000
# Avoid startup latency for new connections.
process_min_avail = 10
}
ssl = required
ssl_cert = </var/lib/acme/live/{{ config.hostname }}/fullchain
ssl_key = </var/lib/acme/live/{{ config.hostname }}/privkey

View File

@@ -1,82 +0,0 @@
import importlib
import qrcode
import os
from PIL import ImageFont, ImageDraw, Image
import io
def gen_qr_png_data(maildomain):
url = f"DCACCOUNT:https://{maildomain}/cgi-bin/newemail.py"
image = gen_qr(maildomain, url)
temp = io.BytesIO()
image.save(temp, format="png")
temp.seek(0)
return temp
def gen_qr(maildomain, url):
# taken and modified from
# https://github.com/deltachat/mailadm/blob/master/src/mailadm/gen_qr.py
info = f"{maildomain} invite code"
# load QR code
qr = qrcode.QRCode(
version=1,
error_correction=qrcode.constants.ERROR_CORRECT_H,
box_size=1,
border=1,
)
qr.add_data(url)
qr.make(fit=True)
qr_img = qr.make_image(fill_color="black", back_color="white")
# paint all elements
ttf_path = str(
importlib.resources.files(__package__).joinpath("data/opensans-regular.ttf")
)
logo_red_path = str(
importlib.resources.files(__package__).joinpath("data/delta-chat-bw.png")
)
assert os.path.exists(ttf_path), ttf_path
font_size = 16
font = ImageFont.truetype(font=ttf_path, size=font_size)
num_lines = (info).count("\n") + 1
size = width = 384
qr_padding = 6
text_height = font_size * num_lines
height = size + text_height + qr_padding * 2
image = Image.new("RGBA", (width, height), "white")
draw = ImageDraw.Draw(image)
qr_final_size = width - (qr_padding * 2)
# draw text
if hasattr(font, "getsize"):
info_pos = (width - font.getsize(info.strip())[0]) // 2
else:
info_pos = (width - font.getbbox(info.strip())[3]) // 2
draw.multiline_text(
(info_pos, size - qr_padding // 2), info, font=font, fill="black", align="right"
)
# paste QR code
image.paste(
qr_img.resize((qr_final_size, qr_final_size), resample=Image.NEAREST),
(qr_padding, qr_padding),
)
# background delta logo
logo2_img = Image.open(logo_red_path)
logo2_width = int(size / 6)
logo2 = logo2_img.resize((logo2_width, logo2_width), resample=Image.NEAREST)
pos = int((size / 2) - (logo2_width / 2))
image.paste(logo2, (pos, pos), mask=logo2)
return image

View File

@@ -26,6 +26,8 @@ http {
gzip on;
server {
listen 80 default_server;
listen [::]:80 default_server;
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
@@ -40,16 +42,6 @@ http {
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
# add cgi-bin support
include /usr/share/doc/fcgiwrap/examples/nginx.conf;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
return 301 https://$host$request_uri;
}
}

View File

@@ -1,4 +0,0 @@
version: STSv1
mode: enforce
mx: {{ config.domain_name }}
max_age: 2419200

View File

@@ -1 +0,0 @@
dkim._domainkey.{{ config.domain_name }} {{ config.domain_name }}:{{ config.opendkim_selector }}:/etc/dkimkeys/dkim.private

View File

@@ -1 +0,0 @@
*@{{ config.domain_name }} {{ config.opendkim_selector }}._domainkey.{{ config.domain_name }}

View File

@@ -1,4 +1,7 @@
# OpenDKIM configuration.
# This is a basic configuration for signing and verifying. It can easily be
# adapted to suit a basic installation. See opendkim.conf(5) and
# /usr/share/doc/opendkim/examples/opendkim.conf.sample for complete
# documentation of available configuration parameters.
Syslog yes
SyslogSuccess yes
@@ -18,9 +21,7 @@ OversignHeaders From
# setup options can be found in /usr/share/doc/opendkim/README.opendkim.
Domain {{ config.domain_name }}
Selector {{ config.opendkim_selector }}
KeyFile /etc/dkimkeys/{{ config.opendkim_selector }}.private
KeyTable /etc/dkimkeys/KeyTable
SigningTable refile:/etc/dkimkeys/SigningTable
KeyFile /etc/dkimkeys/{{ config.opendkim_selector }}.private
# In Debian, opendkim runs as user "opendkim". A umask of 007 is required when
# using a local socket with MTAs that access the socket as a non-privileged

View File

@@ -23,7 +23,6 @@ smtpd_tls_security_level=may
smtp_tls_CApath=/etc/ssl/certs
smtp_tls_security_level=may
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_policy_maps = socketmap:inet:127.0.0.1:8461:postfix
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = {{ config.domain_name }}

View File

@@ -32,7 +32,6 @@ submission inet n - y - - smtpd
-o smtpd_recipient_restrictions=
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
-o milter_macro_daemon_name=ORIGINATING
-o smtpd_client_connection_count_limit=1000
-o smtpd_proxy_filter=127.0.0.1:10080
smtps inet n - y - - smtpd
-o syslog_name=postfix/smtps
@@ -47,7 +46,6 @@ smtps inet n - y - - smtpd
-o smtpd_sender_restrictions=$mua_sender_restrictions
-o smtpd_recipient_restrictions=
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
-o smtpd_client_connection_count_limit=1000
-o milter_macro_daemon_name=ORIGINATING
-o smtpd_proxy_filter=127.0.0.1:10080
#628 inet n - y - - qmqpd

View File

@@ -1,10 +0,0 @@
[Unit]
Description=Postfix MTA-STS resolver daemon
[Service]
ExecStart=/usr/local/lib/postfix-mta-sts-resolver/bin/mta-sts-daemon
Restart=always
RestartSec=30
[Install]
WantedBy=multi-user.target

View File

@@ -1,13 +0,0 @@
host: 127.0.0.1
port: 8461
reuse_port: true
shutdown_timeout: 20
cache:
type: internal
options:
cache_size: 10000
proactive_policy_fetching:
enabled: true
default_zone:
strict_testing: false
timeout: 4

View File

@@ -1,4 +1,4 @@
#!/bin/bash
set -e
venv/bin/pytest tests/online/benchmark.py -vrx
online-tests/venv/bin/pytest online-tests/benchmark.py -vrx

View File

@@ -1,15 +1,10 @@
#!/usr/bin/env bash
: ${CHATMAIL_DOMAIN:=c1.testrun.org}
export CHATMAIL_DOMAIN
echo -----------------------------------------
echo deploying to $CHATMAIL_DOMAIN
echo -----------------------------------------
chatmaild/venv/bin/python3 -m build -n --sdist chatmaild --outdir dist
echo WARNING: in five seconds deploy to $CHATMAIL_DOMAIN starts
sleep 5
venv/bin/python3 -m build -n --sdist chatmaild --outdir dist
venv/bin/pyinfra --ssh-user root "$CHATMAIL_DOMAIN" \
deploy-chatmail/venv/bin/pyinfra --ssh-user root "$CHATMAIL_DOMAIN" \
deploy-chatmail/src/deploy_chatmail/deploy.py
rm -r dist/

View File

@@ -1,23 +0,0 @@
#!/bin/sh
: ${CHATMAIL_DOMAIN:=c1.testrun.org}
: ${CHATMAIL_SSH:=$CHATMAIL_DOMAIN}
set -e
SSH="ssh root@$CHATMAIL_SSH"
EMAIL="root@$CHATMAIL_DOMAIN"
ACME_ACCOUNT_URL="$($SSH -- acmetool account-url)"
cat <<EOF
$CHATMAIL_DOMAIN. MX 10 $CHATMAIL_DOMAIN.
$CHATMAIL_DOMAIN. TXT "v=spf1 a:$CHATMAIL_DOMAIN -all"
_dmarc.$CHATMAIL_DOMAIN. TXT "v=DMARC1;p=reject;rua=mailto:$EMAIL;ruf=mailto:$EMAIL;fo=1;adkim=r;aspf=r"
_submission._tcp.$CHATMAIL_DOMAIN. SRV 0 1 587 $CHATMAIL_DOMAIN.
_submissions._tcp.$CHATMAIL_DOMAIN. SRV 0 1 465 $CHATMAIL_DOMAIN.
_imap._tcp.$CHATMAIL_DOMAIN. SRV 0 1 143 $CHATMAIL_DOMAIN.
_imaps._tcp.$CHATMAIL_DOMAIN. SRV 0 1 993 $CHATMAIL_DOMAIN.
$CHATMAIL_DOMAIN. IN CAA 128 issue "letsencrypt.org;accounturi=$ACME_ACCOUNT_URL"
_mta-sts.$CHATMAIL_DOMAIN. IN TXT "v=STSv1; id=$(date -u '+%Y%m%d%H%M')"
mta-sts.$CHATMAIL_DOMAIN. IN CNAME $CHATMAIL_DOMAIN.
_smtp._tls.$CHATMAIL_DOMAIN. IN TXT "v=TLSRPTv1;rua=mailto:$EMAIL"
EOF
$SSH opendkim-genzone -F | sed 's/^;.*$//;/^$/d'

View File

@@ -1,4 +1,4 @@
#!/bin/bash
venv/bin/tox -c chatmaild
venv/bin/tox -c deploy-chatmail
venv/bin/pytest tests/online -rs -vrx --durations=5 $@
tox -c chatmaild
tox -c deploy-chatmail
venv/bin/pytest tests/online -vrx --durations=5 $@

View File

@@ -0,0 +1,53 @@
import os
import pytest
import chatmaild.dictproxy
from chatmaild.dictproxy import get_user_data, lookup_passdb
from chatmaild.database import Database, DBError
@pytest.fixture()
def db(tmpdir):
db_path = tmpdir / "passdb.sqlite"
print("database path:", db_path)
return Database(db_path)
def test_basic(db):
chatmaild.dictproxy.NOCREATE_FILE = "/tmp/nocreate"
if os.path.exists(chatmaild.dictproxy.NOCREATE_FILE):
os.remove(chatmaild.dictproxy.NOCREATE_FILE)
lookup_passdb(db, "link2xt@c1.testrun.org", "asdf")
data = get_user_data(db, "link2xt@c1.testrun.org")
assert data
def test_dont_overwrite_password_on_wrong_login(db):
"""Test that logging in with a different password doesn't create a new user"""
res = lookup_passdb(db, "newuser1@something.org", "kajdlkajsldk12l3kj1983")
assert res["password"]
res2 = lookup_passdb(db, "newuser1@something.org", "kajdlqweqwe")
# this function always returns a password hash, which is actually compared by dovecot.
assert res["password"] == res2["password"]
def test_nocreate_file(db):
chatmaild.dictproxy.NOCREATE_FILE = "/tmp/nocreate"
with open(chatmaild.dictproxy.NOCREATE_FILE, "w+") as f:
f.write("")
assert os.path.exists(chatmaild.dictproxy.NOCREATE_FILE)
lookup_passdb(db, "newuser1@something.org", "kajdlqweqwe")
assert not get_user_data(db, "newuser1@something.org")
os.remove(chatmaild.dictproxy.NOCREATE_FILE)
def test_db_version(db):
assert db.get_schema_version() == 1
def test_too_high_db_version(db):
with db.write_transaction() as conn:
conn.execute("PRAGMA user_version=%s;" % (999,))
with pytest.raises(DBError):
db.ensure_tables()

View File

@@ -1,90 +0,0 @@
import json
import sys
import pytest
import threading
import queue
import traceback
import chatmaild.doveauth
from chatmaild.doveauth import get_user_data, lookup_passdb, handle_dovecot_request
from chatmaild.database import Database, DBError
def test_basic(db):
lookup_passdb(db, "link2xt@c1.testrun.org", "Pieg9aeToe3eghuthe5u")
data = get_user_data(db, "link2xt@c1.testrun.org")
assert data
data2 = lookup_passdb(db, "link2xt@c1.testrun.org", "Pieg9aeToe3eghuthe5u")
assert data == data2
def test_dont_overwrite_password_on_wrong_login(db):
"""Test that logging in with a different password doesn't create a new user"""
res = lookup_passdb(db, "newuser1@something.org", "kajdlkajsldk12l3kj1983")
assert res["password"]
res2 = lookup_passdb(db, "newuser1@something.org", "kajdlqweqwe")
# this function always returns a password hash, which is actually compared by dovecot.
assert res["password"] == res2["password"]
def test_nocreate_file(db, monkeypatch, tmpdir):
p = tmpdir.join("nocreate")
p.write("")
monkeypatch.setattr(chatmaild.doveauth, "NOCREATE_FILE", str(p))
lookup_passdb(db, "newuser1@something.org", "zequ0Aimuchoodaechik")
assert not get_user_data(db, "newuser1@something.org")
def test_db_version(db):
assert db.get_schema_version() == 1
def test_too_high_db_version(db):
with db.write_transaction() as conn:
conn.execute("PRAGMA user_version=%s;" % (999,))
with pytest.raises(DBError):
db.ensure_tables()
def test_handle_dovecot_request(db):
msg = (
"Lshared/passdb/laksjdlaksjdlaksjdlk12j3l1k2j3123/"
"some42@c3.testrun.org\tsome42@c3.testrun.org"
)
res = handle_dovecot_request(msg, db, "c3.testrun.org")
assert res
assert res[0] == "O" and res.endswith("\n")
userdata = json.loads(res[1:].strip())
assert userdata["home"] == "/home/vmail/some42@c3.testrun.org"
assert userdata["uid"] == userdata["gid"] == "vmail"
assert userdata["password"].startswith("{SHA512-CRYPT}")
def test_100_concurrent_lookups_different_accounts(db, gencreds):
num_threads = 100
req_per_thread = 5
results = queue.Queue()
def lookup(db):
for i in range(req_per_thread):
addr, password = gencreds()
try:
lookup_passdb(db, addr, password)
except Exception:
results.put(traceback.format_exc())
else:
results.put(None)
threads = []
for i in range(num_threads):
thread = threading.Thread(target=lookup, args=(db,), daemon=True)
threads.append(thread)
print(f"created {num_threads} threads, starting them and waiting for results")
for thread in threads:
thread.start()
for i in range(num_threads * req_per_thread):
res = results.get()
if res is not None:
pytest.fail(f"concurrent lookup failed\n{res}")

View File

@@ -1,28 +0,0 @@
import json
import chatmaild
from chatmaild.newemail import create_newemail_dict, print_new_account
def test_create_newemail_dict():
ac1 = create_newemail_dict(domain="example.org")
assert "@" in ac1["email"]
assert len(ac1["password"]) >= 10
ac2 = create_newemail_dict(domain="example.org")
assert ac1["email"] != ac2["email"]
assert ac1["password"] != ac2["password"]
def test_print_new_account(capsys, monkeypatch, maildomain, tmpdir):
p = tmpdir.join("mailname")
p.write(maildomain)
monkeypatch.setattr(chatmaild.newemail, "mailname_path", str(p))
print_new_account()
out, err = capsys.readouterr()
lines = out.split("\n")
assert lines[0] == "Content-Type: application/json"
assert not lines[1]
dic = json.loads(lines[2])
assert dic["email"].endswith(f"@{maildomain}")
assert len(dic["password"]) >= 10

View File

@@ -9,10 +9,9 @@ import itertools
from email.parser import BytesParser
from email import policy
from pathlib import Path
from math import ceil
import pytest
from chatmaild.database import Database
conftestdir = Path(__file__).parent
@@ -72,7 +71,7 @@ def pytest_report_header():
@pytest.fixture
def benchmark(request):
def bench(func, num, name=None, reportfunc=None):
def bench(func, num, name=None):
if name is None:
name = func.__name__
durations = []
@@ -81,7 +80,7 @@ def benchmark(request):
func()
durations.append(time.time() - now)
durations.sort()
request.config._benchresults[name] = (reportfunc, durations)
request.config._benchresults[name] = durations
return bench
@@ -102,9 +101,7 @@ def pytest_terminal_summary(terminalreporter):
headers = f"{'benchmark name': <30} " + fcol(float_names)
tr.write_line(headers)
tr.write_line("-" * len(headers))
summary_lines = []
for name, (reportfunc, durations) in results.items():
for name, durations in results.items():
measures = [
sorted(durations)[len(durations) // 2],
min(durations),
@@ -113,16 +110,6 @@ def pytest_terminal_summary(terminalreporter):
line = f"{name: <30} "
line += fcol(f"{float: 2.2f}" for float in measures)
tr.write_line(line)
vmedian, vmin, vmax = measures
if reportfunc:
for line in reportfunc(vmin=vmin, vmedian=vmedian, vmax=vmax):
summary_lines.append(line)
if summary_lines:
tr.write_line("")
tr.section("benchmark summary measures")
for line in summary_lines:
tr.write_line(line)
@pytest.fixture
@@ -130,16 +117,6 @@ def imap(maildomain):
return ImapConn(maildomain)
@pytest.fixture
def make_imap_connection(maildomain):
def make_imap_connection():
conn = ImapConn(maildomain)
conn.connect()
return conn
return make_imap_connection
class ImapConn:
AuthError = imaplib.IMAP4.error
logcmd = "journalctl -f -u dovecot"
@@ -180,16 +157,6 @@ def smtp(maildomain):
return SmtpConn(maildomain)
@pytest.fixture
def make_smtp_connection(maildomain):
def make_smtp_connection():
conn = SmtpConn(maildomain)
conn.connect()
return conn
return make_smtp_connection
class SmtpConn:
AuthError = smtplib.SMTPAuthenticationError
logcmd = "journalctl -f -t postfix/smtpd -t postfix/smtp -t postfix/lmtp"
@@ -228,20 +195,13 @@ def gencreds(maildomain):
num = next(count)
alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
user = "".join(random.choices(alphanumeric, k=10))
user = f"ac{num}_{user}"[:9]
password = "".join(random.choices(alphanumeric, k=12))
user = f"ac{num}_{user}"
password = "".join(random.choices(alphanumeric, k=10))
yield f"{user}@{domain}", f"{password}"
return lambda domain=None: next(gen(domain))
@pytest.fixture()
def db(tmpdir):
db_path = tmpdir / "passdb.sqlite"
print("database path:", db_path)
return Database(db_path)
#
# Delta Chat testplugin re-use
# use the cmfactory fixture to get chatmail instance accounts
@@ -312,7 +272,7 @@ class Remote:
self.sshdomain = sshdomain
def iter_output(self, logcmd=""):
getjournal = "journalctl -f" if not logcmd else logcmd
getjournal = f"journalctl -f" if not logcmd else logcmd
self.popen = subprocess.Popen(
["ssh", f"root@{self.sshdomain}", getjournal],
stdout=subprocess.PIPE,

View File

@@ -1,6 +1,5 @@
import pytest
import threading
import queue
import smtplib
def test_login_basic_functioning(imap_or_smtp, gencreds, lp):
@@ -24,11 +23,6 @@ def test_login_basic_functioning(imap_or_smtp, gencreds, lp):
with pytest.raises(imap_or_smtp.AuthError):
imap_or_smtp.login(user, password + "wrong")
lp.sec("creating users with a short password is not allowed")
user, _password = gencreds()
with pytest.raises(imap_or_smtp.AuthError):
imap_or_smtp.login(user, "admin")
def test_login_same_password(imap_or_smtp, gencreds):
"""Test two different users logging in with the same password
@@ -41,30 +35,3 @@ def test_login_same_password(imap_or_smtp, gencreds):
imap_or_smtp.login(user1, password1)
imap_or_smtp.connect()
imap_or_smtp.login(user2, password1)
def test_concurrent_logins_same_account(
make_imap_connection, make_smtp_connection, gencreds
):
"""Test concurrent smtp and imap logins
and check remote server succeeds on each connection.
"""
user1, password1 = gencreds()
login_results = queue.Queue()
def login_smtp_imap(smtp, imap):
try:
imap.login(user1, password1)
except Exception:
login_results.put(False)
else:
login_results.put(True)
conns = [(make_smtp_connection(), make_imap_connection()) for i in range(10)]
for args in conns:
thread = threading.Thread(target=login_smtp_imap, args=args, daemon=True)
thread.start()
for _ in conns:
assert login_results.get()

View File

@@ -91,7 +91,7 @@ class TestEndToEndDeltaChat:
lp.sec("setup encrypted comms between ac1 and ac2 on different instances")
qr = ac1.get_setup_contact_qr()
ac2.qr_setup_contact(qr)
ch = ac2.qr_setup_contact(qr)
msg = ac2.wait_next_incoming_message()
assert "verified" in msg.text

View File

@@ -1,31 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title>chatmail instance</title>
</head>
<body>
<h1>Welcome to {{ config.mail_domain }}!</h1>
<h2>Getting started</h2>
<ol>
<li>Install <a href="https://get.delta.chat">https://get.delta.chat</a></li>
<li>Scan or Tap on the invite QR code</li>
<li>Choose Nickname and Avatar</li>
<li>Setup contact with others using <a href="https://delta.chat/en/help#howtoe2ee">
guaranteed end-to-end encryption via QR code scans</a>
</li>
</ol>
<a href="DCACCOUNT:https://{{ config.mail_domain }}/cgi-bin/newemail.py">
<img class="section" src="qrcode.png" />
</a>
<h2>Constraints</h2>
<ul>
<li>You can only send encrypted mails to anyone outside {{config.mail_domain }} </li>
<li>You may send up to 60 messages per minute</li>
<li>Messages are unconditionally removed 40 days after arrival</li>
<li>Max storage per user is 100MB</li>
</ul>
</body>
</html>

Binary file not shown.

Before

Width:  |  Height:  |  Size: 96 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 163 KiB

View File

@@ -1,81 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title>nine.testrun.org - Experimenting with the Future of Email</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>
.wrapper {
width: 100%;
max-width: 596px;
margin: 0 auto;
}
.section {
width: 100%;
max-width: 596px;
}
.text {
box-sizing: border-box;
padding: 9px;
font-size: 18px;
font-family: "Swansea", "Helvetica", sans-serif;
color: black;
}
a {
color: black;
}
h1, h2, h3 {
font-size: 18px;
font-weight: bold;
}
</style>
</head>
<body>
<div class="wrapper">
<img class="section" src="collage-top.png" />
<div class="section text">
<h1>Dear Delta Chat users and newcomers,</h1>
<p>
welcome to the first public "chat-mail instance",
a small and lean e-mail provider for smooth chatting.
Install Delta Chat and then
Tap or scan this QR code to obtain a random e-mail address:
<a href="DCACCOUNT:https://nine.testrun.org/cgi-bin/newemail.py">
<img with=300 src="qrcode.png" /></a>
</p>
<p>
Alternatively, you can manually invent an e-mail address:
<ul>
<li>Tap "LOG INTO YOUR E-MAIL ACCOUNT".</li>
<li>Address: invent a word with <i>exactly</i> nine characters
and append @nine.testrun.org to it.</li>
<li>Password: invent at least 10 characters. The first login sets your password.</li>
</ul>
If the e-mail address is not yet taken, you'll get that account.
</p>
<p>
<img class="section" src="collage-down.png" />
<h2>What's behind it, how does it operate?</h2>
<p>nine.testrun.org is run
by a small group of devs and sysadmins, reachable via root@.
They want to keep this instance running at least until end 2024.
Current limits:
<ul>
<li>Un-encrypted mails can not leave the chat-mail instance.</li>
<li>Use <a href="https://delta.chat/en/help#howtoe2ee">
guaranteed end-to-end encryption via QR code scans</a>
to setup contact with users outside of the chat-mail instance.
</li>
<li>You may send up to 60 messages per minute.</li>
<li>Messages are unconditionally removed 40 days after arrival.</li>
<li>Max storage per user is 100MB.</li>
</ul>
<h2>Why are other email providers 1000 times more complicated?</h2>
<p>¯\_(ツ)_/¯</p>
</div>
</div>
</body>
</html>