The scenarios worked around Postgres 18's data layout change by mounting the
tmpfs at /var/lib/postgresql. The postgres role solved the same problem
differently and better, when it bit us there: it sets PGDATA explicitly, and to
/data rather than anywhere beneath /var/lib/postgresql, because that path is a
VOLUME in the official image and nothing can be mounted under it.
Doing the same here means the scenarios exercise the layout the role actually
deploys, and that a future image changing its own default cannot move the data
directory out from under them.
Ref: https://github.com/docker-library/postgres/pull/1259
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SEH3vxYSQ5SV4N5z61eyGT
`group_vars/matrix_servers` selects postgres whenever postgres is enabled, which
is the default, so postgres is what essentially every deployment runs. The
scenarios were testing sqlite - a path almost nobody is on.
How little that path is used is not a guess: the mautrix-meta bridges could not
start at all under sqlite, and nobody reported it. Testing the engine users are
actually on is worth more than keeping coverage of the one they are not, so no
scenario is left behind on sqlite.
Four of the eight scenarios have a database and are converted; the other four
have none and are untouched.
molecule-shared/tasks/postgres.yml stands Postgres up on the scenario's network,
with the data directory on a tmpfs since it is thrown away with the container.
The image is pinned at the major the postgres role deploys to new installations
and left to Renovate: when a new major lands, the PR bumping that pin runs every
scenario against it, which is the earliest warning we get that a component does
not cope.
Each scenario gives its database and user names that differ from the role's
defaults, so the component reaching the database proves the role built its
connection string out of them. The assertions moved from "a file appeared at the
path we configured" to "these tables exist", which is strictly stronger: tables
can only appear once the component has resolved the hostname, authenticated with
the credentials the role rendered, and run its migrations to completion.
Costs about 10 seconds per affected scenario (115s to 125s locally for
mautrix-whatsapp), on jobs that run in parallel.
Gotcha worth recording: since Postgres 18 the image puts PGDATA in a versioned
subdirectory and refuses to start if it finds a mount at the old
/var/lib/postgresql/data, so the tmpfs is mounted at /var/lib/postgresql.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SEH3vxYSQ5SV4N5z61eyGT
They were hard-wrapped at 80 characters, broke mid-parenthesis, and spent lines
restating what the code below them does.
Rewrapped at natural boundaries instead, with the narration dropped and only the
reasons, gotchas and surprises kept. Section dividers stay - they delineate long
plays rather than narrate them.
Comments only; no scenario behaviour changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SEH3vxYSQ5SV4N5z61eyGT
baibot is the first bot rather than a bridge, and the shape
differs from the appservices: it is a plain Matrix client that logs in with a
password, sets up its profile and then syncs. It publishes no port, so nothing
can be probed over HTTP; what it says about itself in the journal is the only
window into whether the role's configuration reached the process.
What the scenario proves:
- The unit is active with no automatic restarts, AND baibot got past startup
into its sync loop. The second half is what carries the scenario. baibot never
exits when startup goes wrong - it retries the failing step forever with a
growing delay - so the unit sits there `active` with `NRestarts` at 0 while
the bot is permanently half-started. Pointing `user.avatar` at a file that is
not there reproduces exactly that: the unit assertion still passes, the sync
assertion does not.
- The display name the bot announces it wants is the role's `user.name`, which
is neither the role's default nor what the stub reports the account already
has.
- The rendered `logging` string took effect per target: baibot's own records
appear at DEBUG (the role ships `info`) while everything underneath stays at
the `warn` catch-all. The second half is the control, and raising the
catch-all turns 2 DEBUG records into 161.
- The rendered config carries the scenario's homeserver, identity, command
prefix, admin patterns and user patterns, and uses password authentication
exclusively, with the access-token keys rendered as nulls.
- The statically-defined agent survived the provider templating - the
per-provider template rendered to YAML, parsed, merged and nested into the
list - key by key.
- The container runs as the uid/gid the playbook supplies (1234, not the 1000
the base image already has), on the image version defaults/main.yml pins, and
could write its session into the data path.
No AI provider is contacted and none is needed. baibot calls a provider only
when a message asks an agent to do something, so a static agent with a
placeholder key and a base URL that resolves nowhere still has to survive the
bot's startup parsing - which is the part worth testing.
The shared stub grew what a syncing Matrix client needs and an appservice did
not: /sync (with a `next_batch`, and holding the call open for the timeout the
client asked for, or the bot spins the stub in a hot loop), the media config
and upload endpoints a bot setting its own avatar insists on, /keys/upload with
its key counts, and filter creation. Without the media config in particular,
baibot never gets past profile setup.
The shared stub task gained a STUB_VERBOSE knob. The stub already advertised
the environment variable but there was no way to set it from a scenario, and
for a component with no port of its own its request log is the only place to
see what the component is actually asking for.
Note: molecule-shared/homeserver-stub.py also carries a loosened /login match
from another scenario being written in this same tree at the same time; it was
already in the working copy and is not mine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SEH3vxYSQ5SV4N5z61eyGT
Hookshot is the widest HTTP surface of the bridges here - an appservice port
plus a `listeners` list whose entries appear or vanish with independent
switches - so the scenario asserts on that surface rather than on "it started".
The scenario configures NO third-party service. GitHub, GitLab, Jira and Figma
each need an account and a credential somewhere else, which is where a scenario
stops testing this repository (docs/molecule-testing.md). GitLab, which the role
turns on by default, is switched off and its absence from both config.yml and
the registration's namespaces is asserted. The generic webhooks listener needs
no credential from anyone, and is what gets exercised live: a POST to an unknown
hook id comes back as `{"ok":false,"error":"Webhook not found"}`, which only
Hookshot's own generic-webhook handler produces - an Express "Cannot POST" page
there would mean the port is Hookshot's but the service was never mounted on it.
Every port the scenario sets differs from both the role's default and Hookshot's
own, and the run asserts both halves: the webhooks, metrics and appservice ports
it chose all answer, and 9000/9001/9003/9993 - what the role would have used
instead - all refuse. Metrics are off in the role's defaults, so that listener
exists only because the scenario asked for it, and /metrics answers with
`hookshot_*` series that nothing else could have produced.
config.yml and registration.yml are parsed as YAML and asserted structurally, so
a value landing under the wrong key cannot pass as a substring match.
Two things the role made necessary:
- `tasks/main.yml` opens with a block tagged `reset-hookshot-encryption`, and
Ansible runs tagged tasks unless tags are actually selected - so any tagless
play hits it, and it stops `matrix-hookshot.service` before that service
exists. The scenario skips the tag, which is what the playbook effectively
does by always running with `--tags=setup-all,start`.
- The role reads `matrix_host_command_openssl`, which matrix-base defines.
matrix-bridge-appservice-irc reads it too, so it goes in the shared playbook
context rather than into this scenario.
Falsified by pointing the webhooks listener at 127.0.0.1 in the role's config
template - a change no file-level assertion can see. Every config and
registration assertion still passed, and the run failed at "Assert the generic
webhooks service is mounted on that listener" with "Port 9741 did not answer as
Hookshot's generic webhooks service". Reverted afterwards.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SEH3vxYSQ5SV4N5z61eyGT
The third shape: a non-mautrix bridge. It confirms the scaffolding is not
mautrix-specific, and it is laid out differently enough to be worth having -
everything sits directly under the base path rather than in config/ and
data/ subdirectories, the owner and homeserver URL are passed on the command
line rather than through a config file, so the unit is where they can be
checked, and identd optionally binds host port 113.
matrix-base's matrix_server_fqn_* family moved into the shared context;
18 of the roles here read one of them.
Three roles now, three shapes - HTTP receiver, mautrix bridge, non-mautrix
bridge - and the scaffolding took no per-shape special-casing beyond each
role's own variables.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The second scenario, chosen to be a bridge rather than another HTTP
component: bridges are roughly 40 of the 70 roles here, and validating the
shape on role two is cheaper than discovering it on role forty. It found
two things the first role could not.
Bridges read a whole family of variables from outside themselves -
matrix_bridges_relay_enabled, _encryption_enabled, _encryption_default,
_msc4190_enabled, _self_sign_enabled, plus matrix_admin - all defined in
matrix-base. Since every bridge reads them, they belong in shared context
rather than in each bridge's scenario: molecule-shared/playbook-context.yml
now carries them along with the identity and path variables the first
scenario had inline. Note it is loaded through vars_files, which outranks
inventory group_vars, so it is authoritative; a scenario that wants to prove
one of these reaches the rendered configuration should say so explicitly.
Appservices call /whoami while starting and refuse to run if the id returned
is not the bot user they were configured as - mautrix-whatsapp exits 17 with
"Unexpected user ID in whoami call". The shared stub now takes the id it
should claim to be. Expect every bridge and bot to need this.
What the scenario proves: the bridge starts and stays up, opens its
appservice port, its rendered config and appservice registration carry the
scenario's tokens and bot user, it created its sqlite database under the
role's data path as the role's uid, and it runs the version the role pins.
It does not bridge anything and never will.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Components here contact a homeserver while starting up and exit if it is
unreachable, so nearly every scenario will need one. Standing up a real
Synapse per role would dominate the run and drag in Postgres, and these
scenarios are not testing Synapse.
The stub answers the handful of endpoints components touch during startup
with the blandest plausible response, and is deliberately permissive: an
unrecognised path returns {} rather than 404, because the goal is to get the
component past its startup checks. It is not an authentication check or a
room state machine, and a scenario should not assert *about* it - if one
starts needing it to behave like a real homeserver, that scenario has
outgrown what these tests are for.
matrix-alertmanager-receiver now includes it instead of carrying its own
inline copy. Verified green afterwards.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three things that would not have scaled to 70 roles:
- The Python and Ansible dependency pins were about to be copied into every
role. They now live once in molecule-shared/, which scenarios reference
relatively, so they cannot drift apart.
- The helper container images used for probing were hardcoded inline. They
are pinned once in molecule-shared/vars.yml, carry `# renovate:`
annotations, and a custom manager in .github/renovate.json keeps them
current - verified with a local Renovate dry run, which offers
curl 8.11.1 -> 8.21.0 and python 3.13 -> 3.14-alpine. Seventy invisible
hardcodes is the blindness class we have been removing elsewhere.
- Running a scenario meant knowing the venv and cd incantation. `just
molecule <role>` does it, and with no argument lists the roles that have
a scenario.
Molecule is deliberately not wired into prek: a run takes minutes, pulls
images and needs Docker, which is fine on request and not fine per commit.
docs/molecule-testing.md covers how to run and write these, including the
four things a role here needs that a standalone role does not. AGENTS.md
points at it rather than carrying the detail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
renovate[bot]andGitHub
Slavi PantaleevandClaude Opus 5