Troubleshooting

This chapter is organised by what you see. The daemon has no control socket and no status command: it says what it knows in the journal at startup, and everything else is an HTTP answer. Those two are the whole toolbox.

systemctl status muxen-config
journalctl -u muxen-config -n 50            # startup lines and errors
curl -s http://127.0.0.1:12000/health       # ready / starting
curl -s -D - -o /dev/null http://127.0.0.1:12000/deploy   # headers of the deployment
ls -l /etc/muxen /etc/muxen/configuration   # ownership and modes

The service restarts every 30 seconds

Restart=always with RestartSec=30, so a daemon that fails at startup loops on that period. The journal names the reason on each attempt.

Message	Cause	Fix
config: configFolder /etc/muxen do not exists !	the directory is gone	recreate it, then systemctl start muxen-config-perms
config: configFolder /etc/muxen must be writable !	it is not writable by muxen	systemctl start muxen-config-perms
config: swaggerFolder /usr/share/muxen-config/swagger do not exists !	incomplete or partially removed package	reinstall muxen-config
lws init failed	the listening port could not be bound	something else holds 127.0.0.1:12000 — ss -ltnp | grep 12000
Failed to compile regex: …	a malformed route in the build	report it; a stock package cannot produce this

The first three print the usage block after the message and exit with status 1. All of them are visible with journalctl -u muxen-config -n 50.

Nothing answers, or everything answers 404

Work down this list.

1. Is the daemon up at all? curl -s http://127.0.0.1:12000/health. No answer means the service is down — see above. 503 with "status":"starting" means it is still compiling its routing table; until it finishes, every path with a variable in it answers 404. It lasts milliseconds, but a probe that fires at the wrong moment sees it.
2. Are you going through nginx with the right prefix?muxen-api-config.conf maps /api/config/<path> to <path> on the daemon. /api/config/project is right; /config/project and /api/project are not.
3. Does that route exist for that method? Several paths only exist for some verbs. There is no GET for a single setting, a single device parameter, or a commissioning rule — read the whole section instead. The full matrix is in Reference.
4. Is the short name legal? Anything outside [a-zA-Z0-9-] does not match the route at all and comes back 404 or 400. A short name of 64 characters or more is also rejected.

Everything about one project answers 401

That project carries a password setting. Send the hash:

curl -H "Authorization: Bearer $(printf %s 'the-password' | sha256sum | cut -d' ' -f1)" \
     http://127.0.0.1:12000/project/<id>

The daemon compares the second whitespace-separated field of the Authorization header against the stored hash, so a header with no space in it — a bare hash, no scheme word — is refused as well.

If the password is lost, the setting can only be removed by editing /etc/muxen/configuration/<id>.json on the Brain and deleting the password entry from settings[]; run systemctl start muxen-config-perms afterwards.

Every write answers 500

Reads work, writes do not. The response body carries the reason:

$ curl -s -X PUT http://127.0.0.1:12000/project/<id>/device/64
{"error":"save project","errno":30,"code":"EROFS","message":"Read-only file system"}

code	Cause
EROFS	you are editing a template. /usr/share/muxen-config/templates is read-only by design — duplicate it first
EACCES / EPERM	/etc/muxen/configuration is not writable by muxen — systemctl start muxen-config-perms
ENOSPC	the filesystem is full
ENOENT	/etc/muxen/configuration is missing

The template case is the common one and it is easy to miss, because reading a template works perfectly: the project-scoped endpoints fall back to the template directory, so GET succeeds and only the save fails.

The boat still behaves the way it did before

In decreasing order of likelihood:

1. The project was edited but never deployed. Editing writes /etc/muxen/configuration/<id>.json; the boat reads /etc/muxen/deploy.json. PUT /project/<id>/deploy is the step that connects the two.
2. A different project was deployed. Compare: curl -s http://127.0.0.1:12000/deploy | head -20 and look at metadata.id.
3. The client is serving a cached copy. GET /deploy is no-cache with an ETag, so a correct client revalidates — but a proxy or a browser in between may not. Compare the ETag against a fresh digest: sha256sum /etc/muxen/deploy.json.
4. The consumers did not restart. They restart because muxen-deploy.target is restarted by the path unit on /etc/muxen/deploy.json. systemctl status muxen-deploy.target shows when it last did.

GET /deploy answers 404

Two causes, and they look identical from outside:

• Nothing has ever been deployed. /etc/muxen/deploy.json does not exist. ls -l /etc/muxen/deploy.json settles it.
• deploy.json exists but is not valid JSON. The endpoint parses the file before answering and treats a parse failure as 404. Check with python3 -m json.tool < /etc/muxen/deploy.json. This is what a file restored from an interrupted manual copy looks like — deploy again, or copy a backup over it.

/api/deploy.json suddenly returns a much bigger body

nginx fell back to serving the file off disk, which happens when the daemon is unreachable — down, restarting, or mid-upgrade. The fallback serves deploy.json raw, so every embedded image comes with it instead of being stripped out.

It is doing its job: the screens keep working. But it means the daemon is not running. Check systemctl status muxen-config.

The Brain sits at 100 % CPU

If a muxen-configd process is spinning, something is talking to 127.0.0.1:12000 with HTTP keep-alive. The daemon deliberately answers without a Content-Length so that every response closes the connection; a client that negotiates keep-alive anyway can drive the event loop into a busy loop on ARM.

curl -s -D - -o /dev/null http://127.0.0.1:12000/deploy | grep -i connection
connection: close

If that says anything else, or if a development tool (a Vite proxy, an API client, a load tester) is pointed straight at :12000, route it through nginx instead — the snippets force Connection: close.

A project vanished from the list

GET /project skips any file it cannot parse, without a message. A project that disappeared from the list is still on disk:

ls -l /etc/muxen/configuration/
python3 -m json.tool < /etc/muxen/configuration/<id>.json

The usual cause is a hand edit, an interrupted copy, or a restore of a file that was not a project. Restore the project from an export or a backup.

A device lost all its parameters

PUT /project/<id>/device/<deviceId> replaces the device: the handler removes any device already at that id and appends a fresh one carrying only the generated Name parameter (plus the extra parameters for an SFSP interrupter). Re-issuing it on an existing device is a reset, not a no-op.

Only PUT /project/<id>/device/<deviceId>/parameter/<name> and .../parameters change a device without recreating it.

Deleting a switching rule removed more than the rule

That is the cascade, and it is intended. Removing an entry from muxing:

• removes the device parameters that entry's config had written onto the devices,
• removes every commissioning rule of type muxing whose target is that name,
• removes any commissioning group left with no rules at all.

There is no undo. Export the project first if the commissioning section matters.

A transaction answered 500

The response headers say exactly how far it got:

$ curl -si -X POST http://127.0.0.1:12000/project/<id>/transaction \
       -H 'content-type: application/json' -d @steps.json | grep -i x-step
x-step-url-1: /project/hull-042/device/64
x-step-method-1: PUT
x-step-code-1: 201
x-step-url-2: /project/hull-042/nope
x-step-method-2: PUT
x-step-code-2: 404

The first step whose code is outside 200–299 stops the run and the whole request answers 500. Nothing is written: the steps operate on an in-memory copy and the file is saved only after the last step succeeds. Fix the failing step and post the list again.

A step with no url or no method is skipped silently rather than failing, so a step that seems not to have run is worth checking for a typo in those two keys.

Renaming a view answers 409

Another view already has that name. Names must be unique within the project, and the new name must match [a-zA-Z0-9-]+ — the same constraint as a project short name, because the name is a URL segment.

Duplicating a project answers 409

A project with the target short name already exists in configuration/. A duplicate only creates: pick another name, or delete the existing project first (its password applies).

A media upload answers 413

That is nginx, not the daemon. muxen-api-config.conf sets client_max_body_size 50M. Media is stored base64-encoded inside the project JSON, so a large image costs about a third more than its file size in the project and again on every read of the whole project — keep them small.

The backup directory is empty, or has not grown

systemctl list-timers muxen-config-backup.timer
journalctl -u muxen-config-backup -n 20
ls -lt /etc/muxen/backup/

• The script only copies when /etc/muxen/deploy.json differs from the last_backup symlink. A boat whose configuration has not changed produces no new file, indefinitely. That is normal.
• It does nothing at all if /etc/muxen/deploy.json does not exist.
• It keeps the 30 newest copies and deletes the rest, so an old file disappearing is expected.
• If the timer is not listed, it is not enabled: systemctl enable --now muxen-config-backup.timer.

FAQ

I changed the configuration and nothing changed on the boat. Changes are saved into a project. The boat runs the deployed configuration. Someone has to press Deploy (PUT /project/<id>/deploy) before anything on board changes.

Why do the screens blink and things restart when I deploy? Because deploying really does change the boat. Writing the deployed configuration restarts the MUXEN services that read it, so they pick up the new device names, switching rules and pages. It takes a few seconds and it is normal.

Can I get yesterday's configuration back? Yes, if the boat was deployed to since then: /etc/muxen/backup/ holds up to 30 dated copies of the deployed configuration. See Deployment.

Can two people edit the same project at the same time? They can, and they should not. There is no locking: each request reads the whole file, changes it and writes it back, so the last write wins and silently discards the other person's change. Agree on one editor.

What happens if the power fails during a deploy? The deployed file is written next to the target and renamed into place, so it is either the old configuration or the new one — never a mixture. Worst case the boat comes back on the previous configuration and the deploy is repeated.

Why does the project list show entries I cannot edit? Those are templates, marked type: template. They ship read-only on the system partition. Duplicate one to get an editable project.

Do I have to restart the configuration service after editing a file by hand? The daemon reads the project files per request, so no restart is needed for the content. Ownership is the problem instead: a file written as root may no longer be writable by the service. Run systemctl start muxen-config-perms after any hand edit under /etc/muxen.

How big can a project be? Large enough to matter. Media is stored base64-encoded inside the JSON, so a project with images runs to megabytes and every whole-project read pays for it. That is why GET /project/<id> and GET /deploy strip media and page bodies, and why nginx allows 50 MB request bodies.

Where is the API documentation on the Brain itself? The daemon serves the bundled Swagger UI at its root — http://127.0.0.1:12000/, or /api/config/ through nginx. It covers a subset of the routes; the complete list is in Reference.

Tips

Commission from a template rather than building from nothing. Duplicating a template with a commissioning object produces a project already trimmed to the hull: devices, sensors, pages and switching rules for options the boat does not have are removed in one pass, instead of being deleted one by one afterwards.

Export before anything irreversible. GET /project/<id>/backup returns the whole project in one file. Delete, restore and the muxing cascade have no undo.

Batch bulk edits into a transaction. Twenty separate requests parse and rewrite the whole project twenty times and leave nineteen intermediate states on disk. One POST …/transaction writes once.

Deploy at a moment when a restart is acceptable. A deploy restarts the services that read the configuration. Not while someone is manoeuvring.

Check /health, not the TCP port. A listening socket is not the same as a working router; /health reports starting until the routes are compiled.

Do not point development tooling straight at :12000. Go through nginx, which forces Connection: close. Direct keep-alive traffic is the one known way to spin the daemon's CPU on ARM.

Keep media out of the project where you can. Every whole-project read carries it, encoded, and every deploy copies it.

Watch what /etc/muxen/backup actually contains at handover, not just that the timer is enabled. A boat whose configuration never changes produces no new backups, which is correct but indistinguishable from a broken job until you look at the timestamps.

Back up /etc/muxen as a whole. The daily job covers deploy.json only — not the projects, not apps/.

Leave -v off in production. Verbose mode logs a line per request and per commissioning decision into the journal; it is for a bench, not for a boat.

Treat short names as permanent. The short name is the filename and every URL segment for that project. Renaming means duplicating under the new name and deleting the old one, which resets ctm and breaks anything that stored the old id.