Troubleshooting

Almost everything that goes wrong with synapse shows up in one place:

muxen-synapse-ctl list

ENABLED is what was asked for, RUNNING is what the engine has loaded. A row where the two disagree is the row to look at, and NOTE usually says why. This chapter is organised by what you see.

Before anything else: a misbehaving automation cannot damage anything. It runs in its own sandbox with no filesystem and no shell, under a hard time budget, and the engine stops it after five consecutive faults. Diagnose at leisure.

list prints "no automations"

The tool connected to the broker but found no retained synapse/<id> status at all. In decreasing order of likelihood:

1. No automations are deployed. A fresh install ships six templates but starts none of them — the template library on its own runs nothing. Check for a synapse section: grep -A2 '"synapse"' /etc/muxen/deploy.json.
2. The daemon is not running. systemctl status muxen-synapse.
3. The daemon never connected to the broker. journalctl -u muxen-synapse | grep mqtt — you want mqtt connected.
4. You are pointed at the wrong broker. The startup banner prints the one the daemon uses (config: mqtt = …); muxen-synapse-ctl resolves its own from --host/--port, then $SYNAPSE_MQTT_HOST/$MQTT_HOST, then 127.0.0.1:1883.

An automation is in deploy.json but not in list

The instance was rejected at load. The reason is always logged. Check:

journalctl -u muxen-synapse | grep -i "reject\|invalid\|errored"

Cause	What you will see
Invalid instance id	rejecting instance '<id>': invalid id (must match [a-zA-Z0-9-]+) — no underscores, no dots, no slashes
Duplicate id	The first entry wins; the duplicate is rejected and logged
Both template and script, or neither	The instance is invalid and skipped
Deploy "enabled": false	Working as designed: a force-off instance is not loaded and owns no status topic, so it is absent from list by construction, not broken
The deploy.json change was never applied	Synapse has no live reload. It reads the file once at startup — restart the daemon

That last one is the common case. sudo systemctl restart muxen-synapse, or let the config daemon's deploy phase do it.

enabled=yes but running=no

The automation faulted out. Five consecutive callback faults — script errors, watchdog timeouts, or a mix — and the engine stops it rather than let it spin. enabled is deliberately left alone so a frontend can show "enabled but faulted".

NOTE in list carries last_error. For the full picture:

muxen-synapse-ctl journal <id> -n 100

last_error looks like	Cause	Fix
a Lua error with a line number	a bug in the script	fix the code; a restart retries with a fresh fault counter
an "exceeded time budget" abort	a callback used more than 50 ms of CPU	the handler is doing too much work — see A handler keeps timing out below
not enough memory	the automation passed its 16 MB Lua heap ceiling	usually accumulating data in a table forever; bound what you keep
a message about a missing required config key	a required schema field was omitted in deploy.json	supply it and restart

Recovery: fix the cause and either restart the daemon, or muxen-synapse-ctl start <id> — an explicit enable clears the fault state and retries immediately.

An automation is running but does nothing

enabled=yes running=yes, no error, and nothing happens. Work through these in order.

1. Is it subscribed to what you think?

muxen-synapse-ctl debug <id> --what wiring,config

wiring.subscriptions is exactly what the automation asked for. A misspelled variable name in the config produces a subscription to a topic nobody publishes, and that is silent.

2. Is the data arriving, and is it fresh?

mosquitto_sub -t 'app/sensor/<name>' -v

synapse.variable.get returns nil for a value that is stale, and staleness is measured from the payload's own publish time against its own expiry window. Two consequences that surprise people:

• a retained value replayed when synapse subscribes, from a sensor that has since stopped publishing, reads as stale — not as fresh;
• a payload missing metadata.rxTimestamp or metadata.expireAfterSec is treated as permanently stale, so the automation never acts on it at all.

The variables debug section shows what the engine actually holds, with ages.

3. Is the condition an edge that already passed? The astronomical and harbour callbacks fire only at the crossing. An automation enabled after sunset has missed sunset and will not act until the next one. The shipped templates seed their state on start for exactly this reason; a hand-written one may not.

4. Is night "unknown" rather than "false"? synapse.sextant.is_day and is_night are tri-state. With no GPS fix or no muxen-sextant running they return nil, which is falsy in Lua — a bare if synapse.sextant.is_night() then silently treats unknown as "not night". Check sextant is publishing: mosquitto_sub -t 'sextant/sun' -v.

5. Is nmea/motor simply absent? It is published only while a motor is active and is not retained. Silence is the normal state with the engines off, and is what masthead-light-motoring interprets as "not motoring".

A light automation never presses the button

The light templates read the keypad LED before pressing, because a press toggles — pressing a button already in the wanted state flips it the wrong way. If the LED state is unknown or stale, the automation deliberately does not press.

So check the keypad is publishing:

mosquitto_sub -t 'device/0/0/state' -v

You want led<N>Code for the button's index. 0 is off, 1 green-blink, 2 green, 3 red-blink, 4 red.

Two traps in the configuration:

• The IOChannel index is 0-based. Button 1 on the keypad is index: 0. The project's buttonMapping.index is 1-based — do not cross the two.
• deviceId for a keypad is 0 × 64 + instance, so instance 0 is deviceId: 0, instance 1 is deviceId: 1.

A command is sent but the device does nothing

Check the payload vocabulary matches the device. The requestOn/requestOff convention belongs to Bloc8 power outputs (function code 1). The Lighting device (function code 8) decodes only { channel, on, dimming } — a { requestOn = true } payload sent to Lighting is silently ignored and the light does not switch.

On many boats the nav, anchor and steaming lights are wired to keypad buttons rather than to the Lighting device. Drive those with pressButton.

Check the channel numbering. In a raw synapse.device.command payload, channel is 1-based; channel 0 is always rejected and logged. In an IOChannel from a config field, index is 0-based.

Check nothing else is fighting it. Synapse does not arbitrate: two enabled automations commanding the same output is last-write-wins at the device. The engine logs an advisory warning at boot when two enabled instances declare commands to the same target, and the commands debug section lists each instance's recent command targets:

muxen-synapse-ctl debug <id> --what commands

Check the broker was up. Commands are QoS 0, fire-and-forget, and are not queued: one published during a reconnect gap is silently dropped. A well-written automation re-asserts the state it wants rather than relying on a single edge command.

A handler keeps timing out

last_error reports a time-budget abort. Every callback gets 50 ms of CPU time, engine-wide, and a script cannot raise its own budget.

The budget is CPU time, not wall-clock, so a load spike from the kiosk does not cause a false abort — hitting it means the handler really is doing that much computation. Typical causes:

• iterating a large table on every message from a 1 Hz-or-faster topic;
• string building in a loop;
• re-deriving something on every message that could be computed once at the top of the file.

The fix is to do less per callback: move work to a slower synapse.timer.every, compute constants at top level, and subscribe to narrower topics.

A popup never appears

• Is one already pending? Only one popup may be pending per automation. A second prompt returns false and does nothing — it neither replaces nor queues. muxen-synapse-ctl debug <id> --what wiring shows the pending popup.
• Is it snoozed? An ignore answer arms a snooze that persists in state.json and survives both a restart and a disable/enable cycle. While it is active, prompt with the same key resolves immediately as "snoozed" and raises nothing. Check /var/lib/muxen-synapse/state.json for the instance's snoozes.
• Is a frontend subscribed? Synapse publishes the popup in the retained status and nothing more. Confirm with mosquitto_sub -t 'synapse/<id>' -v; if the popup object is there, the problem is downstream.

A popup was answered but nothing happened

The answer is matched on its requestId. An answer whose id does not match the currently pending popup — because it was already answered, cleared, or timed out — is ignored by design.

Popups time out after 300 s by default. On timeout onAnswer is called with the action "timeout", so an automation that only handles "ok" does nothing at all. That is correct behaviour, not a fault.

A state machine came back in the wrong state

State-machine positions are resumed on a daemon restart and reset to initial on an explicit disable. That asymmetry is deliberate: a crash or redeploy should not lose a long sequence, but a deliberate off-then-on should re-derive from live state.

Resume is also self-healing. If the stored state name no longer exists in the redeployed script, or the script's version changed, the machine falls back to initial, fires that state's enter, logs a warning and sets last_error. Seeing that after a template update is expected.

The service will not start

systemctl status muxen-synapse
journalctl -u muxen-synapse -n 50 --no-pager

The startup banner is the first thing to read — it prints the effective deploy path, state path, template dir, broker and time source. If a path in it is not what you expect, a drop-in is overriding it — check systemctl cat muxen-synapse.

RestartSec=30 means a daemon that fails at startup retries every 30 seconds rather than in a tight loop, so give it a moment before concluding it is down.

The unit runs under ProtectSystem=strict with RestrictNetworkInterfaces=lo: the only writable path is /var/lib/muxen-synapse, and the daemon can only reach a broker on loopback. A remote --host will not work from inside the unit.

Getting more detail

journalctl -u muxen-synapse -f              # everything the daemon logs
muxen-synapse-ctl journal <id> -f           # just one automation
muxen-synapse-ctl debug <id>                # full snapshot of one automation
muxen-synapse-ctl debug --json <id>         # the same, for a script
mosquitto_sub -t 'synapse/#' -v             # the whole control plane, live
sudo systemctl kill -s SIGUSR1 muxen-synapse   # one-line-per-instance audit

SIGUSR1 prints a per-instance audit to the journal: run state, effective enabled, consecutive fault count, current Lua heap usage, and any pending popup or last error. It is repeatable and has no side effects.

For more verbosity the daemon takes -v (lifecycle, enable/disable, transitions, popups, watchdog aborts) and -vv (adds per-message dispatch, timer firings, FSM detail and full Lua tracebacks). Production runs without either; raise it with a systemd drop-in or by running the binary by hand.

---

FAQ

Do I have to write Lua to use synapse? No. The six shipped automations cover the common jobs and are configured entirely through the interface — you pick a template, choose the button or output it drives, set a threshold. Writing Lua is for automations nobody has written yet.

I turned an automation off from the interface. Does it stay off after a reboot? Yes. A runtime enable/disable is persisted and survives a restart. The one thing that overrides it is a deployment pushing "enabled": false, which forces the automation off everywhere.

Why did my automation disappear from the list entirely? Two things remove an automation from the list rather than showing it as off: it was deleted from deploy.json, or the deployment forced it off with "enabled": false. A runtime disable keeps it listed, showing enabled: false.

Will an automation fight me if I flip a switch by hand? No. All the shipped light automations act only at the transition — dusk falling, an engine starting — and then hand control back. If you change the light afterwards, they do not change it back; they act again at the next transition.

Why is my tank-level automation not triggering when the tank is clearly low? Most often because the value is stale rather than low. Synapse deliberately refuses to act on sensor data it cannot date: a reading whose publisher has stopped, or one without freshness metadata, reads as unknown and is skipped. Check the sensor is still publishing.

Two automations control the same output. Which wins? The last one to publish. Synapse does not arbitrate and never owns an output. It logs an advisory warning at boot if it spots the overlap, but avoiding it is a deployment decision.

Can an automation brick a device or the boat? It can command devices the same way any other MUXEN client can, so treat what you deploy accordingly. It cannot escape its sandbox: no shell, no filesystem, no process control, no loading code at runtime. And devices clamp what they are told — a dimming level, for instance, is bounded by the device's own configured safe range.

Does an automation cost anything while it is disabled? Almost nothing. A disabled automation has no interpreter at all — it is torn down. Its persisted values and snoozes stay on disk.

How many automations can I run? There is no hard limit in the engine. The practical limit is CPU: every callback is serialised on one loop, so what matters is total work per second, not instance count. Dozens of event-driven automations are unremarkable; dozens all running heavy per-second timers are not.

Why does the log say a config value was clamped? Because the deployed value was outside the range the template declares. The engine corrects rather than rejects — a pump_channel: 12 on a template capped at 8 becomes 8 — so a typo runs safely in range instead of failing later at the device. The warning tells you to fix the value.

---

Tips

At commissioning, deploy one automation at a time and watch it.muxen-synapse-ctl journal <id> -f while you exercise the condition — run the tank down, start the engine, wait for dusk — tells you in seconds whether the wiring is right.

Check list for enabled=yes running=no as a routine health probe. It is the single signal that something faulted. Everything else is usually visible in the interface.

Verify the input before blaming the automation. mosquitto_sub -t 'app/sensor/<name>' -v and mosquitto_sub -t 'device/0/0/state' -v settle most "it does nothing" questions in one command. Confirm the data is arriving and that it carries metadata.rxTimestamp and expireAfterSec.

Prefer template over inline script unless you need to diverge. A template instance follows package improvements automatically across the fleet; an inline script is a frozen private copy that will silently miss every fix.

Give instances descriptive ids. The id is the MQTT topic and the journal tag, and it is what you will be reading at three in the morning. fresh-water-port-cutoff beats auto1. Remember: [a-zA-Z0-9-] only.

Set name and name.<LANG> per instance. Two instances of one template are indistinguishable in the interface otherwise. The Lua is never localised — the translations live in deploy.json.

Watch for two automations reaching for the same output. The boot advisory warning is the cheap check; muxen-synapse-ctl debug <id> --what commands is the detailed one.

Do not build safety interlocks out of snoozes. Snoozes are wall-clock and a GPS clock correction can shorten or lengthen one. Timers and FSM after are monotonic and are what safety logic should use.

Remember there is no live reload. Any deploy.json change needs a daemon restart. If a change appears not to have taken effect, that is the first thing to check.

Keep an eye on the journal after a template update. A changed script version resets stored state-machine positions to initial, which is logged. It is expected, but it explains an automation that "restarted itself" after an upgrade.

When you write your own automation, validate it before saving. The synapse/validate round-trip catches syntax errors, typo'd globals and a missing synapse.meta without any side effects — much cheaper than discovering them from a faulted instance on the boat.