Troubleshooting

This chapter is organised by what you see.

The daemon has no control socket and no status command. Everything it knows it either says in the journal at startup or publishes on MQTT, so those two are the whole toolbox:

systemctl status muxen-nmea2000
journalctl -u muxen-nmea2000 -n 100                 # the startup log
journalctl -u muxen-nmea2000 -f                     # follow it
mosquitto_sub -h 127.0.0.1 -t 'nmea/interface' -v   # CAN link health, every 10 s
mosquitto_sub -h 127.0.0.1 -t 'nmea/status'    -v   # module health, every 30 s
mosquitto_sub -h 127.0.0.1 -t 'nmea/config'    -v   # what configuration took effect
mosquitto_sub -h 127.0.0.1 -t 'nmea/devices'   -v   # who is on the bus
muxen-nmea2000-inventory                            # the same, formatted

Read them in that order. nmea/config answers "is the module on?", nmea/interface answers "is the bus alive?", and nmea/devices answers "is anything talking?". Most reports of "no data" are answered by one of those three before any real diagnosis starts.

The service will not start at all

The daemon refuses several configurations outright rather than running in a state it cannot make sense of. Restart=always with RestartSec=10, StartLimitBurst=5 and StartLimitIntervalSec=300 means a persistent failure loops five times in five minutes and then leaves the unit failed.

Journal line	Cause	Fix
Config: Interface 'canX' does not exist	the interface is not present	ip link show; check MUXEN_NMEA_INTERFACE in the unit or its drop-in
Config: Interface 'X' is not a CAN interface	the name resolves to something else	as above
CAN bus baudrate is N bps, NMEA 2000 requires 250000 bps	the interface is up at the wrong bitrate	bring it down, set 250000, bring it up. See below
J1939 NAME is required (use --name or -n option)	no --name	the unit supplies it from MUXEN_NMEA_NAME; check that variable is not empty
Config: Invalid preferred address: N (must be 0-253)	--address out of range	0–253; 254 and 255 are reserved
Config: Invalid address range: min=A > max=B	--min-addr above --max-addr	swap them
Error: another instance is already running	a daemon already holds the lock	see below
Error: cannot create lock file in …	the runtime directory is not writable	check RuntimeDirectory= and the muxen user
Config: --source-gpsd-transmit requires --source-gpsd	a gpsd option without its prerequisite	GNSS and gpsd

The bitrate check

NMEA 2000 is 250 kbit/s and nothing else. A node at the wrong bitrate does not simply fail to communicate — it corrupts frames for everyone else on the segment. The daemon therefore reads the interface's configured bitrate at startup and refuses to run on anything else.

ip -details link show can1        # look for "bitrate 250000"
sudo ip link set can1 down
sudo ip link set can1 type can bitrate 250000
sudo ip link set can1 up

Interfaces whose name begins with vcan skip the check, which is why the simulator setup works without any CAN hardware.

"Another instance is already running"

The single-instance lock lives in the runtime directory. Two daemons on one interface would claim the same J1939 NAME, which is illegal on the bus, so the second one refuses to start.

The usual cause is a hand-run daemon left over from a test, because a manual run uses the same /run/muxen-nmea2000 the service does. systemctl stop muxen-nmea2000 before running the daemon by hand, and stop the manual run before restarting the service.

Templated instances have their own runtime directory and therefore their own lock, so muxen-nmea2000@can0 and the primary daemon do not collide — More than one bus.

Nothing at all on MQTT

Check	What it means
systemctl status mosquitto	the daemon exits nothing but retries; no broker, no topics
mosquitto_sub -t 'nmea/#' -v returns immediately with nothing	either no broker, or the prefix is not nmea
mosquitto_sub -t 'nmea-can0/#' -v	you are looking at a templated instance's data under the wrong prefix

MQTT failure is not fatal: the daemon keeps running, keeps claiming its address and keeps answering the bus. So systemctl status showing active (running) with no MQTT output is a broker problem, not a daemon problem.

A report topic never appears

Work down this list; it resolves nearly all of them.

1. Is the module on? Read nmea/config, or the startup summary line. A daemon with no report modules prints only MQTT Alarms DateTime in its feature list. Report modules are opt-in — Configuration.
2. Have all four addresses been claimed? No report starts until every virtual device has an address. --verbose-control shows each claim; nmea/devices should list the gateway four times.
3. Is anything transmitting the data? nmea/devices lists the bus. If the instrument is not there, the problem is upstream.
4. Is the report waiting for content? nmea/motor publishes nothing at all while no engine is active, and nmea/ais is an empty set of objects on empty water. Both are correct behaviour.

The screens show stale numbers

Most topics here are retained. A stopped daemon leaves its last payload on the broker, and a screen that subscribes afterwards sees perfectly plausible figures forever.

mosquitto_sub -h 127.0.0.1 -t 'nmea/navigation' -v -W 3

If the payload arrives once and never repeats, nothing is publishing. Compare metadata.rxTimestamp against the current time, and treat anything older than metadata.expireAfterSec as dead. Every payload carries both fields precisely so a consumer can make that judgement.

nmea/motor and nmea/thruster are the exceptions: they are not retained, so they cannot go stale this way.

Removing a decommissioned topic needs an explicit empty retained publish:

mosquitto_pub -h 127.0.0.1 -t nmea/route -r -n

A value is null or zero when the instrument is clearly working

Every report ages its fields out. A field that stops arriving becomes null or 0 rather than keeping its last value, which means the symptom is "the data stopped arriving", not "the daemon lost it".

Report	Field expiry
navigation	2 s for wind, 5 s for everything else
motor	5 s for dynamic values, 5 min for the engine itself
thruster	10 s
route	5 s for steering data, 5 min for the route and waypoints
AIS	60 s for position, 300 s for identity

If the instrument really is transmitting, the next question is whether the daemon is selecting it — see the next entry.

The wrong instrument is being used

Two GPS receivers, or two compasses, and the number on the screen is from the wrong one.

Run muxen-nmea2000-config and look at the source list: it shows every device, the data types each supplies, and the priority in force. Then:

• Priority 1 wins. Lower number is higher priority. An unconfigured source is treated as 50.
• Equal priorities are decided by freshness, with a hysteresis window to stop the selection flapping.
• A source in warm-up is not selectable. A GPS that has just been powered on waits for a fix and then a further 10 seconds.
• A source with an expiration counter at 5 is out of the running until fresh data brings the counter back down.

Set an explicit priority, or block the source you do not want, and press R to make the daemon reload. Details in Navigation.

Two engines show as one

Both engines are shipped at the same NMEA 2000 instance, so they collide into a single entry in nmea/motor. The daemon keys engines by instance and has no other way to tell them apart.

muxen-nmea2000-config can rewrite a remote device's instance over the standard Command Group Function (PGN 126208) with the I key. Change one device at a time and confirm the result before moving on. Some devices refuse the command; those need their manufacturer's tool.

The gateway is not visible to the chartplotter

The daemon presents four virtual devices — navigation, engine, AIS and electrical — because one NMEA 2000 device may only advertise a bounded PGN list, and this gateway handles far more than one device's worth. So a healthy daemon appears four times in a plotter's device list, normally at four consecutive addresses from 128.

If it appears not at all:

1. journalctl for ACD: Failed to claim any address. That means the configured address range is full and the NAME is not arbitrary-address-capable, so there is nowhere to move to. Widen the range with --min-addr / --max-addr — Address claiming.
2. Check nmea/interface: state must be ERROR-ACTIVE.
3. Check the plotter is not filtering by device class.

Some plotters cache their device list; a rescan on the plotter is often needed after a change.

nmea/interface shows errors, or the bus keeps dropping

state is the kernel's view of the CAN controller:

State	Meaning
ERROR-ACTIVE	normal
ERROR-WARNING	error counters have crossed the warning threshold
ERROR-PASSIVE	error counters have crossed the passive threshold; the node has stopped asserting errors
BUS-OFF	the controller has taken itself off the bus
STOPPED, SLEEPING, UNKNOWN	not operational

Anything other than ERROR-ACTIVE, and rising bus_errors or rx_errors, is a wiring problem, not a software one. In rough order of likelihood: missing or doubled termination, a stub too long, a node at the wrong bitrate, a damaged connector, insufficient bus power.

The daemon recovers from a bus that goes away. On BUS-OFF or the interface going down it suspends the virtual devices, clears the Fast Packet reassembly queues and stops the reporting timers; then it retries with an exponential backoff, starting at 1 s and capping at 30 s, indefinitely by default. On a successful retry it rebuilds the sockets, re-registers the I/O watches, rebuilds the address claim state from the stored peer registry and resumes every report.

--recovery-initial-delay, --recovery-max-delay and --recovery-max-attempts tune this. With a non-zero attempt limit, a final failure logs CAN … recovery failed - terminating and the daemon exits so systemd can restart it clean.

The state file complains at startup

The daemon keeps the addresses it claimed in a small binary file so it can ask for the same ones next time. It validates that file hard and deletes it rather than acting on something it does not trust:

State file: Invalid magic number 0x… (expected 0x4A313933)
State file: Checksum mismatch (got 0x…, expected 0x…)
State file: Incomplete read
State file: Validation failed, deleting corrupted file
State file: No saved state found (first run)

None of these is a fault to chase. The daemon falls back to its preferred address and claims normally; the only visible effect is that it may end up at a different address than last time.

State file: Migrating v2 format to v3… is a normal one-off after an upgrade. Full detail in The state file.

Raw PGN topics are missing

nmea/device/<addr>/pgn/<pgn> is off by default. Turn it on with --mqtt-publish-pgn.

It is off for a reason: it republishes every decoded message from every device, which on a busy bus is a large and continuous volume of MQTT traffic. Use it to diagnose, not to run.

FAQ

Why is the navigation page empty? Most likely the navigation report was never enabled for this boat. That is a commissioning setting, not a fault — an installer turns it on in the boat's configuration file and restarts the service.

The chart shows no other ships. Is AIS broken? Three possibilities, in order: the AIS report is not enabled; there is no AIS receiver on the bus; the receiver is there but its antenna is disconnected. An installer can tell them apart in a couple of minutes.

Why does a ship appear with no name for several minutes? That is how AIS works. Position is broadcast every few seconds; the vessel's name, type and destination only every six minutes or so. The triangle appears first and acquires its label later.

The depth reading is frozen. Is the sounder dead? It is not frozen — this daemon never repeats an old value. A depth that stops updating means the sounder stopped transmitting, and after five seconds the field goes empty rather than lying.

Why did the wind reading get smoother after the yard visited? Wind damping was probably turned up. It is a filter that smooths the gusts out of the display without hiding a real wind shift; level 0 switches it off entirely.

Can I acknowledge a MUXEN alarm from the chartplotter? Yes. That is what the alarm bridge is for — press silence or acknowledge on the plotter and the MUXEN system is told. Alarms.

Does this change anything on my instruments? No, with one exception. The daemon reads the bus and republishes; it never commands an autopilot, an engine or a thruster. The exception is deliberate and manual: an installer can change a device's instance number from muxen-nmea2000-config.

Do I have to restart anything after changing the configuration? Navigation source priorities: no, systemctl reload muxen-nmea2000 is enough. Anything in the boat's deploy configuration: yes, and in normal MUXEN operation a deployment restarts the service for you.

Why are there four MUXEN devices in my plotter's device list? Because a single NMEA 2000 device may only advertise a limited list of messages, and this gateway handles far more than that. It presents itself as four cooperating devices. It is one program.

Is it safe to run a second daemon while we are sailing? Yes, on a different interface. On the same bus a templated instance is built so it always loses arbitration to the boat's gateway and never reads the boat's configuration. Stop it when you are done.

Tips

At commissioning, read nmea/config once and keep it. It is the complete record of what the daemon resolved: every module, every limit, every configured navigation source with the address it resolved to. It is faster than reconstructing the same picture from the journal, and it is the thing to compare against when something changes later.

Set navigation priorities before handover, not after a complaint. A boat with two GPS antennas and no priorities picks by freshness, which is stable but arbitrary. Five minutes in muxen-nmea2000-config at the dock avoids an argument at sea.

Give a GPS a minute after power-up. Position warm-up waits for a real fix and then a further ten seconds. A source that "does not appear" thirty seconds after switch-on has not failed.

Check nmea/interface error counters at handover and write them down. On a healthy bus they stay near zero. Knowing the normal value turns them into a diagnostic: a step change with no change to the installation means a connector, a terminator or a new node.

Do not leave --mqtt-publish-pgn on. It is a diagnostic firehose: every decoded message from every device, republished continuously.

Watch out for retained topics when you decommission something. Stopping a module does not remove what is already on the broker. Clear it explicitly with an empty retained publish.

Prefer failover over on when transmitting GNSS. A boat that already has a working GPS does not want a second position source on the wire; failover puts one there only after the bus has gone without a valid position for the failover delay (60 seconds by default), and withdraws it once a bus GPS has been valid again for 10 seconds.

Never enable muxen-nmea2000-from-simulator.service on a boat. It declares Conflicts=muxen-nmea2000.service, so enabling it takes the real gateway down and replaces the boat's data with synthetic data that looks entirely plausible.

Stop templated instances when you are finished. They do not survive a reboot, but until then they are a second gateway on the bus and a second set of retained topics.

Use the simulator for HMI work. muxen-nmea2000-simulator --all vcan0 with a daemon on the same interface gives a full set of live reports with no boat, no bus and no risk.